Prof Claudia Harsch, University of Bremen, Germany
Using the CEFR for aligning teaching, learning and assessment – practices in Cuba and Germany
In my talk, I will focus on how the CEFR has been used as framework for teaching, learning and assessment in higher education in Cuba and Germany. I will present two projects: one from my Languages Centre in Bremen, Germany, were we adapted the CEFR for end-of term achievement tests for all languages taught at CEFR Levels A1 to C1; the other from
the Cuban context, where the Ministry of Higher Education released a new policy requiring English proficiency at CEFR Level B1 for all students at the end of their BA studies, regardless of their subject.
In both contexts, I set up a training series for the language teachers to develop test specifications and tasks that operationalize curricula, learning goals and relevant CEFR descriptors. The training was regularly evaluated by the participating teachers to allow for a needs-based adaptive training approach. I will report how we adapted the CEFR in those two contexts, what challenges we encountered, and how the teachers perceived the training.
I will then discuss implications of these two cases for implementing the CEFR for teaching, learning and assessment.
Biodata
Claudia Harsch is a professor at the University of Bremen, specialising in language learning, teaching and assessment. She has worked in Germany and in the UK, and is active in teacher training worldwide. Her research
interests focus on areas such as language assessment, language and migration, the development of language assessment literacy, and the implementation of the CEFR. She is the current president of the International
Language Testing Association, and was president of the European Association of Language Testing and Assessment from 2016-2019.
Thank you very much for the introduction and I feel very honored uh uh to be here and have been invited to share with you my experiences in Cuba and in Brean and I present here also on behalf of the project teams so we’ll look a little bit into
The CFR and the alignment of language education so that we’re just all on the same page because I’m not sure who is in testing and who is more in education so I’ll just bring you all on the same page then I present you the two contexts and
What we did and why we did it and the challenges we face in adapting the CFR then in both contexts we had teacher training involved and I want to share with you insights that the teachers gave us feedback the teachers gave us on the training and the new assessment and in
Breman I also have feedback from the students and I’ll end with the implications that we may be able to gener generalize beyond the two contexts so first of all when we talk about aligning language education to the CFR maybe I bore you with it but I find it
Ever so important and I think I reiterate in Parts what Zarah shared with us earlier on the CFR is a language policy instrument it doesn’t tell us what to do it sets the frame it’s a policy document and it describes relevant aspects of learning teaching and assessment and the CFR originally
Has nine chapters hands up who has read the nine chapters that’s what I feared it’s worth looking into it it’s still valid it’s still really valuable information and uh it provides a common meta language so that we can talk about uh what our Learners can do and what we
May want them to be able to do but don’t reduce it only to the scales and the scale language in the descriptor is rather abstract it’s context and language independent and that’s what Sarah also said the context matters and I’ll talk a lot about contextualizing so bear that in mind you
Will have to adapt the CFR it’s not there to be used as it is you it’s a tool it’s a workbench so use it and adapt it it provides a reference framework and I like Sarah’s um quote to words it’s not reverence it’s reference and you can inform learning and self
Assessment the CFR can inform teaching and curricular development and it can inform assessment but uh you have to make it work in your context and the proficiency scales that now have been revised in the um CV in the companion volume they are at the heart of the CFR
But the CFR is much more than just the frame system the the scale system but usually when we talk about adapting the CFR usually people mean the scale system their proficiency system and uh when we look at the scales and the descriptors as I just said they’re a
Common framework they provide a common meta language they are referen levels and they’re unspecific you have to look at your context and the languages you’re working in and your learners and they specifics and the scales are not directly usable in teaching learning or assessment there are no rating scales
You cannot go and rate one performance of your learners with the CFR scale so I think at the heart for me is the analysis of the local context you have to look into the local needs the purposes why is the CFR adapted what’s behind it what’s the rationale what’s a
Learning context what are learner specifics what’s the learner group what are the purposes of the education what’s the language used domain and what are the local assessment needs and once you’ve analyzed this and established the Common Grounds then you can go into making the CFR work for your context and
You come up with one possible interpretation and uh you have specific local construct specific tasks that may be influenced by the local assessment or learning and teaching culture you have specific criteria and you may adapt certain descriptors from the CFR but you have to adapt them and then you come up
With differences to the original CFR wording and as long as you make it transparent as long as you share with the world this is my interpretation of let’s say B1 then the world can compare and share their interpretations with you so that’s the approach I take usually
When we adapt and we use the CFR and the constructive alignment cycle Sarah started with a policy I left out the policy I jump right into the curriculum but the policies are behind the curriculum the CFR can inform your learning and teaching goals it can inform which competences
You may want to focus on then you go into the classroom and the CFR can inform the action-oriented approach learner centeredness but it won’t give you the answers you still need competent teachers who then will implement the curriculum in the classroom you may want to assess formatively summatively you
May want to use the CFR to inform your test specifications your task design and you do need training to develop valid tasks that operationalize certain aspects of the CFR and you may also want training for delivery for radar training and you may want to link your feedback
Against CFR inform so that you can actually feed it back into the learning and the teaching and the curriculum revision so the CFR is at the center of the alignment Circle but it is informative it’s not telling you how to do it it can only give you some information towards the elements that
You need to fill and uh if you do a formal alignment process if you want to claim my exam measures student competence at B1 or my textbook is a B1 textbook there is a lot of work behind that and I think it’s worth summarizing it here and um Alti
Alta the British Council ukala anybody else I forgot they all got together P resources and they brought out the CFR alignment handbook and I think that is a really really valuable resource and it complements and widens nicely the existing manual for language for uh linking language examinations because it
Opens um alignment to the whole educational context so and it’s reiterated in the handbook that we have different phases you start with familiarization everybody in the process needs to be familiar with the CFR its categories its levels and then you go on to specification transparently analyze your textbooks your materials your tests
In terms of relevant CFR categories tell the world why you think this is a test or a task or whatever material targeting a B1 learner then we need standardization we need to train everybody involved so that we come to Shared understanding what does B1 mean for us in which context and that’s not
To be doable without training and benchmarking and particularly in high stakes contexts you may want to do an official formal standard setting study where you determine the cut scores yeah where you determine the Judgment when has a learner reached a certain level when has a learner passed a certain exam
And that’s a formal Endeavor with a lot of resources and expertise involved and along with all these phases comes validation you want to collect evidence for all the claims you make and ideally make this transparent bring it out into the open into the public let exam users
Textbook users see what you have done to support your claim so that’s the formal alignment procedure and uh that sets the scene so to speak in this context I now go into excuse me into two different um local contexts where we adapted the CFR for quite different purposes but
Interesting we interestingly we came across similar challenges and similar Solutions so the first context is Cuba um where we start with policy the ministry of higher education released a policy in 20156 they wanted to open up internationalization and they wanted all ba students to leave University with a
General English proficiency at level B1 they adapted the CFR and that was before they contacted me I have nothing to do with this they adopted it and um they seek International recognition they want a certification that is formally aligned to the CFR and uh here training needs
Rise the Cuban um key players have to be familiarized with the CFR they have a curriculum group they developed the curriculum for higher education and we have cross references with this group but I wasn’t involved there but we have people on both projects and I was involved with developing the high stakes
Assessment that should be aligned to the CFR we started in 2017 the ministry dedicated about 40 teachers from all across Cuba and you see uh maybe you don’t see see on the Cuban um um um um map we have red dots for all the different regions and Cuba is a large
Country it’s more than th000 kilometers long and so we had teachers from all the different universities a brief outline of the project it has three strands we have one big strand of teacher training we analyze the local requirements we provided training for the theory and practice in testing and assessment and
How we aligning exam and what it means to be at a CFR level and what a CFR is and all the teachers in the projects were to become trainers in their local region because we wanted to bring the assessment approaches early on into the classrooms and the second strand is
Developing the exam and that’s of course part of teacher training so we train all the teachers and all practical aspects test specifications tests items tasks rating scale we did benchmarking studies and r a training with them and we will formally align to the CFR in the next
Round and along with that we have a research strand to look into all aspects of validation into alignment studies and we have already done a bit of research on how we develop the rating scales and into assessment literacy and once the exam is life we will accompany its impact and the
Sustainability the overall plan we started with planning the training and the project and we started with training then the teachers developed tests and uh tasks and rating scales until 19 we sought external and internal feedback and then covid hit and during Co they still managed locally there to do a
Little round of pre-testing we did benchmarking exams they’re currently revising the tests uh and compiling an item bank and now we’re hopefully going into phase two where we want to do a larger scale piloting and IRT scaling so that we then can align formally to the
CFR I may Reach Out to one or the other of you when I need help for the the standard setting judgments and then they will implement the exam into the classroom and monitor its impact on teaching and learning that’s the plan and we’re halfway there and a little bit
Into the local people we did the first Workshop yeah maybe you can see some of the colleagues here in 2017 and Ila supported this workshop and then we went to LGC with a group where I met Steve saachi the first time in bota that was all a long time ago then the teachers
Went off into Regional groups to do the task development we came back to another Workshop we did about two a year and the teachers worked in between in smaller groups and so far we managed nine workshops um we started face to face then we went uh online because of Co the
Last we tried hybrid but it’s a bit challenging in Cuba with the resources and we did evaluate continually where we are and whether the teachers find it supportive and we had two big evaluation points I’ll talk about that a little bit later we had two big questionnaires to
See whether the teachers uh gained enough knowledge and whether we met the teachers needs so I give you just a little example what we did in Cuba we’ve developed a lot too much that I can share with you here but uh with regard to the test specs we had three major
Sources it’s the curriculum that was locally developed and that was already referencing to the CFR then we went into the original CFR scales and as soon as the companion volume came out we went into those scales and we use the Pearson uh scale of global English the global
Scale of English because they are geared towards higher education context and they also cover the plus levels and just one example for uh the reading um test specs the specific purposes that the test is to assess is the student students can understand the gist of straightforward factual texts related to
Topics they are familiar with you will recognize the CFR that’s for the overall reading comprehension scale they students can scan longer texts um to locate information that’s from reading for orientation but also mentioned in the curriculum we have uh the understanding main ideas and supporting details that comes from the global scale
Of English and the curriculum and we outline nine different specific purposes for reading and we do it for all the other uh skills as well and they are then operationalized into test tasks and the tasks are specified in terms of these specific purposes so that we can transparently show that’s what we think
This task is assessing that’s uh all I can share here for Cuba we come back to Cuba in a minute when we look at how the teachers perceived the training and the delivery let’s move to Breman now the languages Center which I direct there since 2015 completely different context where
In a city state in the north of Germany and we cater for four universities we have about 6,000 students who attend our courses every every year we cater for about 20 languages plus minus and when I came we realized we needed to reform the summative achievement tests we have
Semesters of 14 weeks of teaching so you cover maybe half a CFR level but you cover it not in a comprehensive way you can cover what you can cover in 14 weeks so we are facing not proficiency but achievement tests and uh something said something here Dev AO power
Off I got 53 seconds left all right I hope we manage without this exploding so in briman we use the CFR as the framework we want to constructively align teaching learning and assessment and we take an action oriented approach towards learning and teaching ah Here Comes help whoops and uh give me a
Second so we try and align the learning objectives the teaching uh outcomes and the assessment tasks in a transparent way and uh yeah super many thanks um we started in 2016 we had up to 25 teachers um on a voluntary basis there’s no point me as the boss telling
Them you have to because if they don’t want to then I better not have them on board but we managed to get PE people on board we have three coordinators and researchers and uh we did training similar to Cuba we had workshops we had external input we had teacher
Conferences and also working groups and a similar range of topics but the challenge was multiple languages we have have as I said about 20 different languages what steps did we take in Breman we we chose a collaborative approach between teachers researchers and coordinators and we started re revisiting the learning outcomes and the
Module descriptions we then develop test specifications for all skills and all levels we use German and English that’s our two working languages but the test SPS are valid for all the languages we teach we tried to use the CFR wherever we found it feasible then we developed
Sample tasks and we tried them out low stakes in the classroom to gain insights to the work what do the students think and we specify them all in terms of what CFR features they operationalize then we developed assessment checklists in Cuba I said we have rating scales covering B1 bit above
A bit below in Breman we teach only 14 weeks and the rating scale wouldn’t work I’ll show you an example why it wouldn’t work as a rating scale so we chose a checklist approach I’ll show you one in a minute we then implement it in the
Classroom and we do a did a continuous evaluation we asked the teachers the coordinators the students where we are going whether we on the right track and what they need I’ll talk about that in a minute what was the overall design similar to um Cuba we started with a
Needs analysis teachers and students came up and said this is not working we don’t have comparable exams so we had a real need to revisit then we started with seminars and workshops we did the first questionnaire early on to check what’s the class CL practice on assessment what is actually happening in
The center what needs and concerns do the teachers have then we continued with the workshops and added working groups to actually go into test development and uh specification development and benchmarking and then once a year we had a questionnaire and the other half half year we later we had an interview focus
Group interviews to see how teachers implemented in new practices and what they needed so that we actually could gear the uh training towards the teachers needs and concerns and that went on until 2019 and we had three major questionaire events in these years and I’ll talk about that in a
Minute and I’ll give you now an example for briman if we look at English for academic purposes level B1 the first course on level B1 the overall learning outcomes which are then broken down in further specific uh learning outcomes are also geared towards that CFR level they can read and understand General
Texts related to their own subject listening General discussions clearly structured academic presentations and speaking they can present and exchange factual information participate in academic discussion of course always in their own field writing they can write clearly structured text General topic simple academic texts in their own field and for writing assessment I’ll share
With you part of the checklist we have a checklist which contains two parts one is Task fulfillment and the teachers can add the specifics because it’s an achievement oriented exam so we have to gear it towards what’s happening in the classroom and for the language competences um this is the checklist for
Cohesion and coherence for vocabulary and for grammar and wherever we could we resorted to the CFR descriptors we tried to stick as close as we could to those proficiency descriptors we are aware their proficiency descriptors and we married them with our ACH achievement needs wherever we need it we added space
For the teachers to specify what kind of vocabulary what kind of grammatical structures did they teach in the course and you see we have five um levels in the checklist from very good to good satisfactory sufficient and fail that’s the grand scheme in the German University system and these five levs
Are broken down into 11 grades I’m not going there it’s crazy but I have to use the 11 grade system so the teachers use the checklist they share them with the students and based on the checklist they come to an overall Judgment of the overall grade and we did Research into
It and it does work and it works better than the intuitive grading that they used to do so that’s the German context with a contextual constraints that we had to address now let me sum up the challenges that we faced when adapting the CFR and I exemplify it for the checklist and the
Rating scale Development Across both contexts because we found quite similar challenges we used a descriptor based approach towards checklist and waiting scale development we used a CFR and CV descriptors any assessment scales that we found in CFR aligned exams and the global scale of English we came up with
The first draft then we used an intuitive approach sorting exercises with the teacher sort the descriptors back into the criteria and the level so we see whether we reach a general understanding what the descriptor is actually describing then we did empirical approaches with data collection on benchmarking exercises
Discussing where we digressed and what we have to change so that everybody is on the same page and the challenges here were an abundance of scale in the companion volume it was it was really hard work in the instrument you find similar aspects described in different ways have
Fun so that was a challenge um we found different categorizations in the companion volume and in our context so we had to do something about that that’s not the CV’s fault you just work in different context with different criteria the plus levels aren’t always described and we found inconsistent
Wording across the scales and sometimes within the scales across the levels and again I’m not trying to say this could be done better I know the complexity of the project behind uh compiling the CFR descriptors and we just can improve and continue improving so what were our solutions to these these challenges and
These are published I can give you the references so you can read up in more detail so first of all we reorganize the descriptors from the CFR into our local assessment criteria then we adapted the descriptors we changed the wording to make the levels coherence for us in our
Local contexts we added the scriptures from other sources wherever we found nothing suitable in the CFR and then we added and adapted the scripts to account for the local context and that was both for the broad levels the Criterion levels and the plus levels we were free to change and we documented
Clearly where we changed and where we relied on the CFR now lot of input I know let’s briefly really sum up and briefly uh look at what the teacher thought of the training approach we used a responsive training and evaluation approach we did a needs analysis we delivered the first
Round of training we evaluated we had it evaluated by the teachers we adapted it to suit their needs and their their concerns and that went on and on and on and uh we evaluate we used questionnaires I’m not talking about the interviews today that’s too much for one
Session we focus on the questionnaires today so we wanted uh to look at the outcomes of the training program in both countries and we had open items asking the participants for their perceptions of the program and their needs and we had closed items so that we could look
Into the learning gains and we based that on a publication by milal and we had three points in Breman and two points in Cuba with different numbers of participants and unfortunately I can’t do a longitudinal study we can discuss that later why that didn’t work we analyzed these questionnaire answers for their
Perceptions of the feasibility of the program we did a Content analysis for common themes across both contexts and we looked at learning gains in a descriptive analysis and let’s come to the perception of the training from the open answers the color collaboration was perceived as really really helpful in
The uh working groups the teachers said the interacting sharing and giving feedback was really helping them to develop their assessment literacy and to develop tasks and in Breman across a range of languages in Cuba across a range of universities and I’d like to share one answer from Cuba from the
First questionnaire I learned from my colleagues in my regional group negotiation respect and knowledge is so important in our group so they really took to that working group and that Collaborative Learning and sharing and then we saw positive effects of changes of assessment practices also in the classroom in both context also in
Cuba where we are looking at a large scale exam but the teachers took it to their classrooms and in Breman I have two teacher quotes the one teacher said the discussions in the working group have an effect on my teaching and another teacher in that focus group says
I think the most important effect is that our teaching and our expectations are more or less comparable and are the same that is an important thing for making sure for standardizing yeah they serve Us in all the teaching that we have so all the discussions about assessment had an effect on their
Teaching and we did link it back to the teaching all the time and uh the Practical Focus was stressed as very very important and in Cuba that we wanted the teachers to become trainers so we asked them what was helpful what would they need for training and uh we had from the first
Questionnaire the feedback that the colleagues motivation the engagement in what I was bringing the New Visions of the teaching learning process the tools that I received on the trainings which are now being spread and the feedback that teachers said that was helpful to act in their local context and pass on the
Knowledge what were the challenges of training it didn’t went it didn’t go without challenges and in bremond we had very different challenges from Cuba um the participation rate was a challenge in in Breman I have a lot of hourly paid teachers I cannot pay them and they’re
Not coming for training so that was a challenge timing was a challenge because in my languages Center we have one course or the other running all the time so we found now a nice way of reserving two days per half year per semester to do training and to work together and uh
Finding the right balance between theory and practice and I did learn that with teachers start with practical aspects and the theory sneak it in when needed don’t start with Theory I’m a more theoretical person as you may have guessed in we had very very different challenges it’s travel and
Nexis the country is really in dire needs and uh that was a real challenge the technological limitations for online collaboration proved The Challenge and we had stakeholders on board who are not in the with a with a language teaching background so we had to get them on
Board and that was challenging but I think we managed um just briefly looking at the learning gains in bran um I’m just summing up here now I’m not going into detail we I base it on the last questionnaire and I’ll just give you the five top five learning gain areas that
My teachers reported from the training events it was constructive alignment yeah so the training did work they learned a lot about constructive alignment and then test specifications evaluting evaluating tests developing checklists and theoretical aspects reliability validity and authenticity so theory is not not all without its uh
Place and and from the working groups of course it was more practical aspects test specifications checklist and test items development but constructive alignment and implementing the CFR for assessment are amongst the top five so I hope the training did achieve some of the goals and overall in Breman did we
Reach the goal to increase authenticity of the assessment transparency and comparability we asked students with the writing task particularly and they said that they have a higher level of authenticity now so I think well that’s good that’s what we wanted the teachers and coordinators say the alignment to
The CFR of learning teaching and assessment and the test specs lead to more transparency and comparability that’s good but the new ways of exam development of using the checklist it did take us a lot of time and a lot of effort so we have to build that into the
Program and for our external accreditation body that’s unicer the alignment also adds transparency to our exams and Mak life easier for everybody so for bremond overall I think it worked nicely if we look at CUA we are still in the middle of the of the project we had
The top five areas from the latest questionnaire and they had a fourpoint like it scale one I didn’t gain anything to four I did gain a lot and Cuba the people are very polite and they were very very happy and grateful that I came and worked with them so maybe we have to
Take that with a pinch off maybe it’s too high for what they really learned but maybe not so they learned assessing productive skills development of test specifications application of the CFR and knowledge about the CFR and that was in the workshops and when we look at the
Regional groups the learning gains are a bit lower but still application of the CFR and knowledge about the CFR rank amongst top five so I’m very happy with that outcome and I hope we can continue and I’m going now to the last part of my presentation to the implications what
Can we take away is there anything we may generalize beyond the two very specific contexts in general if you like to embark on such a project I may State the obvious you may all be trainers and very um um um um competent in doing that but familiarization with a local context
With the institutional constraints with the key stakeholders and with existing professional development initiatives is ever so important that you hit the ground running and you know what’s going on in the different contexts then including all the stakeholders from day on day one onwards and make it possible that everybody brings their expertise it
May take more time it may take a several rounds of discussion but it’s important to have everybody on board otherwise you end up with a nice exam and nobody wants it so that’s very important and the time we have to schedule a lot a lot of time
It took all longer that I had envisioned and also the ministry in Cuba had envisioned but we have learned that things take time and if you don’t take everybody on board it’s not going to work and the collaboration as we heard from the teachers the regional groups the collaboration is ever so important
Create spaces online or in in the real world face to face that’s important that everybody can work together see each other and form a community of practice and the experiential starting point the the the Practical starting point is is really really important make it work that the teachers see why they’re doing what
They’re doing and how they can take it into the classroom and provide Theory as you go along and some more specific focus on the CFR how can you implement it what can we take away um I think the familiarization exercises are so important and Sarah pointed out the
Newly structured homepage with all the resources make this available in advance let people do it in their own time in small groups that they can familiarize themselves give them a bit of guidance but allow them to use their own approaches and their own ways with a bit
Of guidance but give time in advance that people aren’t overwhelmed with all the materials that are already there and existence choose a collaborative approach let them bring their um needs and their ways of of of uh working with these materials to the table Hands-On activities are ever so important that
It’s not only Theory here’s a scale and there’s the scale but give them the scales break them down let them work with with the scales and pre-select the scales there’s too much in a CV for a workshop teachers will just back off cuz it’s too much get together with a
Smaller group of organizers of responsible key stakeholders and select a few things that you think are most relevant in that context um use other sources the CFR doesn’t hold all the answers we have a project another project of integrated assessment and in the CFR if it says can
Summarize a text a test I can summarize a text in a summarization scale it doesn’t help me because I want to know what makes a good summary so you may have to look into different scales and different sources of information document everything transparently so that in the end you can
Still say this is where the CFR comes in and this is what I needed in addition and uh Cod develop learning goals curricular and assessment goals do it in in a group do it in different groups but make sure they collaborate with each other and when it comes to developing
Learning activities teaching tasks and assessment tasks also think of them all in a in in a comprehensive way don’t do it in different uh silos and I think that’s all I can share in that little time and here