ESL assessment

Rubrics: a basis for qualitative feedback

Rubric-1

This is one of those things I wished I’d learned about years ago, because it would have made my own life as a teacher so much easier.  I’ve learned about them now, however, so I’m shouting my joy from the rooftops.  Hurray for rubrics!

What is a rubric, one might ask.  A rubric is a means of giving detailed, qualitative feedback to students regarding a given product.  It contains concrete descriptions of the criteria for a well-completed product.

There are different sorts of rubrics.  I’ll explain two sorts of rubrics, using generic sample rubrics I wrote for this purpose.  The first one is a criterion-referenced rubric, and the second one lists success critera for different levels of ability.  I wrote these rubrics for a group project in which the children had to create posters demontrating what they’d learned during the last unit of learning.  They were to use the new words they’d learned in correct sentences.  For the sake of simplicity, I’ve omitted criteria for layout and presentation.  I’ve only included the very basic criteria of content, language, and process.

The first example here shows a criterion-referenced rubric of the sort most people might use.   For teachers, this is an easy form of marking, since the standard for the work remains the same, no matter the ability level of the child.  Also, the criteria for success are clearly described, so all children can know ahead of time what he needs to do in order to pass the assignment.  Another positive aspect is that children get differentiated feedback per criteria heading.  On the downside, it’s perfectly possible for a child to fail the given assignment, as the criteria for success are of the one-size-fits-all variety.  There is no way to allow for differences of ability when using this sort of rubric.Rubric01

That problem can be solved by using a different setup.  The example shown below demonstrates a way to differentiate feedback per ability level.  For instance, if you work in a classroom with a broad difference in language ability, then it might be nice to set up the assessment so everyone has the chance to succeed.  At the same time, this rubric also allows you to set up minimum success criteria per ability group that are just above the actual level of the children so that each child is pushed towards a higher level of ability.  This is called differentiating in output.

In this case, the children should know ahead of time what group they belong to, and they understand that they each have a choice: to succeed at his own level, or to work towards success at a higher level.  The term minimum success criteria is critical here: children should reach the minimum level indicated, but may also choose to work towards a higher level.  Sometimes, if a child needs, he may choose to work at a lower level, but that is a pedagogical decision that you and that child can discuss.  In this rubric, the indicators for ‘process’ are the same for all children, since it would be reasonable to expect all children to work on their social development irregardless of their language development.

Rubric2

Rubric for differentiating in output.  Note that success criteria for the ‘process’ are the same for all children, regardless of ability level.

Of course, no matter what kind of assessment format you use, it’s important that children be aware of the criteria for success so they know what they need to work toward.  As a teacher, I post my rubrics on their electronic bulletin board at school, so students know what they can expect.  It helps them focus their work and gives them the space to make informed decisions when it comes to their own learning.  It also means they have no surprises when they get their grades back, which makes a big difference for everyone involved.

For more information on the use of minimum success criteria and rubrics, feel free to have a look at this site: http://www.assessmentforlearning.edu.au/professional_learning/modules/success_criteria_and_rubrics/success_criteria_landing_page.html

What other topics would you like to see covered on this blog?  Please let me know!

Not all equal, but moving forward all the same

Standardized-testingAt one time or another, we teachers are confronted with the need to assess our children’s learning.  Many of us have thought long and hard about the use of a single, standard test to find out what our children have learned.  There are, of course, things to be said in favor of standardized testing: one gets a view of how children perform compared to other children their age.  That can be very valuable information, providing a basis for differentiated instruction.

However, children who are the weaker learners in the class also need a moment of success, of being “good enough” without always being last in line.  When will these children be allowed to feel like they have learned enough, that they are making progress?  Earlier, I wrote a blog entry about writing group plans for long-term planning.  Based on these semi-annual plans, the language goals for a given theme can be determined.  After that, though, how does one determine when each child has actually made progress at his or her own level?  This is when differentiated outcome rubrics come in handy.

Part of what I do when designing a new theme, is determine which words must be learned by everyone, which words most children should learn, and what words are challenge words.

  • Basic vocabulary: Words everyone should learn.  These generally transfer easily from the mother tongue, are shorter, and used relatively often.
  • Extended vocabulary: Words most children should learn.  These may transfer easily, but may also be longer and used less often than the basic vocabulary.
  • Challenge words: Words some children should learn.  These words may be difficult for a number of reasons, they may be spelled unusually, be seldom used, or longer in length.

Next, I determine some form of end product that the children should work toward in the course of the theme.  In the example below, I want them to do some kind of oral presentation about something we’ve learned.  The weakest children are the the group “Cat”, the strongest are in the group “Chipmunk”, and everyone else are in the group “Bird”  (no particular reason for those names, incidentally, I’ve used “skateboarders”, “snowboarders”, and “kite-surfers” in the past as well).

Finally, I determine what concrete language they should be able to produce for this product, based on the semi-annual plan.  In this differentiated outcome rubric, I show what the minimum expectations are for a presentation that is “good enough.”  Each child knows what group he or she belongs to, and therefore what kind of output is considered “good enough” in order to be considered successful.

In this example, the “Cats” work towards a short presentation in which they use short sentences correctly applying the basic vocabulary.  There is space for some hesitation during the presentation.  “Birds” need to use the extended vocabulary correctly, in longer sentences,with better pronunciation, and so on.

Cat (intensive) Bird (basic)

Chipmunk (talent)

Vocabulary Uses basic vocabulary correctly Uses extended vocabulary correctly Uses challenge vocabulary correctly
Sentence length 3 to 4 words 4 – 7 words 5 – 10 words
Speaking Some errors in pronunciation

Some hesitation

Few errors in pronunciation

No hesitation

Clear diction

Confident

No hesitation

Of course, it is perfectly fine if children decide to try out a more difficult level of work.  Some children get a real “kick” out of performing at a higher level than expected.  Some, however, might wish to try out a lower level, and that’s fine too.  There are plenty of children suffering from performance anxiety who might feel more comfortable operating at a lower, more easily-achieved level.  Others might try out a lower level for fun, find it too easy (and therefore boring), and return to a more challenging level of work.  The important thing is, however, that each child be allowed to succeed at a level appropriate to his or her own level, and a differentiated outcome rubric is good for just that.

Flow charts: visualizing the 20 questions game

4c3cb8e281f009140a83546b1bdd72b9“Does it have legs?” a child asked.  The child in front of the class answered quickly, “yes, it does.”

“Can it fly?” another asked.  “No, it can’t,” was the answer.

It took a little while and a number of yes/no questions, but soon the class knew the animal’s secret identity: a giraffe.

This guessing game is a favorite among many ESL teachers, including myself.  The question I found myself asking was how to make it more challenging for the older learners.  Also, how could I change the format of this game so that every learner could participate, even the shy ones?

 

It took a little searching, but I soon had a viable answer: flow charts.  In essence, a flow chart works just like the verbal version of the guessing game, but visualizes the process of elimination involved.

There are different ways a flow chart can be used in the lesson.

flow-chart-animals-pictograms

A sample visual flow chart for younger learners.

For younger learners, one can make up a poster-sized chart with pictograms on the question blocks and pictures of the vocabulary being sorted out.

For middle learners, one can make a flow chart with simple questions.

The older learners can make up their own flow charts to try out on classmates.

flow-chart-animals

The same flow chart, but now with questions written out.

Flow charts allow children to visually sort information along the lines of simple questions.  The example provided here is about a few animals, but with a bit of creativity, one can help children make their own flow charts about any number of topics one teaches about, for instance modes of transportation, clothing, food, weather, hobbies, and jobs.  By having children sort the information in this fashion, they are also activating their logical-mathematical intelligence, broadening their learning.

Flow charts can also be used to assess a learner’s understanding of the concepts taught.  Can he or she ask questions effectively to find out what the secret word is?  Can he or she formulate the questions correctly?  Can he or she create a flow chart that includes all of the concepts learned during the last few lessons?  These are just a few possibilities that come to mind when connecting flow charts to assessment of our learners.

The examples provided here are, of course, rather straightforward and very simple.  I suppose children in high school or adult ESL learners could make more intricate examples, for example to describe their day or how they prepare a meal.  I wonder if others use flow charts in their ESL classrooms?  If so, how?  I’d like to hear your ideas.

 

 

 

Contemplating portfolios

Two children sit at the table, ready for their role play. 

“What’s the matter?” one of them says.

“It’s my arm.  It hurts,” the other answers.

The two of them carry on for another exchange or two, and then look at me.  “All done!” they grin.  I turn off the camera, and give them a thumbs up.  “Great!  That’s for your portfolio!” I tell them.  

By now, the children know to expect me to show up with the video camera every once in a while.  We make recordings of their work which then go into their portfolio.  It took a few tries to figure out a workable portfolio, and quite frankly, I’m still working on it.  But first, a short history of what I’ve tried out…

cefr_head

The first thing I decided was that I wanted the children’s portfolios to be related to the Common European Framework of Reference (CEFR).  The CEFR had printable checklists that I could easily use to keep track of children’s learning.  From there, I had to decide how to document the children’s learning.  Ideally, I wanted to have some tangible form of proof: videos, pictures, recordings and the like.  A paper checklist alone was not going to be sufficient, that much was clear.  

My thoughts turned to digital documentation.  I scoured the internet for a child-friendly language portfolio.  The closest thing I found was the “Europees Taal Portfolio” (European Language Portfolio).  I decided to try it out with the children.  

europeestaalportfolio

This online language portfolio had a lot of good aspects to it: nice colors and design, space for documents and images, and checklists the children could fill in for themselves.  It also had room for improvement, such as: videos and sound recordings could not be uploaded and the teacher could not add documentation to the children’s portfolios.

After a year or two of working regularly with children on these online portfolios, I had to throw in the towel.  Too many children lost their passwords, the suggested activities for portfolio development weren’t always complete, and to top it off, I accidentally erased my own, years-old portfolio with a single click of the wrong button.  Recovery would cost an entire day, so my portfolio – including any access to those of the children – was scuttled.  Soon after that, the entire project was unexpectedly taken off the air, so all of the portfolios my children had built were gone as well.  

As a result, I renewed my search for an alternative.  Until now, I have yet to find anything that I can really use, and so I find myself making do with an external hard drive and a cd-writer.  Every year, I make certain to get two examples of work from each child.  This work is kept on the external hard drive.  When the children graduate or leave school, I burn their work onto a cd-rom as a good-bye gift.  

video4What kinds of work do I put into their portfolios?  I like to have a balanced sample of reading, writing, speaking and listening skills.  Writing, obviously, is the easiest sample to get: I scan samples of their writing from their project notebooks or save copies of power points.  For reading, I make a sound recording of the child reading.  For speaking, I make videos of the child doing a presentation or taking part in a role play.  I’m still not sure what to do for listening, so that’s something I think about on a regular basis.

I’m still on the lookout for a child-friendly online portfolio that actually allows all forms of digital documentation, so if anyone knows of anything I could use, please let me know!

ESL and the long-term plan

Life-Plan_1

Twice a year, I sit down to write out a semi-annual plan for my teaching.  My table is strewn with all sorts of documents: summaries of the results from the latest round of assessments, lists of children per class, the books I use for each class, and copies of the old semi-annual planning.  A cup of coffee and my laptop complete the picture, and I know I’m in for a long, but productive, sit.

Why would I put myself through this, I’ve wondered.  I could, of course, just follow the book.  Loads of teachers do that, every single day.  Why wouldn’t I just do the same?  And I must admit, it’s a bit tempting to do just that, sometimes.  But then I think about how valuable this plan will be, how it will inform my teaching, and point out the real goals of my teaching in ways no textbook can.

What do I put into each semi-annual planning?  The information is compacted into a table, and usually fits onto one or two pages.  First, I’ll briefly describe what goes into each column, before going into details.

Heading: basic information.  Which class, the period covered by this plan, name of the teacher, subject name, basic topics we’ll be covering, and which book(s) we’ll be using.

Column one: names of the children, listed in order from strongest to weakest.  In general, the strongest 25% of the children become the “talent” group, the middle 50% of the children become the “basic” group, and the weakest 25% of the children become the “intensive” group.  Each group has its own needs that I will need to meet via differentiation.

Column two: language goals for each group.  What listening, speaking, reading, and writing skills do I think the children should learn in the next few months?  I start with the basic group, and tack on a few extra goals for the talent and intensive groups.

Column three: didactic activities.  What activities will we undertake to insure that the children learn the skills listed in column two?  Will these activities be class-wide, small group, paired, or done individually?

Column four: time.  How much time will be available to complete the lessons?

Column five:  pedagogical measures I plan to take to insure that the children are participating – what kinds of feedback will I give and when, what kinds of questions will I ask and how will I stimulate the kind of language use I want the children to learn?

focusWhile I’m filling in this table, I find myself asking and answering hundreds of questions, the process of which informs and focuses my teaching for the months to come.  Another side-effect of this planning is that I find myself energized and inspired with exciting plans for great lessons for children involved in their learning.

One of the first questions I have to answer, of course, is how to group the children.  It’s never, ever, ever, a precise 25 – 50 – 25 split.  I take into consideration where the children were in the last plan: did that “talent” child really do well in that group?  Or did he do better when working with the “basic” group?  Will that “intensive” child do better if placed in the “basic” group?  Or did he really start learning because of the extra time and attention I gave him with the “intensive” group?  This is where the results from the latest assessments come in handy.  How much did each child actually learn during the last few months?  In other words, did that child flourish in the group he was in last time?  If so, and if not, what factors contributed to that?  Will keeping him in the same group help him, or will he need to be placed in a different group?

goalOnce the first column has been filled in, the rest, I find, is quite easy.  I then pull out the tables in which the assessment results for each child are summarized, and draw lines separating each group.  I look to see what skills the entire group is missing, and what might be a realistic next step.  Where can these children go next in their development?  In general, I choose 7 to 8 goals for the basic group, spread over the skills areas.  I add on 2 or 3 for the talent and intensive groups.  I have found that any more goals is simply too many for me to meet in my limited time, while any fewer isn’t ambitious enough.

fb-inspireThe third column, learning activities, is the funnest.  What kinds of activating, co-operative learning structures will we use?  What kinds of pages will we use in our project notebooks?  What kinds of role plays will we practice?  This is the part where I get the most inspiration.

The fourth column is the easiest, of course.  How much time do I actually get to teach each class, and that’s it.  Also, am I teaching the entire class for that time, or is there time for small-group work.

reflection-24The fifth column is where I get to spend time on self-reflection.  What kinds of concrete action should I undertake to insure that each child feels safe enough to try speaking in a language that isn’t his own.  How will I challenge the stronger children to strike out and learn even more English than they already know.  How will I encourage the weaker children to take part, even when they don’t know all of the words or feel uncertain about themselves?  These questions and more poke their heads up while I write.

In the end, I have a document that will guide my teaching for the next few months.  I share this planning with the classroom teacher, so that he or she knows what we will be doing for the next half-year, and keep a copy in my class binder so I can always refer to it whenever I feel the need or plan the next theme.

I wonder what other teachers do, for their planning?

Testing, testing… (continued)

Three children sat at the table, sorting through six laminated cards.  “All right, everybody,” I started.  “Where is frog hopping?”  This is a toughie, since “hopping” in English and “happing” in Dutch sound similar, but the meaning is quite different.  I demonstrate with the frog puppet.  “Look!  Hopping!”  The children catch on, and hold up the picture of frog hopping across the yard.  We continue on, looking at frog eating a fly, frog eating a stick, and frog eating a flower.  “What will happen in the story,” I ask.  The children look at the pictures and start to tell me a story.  I encourage them to use all of the English words they know, resulting in a mix of words one might call “Dinglish,” Dutch and English mixed together.  Then I set up the “walls” – some large books – and pull out my storybook.  It’s time to read aloud so that the children can put the pictures in order.

In this task, I look for language behaviors as they are described in the teacher’s handbook for the ESL program I work with (Note 1).  For instance, can the children understand simple, short sentences?  Can they predict what might happen, by looking at the stories?  Can they put the pictures in the correct order?  And after hearing the story, can they answer questions about the story?

With this task, I assess four separate language behaviors with a group of 2 or 3 children.  Within 10 minutes, I have quite a bit of information already, and we still have time for more fun and games, as I call my tests.

I developed my test after extensive work with the Reynell and Anglia tests.  I noticed that while each test had a number of good points, each also had its weak points which made it unsuitable for my purposes.  Here, I list a few of my considerations regarding each of the tests:

Reynell test, pros:

  1. interesting tasks – the children enjoyed each of the tasks given, the tasks are appropriate for the target age group
  2. norm-referenced – no child can fail, although he can perform above or below the norm for his age

Reynell test, cons:

  1. time-consuming – each test is given individually, so a lot of time is spent introducing each task, and the test can take up to 45 minutes
  2. age limit – the test may only be used for children up to 7 years of age

Anglia test, pros:

  1. efficient use of time – the test is given classically, so the entire class is finished with the listening/reading/writing section within an hour or two
  2. structure – the test is well-structured and easily administered

Anglia test, cons:

  1. criterion-referenced – the test may be passed or failed, but in the case of failing (or superbly passing) it doesn’t give any information about what would have been a more appropriate level for testing
  2. level of testing – all sections tested are tested at the same level, there is no differentiation possible between the levels

I decided I needed a test that combined the good aspects of these two tests, while dealing with the negative aspects.  I ended up with my own system, which I call DRoPP (Digital Record of Pupil Progress).

While testing, I use a paper checklist per child.

While testing, I use a paper checklist per child.

How do I test children?  Step-by-step…

  1. I list the children in order of their ability.  The reason I do this is because I test children in small groups of 2 or 3, and it’s easier to test them if I have similar tasks for all of the children in the group.
  2. I pull out the children’s individual checklists of language behaviors.  The tasks on these checklists are based on the Early Bird curriculum.
  3. I give the group a number of language tasks to complete.  I don’t repeat the stuff they’ve already proven they can do, I only look for new information.  If the children attempt a task and cannot complete it, I put a dot in the space next to that task.  If the child completes it successfully, I put a dash.  I use one color for each assessment period.
  4. After each assessment – 10 to 15 minutes later – I give each child a well-earned compliment.
  5. I input the information from the paper form into each child’s DRoPP file, noting the date of the assessment was.  I often input the information directly into the summary screen, but I may also use a more detailed screen if I like.
The summary screen for a child, indicating name, group, and general development.

The summary screen for a child’s DRoPP file

A detailed list of the language behaviors.

A detailed list of language behaviors

Once the input has been done for all of the children, I run the output program to see just how well the children have done – as individuals, but also as a group.  I use the results to write my group plan for the coming semester.

I will write more about group plans another time…

The program I developed to keep track of pupil progress is now available online, free of charge:  https://sourceforge.net/projects/dropp/


Note 1:  for further reading on the ESL Program I work with:  The Early Bird Curriculum for Primary Schools (www.earlybirdie.nl)

Testing, testing, 1-2-3…

For what must have been the hundredth time, I pulled out the apple, the fish, the key, and a dozen other toy-like attributes.  The kindergartener eyeballed the objects, eager to play.  I pulled out my checklist and pencil, and started: “where is the fish?”  “Where is the chair?”  “Where is the key?”  Each time, the child would point at the object, sometimes uncertain, other times rejoicing in the right answer, gaining in self-confidence each time I nodded at him. So began yet another of one of hundreds of Reynell tests I administered as part of a research project being carried out by my employer.

The Reynell test uses a mixture of attributes that children may handle and colorful pictures.

The Reynell test uses a mixture of attributes that children may handle and colorful pictures.

The Reynell test

The Reynell test itself is well-designed, including practical and interesting tasks for young children in order to assess their listening and speaking skills in English.  The test begins with groups of similar, easy tasks, which become increasinly difficult as the test progresses.  When a child begins to fail at a particular sort of task, the assessor ends that task and moves on to the next set.  If the child fails at that set as well, the test is done.  It is easy, standardized, and informative. The point of the test is to see what developmental age a child has reached in his language abilities; even though a child can perform poorly, he can never fail this norm-referenced test.

This test is, however, also very time-consuming.  Depending on how well a child does, the test may take only 10 minutes, or a full three-quarter hour per child.  After administering the Reynell test a hundred times, I found myself quite ready to throw rabbit and bear out of the window and move on to something – anything – besides rabbit putting the knife under the bed and bear pushing the bed.

One of the dangers of administering the same test a hundred times: the examiner (in this case, me) might start getting bored and make things up, which isn’t allowed, of course, as that would affect the standardized scoring.

Later that year, I administered another sort of test to the older ESL pupils.  This time, there was no teddy or rabbit, but instead, paper, pencil, and a cd with spoken texts.  It was time for the Anglia exam.  

The Anglia exam

The Anglia exam isn’t a single exam; instead, it is a series of exams that begin at a very basic level (A1) and graduate to higher levels of skill (C2)*.  The basic Anglia exam includes listening, reading, and writing skills.  Speaking assessments are separate and cost extra, depending on who scores the test.  The Anglia exam is a criterion-referenced test, which means that a child may fail.  If he fails, then the attempted level was too difficult, but if he passes, then perhaps the attempted level was too easy.  Therefore with this test, it is necessary for the examiner to know two things ahead of time:

1) what exactly is tested at each level (described in detail in the Teacher’s Manual on the Anglia website)

2) what each examinee’s general level of English is (for instance, by using the Placement Test on the Anglia website)

The Anglia exam is a series of leveled exams, starting at pre-A1 and building up to C2.

The Anglia exam is a series of leveled exams, starting at pre-A1 and building up to C2.

Besides that, all of the sections are tested at the same level, regardless of possible differences between a child’s skills in listening, reading, and writing.  The tests are costly, and since children want to pass this exam, the teacher must make a careful estimation of the highest possible test the child will be able to pass, even if much of the examination might be too easy for the child.  

Administering the test itself is simple enough, since that is done classically. The feedback from the assessment is a diploma in which the child (barely) passes, passes well, or passes exceptionally well.  If a child fails, it receives a referral to try again at an easier level.  Personally, I found this feedback to be as effective as measuring the depth of the North Sea with a meter stick:  reliable, but not terribly informative.

A new test

A few years ago, my employee decided it was time to create a new sort of test, an informative assessment that would cover the broad range of ability found at our schools, while being time- and cost-effective.  We spent hours analyzing existing tests, discussing questions like “do we really need to count spelling as part of a listening test?” and “how can we differentiate the material so that we can find a child’s level in each language skills area?”

After that, the test was administered to hundreds of children, as part of the process of creating a normative score.  Basic feedback was given to the teachers, and the children all got certificates stating that they helped in creating this test.  Unfortunately, this test is still not available for regular use, so I am – still – left to my own devices.  Fortunately, I had already been developing my own devices for nearly ten years.

My own assessment

I have been creating my own means of assessing children’s progress in English.  Not only that, but I have developed a system of recording their progress so that I have a long-term picture of children’s development in the ESL program.  I used my experiences with both the Anglia and the Reynell tests to form something more useable for my school.  But more about that in another blog…


* The levels A1, A2, B1, B2, C1 and C2 are part of the Common European Framework of Reference.  More information about the CEFR can be found here:  http://en.wikipedia.org/wiki/Common_European_Framework_of_Reference_for_Languages

More information about Reynell tests:  http://www.ncbi.nlm.nih.gov/pubmed/15587011

More information about Anglia tests:  http://www.anglianetwork.eu/