I came across an interesting research paper from the Journal of Educational Technology & Society on the subject of online assessment last week. This is a key topic in e-learning as along with courseware it’s the most widespread formal use of e-learning in organisations.

Marking Strategies in Metacognition-Evaluated Computer-Based Testing looks at the effect of marking strategies. Catching the title I thought this would be about evaluation of tests and different ways you could go about scoring an online assessment, but not so. Marking in this case refers to the added functionality in an online assessment for a person to mark-up questions they answer as ‘uncertain’ during a test, with the additional option of coming back to it later before they submit their answers.

In addition to this marking-up facility that was at the core of the research the authors also designed a more comprehensive form of feedback, which they call metacognition-evaluated feedback (MEF). Simply put, MEF integrates students’ answers with their mark-ups, together with explicit feedback about a specific choice.

The study in question featured ninth grade (15yr old) participants taking a 30 question multichoice exam in vocabulary and reading comprehension. The authors looked at answering two questions:

• Does mark-up improve student scores?
• Does MEF encourage marking-up and review behaviour?

The authors found that:

• student ability was crucial as to whether marking up improved test results
• mark-up only improved ‘medium’ ability student test scores. Students with higher or lower ability didn’t show any noticeable improvement
• mark-up increased efficiency and effectiveness of self-managed learning
• mark-up did encourage across all abilities the behaviour of reviewing and reflecting on answers
• mark-up facilitated greater metacognition by someone taking the test, actually questioning their own learning and understanding
• MEF encouraged students to use mark-up skills more frequently and to review answer-explanations of test items

Incorporating this richer MEF at the end of the test certainly appears to offer a degree of formative assessment of student performance that feeds in to construct the next bit of learning needed. As the authors state: “Students made predictions about their test results and then observed what happened to check their predictions. If their predictions failed, they tried to determine how these mistakes occurred and then solved their problems.”

The mark-up system implemented was simply the option for a student to mark any given answer as ‘unsure’. This covered off a number of possibilities: 

• sure correct: the student believed they were right, and they did indeed get the answer right
• sure incorrect: the student believed they were right, but actually got the answer wrong
• not sure: in this case the student wasn’t sure, and may have got the answer right or wrong

The results the authors obtained here also provided an insight and/ or confirmation of who the low performers were likely to be: they generally fell into the ’sure incorrect’ category.

As ever effective feedback following submission of results is crucial. As the paper states you should “provide useful adaptive feedback so that students [can] understand their performance, clarify their mistakes, and increase their learning motivation.” In addition to take into account the lowest performers feedback should be written in a way that will “encourage review behavior”, with the suggestion of making this feedback “adaptive and detailed” with specific examples.

This paper provides a good case for improving the basic multichoice assessment found in courseware, as well as providing the design features to include. My experience of courseware is that the majority of embedded tests and assessments aren’t designed with a mark-up function. It certainly looks like it would do no harm to do so.

I’ve just been reading an article on rapid e-learning (more of that in my next blog post), but it got my thinking - which is always a good thing.  There are some things I see time and time again in terms of material presented for ‘conversion’ to
e-learning (well, courseware actually if I’m being specific). A lot of the time this material is presented by an organisation as something they’d like done ’rapidly’.

• More often than not the content structure isn’t that great
• There’s simply too much of it
• It focuses on facts, rather than the learners’ and organisation’s needs
• Its tone and style is totally unsuitable – usually a highly passive ‘formaleese’
• And it’s ‘why use an image, when a 1000 words will do?’

And this is just the tip of the iceberg, but there’s a bottom line issue: you’ve gotta sort out the basics no matter what you’re going to do with your content. So what follows isn’t necessarily best practice for e-learning, I’d say it was best practice for content -and it’ll make turning it into some form of learning/ e-learning material a whole lot quicker and easier.

Sort out a structure
All this needs to be is a simple flow that allows a learner to ‘get it’ and fit the details in with the big picture; maybe content wrapped around a timeline/ scenario. Don’t worry what. As long as it’s got a basic, clear structure – that’s good.

Cut it…
Aim to cut the text in documents to 40% of the original word count with ZERO loss of content. If all you’ve got are PowerPoints… well, first I hope they’re visuals because that’s a good thing. If they’re bullets… that’s not so good. Only thing here is to note down what’s said ‘over’ these PowerPoints when used - and then cut, cut, cut!

Then cut again!
Simply ditch the ridiculous. Take the dreaded listing of procedures for example – DITCH! I fall back when I see this. If you want it as content then give it a separate page to itself, or make it a printable take-away in a pdf so that whoever wants to can have it to hand when they go to DO – but think about what a learner is going to be able to DO based on this list. If nothing – say bye bye.

If it’s an actual procedure – video it! Much more powerful than a written list – I won’t bore you with the details, you get the picture…

And speaking of pictures…
Use images. Say it with pictures. Again, it’ll cut down words and enhance understanding when done appropriately. Slap any words that describe your picture, or even highlight your data, back into the image. And don’t repeat what the image does well in cumbersome text somehere removed fom your image – resist!

The story so far
OK, so we’ve a bit of a structure, 30% of the original word count, and nice informative (and more memorable) images… time to make it snappy by making your remaining words earn their keep. If you’ve not come across it before you should check out an essay by George Orwell that could support the point of view that he was the first writer to offer a style guide for online text! His 1946 essay ‘Politics and the English Language’ has six great tips for knocking-up some e-learning prose:

“Never use a metaphor, simile, or other figure of speech which you are used to seeing in print

Never us a long word where a short one will do

If it is possible to cut a word out, always cut it out

Never use the passive where you can use the active

Never use a foreign phrase, a scientific word, or a jargon word if you can think of an everyday English equivalent

Break any of these rules sooner than say anything outright barbarous”

One more bullet I’d have added to Mr Orwell’s, but maybe a bit too much for 1946: make it chatty. Make your writing, your audio too if you want, warm, friendly, personable – not HAL from 2001.

Now I’m not advocating this for everything, not all the time. It’s not about dumbing down. It’s about simply making things clear and easy so that the channel doesn’t get in the way of the content. There are loads more tips about how to write in a learner/ web friendly way – but that’s something you can chase up if you need to.

You can add some quizzes, interactivity, or even turn it into a simulation – that’s all fine. But that comes after you’ve got your content sorted. Once that’s sorted everything should simply flow into its place, no matter what the pedagogy you decide. A waste of effort this is not – you will need basic content whatever the approach.