I came across an interesting research paper from the Journal of Educational Technology & Society on the subject of online assessment last week. This is a key topic in e-learning as along with courseware it’s the most widespread formal use of e-learning in organisations.

Marking Strategies in Metacognition-Evaluated Computer-Based Testing looks at the effect of marking strategies. Catching the title I thought this would be about evaluation of tests and different ways you could go about scoring an online assessment, but not so. Marking in this case refers to the added functionality in an online assessment for a person to mark-up questions they answer as ‘uncertain’ during a test, with the additional option of coming back to it later before they submit their answers.

In addition to this marking-up facility that was at the core of the research the authors also designed a more comprehensive form of feedback, which they call metacognition-evaluated feedback (MEF). Simply put, MEF integrates students’ answers with their mark-ups, together with explicit feedback about a specific choice.

The study in question featured ninth grade (15yr old) participants taking a 30 question multichoice exam in vocabulary and reading comprehension. The authors looked at answering two questions:

• Does mark-up improve student scores?
• Does MEF encourage marking-up and review behaviour?

The authors found that:

• student ability was crucial as to whether marking up improved test results
• mark-up only improved ‘medium’ ability student test scores. Students with higher or lower ability didn’t show any noticeable improvement
• mark-up increased efficiency and effectiveness of self-managed learning
• mark-up did encourage across all abilities the behaviour of reviewing and reflecting on answers
• mark-up facilitated greater metacognition by someone taking the test, actually questioning their own learning and understanding
• MEF encouraged students to use mark-up skills more frequently and to review answer-explanations of test items

Incorporating this richer MEF at the end of the test certainly appears to offer a degree of formative assessment of student performance that feeds in to construct the next bit of learning needed. As the authors state: “Students made predictions about their test results and then observed what happened to check their predictions. If their predictions failed, they tried to determine how these mistakes occurred and then solved their problems.”

The mark-up system implemented was simply the option for a student to mark any given answer as ‘unsure’. This covered off a number of possibilities: 

• sure correct: the student believed they were right, and they did indeed get the answer right
• sure incorrect: the student believed they were right, but actually got the answer wrong
• not sure: in this case the student wasn’t sure, and may have got the answer right or wrong

The results the authors obtained here also provided an insight and/ or confirmation of who the low performers were likely to be: they generally fell into the ’sure incorrect’ category.

As ever effective feedback following submission of results is crucial. As the paper states you should “provide useful adaptive feedback so that students [can] understand their performance, clarify their mistakes, and increase their learning motivation.” In addition to take into account the lowest performers feedback should be written in a way that will “encourage review behavior”, with the suggestion of making this feedback “adaptive and detailed” with specific examples.

This paper provides a good case for improving the basic multichoice assessment found in courseware, as well as providing the design features to include. My experience of courseware is that the majority of embedded tests and assessments aren’t designed with a mark-up function. It certainly looks like it would do no harm to do so.

Skillsoft, the giant e-learning courses and services business, looks set to be bought. The company, with a turnover c£200 million, is currently courting a $1.1 billion offer.

Clearly the prospective buyer, a new company formed from a group of three private equity firms who are raising over half the cost from the banks, clearly see the future of e-learning in the upcoming years as very rosy indeed. And rightly so: Skillsoft posted higher than expected figures for Q3, and other reports are suggesting that the e-learning sector is in for a boom. (Although this growth in the e-learning sector has been reported practically every year since the late nineties, the global economy this time round, together with the maturing nature of e-learning make this a real prospect.)

The deal has yet to be done… so others may well consider making a move if they believe an acquisition of this type strengthens them and fits their strategy.

Update: Josh Bersin has posted an article on the Skillsoft acquistion with some interesting thoughts that’s certainly worth a read.

As it’s the end of January we thought we’d chip in on what the rest of the year could bring, although rather than predictions we’re adding each of our top 3 wishes – which makes a total of 6!

More of this…

fMRI: the king of man-made technologies for learning
It’s been around for a while, but with it now being applied to see our brain as we learn it’s jaw dropping stuff. At LIMBIC we love the brain. This should and will give us deeper insight into how our brain processes information, and learns – let’s hope we can use these findings as they emerge and package them into some well formulated practical guidelines. Considering we know so much already, and solutions use so little of it… we’re not too optimistic

Neuromarketing meets e-learning
It strikes us that this is a gaping cavern of a missed opportunity – but I’m sure if realised we’ll get a spate of gurus jumping on this next potential bandwagon (don’t you just love ‘em!). Not sure of its potential? Then just do a little bit of research…

Independent expertise
Organisations bringing in someone who is channel neutral, technically savvy, and pedagogically aware. This will add immense value to learning strategy

Less of this…

Learning technologies
This term is so bad. Let’s have a reality check: technology is created by highly capable people with learning having a small impact. When there’s technology we simply need to ask: “can and how does this technology help with learning?”

This is sure to cause a bit of contention, but we don’t see it as a specific branch of technology, I wish we’d get away from that – it’s simply more channels to use. So many times this expression has skewed thinking with people we meet that we avoid it at all costs

Organisational control of learning
This is tricky. It depends on your culture, on your people. Not every organisation has every individual bursting with ambition – another reality check. The so called learning organisation will be driven by the culture the employees create. But to embrace the benefits of collaborative technologies and employee generated content, some seismic shifts need to happen

Branding
More unbranding! Remove the ideas of training, and even learning. LMSs, learning portals, academies, universities, the list goes on. Google, Wikipedia, YouTube – I don’t see the langauge of learning or training here, so let’s learn from this…

I’ve just been reading an article on rapid e-learning (more of that in my next blog post), but it got my thinking - which is always a good thing.  There are some things I see time and time again in terms of material presented for ‘conversion’ to
e-learning (well, courseware actually if I’m being specific). A lot of the time this material is presented by an organisation as something they’d like done ’rapidly’.

• More often than not the content structure isn’t that great
• There’s simply too much of it
• It focuses on facts, rather than the learners’ and organisation’s needs
• Its tone and style is totally unsuitable – usually a highly passive ‘formaleese’
• And it’s ‘why use an image, when a 1000 words will do?’

And this is just the tip of the iceberg, but there’s a bottom line issue: you’ve gotta sort out the basics no matter what you’re going to do with your content. So what follows isn’t necessarily best practice for e-learning, I’d say it was best practice for content -and it’ll make turning it into some form of learning/ e-learning material a whole lot quicker and easier.

Sort out a structure
All this needs to be is a simple flow that allows a learner to ‘get it’ and fit the details in with the big picture; maybe content wrapped around a timeline/ scenario. Don’t worry what. As long as it’s got a basic, clear structure – that’s good.

Cut it…
Aim to cut the text in documents to 40% of the original word count with ZERO loss of content. If all you’ve got are PowerPoints… well, first I hope they’re visuals because that’s a good thing. If they’re bullets… that’s not so good. Only thing here is to note down what’s said ‘over’ these PowerPoints when used - and then cut, cut, cut!

Then cut again!
Simply ditch the ridiculous. Take the dreaded listing of procedures for example – DITCH! I fall back when I see this. If you want it as content then give it a separate page to itself, or make it a printable take-away in a pdf so that whoever wants to can have it to hand when they go to DO – but think about what a learner is going to be able to DO based on this list. If nothing – say bye bye.

If it’s an actual procedure – video it! Much more powerful than a written list – I won’t bore you with the details, you get the picture…

And speaking of pictures…
Use images. Say it with pictures. Again, it’ll cut down words and enhance understanding when done appropriately. Slap any words that describe your picture, or even highlight your data, back into the image. And don’t repeat what the image does well in cumbersome text somehere removed fom your image – resist!

The story so far
OK, so we’ve a bit of a structure, 30% of the original word count, and nice informative (and more memorable) images… time to make it snappy by making your remaining words earn their keep. If you’ve not come across it before you should check out an essay by George Orwell that could support the point of view that he was the first writer to offer a style guide for online text! His 1946 essay ‘Politics and the English Language’ has six great tips for knocking-up some e-learning prose:

“Never use a metaphor, simile, or other figure of speech which you are used to seeing in print

Never us a long word where a short one will do

If it is possible to cut a word out, always cut it out

Never use the passive where you can use the active

Never use a foreign phrase, a scientific word, or a jargon word if you can think of an everyday English equivalent

Break any of these rules sooner than say anything outright barbarous”

One more bullet I’d have added to Mr Orwell’s, but maybe a bit too much for 1946: make it chatty. Make your writing, your audio too if you want, warm, friendly, personable – not HAL from 2001.

Now I’m not advocating this for everything, not all the time. It’s not about dumbing down. It’s about simply making things clear and easy so that the channel doesn’t get in the way of the content. There are loads more tips about how to write in a learner/ web friendly way – but that’s something you can chase up if you need to.

You can add some quizzes, interactivity, or even turn it into a simulation – that’s all fine. But that comes after you’ve got your content sorted. Once that’s sorted everything should simply flow into its place, no matter what the pedagogy you decide. A waste of effort this is not – you will need basic content whatever the approach.

Nice YouTube video that’ll open your eyes to current technology trends and how the whole Web 2.0/ social media world is moving things along… at pace!

It’s also a great viral for promoting the book that the stats are from!