ODE Cupcake

Well, the big experiment has come and gone and was a massive success. The experiment went a little bit like this…

1) We let people know at the door that they were at the right place. This was important.

Wisdom of Crowds - Welcome to odeworld

2) The majority of those invited were Publishers and Editors, but we also sprinkled in a mixture of people from other professions, following the 1st rule of forming an intelligent crowd:

i) It needs to be diverse, so that people are bringing different pieces of information to the table.

ii) It needs to be decentralized, so that no one at the top is dictating the crowd’s answer.

iii) It needs a way of summarizing people’s opinions into one collective verdict.

iv) The people in the crowd need to be independent, so that they pay attention mostly to their own information, and not worrying about what everyone around them thinks.


Firstly we asked people to fill out a questionnaire whilst they waited for a slot on the experiment. It contained the usual stuff we want to know peoples’ opinions on such as VLEs, SCORM, tagging etc and the old chestnut “Anything else you’d like to add?” (mainly due to having a big blank space at the end and running out of proper grown up questions to ask).

We’ll send out the results to the participants very soon but a sample Q&A is below:

Q: Please let us know any thoughts you have about VLEs (Virtual Learning Environments) and the use of disaggregated electronic resources (little bits of learning) in schools/colleges:”

  • “Good idea – reduces the startup time using new content and means new content doesn’t need to carry around duplicated authorisation stuff.”
  • “Seems to ‘fit’ with where most are going and puts the teacher in the ‘driving seat’. I think primary schools are probably still a bit unsure about VLEs themselves but would be positive about the concept as long as there’s no techie things to worry about! As well as bits of content, they still may want a suggested programme or guide for how to link things together and to define where they fit in the curriculum.”
  • “Not to be ignored, although I’m unsure how much they are used by teachers at the moment. As things progress SCORMed content will be more and more central to learning.”
  • “In colleges Moodle is used and becoming default. Tutors put little bits of their own prepared material onto it. Also bits they found elsewhere and found useful e.g. Construction animations of water being pumped through a boiler; an engine working. It is still very variable in depth. Quality is not an issue.”
  • “Think they’re used a lot but not effectively. Teachers waste loads of time surfing for free content and find it doesn’t meet their needs.”

3) We then invited people up in pairs. They were given a brief introduction to what we were trying to do and then told to work individually not collaboratively. The proposal we put to them was: “In your judgement what sort of content would go under what sort of category in ode?” This was to help us decide our content taxonomy and the whole reason for this experiment.

4) We placed 12 categories on the table that, in the teams’ judgement, filled every permutation for the ode content taxonomy. A good taxonomy would provide the users with quick, painless search. A complex or misjudged taxonomy would impede the users progress through ode, thus creating a frustrating and limited experience.

5) We then gave each participant a stack of content examples and asked them to place each one under the category they felt suited it best. There were no right or wrong answers.

Wisdom of Crowds Table

For example would a “10 x spelling activities (Macromedia Flash + accompanying MP3 files)” come under “Pupil Activity”, “Tool”, “Teacher Support” or “Assessment”? There was also the all important “Don’t know” category, which we could then dig deeper into and draw out some further thinking on why that content was difficult to place.

Wisdom of Crowds Card Sorting

Whilst doing all this everyone who volunteered to come along was given a top quality freshly made-on-the-spot smoothie from our friends down at “Shaken Stephens” (below is the titular Stephen who heroically offered to serve the smoothies himself, in case you thought maybe we meant this one although it would have been awesome if we had). And an ode branded cupcake from Caketoppers, who will put anything on a cake that takes your fancy, within reason.

Wisdom of Crowds SmoothiesShaken Stephens Logo

So did the Wisdom of Crowds method work? Well we’ll know more when we have analysed the data properly but first impression is a resounding yes. We appear to have reached, through everyone’s input, as definitive content selling taxonomy as we’re going to get. And the nice thing about asking everyone else to decide for you is that you can blame the stupidity of mob rule if it doesn’t work!

I’m joking of course. See you at the next one?

PS: Sian Smith won the T-Shirt and it fitted perfectly. The two grinning idiots either side of you were an unfortunate by product of presenting the prize but I’m sure it was worth it…

Wisdom Of Crowds Tshirt winner - Sian Smith