As I follow the Olympics, one of the things I see on a LOT of places is the “Medal Count.” For example, here’s a summary taken from Yahoo! Sports (8/19/2008):
| Country |
Gold |
Silver |
Bronze |
Medal Count |
| United States |
26 |
26 |
27 |
79 |
| China |
43 |
14 |
19 |
76 |
| Russia |
10 |
14 |
18 |
42 |
| Australia |
11 |
12 |
12 |
35 |
| Great Britain |
16 |
9 |
8 |
33 |
| France |
4 |
11 |
14 |
29 |
| Germany |
11 |
8 |
9 |
28 |
| South Korea |
8 |
10 |
6 |
24 |
| Japan |
8 |
6 |
8 |
22 |
| Italy |
6 |
6 |
7 |
19 |
And, looking at the comments given by visitors to various Olympics blogs, these medal counts are a popular way of arguing America’s preeminence in the sporting world. However, as I looked at tables like the one above, it began to seem very familiar. At Primary Intelligence, we often ask respondents to rank criteria pertaining to the sales experience in order of importance, first through third. In the past, we would simply sort the criteria by the number of mentions they received to get an overall sense of what was important to our respondents. The results took a form very similar to a medal count table.
A potential problem we found with this system was that sometimes, the number of mentions was not the best indicator of importance—in some cases, it was which ones that were mentioned first that was the key. So, in these cases, we applied a standard weighting system to our responses (where 1st mention equaled 3 points, 2nd mention equaled 2 points, etc.), which sometimes gave us remarkably different results.
This seems to be the case with our Olympics example; if we use the weighting system, we get the following:
| Country |
Gold |
Silver |
Bronze |
Medal Count |
Weighted Score |
| China |
43 |
14 |
19 |
76 |
176 |
| United States |
26 |
26 |
27 |
79 |
157 |
| Russia |
10 |
14 |
18 |
42 |
76 |
| Great Britain |
16 |
9 |
8 |
33 |
74 |
| Australia |
11 |
12 |
12 |
35 |
69 |
| Germany |
11 |
8 |
9 |
28 |
58 |
| South Korea |
8 |
10 |
6 |
24 |
50 |
| France |
4 |
11 |
14 |
29 |
48 |
| Japan |
8 |
6 |
8 |
22 |
44 |
| Italy |
6 |
6 |
7 |
19 |
37 |
Using this particular method, America’s dominance in the Olympics is not so self-evident, which probably explains why most American news sources use the first kind of table.
However, this isn’t an argument about sports, politics, or what is on Michael Phelps’ iPod*. What is of interest here is that how you decide to calculate scores and sort those scores will have a significant impact on your final results. The key is that you must not simply calculate importance (or any other score) in a number of ways and decide which finding you like best, but instead, think carefully before you begin your study and decide which method will best answer your research questions. For example, if you are interested in learning what product features are “top of mind” for respondents, then asking them to name three features and then calculating their order by number of mentions would probably work well. However, if you are interested in which product feature they couldn’t live without, then you might want to consider making sure the responses are ranked when asked, and weighted when presented. By making these decisions early on in the study design, you’ll have more effective and appropriate results, and you won’t be accused of bias in your findings.
*Sorry, I had to mention Michael Phelps. It’s mandatory in any blog entry that mentions the Olympics. Not sure why.
Add to the following social sites:


