« Pluto Clears Up| Main | Recommendation-Consumption Impedance Mismatch »

Voting up or down is dead

| | Comments (9)

Look. I know the arguments. Let's say you're creating a social web site, that is, a place where people can do stuff and other people can recognize them for what they are doing (even if it is just scratching their elbow). Sooner or later, you're going to want some kind of score, some numerical system where users can find the "good stuff".

So you put in Diggs, or thumbs-up and down, or arrow votes up or down, or smiley or frowny faces. Maybe you add a little score number to show how much cowbell it has.

Hey, the argument goes, people aren't going to have time to do much more than a simple yes/no, people hate complicated interfaces, a simple score can let folks see how well they're doing. What could go wrong with a simple scoring system, a simple number, and an algorithm that's invisible to the user?

What could go wrong? Let's find out.

A list of the vote for this sites
Reddit has been overrun with "vote for me!" posts

  • You're not measuring the perceived value to the reader

  • Sample list from the site DZone
    DZone has a much more intricate control system,
    although it still all revolves around up or down. Of course,
    DZone also has a much more technical audience

  • It makes the site a game instead of a resource

  • People use it as a way of saying "heck yeah!"

  • Vote up or down means different things to different people - Am I agreeing, wanting other people to see it? Endorsing it as my point of view? Announcing that even though I disagree with it I find it interesting? (A sophisticated argument that seems lost on most folks).

  • Most people don't participate

  • Typcial Bush entry
    A typical political article on Digg
    It's left as an exercise to the reader to see if protests are really
    outlawed or if the article deserved the number of Diggs it received

  • Titles are slanted to get the most attention instead of being honest

  • There's a difference between "Wisdom of the Crowds" and "Mob rule"

Sample shot from YC News
Sample screen shot from news.yc. Like all social sites,
I predict that content will slowly drift from "Hacker News"
to "titles that can get me karma", since the rating system is part
of the way such sites are played.

I've had this design discussion a number of times. The social rating system in place in most web outlets is broken.

Like most things, good design is always a trade-off between opposites that are pulling at you. On one side is the absolute need for simplicity, and the need to keep the screen as uncluttered as possible. On the other side is runaway conditions like we're seeing on some of these sites. Quite frankly, Digg has become a cesspool. It used to be I got a hit of 1 out of every 5-10 articles, that is, somewhere around 10% of the articles were interesting to me. Any more, I go through 30-50 articles looking for something of value. It's enough to believe that the system is being gamed. But my opinion is that it's not broken people, it's a broken system. People will do what the system rewards -- to a fault.

UPDATE: This discussion continues with an overview of various solutions here.


I was hopping to find the "digg me" button somewhere haha

The scary thing is that the video game mentality sucks you in even if your motives are pure.

This problem is something I've been thinking about working on to some degree or another for a couple of years now. So it's not like I just made the article up to get hits. I just wanted to re-hash some of the good stuff I have been reading on YC and add some of my own analysis.

But once I submitted it, this strange urge crept over me. I kept hitting refresh over and over again, looking to see how many votes I had. Then I started checking the server logs, wondering if anybody from YC was dropping by to read the article. Finally I went over to YC and monitored the comments, seeing if I was getting trashed or not.

I can see where very easily people could just start looking at their "score" on various systems and the amount of traffic they get. Once the score becomes more important that the perceived quality to the user, you're all out of whack. That's why these things spin out of control.

And yes, I used to have some buttons like that -- I am debating putting them back. If the system is hosed up, might as well enjoy it!

The problem is not the notion of popularity itself - that's hardwired into us. Instead, the problem lies with large audiences. What we need is to make it harder and harder to gain large audiences. Communities need to fragment to avoid turning dysfunctional.


A hundred years from now I predict the broadcast era will be a blip in our history. Our past was multicast, and the future will be too. Audiences will just not be geographically localized.

You're right, but you're missing the underlying reason why. The correct way to go about it, which is even simpler for the user than rating up/down, is to just drive things from implicit data. No work for the user at all.

I wrote about this a little while back:

I have to agree with you, Udi, implicit does kick explicit's ass.

But only for certain things. As you pointed out, "You just have to generate and track the right kind of metadata. " -- which seems just as important.

The problem with implicit-data-only tracking is that the structure of the implied knowledge may not reflect the value to the user. Look at Google: great implicit engine -- that is, until people started gaming the structure. People are going to game whatever system you create. As popular as it is, Flickr is probably still under the radar and their system harder to game. There's a cost-benefit decision that goes on with game players. Perhaps the goal is to make difficulty of hacking the system proportional to the number of users.

Great article, though!

Maybe there needs to be something like "Quality" ratings in addition to the simple "Yes" or "No" metric.

Y'know, something completely independent of whether you find the article interesting or not, just whether it's quality info or mind-junk food.

Voting systems exist to raise up quality content and hide poor content. I think you're right when you say that systems don't work; they sink into promoting the lowest common denominator.

Any suggestions how to do it better?

I think as Udi points out, there is a place for implicit knowledge and reasoning. But explicitly indicating something about the content has it's place too.

I have a lot of ideas, Peter. I'd like to be able to test a few of them in a startup format rather than just speculating without adequate data.

I guess in general I'd be curious as to the answer to the question: "aside from social sites, what were your experiences finding new content on new sites that you really enjoyed?"

I'm sure most woud say something about "an email from a friend", and that seems to be the bedrock for most of these engines. But I'm not sure that is the total story. Unpacking what goes on with that new cool content experience is going to be key, after all. That's the thing we want to replicate.

Up/down isn't necessarily that bad. Using the raw data directly certainly is. Also tactical voting becomes a problem.

A better system would use the data indirectly like Amazon's recommendation system, which works frighteningly well. So, you would only vote for yourself (because downright honest votes yield the best results) and people who happen to think alike.

If someone votes all the time for some random "vote me" bullcrap, then his/her personalized list will feature more of the same in a prominent place, but other personalized lists won't be affected (unless you share the same obscure interests, that is).

Additionally the incoming queue could be presorted based on a few votes by people whose interests overlap with yours.

If anyone implements this, buy me a house or something after you got your first 50 million dollars. ;)

Leave a comment

About this Entry

This page contains a single entry by Daniel published on October 14, 2007 11:44 PM.

Pluto Clears Up was the previous entry in this blog.

Recommendation-Consumption Impedance Mismatch is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

Powered by Movable Type 4.23-en
Daniel Markham