Wednesday, August 13, 2014

Fact ranking quiz

We have developed a fun and exciting quiz which will allow us to tackle one scientific problem and we would love for you to participate and help us out!

The problem is that, as we all know, there is too much information available on the web. How do we decide which facts are important or most relevant for a certain object?

Many fact ranking systems have already been developed, trying to solve this issue. However, the difficulty is that there is no gold standard corpus which could serve as a ground truth for evaluating their performances.

How do you determine, which information or fact is more important or relevant than another? Just a quick example. Let's assume we have the following two facts:

  • (1) Albert Einstein is a physicist.
  • (2) Albert Einstein is a Vegetarian.

Which of the two facts is more important or relevant? Yes, this is difficult to answer, simply because the truth often lies in the eye of the beholder. For a vegetarian, maybe the second fact is more important. But, what about the most common opinion? What would the mainstream think? Probably, most people would say that fact (1) in general is more important.

You might interrupt your rating of the presented facts at any time you like and continue later. To make it a bit more interesting, you can also score points and of course there is a highscore list. We would really appreciate your help in this task. Please do also spread the word. The more participants, the more valid our ground truth will be.

While you are playing this game, your inputs (which are greatly appreciated) will contribute to our scientific experiment. Together, we will build a first corpus that will help our scientific community in evaluating different fact ranking strategies!

This ground truth is achieved with this little 'voting' application [1]. You just have to register with the tool and then the task will be explained to you in detail.

We took 500 popular concepts from Wikipedia and you have (1) to think about the most important facts about these concepts that come to your mind and then (2) rate the (new) facts presented to you according to their relevance. There is no right or wrong answer. Just vote as you think it seems right for you. Afterwards, we will aggregate all votes from all participants to determine the general (mainstream) relevance of the presented facts.