15

Stack Exchange recently started a collaboration project with the University of Melbourne, in an attempt to improve the automatic detection of duplicate questions.

More information on the project and a call for help from the community can be found here: Project Reduplication of Deduplication Has Begun!

In it, Tim Post mentions that Doris Hoogeveen (me) will post a message on the different meta sites to let you know that we're ready for help. This is it!

As a brief summary, the project is about finding ways to automatically determine if two questions are duplicates, and also to detect the opposite: falsely tagged duplicates. To validate our methods, we need manual judgements from people who really know the data. This is where we need your help. It is impossible to manually annotate all the data from Stack Exchange, and so we have applied several filters to make sure we only show you questions pairs with a high likelihood of being duplicates.

Participation is completely optional, and there's no minimum time requirement. If you are interested in participating in the project, please head over to the annotation interface, which can be found here: http://hum.csse.unimelb.edu.au/se-annotate/

You can log in with your Stack Exchange account, so no sign up is required. The site also contains some more information on the project, which might answer some questions you may have. And I'm also happy to answer them here.

Thank you!

1
  • 1
    Hi everyone, thanks to everyone who has helped me in this project so far. I have improved the interface based on feedback from you. You can now see tags, the voting has directionality, and you can optionally see the comments and answers to questions. Your previous work has all been saved, so you will not have to start from scratch.
    – Monozygotic
    Commented Nov 10, 2016 at 22:15

3 Answers 3

Reset to default
7

I've been working on reviewing these and I've noticed that I've picked the "related, but not a dupe" option far more than saying that questions are duplicates. I think part of that is the nature of what we consider a good question here. In order to have a question that is answerable, one needs to fully describe the environment in which one is working. Because of some difference, the answers would be interesting, and perhaps relevant, but don't meet the Stack Exchange definition of duplicate. This may be particular to the nature of this site and topic, but I'm not sure what impact it will have on your data.

Thinking about it some more, I think some of this has to do with audience. When I'm looking at these question examples, I'm thinking to myself: "Would I give the same answer to both of these questions, and would that answer be helpful to the asker?" In some cases, on the surface, the answer is "yes". However, when I look at the details of the question, I can see a difference in education level, experience, or specific role of the asker. As such, they may be asking the same thing, but the answers would not be duplicates - they would need to be appropriately tailored to the asker. And I'm not sure that is something that should be called a duplicate.

I'm almost done annotating. I have less than 75 left for this site. This is how I've been using my options, based on my understanding:

  • Yes - The answers to both questions would be the same.
  • No - The answers to both questions would not be the same, and seeing one question and its answers would not be helpful to someone asking the other question.
  • Related, but not duplicate - The answers to both questions would be different, but seeing one question and its answers would be helpful to someone asking the other question.
  • Really can't tell - I'm not sure what the case is, based on the questions.

When annotating, I didn't pay attention to determining if the question was a good fit or not. I only looked at the question and its answers, along with the equivalence between those two things.

4
  • Hi Thomas, You've touched on an interesting aspect. I hadn't thought about the answers being different due to the expertise of the question asker yet. It's not about the answers being duplicates though, but about the information need of the question askers. Two people with a very different level of expertise might be asking about the same thing, even though the more experienced person would like to have more details in their answer. In such cases I would still consider the questions to be diplucates.
    – Monozygotic
    Commented Oct 30, 2016 at 22:17
  • @Monozygotic I don't think that fits the definition of duplicate, though. At least how it's used here. If you have Question A and Question B, Question B is a duplicate of Question A if and only if the answers that would be provided to Question B are the same as the answers to Question A. What you describe is what I would call a related question - the answers to one may help the other, and people answering would likely be interested in both, but they aren't necessarily helpful to the askers by itself.
    – Thomas Owens Mod
    Commented Oct 30, 2016 at 22:21
  • 1
    @Monozygotic I think something to consider is that each SE site is a community. What we consider to be a duplicate here may not be the same definition used on Stack Overflow or the same definition used on The Workplace. The same goes for what is considered a good question. There are likely to be commonalities, but every community is slightly different.
    – Thomas Owens Mod
    Commented Oct 30, 2016 at 22:23
  • 1
    Hi Thomas, that's completely fine. I would like to work with the definitions as used by the separate communities. This ensures that the provided annotations are in line with the existing duplicate question tags for the different sites. So if the Software Engineering community thinks a case like the one you highlighted is not a duplicate, but only related, then please annotate it as such.
    – Monozygotic
    Commented Oct 30, 2016 at 23:00
4

I've gone through a bunch of these questions (about the first hundred or so).

I think I marked only a handful as actual duplicates (maybe 5?). A handful were related but it felt like a large percentage of them were just straight up not even close.

A lot of the questions that were included seemed like bad questions for SESE to begin with, too. At least one (the "what would you like to see as part of PHP in the future") showed up a ton too and is a pretty bad question to begin with.

I got a bit demotivated after this, stuff like this question is just a bad question.

3
  • Hi enderland, the questions that we show are selected automatically. You can read more about it here: people.eng.unimelb.edu.au/tbaldwin/pubs/sigir2016-webqa.pdf The selections are far from perfect, which is exactly why we are doing this project! If I could show you only high quality questions and duplicates of them, then there would be no need for the annotations. So your feedback is valuable exactly to try to build a system that avoids showing such bad questions as suggestions in the future.
    – Monozygotic
    Commented Oct 30, 2016 at 22:20
  • 1
    @Monozygotic I guess my concern is that when questions that are not ever on topic here get put into the group, the correct response is something like "off topic" and none of the selection options really fit that.
    – enderland
    Commented Oct 31, 2016 at 12:58
  • I don't think that's something you have to be worried about. My main focus will be to distinguish duplicate questions from non-duplicate questions. The non-duplicates can be non-duplicates for many different reason, one of which could be that one of the two questions is off topic.
    – Monozygotic
    Commented Nov 1, 2016 at 0:36
2

The project is drawing to a close, and I am planning to take down the annotation interface this weekend.

Thank you very much to everyone who helped out, and thank you for all the valuable feedback and suggestions. Hopefully the collected annotations will lead to the development of better duplicate question detection systems, especially for StackExchange.

2
  • 2
    Can you tell us when we'll be hearing more about the outcome of this project? If you make a post somewhere else, could you come back here and send us a link to it, or cross-post the update here? I'm sure there are at least a few people here who would be interested to know how things turn out.
    – Thomas Owens Mod
    Commented Jan 12, 2017 at 10:24
  • 1
    @ThomasOwens, yes! As with all research, I have no idea what results we will get, or when they will be available, but I can definitely cross-post the update here when I have one.
    – Monozygotic
    Commented Jan 13, 2017 at 3:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .