13

I won't belabor the research by repeating it here. It's been described in exhaustive detail at this post.

The executive summary:

Closing, editing and reopening all become more effective.

Can we do the same here?

20
  • 3
    I've applied the status-review to escalate this to staff per the meta escalation/response process.
    – Thomas Owens Mod
    Commented May 27, 2020 at 16:33
  • 11
    I am clearly against it. IMHO we have still too many community members here who close-vote (and downvote) each and every question which has only some minor issues, even if most part of the question is answerable and on-topic.
    – Doc Brown
    Commented May 27, 2020 at 20:14
  • 2
    @DocBrown: [shrug] Stack Exchange didn't even ask the Stack Overflow community whether they wanted it or not. They ran some experiments, gathered data, came to conclusions and made up their own damned mind. In other words, they made the decision based on science, not gnat's closing proclivities. Remember, opening gets easier too.
    – Robert Harvey
    Commented May 27, 2020 at 23:50
  • 3
    @DocBrown Depending on what those minor issues are, closing quickly before answers are given so that they can be resolved and clarified is preferred. Otherwise, the chances of getting answers that would be invalidated by edits increases. I know that I don't want to spend a bunch of time writing an answer that's only going to be invalidated by an edit - I want the right question written first. Lowering the threshold for both closing and reopening would make it easier for the community to take these actions without moderator intervention.
    – Thomas Owens Mod
    Commented May 27, 2020 at 23:53
  • 1
    @RobertHarvey: so you think the conclusions drawn from the data collected on Stack Overflow can just be applied here to Software Engineering and its community, too?
    – Doc Brown
    Commented May 28, 2020 at 4:31
  • 2
    ... IMHO the situation has not really changed since the 3 votes for closing/reopening experiment 2015 - and your's and Thomas opinion since that time did not have changed, either. That's ok, I accept it, but I have a different one about this topic.
    – Doc Brown
    Commented May 28, 2020 at 4:40
  • 2
    @ThomasOwens, in my view, those minor issues are that a question contains more than one question mark or that it needs more than a single paragraph for an answer. That seems nowadays about the limit before a question gathers close votes for "Needs more focus".
    – Bart van Ingen Schenau
    Commented May 28, 2020 at 9:51
  • 1
    I am with @DocBrown in that previous experiment results didn't look very impressive to me. However I would be interested in re-running experiment - because in more than 4 years that passed scope of our site seems to be much solidified which may (or may not, there is no way to tell without trying) lead to better results in closing and reopening than what we got back then
    – gnat
    Commented May 28, 2020 at 9:57
  • 2
    @DocBrown: I've heard a lot on this meta site about how we should be more liberal about keeping questions open. It's based on a false premise: that closing questions early prevents them from being rehabilitated when, in fact, the opposite is true. The evidence that I've seen (while five votes exist and I don't have a diamond) is that the front page continues to be littered with crap questions, very few of which ever get rehabilitiated. At this very moment, there are fifteen questions on the front page with negative votes.
    – Robert Harvey
    Commented May 28, 2020 at 16:02
  • 1
    @DocBrown: Here's one to sink your teeth into: softwareengineering.stackexchange.com/q/410753. It's a perfectly good question, but it already has two downvotes (no close votes, though), and it's going to die on the vine unless something happens to save it. There are people on the site qualified to answer it.
    – Robert Harvey
    Commented May 28, 2020 at 16:40
  • 1
    @RobertHarvey: now I have really trouble to follow your logic - how would "3 close votes" help that question to get better answers, for example?
    – Doc Brown
    Commented May 28, 2020 at 18:26
  • 2
    @DocBrown: Questions that stay open have no incentive to improve. Questions that get closed are more easily removed from the front page while they get rehabbed. The purpose of closure is not to kill questions; it's the first step in the rehabilitation process. The purpose of deletion is to remove questions that have no hope of being rehabilitated. Much of the angst on this site is due to a misunderstanding of these fundamental principles.
    – Robert Harvey
    Commented May 28, 2020 at 18:50
  • 2
    The reason I pointed that particular question out is that folks are quite happy to downvote questions, but not offer advice to improve. If we're not going to close questions, but we're also not going to help people improve their questions, then what's the point of downvoting?
    – Robert Harvey
    Commented May 28, 2020 at 18:54
  • 2
    This community wants questions to stay open? Fine. But if you're going to insist on having 15 negatively voted questions on the front page at all times, at least provide a reasonably good story for doing something about those questions. Simply leaving them on the front page to rot doesn't seem like a viable strategy to me.
    – Robert Harvey
    Commented May 28, 2020 at 18:57
  • 2
    @DocBrown: There's a bigger picture here. If you want questions to be handled better, convince the community to handle them better. Otherwise, three close votes rules the day; the science says so.
    – Robert Harvey
    Commented May 28, 2020 at 20:11

3 Answers 3

Reset to default
7

Some of you may have noticed the Meta Stack Exchange post - Testing three-vote close and reopen on 13 network sites (it's linked in the featured on meta sidebar) - we've finally got this project under way and Software Engineering is one of the sites we'll be running the test on.

Starting tomorrow, I'll be changing the site setting and closing and reopening will require only three votes. This test will run for 45 days and will be turned back to five votes to close and reopen while I review the data from the 13 sites. After we've seen the impact, I'll be posting results and, if there aren't negative impacts, we will change the setting to three permanently.

A few weeks into this, I'll be posting a question here on meta to ask for your thoughts about this change, so you will have an opportunity to discuss the impact.

Thank you so much for your patience while we got this prioritized and scheduled. There's a lot more information in the MSE post, so please review it.

I understand that this is a re-test for y'all - we did this a while back and the results weren't awesome. What I will say is that we're looking at slightly different things now and it's completely possible that the problem y'all are facing is different than it was at the time. Over a 60 day period, only 38% of posts that were flagged as close-worthy (received at least one vote or flag to close) completed review - this includes posts that were closed and posts that were marked "leave open".

The big concern I have here is what that means about the 62% that were unhandled. They could be close-worthy or they could be just fine but, because there aren't people reviewing, we don't actually know. It would be bad if more stuff that shouldn't be closed got closed and, to some degree, that's a matter (which is not easy) of getting the community on board with when a question should be open or closed.

This test won't make more people review. We've been making changes to the review queues that we're hoping will make reviewing more interesting and easy for people with the privilege to do it - but reviewing is not an easy process, even less so on sites where there's confusion or disagreement about what should be open or closed. So we'll end up on relying on the opinions of only three people - which may just be all that are willing to participate in review.

Happy to hear what y'all think about the test either here, on MSE or on the midway post I'll put up in a few weeks.

8
  • don't be disappointed if you notice that 62% unhandled weren't much affected by this experiment. I visit close review time to time and many questions in there seem to be simply too far into gray area to encourage voting one way or another. Making decisions on these is maybe even too difficult to attract those who would be otherwise interested in getting review badges. I'd rather expect that we will get a bit livelier reopen queue if we're lucky (I also check it sometimes) - but only a little bit livelier
    – gnat
    Commented May 5, 2021 at 20:19
  • I'm not going into this with any specific expectations - as I noted in the MSE question - there's definitely room to take some time to understand why posts are going unreviewed, so a next step may be to look at whether things are aging out due to no reviewers or due to being skipped regularly - if we can come up with a list of the latter, y'all might be able to use it to do case studies into how to handle more confusing posts that may be borderline in/out of scope. If that sounds like an interesting option, feel free to post it on the MSE post.
    – Catija Staff
    Commented May 5, 2021 at 20:39
  • 1
    this option is primarily interesting to those fond of deepest matters of statistics :) because these things correlate so much that it makes a hard task to separate one from another. At least this is how it is for me: first I noticed that I skip too much, then I gradually lost interest because of that and started visiting infrequently letting close reviews age away. You see, what currently looks like aging out is under cover a result of skips, how to account for these separately
    – gnat
    Commented May 5, 2021 at 20:47
  • 3
    I started this project with Shog and he probably would have thought of things like this - so, without him around, I'm reliant on people who think of things this way to help me identify alternative explanations for what's happening - I've been around for a while, but I'm still learning about how to do data analysis and how to think about this, so it's really helpful to have these sorts of discussions. I want the results of this test to be really useful and help me identify where this change can be effective and where it can't - and if a high number of skips is an indicator, that's good to know!
    – Catija Staff
    Commented May 5, 2021 at 20:52
  • speaking of aging out, out of curiosity I checked some reviews history and it looks like they expire from queue in 4 days, like at Stack Overflow. For smaller sites this may be indeed too quick, consider making them age away in 14 days. Or, if you prefer generic approach without site-specific conditions, make questions age away in 14 days if size of the close queue is under 100-200
    – gnat
    Commented May 7, 2021 at 10:35
  • @gnat I share your experience about the queues. A lot of stuff on this site is quite borderline, so skipping it is often the most reasonable response – and making these decisions is extremely exhausting. Personally, I lost interest in curation activities in 2019 (after the election, but September strengthened my disinterest into a continued boycott) and have since ad-blocked the nagging review queue icon in the site header. But I think by now this site's curator community has dwindled so far that even three votes might be hard to get without moderator assistance.
    – amon
    Commented May 11, 2021 at 20:26
  • 3
    In the short period of time, the number of posts that I've closed (either because they came up in my RSS-to-email feed or because of a flag) is greatly reduced. In fact, at least twice now, a third community vote was cast when I was in the question attempting to close it. I personally think that this is already successful - the need for moderator involvement is greatly reduced. The next step is to figure out how to get more people invested in curation. Perhaps with fewer mod flags to deal with fast closure of bad questions, I'll have more time and inclination to do so myself.
    – Thomas Owens Mod
    Commented May 13, 2021 at 14:26
  • 1
    @ThomasOwens: I am specifically interested in how getting those "lets-close-everything-what-looks-suspicious-and-don't-tell-people-why-even-when-they-ask-for" fraction of our community into more curation work (latest example here).
    – Doc Brown
    Commented May 14, 2021 at 10:04
5

As I wrote in the comments above, I first was very sceptical about this change when Robert asked for it. In the meantime, however, my opinion changed a bit and I think the positive effects of the change most probably outweigh the negative ones.

Don't get me wrong, I still have the strong opinion there are currently too many community members around here who don't behave nicely:

  • downvoting and closing question with minor issues or no apparent issues (at least not apparent to me, so maybe my fault?)

  • refuse to give constructive or specific feedback to askers which are willing to improve their question (last example here, where after a full week of silence Thomas Owens finally was gracious enough to write a kind of answer I had expected to get from one of the initial high-rep close-voters).

  • stay away from curating questions actively, though they have more than enough rep for being able to do so

  • stay away from any reflective discussion like this one about the groups self-moderating style

  • mark questions frequently as duplicates of older questions which are way-too-general (or simply unsuitable) for giving the asker a helpful answer

  • give me the impression the only moderation tools they know are the downvote and the close vote button.

My fear was that the influence of those community members would increase even more when they now get a chance to close reasonable questions with just three votes, and probably it has become now. And I still don't buy Robert Harvey's argument that quick closing will motivate more askers to improve their question, so 3 reopen votes will actually lead to more improved, reopened questions, balancing the formerly described, IMHO abusive behaviour.

On the other hand, I see this site being still flooded with way more unsuitable questions than ones which might be salvaged, and these kind of posts now vanish a lot more quickly than before from the site. So there is probably some collateral damage I have to accept here. And the fact Thomas gets a little bit unburdened from cleaning up cumbersome stuff is definitely a positive one.

Of course, all what I wrote above is not based on any statistics, just my personal impression, so I am looking forward to see a statistical summary at the end of the experiment.

2

I can understand why you are willing to try the way that has proven to work so well at Stack Overflow. I am observing it for about half year now and it looks really impressive and, which is especially promising, it seems to be free from (serious) negative side effects.

On the other hand, there is no guarantee that this will work well here. Rather opposite, I think we better be sceptical because as was pointed in comments we already experimented with such a change about five years ago and back then, results weren't encouraging.

Speaking of that prior experiment, I decided to re-visit and study past discussions about how it went (here, here and here) to see what we can learn from it. I found lots of insightful considerations posted back then and I strongly recommend checking these.

That said, studying these prior discussions left me uncertain about whether it is worth having this change now or not. Some points for or against it I've seen were apparently relevant back then but seem to be no longer applicable. It was particularly striking to discover that my own reservations against this change no longer hold.

It looks like in the years that passed site has changed too much to rely on analysis and conclusions made back then.

We had site name change (which seemed to have much more profound impact than I anticipated), we had a noticeable change in the way how diamond moderators approach blatantly off-topic questions. I think we even had some shift in topicality due to successful rise of the sites that handle law and open source topics (can't say for others but to me this changed alot in the way how I approach these topics at our site).

Suming up above, I think we could give this change yet another try - run an experiment with 3 close / reopen votes for a month or two and study how it works. Maybe this time it will help getting reopen votes work like it did at Stack Overflow (it failed in our previous experiment but reasons for that seem to no longer hold).

2
  • I've seen the old discussions; in fact, I participated in them. Stack Exchange interpreted the facts to fit their own conclusions. It seems unlikely to me that SE would be willing to repeat an experiment they already conducted and concluded to their satisfaction.
    – Robert Harvey
    Commented Jun 1, 2020 at 19:44
  • @RobertHarvey as opposed to how it was 5 years ago this is currently said to be a matter of a simple flip of a switch and it is going to be rather hard to justify rejecting this low effort change in the light of fairly strong evidence that results of 5 years old experiment aren't compelling anymore. Though frankly point of my answer is not to convince powers that be but primarily to summarise what I think about this proposal
    – gnat
    Commented Jun 1, 2020 at 21:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .