38

This experiment is over. I will be adding an answer to show the results and suggest a plan for going forward.


For the next 30 days, closing a question will require just 3 votes. Also 3 votes will be sufficient to reopen a closed question. In essence, each close and reopen vote has 5/3rds the normal value. After 30 days I will revert the site setting back to 5 and evaluate the results.

I have tested this change on a development instance and questions that currently have 3 or 4 close votes will not be closed automatically. Rather they will be closed when the next close vote comes in.

Background

I think it's fair to say there's a lot of dissatisfaction with the question closing system on Programmers. One proposed solution is to increase the number of close votes available to 10k users. Some of us on the community team have considered various schemes to scale close votes with reputation under the theory that people most invested in the site (as measured by reputation) will have the most incentive to curate questions. That might still happen, but we'd like to be pretty sure that it will solve the various problems with question closing.

It's pretty clear that organic closing (via the close votes queue or people voting to close questions they find naturally) isn't working very well. To demonstrate, here's how close votes on questions asked in the last 30 days are distributed on Super User:

Super User close vote distribution

Most questions have no close votes (not shown). 461 questions have just one (the yellow circle on the left) and the number slopes down to 142 with 4 close votes. 5 votes is a natural attractor because people will keep voting to close off-topic and otherwise closable questions until they reach that limit. There's a long tail beyond 5 close votes (also not shown) due to votes expiring, close wars and so on. In essence, questions with at least 1 close vote, but not 5, are in a grey area in terms of closablity. This general pattern turns out to be extremely common across a variety of sites.

In contrast, here's the same graph for Programmers—Stack Exchange:

Programmers close vote distribution

Remember this is only showing questions that have been asked in the last 30 days; we aren't talking about Not Programming Related questions. Instead of a fairly smooth gradient between questions that should clearly be open to questions that clearly should be closed, this graph shows something closer to a binary state. My hypothesis is that since the people contributing the most close votes feel the need to conserve votes, they don't vote to close borderline questions. There's too great a risk those votes will expire impotently.

I've also included the counts of questions that have been edited or deleted in each close vote bin. Relative to other sites, questions on Programmers with close votes are more likely to be deleted and less likely to be edited. It's entirely possible that a greater proportion of questions are unsalvageable on this site. But I suspect the underlying problem is that the questions that could most benefit from an improvement are not getting close votes in the first place. That might not be a problem, but close votes can provide incentives for askers to fix problems even before their questions are closed. Close votes also put questions in front of reviewers who might be inspired to improve them.

Another problem that we discovered very dramaticly is that closed questions are rarely reopened. Most closed questions can't (and shouldn't) be reopened, but statistically speaking, we expect more than 1-2% will be. My hypothesis is that it takes so much effort (and coordination) to close questions that people are not interested in undoing that work even if there's been an edit to fix a question's problems.

To be clear, the breakdown in the close voting system is not limited to Programmers. It is, however, rare across the network for the division between open and closed questions to be so stark. We hope this test will help us understand how the close vote system breaks in order to (hopefully) fix it.

Why 3 votes?

Initially users with enough reputation could unilaterally close and reopen questions on Stack Overflow. In December 31, 2008, the close vote system was created and required 3 votes. Between then and the next April, the limit was increased to 5. (Shog9 remembers 3 lasting a few weeks.) The documentation of the site setting reads:

KEEP THIS AN ODD NUMBER

So we need an odd number greater than 1 and less than 5.

How will this change be evaluated?

There's a nontrivial chance something will break requiring me to revert the change. If we continue for the full 30 days, we will compare before and after the change:

  • Close rate

    Obviously, it's now easier to close questions. Insofar as bad questions are put on a path toward either corrective edits or deletion, that's a good thing. But this site closes well over 50% of new questions, which is the highest rate on the network. (Considering total questions closed to questions asked in a time period, the rate is more like 70%.) Paradoxically, I'd like to see the close rate remain steady or decrease.

  • Reopen rate

    Reopening questions is one way to mitigate against a high close rate. Since the bar for reopening is now lower, we'd expect more questions will be reopened. Currently the reopen rate is about 2% of closed questions.

  • Edit rate on closed questions

    In general, the only path to reopening a question is editing. It's also the most effective way to prevent questions from being closed in the first place. Since it's easier to reopen questions, there's even more reason for askers to fix their own closed questions. In the last 30 days 9% of closed questions were subsequently edited.

  • Deletion rate

    About 65% of questions that get at least one close vote on Programmers are deleted. We aren't exactly looking at the cream of the crop here, but that's noticeably more than other sites. (Again, this only counts questions asked and deleted within the last 30 days.) We can't force people to ask questions that don't deserve deletion, but hopefully more questions can be salvaged with edits.

Finally, I'm curious if this change helps people enjoy the site more overall. At the end of the experiment, I'll post my results and ask for any followup thoughts you as a community have.

How you can help

Relatively few people are active in closing, reopening and deleting questions. Even fewer are active in editing. In order for this experiment to improve the quality of content on this site, more people need to be involved in identifying questions with potential to be unlocked with a judicious edit. Fixing tags and removing signatures are good first steps. But title and (occasionally heroic) body edits might be needed to pull some questions out of mediocrity. Remember: once the question is posted, it belongs to the community as much as it does the original author.

In addition, please use the answers below to let me know how the experiment is going.

  • 6
    Words can't even begin to express how awesome this experiment is. I mean, it's all about Science!!! – user53019 Nov 9 '15 at 17:19
  • 3
    For science!!!! – enderland Nov 9 '15 at 17:19
  • 4
    I particularly like how your X axes make allowances for the possibility of 5.5 close votes :) – Lightness Races in Orbit Nov 9 '15 at 17:24
  • 2
    @LightnessRacesinOrbit: Unfortunately, SEDE doesn't honor the data type of seelcted numbers. I'm sure a pull request would be welcome. ;-) – Jon Ericson Nov 9 '15 at 17:27
  • 7
    @JonEricson: I find my time is better spent making disparaging but well-humoured remarks. :) Anyway, I don't use Git. – Lightness Races in Orbit Nov 9 '15 at 17:28
  • 2
    "It's entirely possible that a greater proportion of questions are unsalvageable on this site." - I've run this experiment a few times. Most recently What questions recently closed can be salvaged? – user40980 Nov 9 '15 at 17:44
  • 1
    @MichaelT: One of the difficulties I think people face when looking at a pile of already-closed questions is that they are primed to see more of the same. It could very well be that there are no gems hidden in those questions. It could also be the case that presenting them in a long list is an inefficient way to find them. In any case, one potentially positive result of this change is that fewer people need to read questions in order to close them. – Jon Ericson Nov 9 '15 at 18:08
  • 3
    interesting experiment. As a side effect it (temporarily, and only at Programmers) addresses a feature request At smaller sites, penalize hot questions having 3-4 close votes. Less janitorial load at both ends of the troublesome spectrum over here may very well end up with increased focus on salvaging and reopening. People have to spend their energy somewhere anyway – gnat Nov 9 '15 at 19:18
  • 3
    Good point, @gnat; if we end up doing this long-term anywhere we'll want to add a multiplier to offset the resulting reduction of embarrassing hot questions. – Shog9 Nov 9 '15 at 19:40
  • 2
    @Shog9 I think you'll have problems with hot questions no matter what you try until you'll learn to shuffle them right. Static tweaks with multiplier, aging factor, whatever else may work for a while but will inevitably break at scaling, like everything else broke so far. Shuffling 100 questions to 100K visitors works, shuffling same amount to 1M will break – gnat Nov 9 '15 at 19:46
  • 4
    Curious... why does the number need to be odd? – David Grinberg Nov 11 '15 at 14:41
  • 7
    @DavidGrinberg This is probably one of these things that no one remembers why it is as it is, but also no one dares change because of the big warning in the docs. – yannis Nov 12 '15 at 10:34
  • 3
    Please consider trying this experiment on other graduated sites, because the problem of having too many not-quite-closed questions affects more than just Programmers. – 200_success Nov 13 '15 at 17:51
  • 2
    @DavidGrinberg Because it eliminates ties. – Robert Harvey Nov 14 '15 at 16:45
  • 2
    @JonEricson aside, could you check on the time it takes to go from 3 votes to 5 votes for questions that ultimately get five votes for awhile now that we're back to 5? – user40980 Dec 10 '15 at 3:39
13

I'm putting this out there as a possible alternative interpretation of the flat and peak graph for Programmers.SE close votes.

Our scope is more cleanly defined than we are given credit for.

The close votes that are cast, and the ultimately deleted questions are done with a great deal of consistency. The questions that are closed really are off topic or not a good fit for the Q&A format. The majority of the people who actively cast close voters recognize this and vote in accordance with this scope.

The problem that the site faces that requires additional close votes isn't so much "we're holding back on borderline questions" but rather "we are deluged with off topic questions and questions that presuppose this site is for opinions and discussion."

Under this perspective, Programmers.SE is open to those well written and answerable questions that come our way. This also acknowledges the ongoing challenge of actually presenting a well written and answerable question, though that is one that may be easier to address if one has more time to guide those questions without having to also spend as much energy on the questions asking for book recommendations and the like.

Unfortunately, the way to differentiate the "we're holding back on close votes on borderline" and the "we are accepting of the well written answerable question that pushes the boundaries without incurring excessive drama and discussion" is to look objectively at the questions (or a sufficiently representative sample) we close or have cast close votes on and then see:

  • Are there borderline questions with 1 or 2 close votes?
  • Are there hidden gems that have been closed?
  • What percentage of the incoming questions are unsalvageable?

As we don't have triage, that last point is an awkward one here. It might be useful for the community manager team to grab a random sample of the questions that we close... say 100 posts and do a 10 second glance at each and determine which pile it should go to (the 'unsalvageable', 'needs improvement', or 'looks ok' bins).

Whatever the case, a faster turnaround time on close votes to closed question means faster feedback for the OP (and in cases where its a good question for Stack Overflow, a faster migration to there). When questions sit for hours before they are able to muster sufficient close votes, the OP may well have lost interest in fixing the question (that would be another query to run - OP engagement with a question over time and see if moving the closed question to within that timeframe allows for the OP to be more active in fixing those borderline questions that do get closed).

  • 1
    I am sampling closed questions today. Mostly I'm looking for ways that the setting change might have broken things. But I am looking at the content of the questions as well. Scope definition is certainly part of the equation; perhaps the primary term. Note that it doesn't take much to improve the recovery rate. We are talking about one extra question in hundred, which just might become the sand for an amazing answer. The system is built on chance and not certainty, so I'm going to evaluate primarily on the statistics. – Jon Ericson Nov 9 '15 at 21:00
  • 2
    @JonEricson out of pure curiosity, any preliminary thoughts from your sampling yesterday? – user40980 Nov 10 '15 at 21:23
  • 3
    I see a lot of well-written, well-intentioned, and even well-formulated questions closed for non-obvious reasons. I flip over to the on topic help and I can kinda see the reason if I squint. Obviously there's junk too. (It's easy enough to identify those from the title alone.) On the plus side, I do see a lot of folks working with OPs to get their questions reopened. – Jon Ericson Nov 11 '15 at 0:34
  • 2
    @JonEricson while it isn't the help center, we do have Why was my question closed or down voted? which tries to go a bit more into the nuances of why we close some things with certain reasons. Its one of the "if we could write a book and put it in the help/on-topic, we'd get even fewer people to read it." I would be curious to have this discussion (maybe in chat some time?) about particular questions and the reasons that it got closed. – user40980 Nov 11 '15 at 2:53
  • 1
    I agree with @JonEricson. I have seen many questions that get closed or down-voted even though they are reasonably answerable and interesting. The mods here seem to be in unison (a good thing), but I believe a lot of questions are closed or down-voted to join the bandwagon. If my belief is true, then it's unfair to the askers because some questions can get downvoted/closed simply because one mod doesn't like it, or thinks it's too broad because they aren't familiar with the topic... then the others follow suit and a reasonable question gets buried. – Kasey Speakman Nov 11 '15 at 14:50
  • 1
    @KaseySpeakman if you have a question about a specific closed or down voted question, asking about it here, on meta, can help either explain or reopen the question. It can be difficult to deal with otherwise nebulous too many things get closed or down voted. Specific questions can help us recognize the direction that closing needs to take. – user40980 Nov 11 '15 at 15:10
  • @MichaelIT, I have been trying to get a feel for how things work before arguing meta. My approach up to now has been: If I feel a down-voted/hold question is reasonable and answerable (and not a homework problem), I just go ahead and answer it anyway. It could still at least help the user even if it gets buried in the list. 1 2 – Kasey Speakman Nov 11 '15 at 15:30
  • 1
    @KaseySpeakman an important consideration that many make is the ability to help the next person with the question. Part of the mouseover for a down vote is "it is not useful". In trying to make the site one with a high signal to noise ratio - that the questions and answers are ones that are helpful to a wider audience, many people will down vote or close questions that are not likely to be useful to the wider community. We also endeavor to close questions that will draw too many answers, or answers that are incomplete by themselves. – user40980 Nov 11 '15 at 16:59
  • @MichaelT. I suppose it is a philosophical question "Where does this site fall on a scale between developing a Q/A showcase or helping users?" I'm not going to argue the point in these comments. I only referenced some questions so that my statements were not as nebulous. – Kasey Speakman Nov 11 '15 at 17:14
  • 2
    @KaseySpeakman Meta isn't your only option when it comes to re-opening questions. Sometimes all it takes is editing the question to shape (see: meta.stackexchange.com/a/196078/162704). You also have the option of dropping the question into our chat room, and discussing the closure with the regulars there. And you can always use a custom flag to ask a diamond to review a closure (we'll probably redirect you to Meta for the non clear-cut cases though). – yannis Nov 11 '15 at 20:11
  • @Yannis, I appreciate the comment. It is good to know how the mechanics work. I would be curious to see numbers on how many questions are actually edited back into shape (especially by not-the-author). It seems a daunting task to do properly. – Kasey Speakman Nov 11 '15 at 21:03
  • @KaseySpeakman its easier to do when the question is closed and hasn't been answered... and there is a core of a good question there. See also What is needed to really fix a question (an Atwood transform). However, when the question is one of the fundamental off topic reasons (polling ones) it is very difficult to do. ... and then sometimes you get the OP who returns to the question and makes hostile revisions too. Though yes, we do encourage it, and its hard. – user40980 Nov 11 '15 at 21:13
  • 2
    @KaseySpeakman I would encourage you to drop by the site chat if you have questions you feel are unfairly closed and need reopen votes. While you might get disagreement, any unfairly closed question which is actually on topic will almost assuredly get reopen votes from the community there. It does happen when people edit their questions to provide more information and they get reopened. But frankly most questions closed never get more information from the poster - as a community we need that info to make a useful/answerable question. – enderland Nov 12 '15 at 17:10
  • wonder why they don't compare Programmers to Server Fault. As far as I can tell, folks over there eventually settled on site scope and defend it quite firmly – gnat Nov 16 '15 at 20:06
  • 1
    @gnat I want to say I suspect that there recent crisis of community moderation and scope influenced them just as the early scope change here did (which relied very much on the mods to change the direction). It also burnt out a mod on each site. However, they don't have people looking back at recent history of the site as the "good old days" complicating matters. Thus, if one is to look at sites, its necessary to pull one that has a large enough close queue to see a distribution of votes - and SF's mods are actively directing the site and keeping the queue low enough for the community. – user40980 Nov 16 '15 at 20:55
11

I wasn't going to update the experiment this week, but the results this week were so encouraging I can't help myself:

Week posted Qs  Closed Close Rate Closed->Edited Reopened Cl->Ed->Re Deletion Rate
----------- --- ------ ---------- -------------- -------- ---------- -------------
2015-10-12  325    190      58.5%           8.9%     2.6%      11.8%         40.9%
2015-10-19  349    205      58.7%           8.8%     2.4%       5.6%         39.8%
2015-10-26  319    207      64.9%           8.7%     1.4%      11.1%         46.1%
2015-11-02  362    218      60.2%          13.3%     3.7%      10.3%         44.2%
2015-11-09  286    206      72.0%          20.9%     3.9%       7.0%         41.3%
2015-11-16  340    205      60.3%          17.6%     9.3%      30.6%         35.6%

Not only has the close rate dropped to something like the pre-experiment rate, the reopen rate has rather dramatically increased. Much of this is due to more edited questions getting reopened than in previous weeks. (It might be too early to tell if it's statistically significant, but more closed questions are being edited since the test began.)

I also spent some time last week trying to learn what prompted people to close so many questions on Programmers. While there are many detailed reasons, two themes emerged in my mind:

  1. Questions that remain open are signposts that prompt new questions of the same type.

  2. It's difficult to maintain the quality of answers on certain types of questions.

The first didn't surprise me. It's a variation of the broken window theory (which is different than "zero-tolerance policing" by the way.) Since it's often difficult to curate a large corpus of answers, it's a lot more convenient to curate the questions instead. Some sites, however, pull it off. Skeptics, for instance, has strict rules about what sort of questions can be asked (notably, notable ones) and what is required of answers (particularly that they be referenced). As a result, Skeptics deletes a large percentage of answers that have a score > 0:

answers deleted % edited % locked % notice % Q deleted % Site Name 
------- --------- -------- -------- -------- ----------- --------- 
   7634     35.5     19.64     0.05     0          97.16 Area 51 Discussions
   8288     13.61    55.68     0.1      6.03       28.63 Skeptics
   5782     10.26    46.49     0.09     0.02       48.74 Islam     
 141392      9.38    25.01     0.45     0.02       78.15 Programmers
   4810      6.63    52.45     0.12     0.46       27.9  Biblical Hermeneutics
  15660      5.86    45.84     0.03     0.37       48.31 Christianity
  22572      5.08    28.81     0.33     0.02       63    Web Apps

I cut the list off at sites that delete at least 5% of positively-scored answers. Area 51 is interesting because 97% of "good answers" get deleted when their question gets deleted as a result of their failed proposal being deleted. But the next in line is a real site: Skeptics. Their policies encourage deletion not only of entire questions, but individual answers as well. (Skeptics is unique in its extensive use of post notices.) Islam, Christianity and Biblical Hermeneutics have fairly strict requirements that answers be well-supported. Those sites also emphasize narrow questions that allow moderators to delete many answers as non-answers.

I'm less familiar with Web Apps, but I believe they had a major change in scope not unlike what happened on Programmers. When you shift the topic of a site, that will leave many now-off-topic questions that need deleting along with all their answers. As a corollary, there's a real need to lock questions and their answers. But look what happens when we limit to answers posted in the last year:

answers deleted % edited % locked % notice % Q deleted % Site Name 
------- --------- -------- -------- -------- ----------- --------- 
  10617      4.53    34.44     0.01     0.03       66.53 Programmers
   2999      1.93    27.91     0.23     0          48.28 Web Apps  

Web Apps has settled into a fairly normal pattern of deleting relatively few upvoted answers. It's also split fairly evenly between deleting those answers individual and as a result of deleting the question. Programmers has become a little less extreme in terms of positively-scored-answer deletion, but it still ranks near the top of the list with Area 51 (31%), Skeptics (11%), Islam (4.5%) and Christianity (4%). More recent additions include Hinduism (4%) and Health (4%).

Ok. What does this all mean?

Truthfully, I'm not sure. My current guess is that people are concerned that forces outside of their control will flood the site with embarrassing or just-plain-wrong answers. (This relates to another theme I heard a lot of: the Hot Network Questions list.) Good Subjective, Bad Subjective gets cited a lot here. However, I wonder if the post gives the false impression that burden falls solely on the question. Certainly the way a question is asked increases or decreases the odds it will receive crowd-pleasing nonanswers. But ultimately, we need to hold answers to that standard too.

The other thing that kinda gets lost when talking about Good Subjective is that it isn't that opinions aren't allowed. Rather naked opinions are noise. The post even suggests:

Use your specific experiences to back up your opinions.

It seems to me that software engineering is one part science and one part craftsmanship, so it should be surprising to see some answers backed by experience rather than data. This is doubly true since the profession is incredibly young. Given quality of users on this site, there's no reason opinions (backed by research and experience) shouldn't be useful to other programmers.

Finally, it's a lot easier to delete popular answers if they don't actually answer the question. One technique that seems to work is requiring questions focus down on a specific domain. For instance, instead of Do TODO comments make sense? maybe it would be better to ask "Should TODO comments be removed when a project is in maintainance mode?" That way the focus of the question isn't a holy war between pro-TODO and anti-TODO, but about the specific utility of those comments. (Obviously the asker should specify the situation they are actually facing.)

  • Also worth noting that the reopen rate increased for the week you'd previously reported, suggesting that it may take a bit of time for questions to "settle". Or that your reporting is influencing the results. Probably the former. – Shog9 Nov 25 '15 at 5:35
  • wonder what do you think about deleted questions. Did you have time to estimate how many unsalvageable / blatantly off-topic questions site gets? – gnat Nov 25 '15 at 7:45
  • @gnat: The close statistics include deleted questions. (I use those numbers to double check my queries.) For the last 90 days, it looks like ~30% of closed questions are closed as some sort of off-topic. Whether or not they are unsalvageable or not, I don't know. But those questions are far less likely to be edited or reopened and questions closed for other reasons. So it's kinda a lot. I'm on vacation this week, so I can't compare to other site right now, however. – Jon Ericson Nov 25 '15 at 18:03
  • Jon, wrt clear cut off-topics, things are now simpler (much simpler) than in the past, these are currently easy to spot even for an outsider. Debugging, career and education advice, software and off-site resource recommendations, what project / technology to take up next. That's it. (not incidentally, all these are reflected in custom close reasons over here) Consider checking the closed (and deleted) questions from this perspective – gnat Nov 25 '15 at 20:03
  • ...on another note, since this time you poked into deletion of positively-scored answers, I wonder if your stats include answers involved in Recent Trouble With Popularity? And if yes, you better be prepared to explain how specifically these are valuable. And you better keep in mind that presenting the value as entertaining for HNQ lemmings is unlikely to bring you much friends over here. "This site is not about roses" – gnat Nov 25 '15 at 20:57
  • 2
    Thank you for acknowledging that our newfound ability to close all the crap is resulting in more editing and reopening. Your last answer had me really worried since it sounded a lot like "They used their new powers to clean up the site, how dare they!". Regarding our still-high deletion rates, I assume I'm one of not very many active delete voters, and the stuff I delete definitely falls into the "broken windows" category. As for WebApps, it seems to me like their scope "change" was more of a narrowing, which probably retro-invalidated far fewer Qs&As than our sea change did. – Ixrec Nov 26 '15 at 23:45
  • 3
    You note that Skeptics deletes answers with upvotes. This would suggest a higher level of moderator curation. With only 5% of the size of the answer count that we have, this might be doable with diamonds actively curating content (except in exceptional situations). Many sites, the diamonds specifically avoid curating content with the delete button. Is there any thought of a way to give the community a better ability to curate the answers? Outside of vote brigading for down votes on answers, it is very difficult for the community to delete answers that were once popular. – user40980 Nov 30 '15 at 0:31
  • A bit on the broken window - we often get people referencing old questions. Even closed ones. (example). These exchanges happen a few times a week and are largely hidden to people who are not closing questions on a semi-regular basis. This is part of what can drive some of the conflict over closing old questions (and keeping them closed - or deleted). It is draining on those few of us who are regularly closing to have this conversation of broken windows over and over. – user40980 Dec 3 '15 at 14:22
  • @MichaelT: Neither of those questions are broken windows in the sense that they are eyesores that encourage disorder. (I've been reading about the theory lately. :-) Those are fine questions that y'all just aren't interested in answering. Short of deleting huge swaths of history, I really don't see how this problem could possible be fixed. I notice that this particular question was closed by three users and left closed in the reopen queue by two others. (There was one user who did both.) Doesn't seem like there was much in the way of conflict this time around. – Jon Ericson Dec 3 '15 at 19:15
  • 3
    @JonEricson Its more a "we get the most tenuous suggestions to leave posts open" - the "this other post was open for a month, couldn't you leave this one open for a month too?" Which is in part why a quicker close is important. It is also a "this is a thing that the people who often close vote frequently see - the 'this other thing is open (for a period of time)'". The "things were different then" is often difficult to impress upon new users. This post isn't a badly broken window - my comment was about the comments that we deal with about broken windows on a fairly regular basis. – user40980 Dec 3 '15 at 19:20
  • 1
    ... and that even closed questions that were open for a period of time are acting as a sign post for new users to ask a similar question as they are interested in getting answers similar to how the other one did for a time. – user40980 Dec 3 '15 at 19:22
  • 4
    @JonEricson Regarding this question, while it may not seem to you like an eyesore, it is a fairly blatant Big List question. You're right; we don't particularly care for these kinds of questions; we prefer that folks come to us with a question that can actually be answered, not debated. You will note in the comments below the question that the OP did, in fact, cite another similar, closed question as evidence that he could ask his. – Robert Harvey Dec 5 '15 at 15:54
6

Getting more questions rehabilitated is a laudable goal, but having the ability to close clearly off-topic questions more quickly and get them off the front page is much more critical.


I would like to address something that you stated in the comments, in detail.

I see a lot of well-written, well-intentioned, and even well-formulated questions closed for non-obvious reasons.

It would be helpful if you could point out a few of these questions, so we can discuss them specifically.

In no small part because of the "Not Programming Related" history of Programmers and its subsequent scope change, those of us who work together to try and keep the site clean have all worked very hard over an extended period of time to try and clarify the scope of Programmers to make it easier for everyone to understand what is on-topic here.

We are therefore understandably curious why people still remain confused about the scope, especially when they visit from SE corporate. The scope is not that hard to figure out; Programmers is about Software Design.

I flip over to the on topic help and I can kinda see the reason [the question was closed] if I squint.

This is the same process that new users employ when they ask a question for the first time here: they ask first, and then read the Help Center (if they read it at all). I once asked someone who had just asked a clearly off-topic question how he managed to do that after clicking through the "yes, I understand" page. His reply was "Bah. Nobody reads those things."

So why do so many folks ask so many off-topic questions here?

  1. They look at the name of the site. While I have always pushed back against the "let's change the title of the site" folks (largely because nobody can agree on a good, descriptive title), they do have a point. People ask all kinds of questions here because the title of the site is "Programmers."

  2. They've seen a question posted previously on programmers that is similar to their question. This happens much more often than most people realize. It is the single most important reason that closing questions on Programmers needs to be easier: to get the newly-posted crap off the site, quickly.

  3. They don't really care about the site scope. These are the cross-posters, the ones who are crowdsourcing their work, the ones trying to start a conversation, argue a pedantic point or start a debate, the ones asking us to refer them somewhere else.

  4. They were referred here from Stack Overflow by well meaning, but ultimately uninformed folks who don't understand Programmers' scope. We have a bot called Duga that catches those referrals; I'd say about 95 percent of those questions are both off-topic on Programmers, and closeable on Stack Overflow under the current rules.

In addition, there are user behaviors that work against questions being rehabilitated and reopened. You ask for clarifications, but get no response. You link to an article that will help them, but they don't read it. You explain a point to them, but they argue about it. The vast majority of askers simply will not cooperate with you long enough to rehabilitate their question.


Which just leaves one remaining question: why are questions being closed that some folks at SE corporate believe should stay open?

Well, firstly, there are a very small percentage of questions that are asked that, while they don't fulfill the letter of the law on Programmers, certainly fulfill the spirit, which is to say that they are interesting questions from a Software Design perspective. We should be able to freely engage such questions, without being hampered by either folks with their own agendas or slavish adherence to the rules.

I defend such questions vigorously, argue for their existence in The Whiteboard, edit them if necessary to bring them in scope, and ask for their reopening. The three vote rule will help such questions, because it will make them easier to reopen.

The concern I have is that I believe that the pool of "technically off-topic, but still interesting from a software design perspective" questions is much smaller than you think it is, and that we will lose a valuable tool because our evaluation metric cannot be satisfied.

  • As a sufferer of ClearCase, I thought this question would have been better to edit than to close. This is doubly true since closing it deleted several useful answers. (This is the result of the question having been migrated here.) The only clue I have for why it was closed is the revision history that removed [ips-and-tricks]: http://meta.programmers.stackexchange.com/questions/7285/structured-tag-cleanup-initiative-phase-ii. I get why that's a poor tag, but why not fix the question body too? – Jon Ericson Nov 15 '15 at 1:35
  • That question has a long history... It got migrated here two years ago from Stack Overflow, but didn't close here until today, which rejected the migration and deleted all of the answers. I've undeleted it from Stack Overflow, and asked a mod to delete it from here. – Robert Harvey Nov 15 '15 at 1:52
  • 4
    Not sure what you could do to make it less broad, though. It's the epitome of a Big List question. Historically, it's been easier to categorically disallow Big List questions on Programmers than it is to argue about which ones are going to be productive, and which ones are not. – Robert Harvey Nov 15 '15 at 2:04
  • It got migrated here two years ago - just to clear it up. From stackoverflow.com/posts/449549/revisions - it was asked on SO in '09. Migrated to Programmers.SE in '11 (before the old question limit was in place?) and finally noticed when a new question with the tips-and-techqiues tag was asked recently (it was one of only two unclosed questions in the tag - the other being this poll). – user40980 Nov 15 '15 at 15:58
  • 2
    And the core question in the post is an example of a "too broad, big list question" that has people providing opinions about what they like and dislike about it. There can be no right answer. Any attempt to edit the body of the question into a core problem (putting a substantial amount of guessing into the body of the question) would likely invalidate all the answers based on how the question was structured and the answers answered ("I agree all the positive feedbacks about CC, BUT I hate its GUI and Extensibility"). It would take herculean efforts to fix the question and all answers. – user40980 Nov 15 '15 at 16:04
6

After one week, the results are not encouraging:

Week posted Qs  Closed Close Rate Closed->Edited Reopened Cl->Ed->Re Deletion Rate
----------- --- ------ ---------- -------------- -------- ---------- -------------
2015-10-12  324    190      58.6%           8.9%     2.1%       5.9%         40.7%
2015-10-19  349    204      58.5%           8.8%     2.0%       5.6%         39.5%
2015-10-26  319    207      64.9%           8.7%     1.4%      11.1%         45.8%
2015-11-02  362    218      60.2%          12.8%     3.7%      10.7%         42.3%
2015-11-09  286    205      71.7%          20.0%     3.4%       4.9%         36.0%

Most of this is in flux as weeks refer to when questions were asked. So questions asked in the week of Nov. 2 might still be closed or deleted while questions asked last week could still get edited and/or reopened. Also, a week is probably not enough time to spot trends since the numbers vary quite a bit from one week to the next. The one thing we can be confident about is that it's a lot easier to get questions closed now:

Only 3 close votes needed.

Pro forma closing

Watching the questions that have been closed in the past week reminds me of my (very short) stint creating UML diagrams: theoretically necessary, but of dubious practical value. It seems clear to me that a lot of closing on Programmers isn't because certain types of questions cause problems, but because they might. In effect, questions are closed because they have a particular form with little regard to the substance and purpose of the question. I encourage you to look at the list of recently closed questions and consider what actual problems these questions cause today.

Let me cherry-pick some recent examples:

  • How narrow or broad should constant names be? (Closed as "primarily opinion-based".)

    Is this the form of a question that could devolve into religious warfare? Sure. Has it? Well there are 3 plausible answers and one that's meh. It was open for 6 hours. Unfortunately, I can't add my answer based on page 186 of my edition of Code Complete until the question is reopened. (Hint: there's been real science done on this question.)

  • Passing fields (instance variables) by arguments (parameters) inside an object - does it make sense? (Closed as "primarily opinion-based".)

    This was immediately closed and only one answer slipped through. I'm not anything like an expert in OOP, so I can't judge the answer. Maybe it's heresy somewhere, but it sounds reasonably uncontroversial and grounded in objectivity to me. The close seems to be triggered by the final (unimportant) paragraph:

    What are the advantages and disadvantages of these solutions? Which is considered as better practise? Which one do you prefer?

  • What are the advantages of a 'traditional' 'backend' architecture for a standard CRUD app? (Closed as "too broad".)

    Two comments on the question are instructive:

    recommended reading: Why do 'some examples' and 'list of things' questions get closed?

    Asking for "the advantages" of X is almost always a list question ([the earlier comment] probably should've linked the "pros and cons" post for this one). As the existing answers have already demonstrated, you end up with a list of example applications for which X is a good thing, and then everyone has to debate in the comments whether those examples are valid because the original question wasn't sufficiently well-defined to make it clear what would or wouldn't count.

    There are exactly as many comments arguing about whether the question should be closed as there are about whether the examples are valid.

I could go on, but I don't want to get bogged down into an argument because I picked a bad example. My point in bringing these up is not to get them reopened, but encourage you all to think a bit about why questions need closing. One advantage of lowering the bar for closing is that it gives this community freedom to field more potentially problematic questions. If things get out of hand, it's far easier to shut questions down than it used to be.

To be completely clear, I have no intention of keeping this site setting if it accelerates the closing of questions. It is my firm conviction that a site which closes 70% of new questions is already failing. That's one experiment I don't want to try.


I have been encouraged by valiant and occasionally successful attempts I've seen this week to fix and reopen questions. Whatever else I might imply above, I truly do admire the care and thoughtfulness you all have exhibited over the years. It's not easy to manage a radical change in scope as the one this community has pulled off. Let me restate that: Programmers has successfully corrected itself from a very rough beginning. That was accomplished not by the Stack Exchange software, but by the efforts of users such as the readers of this post. This site is now useful to actual programmers whereas the initial scope was not. So build off of that success.

  • 2
    How narrow or broad should constant names be? -- I voted to reopen. This is one of those questions that meets the spirit of the site, if not the letter. – Robert Harvey Nov 17 '15 at 22:46
  • 3
    The close seems to be triggered by the final (unimportant) paragraph: What are the advantages and disadvantages of these solutions? Which is considered as better practise? Which one do you prefer? -- That's exactly why this question was closed. Note that the answer posted essentially said "it's up to your personal preference," lending credence to the "Primarily Opinion-Based" close reason. But I'll grant you, this one is right on the edge. – Robert Harvey Nov 17 '15 at 22:52
  • What are the advantages of a 'traditional' 'backend' architecture for a standard CRUD app? -- A good answer to this one would be quite lengthy, I think. Nevertheless, I have voted to reopen. – Robert Harvey Nov 17 '15 at 22:55
  • 7
    It's probably worth mentioning that the vast majority of users posting new questions on Programmers do not cooperate when we ask them to clarify their question or make it more specific. Consequently, the regulars have evolved a "close first, ask questions later" mentality. That's not necessarily a bad thing; without fast closures, there's literally zero motivation for a user to improve their question. – Robert Harvey Nov 17 '15 at 22:58
  • 4
    It is my firm conviction that a site which closes 70% of new questions is already failing -- Then perhaps we should try to find a way to discourage people from asking their categorically off-topic questions here in the first place. In my experience, at least 50% of new questions routinely asked on Programmers are categorically, unambiguously off-topic; the number of votes to close doesn't change that essential fact. (the vast majority of those questions are either code troubleshooting, too vague or unclear to be answerable, or not within Programmers' subject matter scope) – Robert Harvey Nov 17 '15 at 23:05
  • 2
    @RobertHarvey: A quick close is way better than a slow close (or no close) for unanswerable or problematic questions. That's one exciting thing about reducing the number of CVs required. I see lots of junky questions in the recently closed list. For those, the change is a clear win. But it's sorta shocking how many extra closed questions the change caused. I gotta assume that most of the extra closures are like the ones I called out here: not cringeworthy. – Jon Ericson Nov 17 '15 at 23:18
  • @JonEricson some of this is a "we've gotten burnt by this before" - when we haven't closed the interesting questions that hit HNQ, the question rages out of control and requires a mod to come in and delete all the comments and lock it for 24h for people to cool off. We don't want that - so it gets closed early. With regular users lacking the tools to be able to moderate a question after it hits that level of popularity, the only tools we have are the preemptive ones (which lead to questionable closes). ... – user40980 Nov 17 '15 at 23:26
  • ... we've really tried to avoid the need for activist moderators like what happened on SF and is currently having some hubbub on meta.SE and the other P.SE site. We don't want to need that amount of diamond moderation where the community can do it instead. For the community, this means closing some of the borderline questions early if we foresee a holy war and excessive comments on the horizon. – user40980 Nov 17 '15 at 23:28
  • 1
    @RobertHarvey worth noting that "close first..." approach appears to be popular since at least 2011. This popularity seems to be network wide, not only at Programmers – gnat Nov 18 '15 at 11:25
  • I would be curious to note the numbers if you exclude users who are not very active normally in closing/reopening, and are only active now as a result of the attention this meta post is giving. I'm one of them... I no longer consider myself an active participant, but I did some reviews as a result of seeing this meta post. I am glad to help try and reopen things as I don't see many users voting to reopen on this site, so know my vote can be very useful right now, however I normally am not an active user here. – Rachel Nov 18 '15 at 16:14
  • 2
    @Jon, this is the response I expected. We have an excess supply of close votes, such that people are using them on borderline and even old questions, on the theory that they "provide poor examples" to new question askers. To incentivize the behavior you want, my suggestion is to limit the number of close votes per person well below current levels, but allow people to earn more by casting reopen votes after edits. – Karl Bielefeldt Nov 18 '15 at 16:45
  • @KarlBielefeldt what is wrong with closing broken windows from old questions? Are you expecting all the questions form 2012 to remain open indefinitely? There are users posting tooling questions based on the existence of a tag and open questions there. This is not a hypothetical. – user40980 Nov 18 '15 at 17:36
  • 2
    Unfortunately the mere existence of a tag can serve as "proof" to a new user that questions related to the tag are on-topic. – Robert Harvey Nov 18 '15 at 19:05
  • 2
    I'm not arguing against the concept. I'm objecting to the injudicious application of the concept. You guys have taken a good idea to a ridiculous extreme. – Karl Bielefeldt Nov 18 '15 at 20:38
  • 2
    Jon, if you want the kind of softer / broader of questions over here, then I think you also need to choose whether you want them for "knowledge" or for "entertainment". And if you want them for the former, you better help us figure how to avoid clashing of their goal with "entertaining" hot questions – gnat Nov 19 '15 at 18:33
5

Another aspect you may not have considered, which I believe is also unique to PSE, is the number of close votes being used on old questions. I have a SEDE query I run occasionally that frequently catches questions closed from several years ago. Even relatively recent questions that were fairly well received, are often closed after a delay of a week or so, when its defenders have moved on.

These are questions that in my opinion do not need editing, for the most part. They are being closed under the radar by people who are retroactively applying stricter and stricter standards to the site, without having to face the scrutiny of the consensus that existed during the prime of that question's activity.

I would simply like a notification if a question I endorsed by upvoting or answering is closed later. The defenders of a question should receive as much consideration as the close queue gives its detractors.

  • 3
    I remain very disappointed at the categorical reopen votes cast upon questions that you have answered without any attempt to address the reason that the question is closed. Old questions do not merit being grandfathered into permanent open status. – user40980 Nov 17 '15 at 23:15
  • @MichaelT For the record, I didn't agree with the reason the questions were closed, so I didn't really see anything that needed "fixing". Actually there was one I admitted may have been broad if it wasn't for the very nice answer and link to a much more detailed post on another site, which I thought made it a very useful question worth keeping. – Rachel Nov 18 '15 at 2:44
  • Look at them, and consider that the closings were made in good faith. If you have questions about why they were closed, ask. Please do not regard every answer you have made to be something that should remain open. If there is a post that has part of the question saying "So, where do I draw the line? Do you feel that hiding details is unethical or just a normal job function of a developer?" - that is asking for opinions. – user40980 Nov 18 '15 at 2:54
  • 3
    @MichaelT: One of the things that makes this conversation so frustrating is that there seems to be such a hard line on questions here. On most sites, those sentences would be ignored or edited out. Here they trigger close votes, down votes and, often, deletion. Maybe you need a zero tolerance stance. I don't know. But the way it's expressed around here just sounds so dogmatic and hostile to outsiders. Your initial comment to this question borders on disrespectful. Why is it wrong for someone else to exercise a privilege they have earned? – Jon Ericson Nov 18 '15 at 20:31
  • 1
    @JonEricson It has been an open issue for some time. It was directly addressed a year and a half ago and a few times in chat before and after then. I am very concerned that there is a perception that questions from before ${time ago} should remain open and not be bothered if they are open now. Even though it has been shown that these broken windows cause problems for new users. I am disappointed that one cannot take the feedback of several users and examine the question again with a critical eye to see if something can be improved. – user40980 Nov 18 '15 at 20:37
  • 3
    @MichaelT: It seems (from looking at the votes) that your meta question does not represent attitude of the community at large. Is it not possible for people to disagree about what's best for this site? If you are expressing your preferences via close vote, why can't others express their preferences though reopen votes? – Jon Ericson Nov 18 '15 at 20:49
  • 2
    @JonEricson If there is to be the accepted community policy that we don't bother with questions from years ago, please look at communicating this with the moderators here, closing the STCI, and going from there and make a policy statement about it. There is no 'closing under the radar' as suggested here - everything is in the review queue. If you believe that this is how the closing should work, please say so, so that we can allocate our votes and time in a way that is more in line with how SE wants this site to be run. – user40980 Nov 18 '15 at 21:06
  • 2
    @MichaelT: I would vastly prefer the community run the site together. I think you should continue to vote to close the way you prefer. But in exchange, I'm asking that you allow others to vote they way they prefer even if it's not a good idea. If you want to influence their preferences, please be respectful about it. I'll be back in a few weeks to wrap this experiment up. – Jon Ericson Nov 18 '15 at 21:21
  • @JonEricson indeed. Just noting however that if we are being asked to reconsider reopening some borderline questions, it would also be nice if people look at them critically and see if there was a reason that it was closed that way that could be changed to make it more likely to get reopened. This is especially useful if the corresponding answer(s) were adjusted to address the modified question. – user40980 Nov 18 '15 at 21:31
  • 2
    ... Leaving text such as ""So, where do I draw the line? Do you feel that hiding details is unethical or just a normal job function of a developer?" in the question when voting to reopen really feels like the question is asking for opinions on ethics. Removing that and refocusing the question rather than just casting reopen votes can go a long way - especially when its one's own answer that can get changed. – user40980 Nov 18 '15 at 21:32
  • 2
    @JonEricson "On most sites, those sentences would be ignored or edited out. Here they trigger close votes, down votes and, often, deletion" - could you look at meta.programmers.stackexchange.com/q/7737 (and the corresponding main question). This has one of those problematic tool recommendation sentences in it. Sure, we can edit it out - but we don't have the tools to make the answers consistent with the question - because the accepted answer is only a tool recommendation and is up voted. How do we fix these questions? – user40980 Nov 18 '15 at 22:58
  • ... and if we don't remove the answer about a three and a half year old (and three major revision out of date) software product, someone else will come by with another software recommendation that is a valid answer (maybe even the current version). But its a valid answer (and accepted) and so the mods are unlikely to delete it with a flag necessary to fix the question. What course of action can we (community - not mods) take so that we can edit out the problematic sentence to avoid deletion and possibly reopen it? – user40980 Nov 18 '15 at 23:02
1

It does seem to me that the state of affairs on PSE is exactly what you'd want - questions are either good or bad and voted on accordingly. Superuser seems much more vague with questions that might be good, or might be bad, or might be meh and as such you see that voting pattern.

I don't think the issue is one of close vote quantity then, but question quality.

It would be interesting to see the time taken to close a question on both sites - I think here on PSE we tend to stomp on off-topic questions pretty quickly.

One suggestion for close votes if you were to change them, is to weight votes according to rep. So someone with 2k rep gets their close vote to count as 1, and someone with 20k rep counts as 2. That'd get them closed quicker.

  • 1
    One of those "I wonder what it would look like" things... voteWeight = f(rep). And look at how the distribution of vote weight for active close voters for f() being linear, logarithmic, or sigmoid. So Robert would always have a weight of 2, and a 3k user would always be 1, and everyone else in between. – user40980 Nov 17 '15 at 15:22
-1

This experiment appears to be a dead end. The problem it revealed can't be fixed by quick closes... it can't be fixed by closes at all.

Per my observations, since the experiment started, we've got lively and interesting reopen queue. To me this indicates that solid amount of closed questions are worth improving and reopening.

Thing is though, when I tried to find these questions myself, not through reopen queue, it turned out very difficult. Site is flooded with blatantly off-topic, unsalvageable crap that hides questions deserving a closer look.

  • Guess we need to thank SE Quality Project which (finally) made it really difficult for low quality questions to get answers at SO. As a result, askers are now trying their luck at other sites. (I wonder by the way why they put "Stack Exchange" in the name of that project. Only site that benefits of it is Stack Overflow; at Programmers it only brings pain and suffering.)

Salvageable questions being visible only through reopen queue is very bad thing. It means that only very small fraction of site users can participate - 3K users who frequent that queue plus few others who were lucky enough to catch a question before it was buried under heaps of unworthy garbage.

This in turn means very low chance for questions to improve and reopen, and no matter how fast we close, we can't change that - the flood of hopeless stuff simply hides questions worth looking at.

Above makes me think that "Server Fault experiment" may be a better way to go for us. Folks over there recently elected moderators who seem to be quickly closing and most importantly, deleting, questions that are clear cut off-topic. This way we can get more usable site front page where every user can find gray area, incorrectly closed, salvageable questions and take care of these.

(On a historical note, I couldn't even imagine proposing above option two years ago. Back then, site scope wasn't settled firmly enough to allow such measures.)


Meanwhile, an advice to those who want to continue reopening efforts no matter what.

When searching through closed questions, skip most recent ones and proceed to those posted 3-4 days ago. At this age, some of the worst off-topics are cleaned up by delete votes of 20K / 10K users.

You may get even cleaner selection at 10 days old questions which are additionally cleaned by roomba, but since that script is automatic, it may also delete some salvageable questions.

  • 2
    "curious if this change helps people enjoy the site more overall" -- that part of experiment worked for me – gnat Nov 20 '15 at 22:02
  • One advantage is that a quick close means any OP-edits bump the post into the reopen queue, and more visibility. With a slower close those edits might be between CVs 4/5 or something and much fewer people see the post in its new form. – enderland Nov 23 '15 at 3:20
  • @enderland agree, I noticed that too. That's certainly an advantage over what we had with slower closes – gnat Nov 23 '15 at 4:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .