The number of votes required to close and reopen questions is temporarily reduced . . . for science!

Question

This experiment is over. I will be adding an answer to show the results and suggest a plan for going forward.

---

For the next 30 days, closing a question will require just 3 votes. Also 3 votes will be sufficient to reopen a closed question. In essence, each close and reopen vote has 5/3^rds the normal value. After 30 days I will revert the site setting back to 5 and evaluate the results.

I have tested this change on a development instance and questions that currently have 3 or 4 close votes will not be closed automatically. Rather they will be closed when the next close vote comes in.

Background

I think it's fair to say there's a lot of dissatisfaction with the question closing system on Programmers. One proposed solution is to increase the number of close votes available to 10k users. Some of us on the community team have considered various schemes to scale close votes with reputation under the theory that people most invested in the site (as measured by reputation) will have the most incentive to curate questions. That might still happen, but we'd like to be pretty sure that it will solve the various problems with question closing.

It's pretty clear that organic closing (via the close votes queue or people voting to close questions they find naturally) isn't working very well. To demonstrate, here's how close votes on questions asked in the last 30 days are distributed on Super User:

Most questions have no close votes (not shown). 461 questions have just one (the yellow circle on the left) and the number slopes down to 142 with 4 close votes. 5 votes is a natural attractor because people will keep voting to close off-topic and otherwise closable questions until they reach that limit. There's a long tail beyond 5 close votes (also not shown) due to votes expiring, close wars and so on. In essence, questions with at least 1 close vote, but not 5, are in a grey area in terms of closablity. This general pattern turns out to be extremely common across a variety of sites.

In contrast, here's the same graph for Programmers—Stack Exchange:

Remember this is only showing questions that have been asked in the last 30 days; we aren't talking about Not Programming Related questions. Instead of a fairly smooth gradient between questions that should clearly be open to questions that clearly should be closed, this graph shows something closer to a binary state. My hypothesis is that since the people contributing the most close votes feel the need to conserve votes, they don't vote to close borderline questions. There's too great a risk those votes will expire impotently.

I've also included the counts of questions that have been edited or deleted in each close vote bin. Relative to other sites, questions on Programmers with close votes are more likely to be deleted and less likely to be edited. It's entirely possible that a greater proportion of questions are unsalvageable on this site. But I suspect the underlying problem is that the questions that could most benefit from an improvement are not getting close votes in the first place. That might not be a problem, but close votes can provide incentives for askers to fix problems even before their questions are closed. Close votes also put questions in front of reviewers who might be inspired to improve them.

Another problem that we discovered very dramaticly is that closed questions are rarely reopened. Most closed questions can't (and shouldn't) be reopened, but statistically speaking, we expect more than 1-2% will be. My hypothesis is that it takes so much effort (and coordination) to close questions that people are not interested in undoing that work even if there's been an edit to fix a question's problems.

To be clear, the breakdown in the close voting system is not limited to Programmers. It is, however, rare across the network for the division between open and closed questions to be so stark. We hope this test will help us understand how the close vote system breaks in order to (hopefully) fix it.

Why 3 votes?

Initially users with enough reputation could unilaterally close and reopen questions on Stack Overflow. In December 31, 2008, the close vote system was created and required 3 votes. Between then and the next April, the limit was increased to 5. (Shog9 remembers 3 lasting a few weeks.) The documentation of the site setting reads:

KEEP THIS AN ODD NUMBER

So we need an odd number greater than 1 and less than 5.

How will this change be evaluated?

There's a nontrivial chance something will break requiring me to revert the change. If we continue for the full 30 days, we will compare before and after the change:

• Close rate

  Obviously, it's now easier to close questions. Insofar as bad questions are put on a path toward either corrective edits or deletion, that's a good thing. But this site closes well over 50% of new questions, which is the highest rate on the network. (Considering total questions closed to questions asked in a time period, the rate is more like 70%.) Paradoxically, I'd like to see the close rate remain steady or decrease.

• Reopen rate

  Reopening questions is one way to mitigate against a high close rate. Since the bar for reopening is now lower, we'd expect more questions will be reopened. Currently the reopen rate is about 2% of closed questions.

• Edit rate on closed questions

  In general, the only path to reopening a question is editing. It's also the most effective way to prevent questions from being closed in the first place. Since it's easier to reopen questions, there's even more reason for askers to fix their own closed questions. In the last 30 days 9% of closed questions were subsequently edited.

• Deletion rate

  About 65% of questions that get at least one close vote on Programmers are deleted. We aren't exactly looking at the cream of the crop here, but that's noticeably more than other sites. (Again, this only counts questions asked and deleted within the last 30 days.) We can't force people to ask questions that don't deserve deletion, but hopefully more questions can be salvaged with edits.

Finally, I'm curious if this change helps people enjoy the site more overall. At the end of the experiment, I'll post my results and ask for any followup thoughts you as a community have.

How you can help

Relatively few people are active in closing, reopening and deleting questions. Even fewer are active in editing. In order for this experiment to improve the quality of content on this site, more people need to be involved in identifying questions with potential to be unlocked with a judicious edit. Fixing tags and removing signatures are good first steps. But title and (occasionally heroic) body edits might be needed to pull some questions out of mediocrity. Remember: once the question is posted, it belongs to the community as much as it does the original author.

In addition, please use the answers below to let me know how the experiment is going.

Answer 1

I'm putting this out there as a possible alternative interpretation of the flat and peak graph for Programmers.SE close votes.

Our scope is more cleanly defined than we are given credit for.

The close votes that are cast, and the ultimately deleted questions are done with a great deal of consistency. The questions that are closed really are off topic or not a good fit for the Q&A format. The majority of the people who actively cast close voters recognize this and vote in accordance with this scope.

The problem that the site faces that requires additional close votes isn't so much "we're holding back on borderline questions" but rather "we are deluged with off topic questions and questions that presuppose this site is for opinions and discussion."

Under this perspective, Programmers.SE is open to those well written and answerable questions that come our way. This also acknowledges the ongoing challenge of actually presenting a well written and answerable question, though that is one that may be easier to address if one has more time to guide those questions without having to also spend as much energy on the questions asking for book recommendations and the like.

Unfortunately, the way to differentiate the "we're holding back on close votes on borderline" and the "we are accepting of the well written answerable question that pushes the boundaries without incurring excessive drama and discussion" is to look objectively at the questions (or a sufficiently representative sample) we close or have cast close votes on and then see:

• Are there borderline questions with 1 or 2 close votes?
• Are there hidden gems that have been closed?
• What percentage of the incoming questions are unsalvageable?

As we don't have triage, that last point is an awkward one here. It might be useful for the community manager team to grab a random sample of the questions that we close... say 100 posts and do a 10 second glance at each and determine which pile it should go to (the 'unsalvageable', 'needs improvement', or 'looks ok' bins).

Whatever the case, a faster turnaround time on close votes to closed question means faster feedback for the OP (and in cases where its a good question for Stack Overflow, a faster migration to there). When questions sit for hours before they are able to muster sufficient close votes, the OP may well have lost interest in fixing the question (that would be another query to run - OP engagement with a question over time and see if moving the closed question to within that timeframe allows for the OP to be more active in fixing those borderline questions that do get closed).

Answer 2

I wasn't going to update the experiment this week, but the results this week were so encouraging I can't help myself:

Week posted Qs  Closed Close Rate Closed->Edited Reopened Cl->Ed->Re Deletion Rate
----------- --- ------ ---------- -------------- -------- ---------- -------------
2015-10-12  325    190      58.5%           8.9%     2.6%      11.8%         40.9%
2015-10-19  349    205      58.7%           8.8%     2.4%       5.6%         39.8%
2015-10-26  319    207      64.9%           8.7%     1.4%      11.1%         46.1%
2015-11-02  362    218      60.2%          13.3%     3.7%      10.3%         44.2%
2015-11-09  286    206      72.0%          20.9%     3.9%       7.0%         41.3%
2015-11-16  340    205      60.3%          17.6%     9.3%      30.6%         35.6%

Not only has the close rate dropped to something like the pre-experiment rate, the reopen rate has rather dramatically increased. Much of this is due to more edited questions getting reopened than in previous weeks. (It might be too early to tell if it's statistically significant, but more closed questions are being edited since the test began.)

I also spent some time last week trying to learn what prompted people to close so many questions on Programmers. While there are many detailed reasons, two themes emerged in my mind:

1. Questions that remain open are signposts that prompt new questions of the same type.

2. It's difficult to maintain the quality of answers on certain types of questions.

The first didn't surprise me. It's a variation of the broken window theory (which is different than "zero-tolerance policing" by the way.) Since it's often difficult to curate a large corpus of answers, it's a lot more convenient to curate the questions instead. Some sites, however, pull it off. Skeptics, for instance, has strict rules about what sort of questions can be asked (notably, notable ones) and what is required of answers (particularly that they be referenced). As a result, Skeptics deletes a large percentage of answers that have a score > 0:

answers deleted % edited % locked % notice % Q deleted % Site Name 
------- --------- -------- -------- -------- ----------- --------- 
   7634     35.5     19.64     0.05     0          97.16 Area 51 Discussions
   8288     13.61    55.68     0.1      6.03       28.63 Skeptics
   5782     10.26    46.49     0.09     0.02       48.74 Islam     
 141392      9.38    25.01     0.45     0.02       78.15 Programmers
   4810      6.63    52.45     0.12     0.46       27.9  Biblical Hermeneutics
  15660      5.86    45.84     0.03     0.37       48.31 Christianity
  22572      5.08    28.81     0.33     0.02       63    Web Apps

I cut the list off at sites that delete at least 5% of positively-scored answers. Area 51 is interesting because 97% of "good answers" get deleted when their question gets deleted as a result of their failed proposal being deleted. But the next in line is a real site: Skeptics. Their policies encourage deletion not only of entire questions, but individual answers as well. (Skeptics is unique in its extensive use of post notices.) Islam, Christianity and Biblical Hermeneutics have fairly strict requirements that answers be well-supported. Those sites also emphasize narrow questions that allow moderators to delete many answers as non-answers.

I'm less familiar with Web Apps, but I believe they had a major change in scope not unlike what happened on Programmers. When you shift the topic of a site, that will leave many now-off-topic questions that need deleting along with all their answers. As a corollary, there's a real need to lock questions and their answers. But look what happens when we limit to answers posted in the last year:

answers deleted % edited % locked % notice % Q deleted % Site Name 
------- --------- -------- -------- -------- ----------- --------- 
  10617      4.53    34.44     0.01     0.03       66.53 Programmers
   2999      1.93    27.91     0.23     0          48.28 Web Apps  

Web Apps has settled into a fairly normal pattern of deleting relatively few upvoted answers. It's also split fairly evenly between deleting those answers individual and as a result of deleting the question. Programmers has become a little less extreme in terms of positively-scored-answer deletion, but it still ranks near the top of the list with Area 51 (31%), Skeptics (11%), Islam (4.5%) and Christianity (4%). More recent additions include Hinduism (4%) and Health (4%).

Ok. What does this all mean?

Truthfully, I'm not sure. My current guess is that people are concerned that forces outside of their control will flood the site with embarrassing or just-plain-wrong answers. (This relates to another theme I heard a lot of: the Hot Network Questions list.) Good Subjective, Bad Subjective gets cited a lot here. However, I wonder if the post gives the false impression that burden falls solely on the question. Certainly the way a question is asked increases or decreases the odds it will receive crowd-pleasing nonanswers. But ultimately, we need to hold answers to that standard too.

The other thing that kinda gets lost when talking about Good Subjective is that it isn't that opinions aren't allowed. Rather naked opinions are noise. The post even suggests:

Use your specific experiences to back up your opinions.

It seems to me that software engineering is one part science and one part craftsmanship, so it should be surprising to see some answers backed by experience rather than data. This is doubly true since the profession is incredibly young. Given quality of users on this site, there's no reason opinions (backed by research and experience) shouldn't be useful to other programmers.

Finally, it's a lot easier to delete popular answers if they don't actually answer the question. One technique that seems to work is requiring questions focus down on a specific domain. For instance, instead of Do TODO comments make sense? maybe it would be better to ask "Should TODO comments be removed when a project is in maintainance mode?" That way the focus of the question isn't a holy war between pro-TODO and anti-TODO, but about the specific utility of those comments. (Obviously the asker should specify the situation they are actually facing.)

Answer 3

Getting more questions rehabilitated is a laudable goal, but having the ability to close clearly off-topic questions more quickly and get them off the front page is much more critical.

---

I would like to address something that you stated in the comments, in detail.

I see a lot of well-written, well-intentioned, and even well-formulated questions closed for non-obvious reasons.

It would be helpful if you could point out a few of these questions, so we can discuss them specifically.

In no small part because of the "Not Programming Related" history of Programmers and its subsequent scope change, those of us who work together to try and keep the site clean have all worked very hard over an extended period of time to try and clarify the scope of Programmers to make it easier for everyone to understand what is on-topic here.

We are therefore understandably curious why people still remain confused about the scope, especially when they visit from SE corporate. The scope is not that hard to figure out; Programmers is about Software Design.

I flip over to the on topic help and I can kinda see the reason [the question was closed] if I squint.

This is the same process that new users employ when they ask a question for the first time here: they ask first, and then read the Help Center (if they read it at all). I once asked someone who had just asked a clearly off-topic question how he managed to do that after clicking through the "yes, I understand" page. His reply was "Bah. Nobody reads those things."

So why do so many folks ask so many off-topic questions here?

1. They look at the name of the site. While I have always pushed back against the "let's change the title of the site" folks (largely because nobody can agree on a good, descriptive title), they do have a point. People ask all kinds of questions here because the title of the site is "Programmers."

2. They've seen a question posted previously on programmers that is similar to their question. This happens much more often than most people realize. It is the single most important reason that closing questions on Programmers needs to be easier: to get the newly-posted crap off the site, quickly.

3. They don't really care about the site scope. These are the cross-posters, the ones who are crowdsourcing their work, the ones trying to start a conversation, argue a pedantic point or start a debate, the ones asking us to refer them somewhere else.

4. They were referred here from Stack Overflow by well meaning, but ultimately uninformed folks who don't understand Programmers' scope. We have a bot called Duga that catches those referrals; I'd say about 95 percent of those questions are both off-topic on Programmers, and closeable on Stack Overflow under the current rules.

In addition, there are user behaviors that work against questions being rehabilitated and reopened. You ask for clarifications, but get no response. You link to an article that will help them, but they don't read it. You explain a point to them, but they argue about it. The vast majority of askers simply will not cooperate with you long enough to rehabilitate their question.

---

Which just leaves one remaining question: why are questions being closed that some folks at SE corporate believe should stay open?

Well, firstly, there are a very small percentage of questions that are asked that, while they don't fulfill the letter of the law on Programmers, certainly fulfill the spirit, which is to say that they are interesting questions from a Software Design perspective. We should be able to freely engage such questions, without being hampered by either folks with their own agendas or slavish adherence to the rules.

I defend such questions vigorously, argue for their existence in The Whiteboard, edit them if necessary to bring them in scope, and ask for their reopening. The three vote rule will help such questions, because it will make them easier to reopen.

The concern I have is that I believe that the pool of "technically off-topic, but still interesting from a software design perspective" questions is much smaller than you think it is, and that we will lose a valuable tool because our evaluation metric cannot be satisfied.

Answer 4

After one week, the results are not encouraging:

Week posted Qs  Closed Close Rate Closed->Edited Reopened Cl->Ed->Re Deletion Rate
----------- --- ------ ---------- -------------- -------- ---------- -------------
2015-10-12  324    190      58.6%           8.9%     2.1%       5.9%         40.7%
2015-10-19  349    204      58.5%           8.8%     2.0%       5.6%         39.5%
2015-10-26  319    207      64.9%           8.7%     1.4%      11.1%         45.8%
2015-11-02  362    218      60.2%          12.8%     3.7%      10.7%         42.3%
2015-11-09  286    205      71.7%          20.0%     3.4%       4.9%         36.0%

Most of this is in flux as weeks refer to when questions were asked. So questions asked in the week of Nov. 2 might still be closed or deleted while questions asked last week could still get edited and/or reopened. Also, a week is probably not enough time to spot trends since the numbers vary quite a bit from one week to the next. The one thing we can be confident about is that it's a lot easier to get questions closed now:

Pro forma closing

Watching the questions that have been closed in the past week reminds me of my (very short) stint creating UML diagrams: theoretically necessary, but of dubious practical value. It seems clear to me that a lot of closing on Programmers isn't because certain types of questions cause problems, but because they might. In effect, questions are closed because they have a particular form with little regard to the substance and purpose of the question. I encourage you to look at the list of recently closed questions and consider what actual problems these questions cause today.

Let me cherry-pick some recent examples:

• How narrow or broad should constant names be? (Closed as "primarily opinion-based".)

  Is this the form of a question that could devolve into religious warfare? Sure. Has it? Well there are 3 plausible answers and one that's meh. It was open for 6 hours. Unfortunately, I can't add my answer based on page 186 of my edition of Code Complete until the question is reopened. (Hint: there's been real science done on this question.)

• Passing fields (instance variables) by arguments (parameters) inside an object - does it make sense? (Closed as "primarily opinion-based".)

  This was immediately closed and only one answer slipped through. I'm not anything like an expert in OOP, so I can't judge the answer. Maybe it's heresy somewhere, but it sounds reasonably uncontroversial and grounded in objectivity to me. The close seems to be triggered by the final (unimportant) paragraph:

  What are the advantages and disadvantages of these solutions? Which is considered as better practise? Which one do you prefer?

• What are the advantages of a 'traditional' 'backend' architecture for a standard CRUD app? (Closed as "too broad".)

  Two comments on the question are instructive:

  recommended reading: Why do 'some examples' and 'list of things' questions get closed?

  Asking for "the advantages" of X is almost always a list question ([the earlier comment] probably should've linked the "pros and cons" post for this one). As the existing answers have already demonstrated, you end up with a list of example applications for which X is a good thing, and then everyone has to debate in the comments whether those examples are valid because the original question wasn't sufficiently well-defined to make it clear what would or wouldn't count.

  There are exactly as many comments arguing about whether the question should be closed as there are about whether the examples are valid.

I could go on, but I don't want to get bogged down into an argument because I picked a bad example. My point in bringing these up is not to get them reopened, but encourage you all to think a bit about why questions need closing. One advantage of lowering the bar for closing is that it gives this community freedom to field more potentially problematic questions. If things get out of hand, it's far easier to shut questions down than it used to be.

To be completely clear, I have no intention of keeping this site setting if it accelerates the closing of questions. It is my firm conviction that a site which closes 70% of new questions is already failing. That's one experiment I don't want to try.

---

I have been encouraged by valiant and occasionally successful attempts I've seen this week to fix and reopen questions. Whatever else I might imply above, I truly do admire the care and thoughtfulness you all have exhibited over the years. It's not easy to manage a radical change in scope as the one this community has pulled off. Let me restate that: Programmers has successfully corrected itself from a very rough beginning. That was accomplished not by the Stack Exchange software, but by the efforts of users such as the readers of this post. This site is now useful to actual programmers whereas the initial scope was not. So build off of that success.

Answer 5

Another aspect you may not have considered, which I believe is also unique to PSE, is the number of close votes being used on old questions. I have a SEDE query I run occasionally that frequently catches questions closed from several years ago. Even relatively recent questions that were fairly well received, are often closed after a delay of a week or so, when its defenders have moved on.

These are questions that in my opinion do not need editing, for the most part. They are being closed under the radar by people who are retroactively applying stricter and stricter standards to the site, without having to face the scrutiny of the consensus that existed during the prime of that question's activity.

I would simply like a notification if a question I endorsed by upvoting or answering is closed later. The defenders of a question should receive as much consideration as the close queue gives its detractors.

Answer 6

It does seem to me that the state of affairs on PSE is exactly what you'd want - questions are either good or bad and voted on accordingly. Superuser seems much more vague with questions that might be good, or might be bad, or might be meh and as such you see that voting pattern.

I don't think the issue is one of close vote quantity then, but question quality.

It would be interesting to see the time taken to close a question on both sites - I think here on PSE we tend to stomp on off-topic questions pretty quickly.

One suggestion for close votes if you were to change them, is to weight votes according to rep. So someone with 2k rep gets their close vote to count as 1, and someone with 20k rep counts as 2. That'd get them closed quicker.

Answer 7

This experiment appears to be a dead end. The problem it revealed can't be fixed by quick closes... it can't be fixed by closes at all.

Per my observations, since the experiment started, we've got lively and interesting reopen queue. To me this indicates that solid amount of closed questions are worth improving and reopening.

Thing is though, when I tried to find these questions myself, not through reopen queue, it turned out very difficult. Site is flooded with blatantly off-topic, unsalvageable crap that hides questions deserving a closer look.

• ^{Guess we need to thank SE Quality Project which (finally) made it really difficult for low quality questions to get answers at SO. As a result, askers are now trying their luck at other sites. (I wonder by the way why they put "Stack Exchange" in the name of that project. Only site that benefits of it is Stack Overflow; at Programmers it only brings pain and suffering.)}

Salvageable questions being visible only through reopen queue is very bad thing. It means that only very small fraction of site users can participate - 3K users who frequent that queue plus few others who were lucky enough to catch a question before it was buried under heaps of unworthy garbage.

This in turn means very low chance for questions to improve and reopen, and no matter how fast we close, we can't change that - the flood of hopeless stuff simply hides questions worth looking at.

Above makes me think that "Server Fault experiment" may be a better way to go for us. Folks over there recently elected moderators who seem to be quickly closing and most importantly, deleting, questions that are clear cut off-topic. This way we can get more usable site front page where every user can find gray area, incorrectly closed, salvageable questions and take care of these.

(On a historical note, I couldn't even imagine proposing above option two years ago. Back then, site scope wasn't settled firmly enough to allow such measures.)

---

Meanwhile, an advice to those who want to continue reopening efforts no matter what.

When searching through closed questions, skip most recent ones and proceed to those posted 3-4 days ago. At this age, some of the worst off-topics are cleaned up by delete votes of 20K / 10K users.

You may get even cleaner selection at 10 days old questions which are additionally cleaned by roomba, but since that script is automatic, it may also delete some salvageable questions.