How to reconcile guidelines, community opinion and moderation

Question

First of all, I realize that this topic has already been discussed ad nauseam, so I apologize in advance if nothing fruitful comes of this. I strongly feel that there is a large disconnect between the community's understanding of the site's purposes and the collective understanding of the moderators. I'm bringing this up again because I feel that previous answers on this topic have been a bit dismissive, and I was hoping that a more data-driven approach might elicit a more meaningful discussion.

I did a few queries on data.stackexchange.com to get some statistics about the number of questions being closed on each site. Unfortunately, saving the queries is not working properly for me, or I'd link directly to them. Instead, I've tabulated the results in this spreadsheet. I found the following:

Site           | % Closed with Score <= 0 | % Closed with Score >= 5
---------------+--------------------------+--------------------------
Stack Overflow | 66.35%                   | 5.08%
Programmers    | 38.5%                    | 23.3%
English        | 47.47%                   | 6.27%
Gaming         | 47.55%                   | 9.18%

The percent columns represent the percentages of all closed questions with scores less than or equal to zero, and greater than or equal to five, respectively. Overall, a relatively high percentage of questions are closed on Programmers compared to other Stack Exchange sites, but I'm not going to give specific numbers on that because to do so would probably not be constructive.

Please note some caveats about these numbers: the community on each site is different and there is no accepted or target ratio for any of these values. The numbers represent all non-deleted questions from the beginning of each site's existence. It's difficult to make proper comparisons because of differences in the number of users and questions on each site, as well as how mature each site is. I am using these numbers merely to illustrate why I think there is room for improvement.

Without wishing to sound whiny, from a personal perspective I'm rather uncertain about asking any questions on Programmers. Even after reading the FAQ and the Good Subjective, Bad Subjective post, I don't feel like I really have a good understanding of what kinds of questions are acceptable. I think the guidelines do a decent job of explaining what not to ask, but are not nearly specific enough on what should be asked. (For instance, there are four examples of bad questions, but not a single example of a good question.)

Now, I am not suggesting that the mods are wrong, or that we should accept questions which do not fall within the guidelines. What I am saying is that these data, to me, make it very clear that the community and the mods are not in complete agreement about the purpose of the site or about how to interpret the question guidelines.

What I am asking for are suggestions as to how we can:

• Bring the community's idea of the site in line with official Stack Exchange policies and guidelines.
• Reduce the number and percentage of closed questions (and resultant disgruntled users).
• Maintain a friendly and mutually respectful atmosphere between moderators and users.

Thanks!

Answer 1

I see several types of closed questions on the Programmers' stack exchange that concern me.

1. Non-duplicate-duplicates: Questions closed as duplicates that are at least some degree different from the question that is cited as a duplicate. (The concern here is that a good question, when asked in a different light, can elicit different, possibly better answers).

2. Poor questions: Questions closed as not fulfilling enough of the 6 guidelines for a good question that are "on the border" of fulfilling enough of the guidelines that, to the casual observer, appear to be good questions. (The concern here is that closing good questions makes the exchange less valuable as a medium for exchanging information about, in this case, the programming profession).

3. Mis-labeled closed questions: Questions closed as not fulfilling guidelines that are actually good questions, but clearly off-topic, such as this one on what is an iteration, or this one from a 14-year old beginner. (The concern here is that newbies can become confused between what it means to stay on topic and how to ask good questions. Another concern is that moderation can take the appearance of being incompetent when it lists the wrong reason for closing a question.)

4. Marginally off-topic questions: Questions that are closed as being off topic that, while not being exactly on topic, have the casual appearance of being on topic, such as this one about interruptions. (The concern here is that closing good questions makes the exchange less valuable as a medium for exchanging information about, in this case, the programming profession. Another concern is that moderation can take the appearance of being in the control of mindless bureaucrats rather than reasoned, pragmatic elders.)

One way to handle this is to ask the moderators to take some extra time to explain close decisions.

For instance, noting that Programmers' stack exchange is for expert programmers would be sufficient to close this one and this one also.

For ***poor questions***such as this one on GIT or this one on regex/regexp, ticking off the guidelines that are not present would really help casual users understand the elements of a good question better. For instance, the regex/regexp question could be closed because it is merely mindless fun, it cannot be answered through facts or references, it does not invite sharing of experiences, it is not asked in an impartial tone, it only elicits short answers, and it will not inspire good answers. (Note -- I am curious about why the regex/regexp question is not a good question...

The entire community can help out with mis-labeled closed questions by flagging then for re-labeling.

When closing, marginally off-topic questions -- either close them with the caveat that they be reworded to apply to expert programmers, or explain why the question is considered off-topic. For instance, the question about interruptions is a real problem for programmers as interruptions impact the productivity of programmers differently than they impact most other professions, so either it can be reworded to apply to programmers, or it should be explained that this is off-topic because other professions (such as mathematicians) also have the same problem with interruptions, so this question will always be considered off-topic in this stack exchange.

I do not know the answer to the non-duplicate-duplicate issue, this one is heavily discussed in meta.stackoverflow as well. It is just one of the minor annoyances of using stack exchanges, I guess.

The moderators for this stack exchange have been very busy lately. I do not know if it is fair to ask them to shoulder the additional thought and care that the above recommendations would entail.

Answer 2

I am very unhappy with the amount of closing and the types of questions being closed.

I feel very strongly that many of the questions being closed are very appropriate for this site, and the # of upvotes and interest these questions often retain would signify that others agree.

I am using this site considerably less these days due to the narrow range of topics permissible. It is adding significantly less value than it used to.

Answer 3

Update - October 20th, 2011

Current statistics:

• Total Questions: 13,440
• Total Closed but not deleted: 1,321
• Closed Percentage: 9.38% (▼ 7.56% from September 20th, ▼ 10.81% since this post was written)
• SE 2.0 Closed Percentage (no Trilogy): 5.36% (▼ 0.72%)
• Network Closed Percentage (w/Trilogy): 2.56% (▼ 0.04%)

At this point, we have a lower close percentage than Web Applications, Android, English, and Webmasters.

There was some concern that we're bleeding users: not sure how long they've been tracking it, but the Stack Exchange site tracks new 200+ users over a rolling two weeks period: Programmers has consistently had the second highest number of 200+ users (only Stack Overflow itself has beaten us), and the number has gone up since I've been tracking it.

We've also updated the FAQ to be much more straightforward, which will hopefully alleviate some confusion about what is and isn't on-topic here.

---

While I don't think it's particularly constructive to approach a problem this way, I think it's important, if we do, to work with accurate numbers.

The numbers

As of right now, there are 2,816 closed—but not deleted—questions on Programmers. There are 13,925 questions total on Programmers. This makes the closed, but not deleted ratio of questions 20.2%. In the 10 days since you measured this, it's gone down 3%.

Focusing on the same metrics you did your post, here are number closed with at least 5 score and number closed with less than or equal to 0 score:

• = 5 score: 774 (27.5%)

• <= 0 score: 265 (9.4%)

This puts the ratio of 5+ score questions closed higher than what you had and the ratio of <= 0 score questions much lower than what you had.

To understand why, we need to include data you didn't have access to when you did this survey: the number of deleted questions.

The number of deleted questions on Programmers.SE is 2,975: more than the number of closed questions you see. If we also include these numbers into the mix, we get a different story:

• = 5 score: 891. % of closed or deleted: 15.3%

• <= 0 score: 2,223. % of closed or deleted: 38.3%

This paints a bit of a different picture than measuring just the closed questions and the results presented in your survey. Why?

It's because we routinely delete low-scoring questions as they are uncontroversially bad for the site, and the scores on those questions agree.

The reason we don't delete nearly as many high-scoring answers is because "closed" doesn't mean "bad". It means the question doesn't meet the site's guidelines in some way, but there's some value, and it could even be improved and reopened.

That is, deletion—not closure—is where bad questions go to spend eternity in Hell. Closure, on the other hand, is the purgatory of questions. They're not on-topic, but they're providing some value: enough to stick around while we figure out what to do with them.

A Comparison

But now that we're working with accurate numbers, how do they compare to other sites, particularly Stack Overflow?

From a purely quantitative standpoint, Stack Overflow's numbers are in a different league:

• Publicly viewable questions: 1,997,856
• Closed, but not deleted: ~39,048 (2%)
• Deleted: 194,008 (8.9%)

• = 5 score:

  • Closed (not deleted): ~1,964 (5%)
  • Closed + deleted: ~4,142 (1.8%)

• <= 0 score:
  • Closed (not deleted): ~26,123 (67%)
  • Closed + deleted: ~199,326 (85.5%)

Like Programmers, Stack Overflow's ratios skew a lot higher when deleted low-scoring questions are added, and lower when it deleted high-voted questions are added. This isn't coincidence: the same moderation style is used on both sites.

Some context

But it's important, from a qualitative standpoint, to put these numbers into context. Stack Overflow is not qualitatively similar to Programmers.SE, and comparing their closed/deleted percentage to ours is about as meaningless as comparing our percentages to Quora or Yahoo! Answers. We just simply don't have the same issues or types of questions they have. If we did, there wouldn't be a separate site: we'd just be part of Stack Overflow.

It's also important to put Stack Overflow and Programmers.SE into historical context:

Stack Overflow was the first, and for a long period of time, the only Stack Exchange site. It did not, for much of its history, have the benefit of knowledge gleaned from the year of trial and error with the Stack Exchange 2.0 project. And notably, it did not have the benefit of 40+ other sites with 40+ other sets of problems to crystalize what makes and doesn't make a good question.

This difference in background and two year head start is a constant cause of problems: 27% of all questions migrated here from Stack Overflow are closed and/or deleted. Thousands upon thousands of questions that should be off-topic there and on-topic here never get closed or migrated.

Because of its history as the flagship Stack Exchange site, it also has enormous scaling issues. They have ~10 volunteer moderators for 2 million questions: one for every 200,000 asked. We have 4 for 16,000: one for every 4,000. We are just better equipped to close bad questions.

The history of Programmers.SE

But going into our history a little bit: Stack Overflow had a serious problem with popular questions that didn't really belong anywhere: "What's your favorite programmer cartoon?", "What do you like about programming?", etc. When Stack Exchange 2.0 and the Area 51 engine launched, one of the first proposals to gain traction was Not Programming Related, intended to house all of these types of questions. It launched on September 1st, 2010 and for a period of 3 weeks, surged in popularity as all the questions repressed on Stack Overflow were asked on this new site.

Unfortunately, the quality was all over the place: the site had some of the worst questions and some of the best questions. After a few weeks of this, new guidelines were put into place to try to raise the level of questions.

For the rest of the beta period, these guidelines were enforced sporadically as the community tried to come to terms with its original charter and what it would take to make a useful site.

On December 16, 2010, Programmers.SE was launched under its new mission, having been the test-bed for a number of guidelines about questions that later generations of Stack Exchange sites benefitted from.

This left 3.5 months of questions asked during a particularly formative point in the Stack Exchange network: more than a quarter of our site's lifespan. Compare this to Stack Overflow, which has had the benefit of 2.5 years of maturity and wasn't originally created to be the toilet bowl of a different site.

While we're slowly resolving those initial growing pains, it's going to take some more time. The ratios will normalize, but they aren't going to normalize in the next month: it going to take several months to a couple of years to stabilize into a site as mature as the currently-three-year-old Stack Overflow.

Conclusion

The situation isn't as dire as it might seem from one reading of the numbers. It was dire at one time, but that was last year. Now, we're moving forward: our traffic is increasing and our closed ratio is decreasing. More and more, people are taking ownership of Programmers.SE as something to be proud of, and the quality of questions and answers is noticeably rising.

I think there's a lot we can improve, but it's going to be far more constructive and productive to isolate specific failings and correct those, not to make generalizations about the site as a whole. If we stick to generalizations, we might as well just shut this site down and start over.

Answer 4

I think it's pretty simple and nearly inevitable. Most of the sites have a single, simple rule : "No subjective questions." P.SE replaces that with a half dozen guidelines about what constitutes a reasonable subjective question.

For a few people who use the site extremely regularly, that's not (much of) a problem, but for most others, it's harder to be sure they're following the guidelines -- especially since most of the guidelines are themselves at least somewhat subjective.

Don't get me wrong: I think the situation is probably open to at least some improvement (what isn't, after all?) but I think it's normal and probably expected that a higher percentage of questions here will be closed, migrated, need editing, etc., than on most of the sister sites.

Answer 5

Mitch, that's an interesting set of figures and the differences between behaviour on StackOverflow and Programmers.Stackexchange are particularly disconcerting as an indication of the lack of line up between the community behaviour and moderation on Programmers.SE. It says to me that StackOverflow is getting its buyin right in a way that Programmers.SE is not.

I'm obviously not privy to what the site owner wants for the site, but when you build a community based site, you can only exert a certain amount of control over the behaviour of that community before you change the community utterly. In some cases, by getting rid of some poor user behaviour, you can cause the site to grow; in other cases you just kill the site. Or you may not wind up with the community you want. I'm reminded by one comment here above that this is a site for expert programmers. The questions, however, are not expert level questions, and a lot of the more nuanced questions seem to be getting closed because nuance is being missed. This is a key criticism I have about one of my questions getting closed, incidentally. Duplicates are not always duplicates and the list of duplicates which Mark Trapp provided for this question are absolutely not duplicates. They are related, yes, but not duplicates.

Mitch - I am not sure what can be done in response. When there is that level of disconnect between what the site providers what to provide and what the community would like to gain, it's hard to tell any audience "we're not giving you this, so have that instead" because communities just move on. I think there is an issue in that for the most part, elements of the community here see this site as having a hefty element of discussion - but that the questions/answers format is enforced in a rigid manner which tends to kill off discussion for questions to which there are non-binary answers. The response seems to be that those questions which have no straight answers and which thus, generate a large number of answers, are not necessarily good questions per the preferences of the site. While I can see the site's rationale for that, I don't agree that it's realistic. Unfortunately.

Jeff - in response to your answer - given that there are some question marks over mod decisions in terms of closing questions, I would not necessarily be in favour of them being responsible for deciding what the best questions are. There's a voting protocol in place for questions and I'd be inclined to wonder why that would need to be over-ridden - again it boils down to the community not being in tune with the site owners, right?

Answer 6

I'll start the discussion by stating that I'd like to see examples of good questions in the FAQ, along with explanations of why those questions are good. I don't mean any disrespect to the hard work that has already gone into the FAQ, but it's clear that a large percentage of community members are finding it an insufficient resource to determine what kinds of questions they should be asking. Examples of existing questions which are on-topic and strictly adherent to Stack Exchange subjectivity guidelines would be very helpful.

Answer 7

Perhaps part of the disconnect is that the site is called Programmers.StackExchange, so people assume the site is about Programmers.

In reality, the FAQ says the site is about "conceptual questions about software development", and questions that aren't related to that get closed.

So we have a community that thinks the site is about Programmers, and moderators that think the site is about conceptual questions on software development.

It probably wouldn't be a popular solution, but if you want a site that's about software development I feel it should be called something like softwaredevelopment.stackexchange.com, not programmers.stackexchange.com. Programmers.stackexchange should be a site about programmers.

And personally, I want to participate in a site about programmers, not one about software development :)

Answer 8

Right off the top of my head, go to the top questions by month:

https://softwareengineering.stackexchange.com/?tab=month

Rule out any [closed] questions, obviously -- and possibly anything that was contentious enough to get closed and reopened based on the revision history of the question -- and there's your list of good, on-topic questions.

I'm sure the mods could hand-pick favorite "best of" here, but is that really necessary?