I asked a question about whether an oft-quoted piece of programmer wisdom has empirical evidence to back it up.

My question was closed within seconds using the close reason normally used for lazy questions asking for laundry lists of libraries and other resources.

From the comments, it's not clear what the reasoning is - a comment says "It looks like you're asking for a paper or book", but I'm not, I'm asking about the currently available evidence on a topic, the results of research testing a claim.

  • Are questions about empirical evidence off topic here? I see a lot of answers that are just one line, "Someone said '[some witticism]'" which get a surprising number of upvotes. Does this site favour anecdotes over evidence?
  • If not, how do I need to rephrase my question to allow someone to post an objective, factual answer like "This has been investigated by researchers in [discipline], the evidence points towards [X] with [Y] caveat, the most recent peer-reviewed literature review / the largest study to date is [citation]"?

This meta thread appears to suggest that questions about empirical evidence are on topic so long as they are about the state of the evidence, not "What's the best book on X". That is absolutely 100% the intention of my question. Is this still the policy?

If so, how do I need to re-phrase my question to stop people misreading it as "pleez can I has teh books"?

Incidentally, this is the first time in 4+ years on stack exchange that I've run into problems for a question being too evidence based.

From the discussion I've seen so far, I'm starting to get the impression that:

  • What does real empirical research say regarding Brook's law? would be of topic because when evidence is peer reviewed, it becomes an "off site resource" rather than a user's opinion
  • Do you guys think Brook's law is true? would somehow be more on topic because it relies on experience of users of the site?

Experienced programmers use research relevant to their field and are in a better position than anyone to see through methodological problems and see what is practical research and what is flawed.


2 Answers 2


My question was closed within seconds using the close reason normally used for lazy questions asking for laundry lists of libraries and other resources.

It's not about calling it a "lazy question". That's the standard close reason for off-site resource requests. And that's why the final item in the list is there:

or favorite off-site resource

If I rephrased your question to "Does Programmers.SE have any empirical evidence backing Brooks' claims?" The answer is "No, it doesn't."

And if we use a loose form of the question you asked "Is there any empirical evidence backing Brooks' claims?" The answer is "we don't know."

If you haven't already, have a read of this meta answer that goes into detail why off-site resource questions are closed. And yes, there is a portion in that answer that directly addresses the main point of your meta question.

  • 2
    "And if we use a loose form of the question you asked "Is there any empirical evidence backing Brooks' claims?" The answer is "we don't know." Sorry but this makes no sense. Are you ruling out the possibility of anyone in the P.SE community having expert knowledge on this topic? Why can't you believe that someone could say "I've read the research on this topic. The evidence generally supports the claim, for example X study and Y review, but with important caveats A, B and C. This matches my experience, for example D."? Aug 28, 2015 at 17:42
  • 2
    Please go read the linked meta answer. It does answer your question. Once you grok why those types of questions are not good for the site's long term health, you'll understand why the answer to "Are you ruling out the possibility of anyone in the P.SE community having expert knowledge on this topic?" is immaterial. Failing that, please feel free to ping me in Chat in The Whiteboard. Software Engineering Chat is the best place for extended discussion, not comments.
    – user53019
    Aug 28, 2015 at 17:45
  • 1
    I've read it twice already. It could only be considered relevant by someone who doesn't know what empirical research is. Can you please answer my question? Why can't you believe that a user of this site who is an experienced software project manager would have expert knowledge and experience about the relevance of research on their area of professional expertise? Aug 28, 2015 at 17:50
  • @user568458 Please join me in The Whiteboard
    – user53019
    Aug 28, 2015 at 17:51

I had an hour-long discussion with moderators and senior users. As an experienced SE user, the results were very surprising, but this isn't my site, so I'll accept your policies as they are.

TLDR; evidence-based questions and answers are not held in the same esteem here as on other sites. It's expected that questions are answered from individuals' experience.

After an hour, the moderators couldn't find any way to ask about the state of evidence on an on-topic topic that wouldn't be counted as a "reference request".

It was accepted that this type of question is routine on other sites on the network, but is apparently not allowed here.

Here's a brief summary of a long, long chat.

Programmers does have a very different policy on evidence to other sites. The preference is for personal ancedotal experience.

GlenH7: Asking for empirical evidence (aka research on a subject) will inherently consist of links off-site. That puts Programmers in the position of being research librarians, which isn't terribly interesting to the community.

After explaining that, on most sites on SE, answerers provide expert summaries of the most relevant research, sometimes with reference to personal experience, it was explained that Programmers is different

Me: Look at any good answer on Skeptics.SE. It's an expert-based summary of research, WITH references, not a list of references. Look at any good answer on biology, chemstry, physics, cogsci - even good quality answers on UX

enderland: Skeptics is not Programmers

After pointing out an example of an excellent evidence-based answer that wasn't simply a reference dump, it was immediately decided to close that other question, and the evidence-based answer was described as being no better in content than other opinion-based answers, beyond being well written.

Me: gnat has a great evidence-based answer here. You can see clearly how his (?) expertise helped inform the elements of the study he picks out. Doesn't look like anyone's in a rush to delete it. programmers.stackexchange.com/…

enderland: I expect that will end up closed in.. a short while

MichaelT: You are also missing seeing the deleted answer and that more answers would be equally right.

Essentially, emperical evidence appears to be something this community has chosen not to be interested in. That's fair enough, it's your community, it's just very unusual for SE.

GlenH7: I think Brooks' Law is better asked on Skeptics. The Progs community tends to be too pragmatic to worry too much about making sure there's backing research when firsthand experience backs the principle

Thomas: ...I read books. I read IEEE Software and the ACM magazines. But I don't read the hardcore research and whitepapers, because it's not likely to help me do my job. Anecdotes are pretty powerful to me, though. I have enough experience to take someone writing about their situation, pick out the things that apply to me, and reason if they will help me.

Thomas: we aren't a science site. We're a bunch of people in the trenches.

Thomas: sharing a personal experience is immediately sufficient for supporting evidence

Thomas: You need to ask us a question that you'd ask a coworker in a conference room or at a whiteboard.

Jimmy: oh goodness we don't bother with that sort of thing. You want methodologies and google scholar? Hit up CS

MichaelT: The trade magazines we read are certainly not peer reviewed.

  • I understand that the chat wasn't an enjoyable experience for anyone involved, but if you disagree with this summary, please simply state clearly what you disagree with. Aug 28, 2015 at 20:43
  • 1
    as far as I can tell, this type of question is frowned upon at Stack Overflow. I also think that approach to asking and answering at Programmers is quite close to one at SO, only topics differ (though even topics noticeably overlap)
    – gnat
    Aug 28, 2015 at 22:38

Not the answer you're looking for? Browse other questions tagged .