Is empirical evidence off topic here? [duplicate]

Question

I asked a question about whether an oft-quoted piece of programmer wisdom has empirical evidence to back it up.

My question was closed within seconds using the close reason normally used for lazy questions asking for laundry lists of libraries and other resources.

From the comments, it's not clear what the reasoning is - a comment says "It looks like you're asking for a paper or book", but I'm not, I'm asking about the currently available evidence on a topic, the results of research testing a claim.

• Are questions about empirical evidence off topic here? I see a lot of answers that are just one line, "Someone said '[some witticism]'" which get a surprising number of upvotes. Does this site favour anecdotes over evidence?
• If not, how do I need to rephrase my question to allow someone to post an objective, factual answer like "This has been investigated by researchers in [discipline], the evidence points towards [X] with [Y] caveat, the most recent peer-reviewed literature review / the largest study to date is [citation]"?

This meta thread appears to suggest that questions about empirical evidence are on topic so long as they are about the state of the evidence, not "What's the best book on X". That is absolutely 100% the intention of my question. Is this still the policy?

If so, how do I need to re-phrase my question to stop people misreading it as "pleez can I has teh books"?

---

Incidentally, this is the first time in 4+ years on stack exchange that I've run into problems for a question being too evidence based.

---

From the discussion I've seen so far, I'm starting to get the impression that:

• What does real empirical research say regarding Brook's law? would be of topic because when evidence is peer reviewed, it becomes an "off site resource" rather than a user's opinion
• Do you guys think Brook's law is true? would somehow be more on topic because it relies on experience of users of the site?

Experienced programmers use research relevant to their field and are in a better position than anyone to see through methodological problems and see what is practical research and what is flawed.

Answer 1

My question was closed within seconds using the close reason normally used for lazy questions asking for laundry lists of libraries and other resources.

It's not about calling it a "lazy question". That's the standard close reason for off-site resource requests. And that's why the final item in the list is there:

or favorite off-site resource

If I rephrased your question to "Does Programmers.SE have any empirical evidence backing Brooks' claims?" The answer is "No, it doesn't."

And if we use a loose form of the question you asked "Is there any empirical evidence backing Brooks' claims?" The answer is "we don't know."

If you haven't already, have a read of this meta answer that goes into detail why off-site resource questions are closed. And yes, there is a portion in that answer that directly addresses the main point of your meta question.

Answer 2

I had an hour-long discussion with moderators and senior users. As an experienced SE user, the results were very surprising, but this isn't my site, so I'll accept your policies as they are.

TLDR; evidence-based questions and answers are not held in the same esteem here as on other sites. It's expected that questions are answered from individuals' experience.

After an hour, the moderators couldn't find any way to ask about the state of evidence on an on-topic topic that wouldn't be counted as a "reference request".

It was accepted that this type of question is routine on other sites on the network, but is apparently not allowed here.

---

Here's a brief summary of a long, long chat.

Programmers does have a very different policy on evidence to other sites. The preference is for personal ancedotal experience.

GlenH7: Asking for empirical evidence (aka research on a subject) will inherently consist of links off-site. That puts Programmers in the position of being research librarians, which isn't terribly interesting to the community.

After explaining that, on most sites on SE, answerers provide expert summaries of the most relevant research, sometimes with reference to personal experience, it was explained that Programmers is different

Me: Look at any good answer on Skeptics.SE. It's an expert-based summary of research, WITH references, not a list of references. Look at any good answer on biology, chemstry, physics, cogsci - even good quality answers on UX

enderland: Skeptics is not Programmers

After pointing out an example of an excellent evidence-based answer that wasn't simply a reference dump, it was immediately decided to close that other question, and the evidence-based answer was described as being no better in content than other opinion-based answers, beyond being well written.

Me: gnat has a great evidence-based answer here. You can see clearly how his (?) expertise helped inform the elements of the study he picks out. Doesn't look like anyone's in a rush to delete it. programmers.stackexchange.com/…

enderland: I expect that will end up closed in.. a short while

MichaelT: You are also missing seeing the deleted answer and that more answers would be equally right.

Essentially, emperical evidence appears to be something this community has chosen not to be interested in. That's fair enough, it's your community, it's just very unusual for SE.

GlenH7: I think Brooks' Law is better asked on Skeptics. The Progs community tends to be too pragmatic to worry too much about making sure there's backing research when firsthand experience backs the principle

Thomas: ...I read books. I read IEEE Software and the ACM magazines. But I don't read the hardcore research and whitepapers, because it's not likely to help me do my job. Anecdotes are pretty powerful to me, though. I have enough experience to take someone writing about their situation, pick out the things that apply to me, and reason if they will help me.

Thomas: we aren't a science site. We're a bunch of people in the trenches.

Thomas: sharing a personal experience is immediately sufficient for supporting evidence

Thomas: You need to ask us a question that you'd ask a coworker in a conference room or at a whiteboard.

Jimmy: oh goodness we don't bother with that sort of thing. You want methodologies and google scholar? Hit up CS

MichaelT: The trade magazines we read are certainly not peer reviewed.