Do significant claims require evidence?

Question

I refer to this answer from Too much version control and bug tracking overhead per change? for which the question itself was actually the subject of a previous meta question. Quoted here in the unlikely event of changes:

Heavy processes are common, unfortunately. Some people - especially management - religiously imagine that processes produce products. So they overdo the processes and forget that it's really a handful of hard-working, smart people who actually create the products. For upper management, it's frightening to even think that their business is in the hands of few geeks, and so the close their eyes from the reality and think of their dear "process" instead, which gives them the illusion of control.

That's why agile startups with a handful of good engineers can beat big, established corporations, whose workers spend 95 % of their energy on process and reporting.

This answer is making grand assertions left and right without a single shred of evidence to back any of them up. Paraphrased from my comments:

• Which people are "some people"?
• What evidence is there for managers believing it more often than anyone else? Why would this be the case?
• Why liken it to a "religious belief"? Is there no factual/logical evidence in favour of the belief? How can we be sure? Have those people been questioned about their belief?
• Who claimed/claims that "processes produce products" and what was the context of that claim? If the answer, as I suspect, is "nobody", then why mention this at all?
• What does "overdo the process" mean? At what point does a process become "overdone"?
• Is it really true that the "business is in the hands of a few geeks?" Is this true in the 85% of programming jobs that are not in the software publishing industry, which is presumably where most of these heavy processes are being instituted?
• What does the phrase "close their eyes from the reality" mean? Do these people presumably have no accountability to their own managers, shareholders, board of directors, etc.? How would they justify this?
• What does "illusion of control" mean? How is it different from actual control? What provides the illusion and how is it supposed to keep people fooled?
• Although the answer technically only says that "agile startups" can beat "big established corporations", it's misleadingly worded as to imply that this is routine. To what degree does this actually happen and is there evidence that it's any more than chance, as opposed to better management?
• Where was the 95% figure derived from? Was this measured somewhere?

As much as I hate to pick on one person/post, this isn't information, it's entertainment. It's mindless, populist drivel clearly written with one purpose: to provoke an emotional response and farm upvotes.

And in this community, it works. Every single time. With 84 upvotes and just 1 downvote (mine, obviously), this is clearly not only the best answer, but many times better than any of the other answers, which the author stubbornly uses as a justification for why it doesn't need to be improved (can't blame him, I guess, I wouldn't look a gift horse in the mouth either).

But the most irritating part is the most recent comment:

This forum is not a scientific paper. Nobody, not you nor me, is going to provide an explanation for every single sentence he writes.

Call me old fashioned, but I thought that the whole point of the Subjective Question Guidelines was to elicit answers that would Back It Up:

Back It Up! means that your answers must be based on either:

• Something that happened to you personally
• Something you can back up with a reference

I realize that we're not Wikipedia. We're not even Skeptics.SE. We can afford to be a little more lax.

But who is correct, him or me? Do we/should we have no standards whatsoever for evidence, or are answers here supposed to be more than just statements of opinion?

Answer 1

It depends on what you think Programmers.SE is for.

If it's a site for expert programmers who are interested in conceptual questions on software development, then answers had better be backed up by references. Even for the kind of subjective questions that make up the meat of the site, some kind of justification should be provided — real experts don't just say “yes” or “no”, they can explain why.

If Programmers.SE is Stack Overflow's toilet bowl, then answers aren't for reading, you don't know where they've been (or rather you do!), so who cares.

Answer 2

In all areas of life extraordinary claims should require not just evidence but extraordinary evidence.

If this doesn't happen, this shouldn't actually be a problem that needs a specific solution. You'd hope that an answer that makes such claims without backing them up is a poor answer and should be treated as such by those voting and commenting, at least that's the theory.

But the nature of PSE and the audience it has means it doesn't always work that way. There are some answers which get away with some fairly outrageous claims because the answer presents a broadly popular perspective. Suggest that someone quit a shitty job and you can get away with almost anything else in the same post. Similarly blame management for getting in your way and the detail will likely be ignored.

While that's annoying it's not actually that much of a problem. For the most part I'd look to read not just the top answer, but the top three or so answers. Generally where you get one answer playing to the crowd, somewhere close to it you get a contrary view point.

And it's important not to discard the populist viewpoint. It may not be what you want to do but you should use it to inform your understanding of the world and the culture you work in. One place I work lived by the phrase "the perception is reality" - essentially saying that even if something wasn't true, if enough people believed it, it was something you had to deal with and I think that's the case with these answers.

Read all the answers, use your own judgement to work out where you think the truth lies but try and learn from all the views that have been put forward as they will all tell you something, even if it's not exactly what the poster was trying to say.

(I speak as someone who is most proud of some of my answers that have got next to no up votes...)

EDIT (from a comments discussion with Aaronaught):

So just to be clear from the outset - I wish all posts drew on research and personal experience and provided links to reputable sources where the claims warranted them. To me that's a good post and what we should aspire to.

But what we have is a situation where some popular answers don't meet that standard, and a question about what that means and whether it's a big deal.

To me the nature of subjective questions means that the most useful "answer" isn't a single answer (whether the top voted or not) but is the amalgamation of all the great information contained in all the answers that have been provided. Subjective questions by their nature don't have a single answer so for me what the upvotes are saying is the community indicating "here is something you should have a think about while you're working out how to handle the specific situation you find yourself in".

In that context a glib unsupported popular answer with a mass of up votes is useful information. Sure it would be more useful if it was better supported but if you get 40+ people saying "I agree with this" it is absolutely something you should consider even if it's categorically wrong.

The reason for that is that right and wrong are not the only thing that matters. If you're in a shitty job and you post about it you will get an answer with a bunch of votes saying "Quit now!" sure as the sun will rise tomorrow morning. While on one level that's a bad answer it gives you useful information - that you work in an industry where a significant number of people believe that that is an acceptable approach to the problem. While I and many others would argue it's not the most constructive approach that doesn't change the reality that many people believe this and act on this and it is something that you will see and have to deal with in the industry.

I think what I'm saying (because I'm kind of thinking aloud here) is a significant number of up votes are evidence in themselves. Not perhaps of factual correctness, but of a perception that is frequently so strong and so prevalent that it has to be understood and considered. As programmers we sometimes forget that factual correctness isn't everything and that situations are more complex than that, that the softer and more emotive side of things is equally important and that any answer which ignores that may be factually correct but isn't the whole story.

Obviously this requires the reader to apply some thought to the answers being provided and understand a wider context but I hope and believe that the people we're talking to are capable of doing that. If they're not then I suspect there's not much we can do for them anyway.

So I'm not saying we should encourage bad answers, just that very popular bad answers provide useful information we can learn from and that we shouldn't be so quick to dismiss.

Answer 3

The short answer is "yes", but that's only (as you say) for significant claims.

Anybody can make a claim in an question or answer, but without any evidence to back it up, but a good question or answer will have that claim backed up with links, personal details etc. That's not to say that a question or answer without back up is a bad question.

You're right to say we don't need (nor should we insist on) the same level of detail as other sites, but if you are quoting figures then this must be backed up.

Answer 4

I've read through all of this discussion, and the one thing that keeps popping out at me is the disconnect between both sides on who has the burden of proof.

Self-evidence

The crux of the argument in support of these types of answers, as I understand it, is that they aren't providing anything that necessitates detailed evidence, and such a requirement would be onerous on otherwise self-evident (or commonly-held) ideas.

Something that's self-evident is, by definition, something any reasonable person would know. By providing answers that are self-evident, you don't provide any expert value.

Take the classic example of someone coming to the IT guy for help with a computer issue they're having, and the IT guy responding, "Did you plug it in?" It's self-evident that a computer must be plugged in for it to work, but the exchange speaks to anyone who's had to deal with really trivial computer questions; it's functionally equivalent to saying "You're wasting my time by asking me this question." It's been lampooned in many a comedy sketch because it's exactly the type of response nobody wants to receive when they're coming to someone for help.

Massively up-voted answers like the ones discussed here are our moral equivalent of "did you plug it in?" Someone asks a seemingly trivial question, someone else leaves the obvious response, and it gets massively up-voted not because the answer actually provided value or great insights, but because it said what we're all thinking.

That is decidedly not what Stack Exchange is about. If you don't think a question is worth answering with a thoughtful, well-reasoned response, there are several options available:

• Leave a comment explaining why the question is trivial and basic things the person can do
• Down-vote the question as showing no research or effort
• Vote to close the question as not constructive, too localized, or not a real question depending on the circumstances

But beyond that, the main premise behind Stack Exchange is that there are experts who are willing to help others who might not know as much as they do. It's important, crucial even, that self-evidence is just taken off the table entirely. The answer is not self-evident: if it was, it wouldn't have been asked.

Preferences vs. belief

Beyond the question of self-evidence is the issue of what constitutes subjectivity, and subjective answers can be roughly categorized into two types:

• Personal preferences
• Warranted beliefs

A personal preference is something you personally like or favor for whatever reason: I like mint chocolate chip ice cream. I enjoy sleeping in late. I want a slice of pizza for lunch today. These are unassailable: provided I'm not lying to you, you can't prove I don't like mint chocolate chip ice cream, no matter how hard you try. I said it, it's true.

A personal preference is the appropriate answer to a question asking for a poll: "What is your favorite ice cream flavor?", "What's your favorite book?", "Which programming language do you like the most?" Those questions, however, are straight up prohibited here and across the network. Personal preferences have no place in an answer, and are the smell of a bad question.

A warranted belief, on the other hand, is something you personally believe based on something you personally know to be true. That is, "I know X, and I know it because of Y."

This, and only this, is what subjectivity is all about on the Stack Exchange network, and it's the heart of the "back it up" mantra. We want—no, crave—answers that provide a belief of yours that you hold because of some special, personal insight.

But it's not enough to just provide the belief: how do I know what you're saying is true or credible (remember, self-evidence is off the table)? Tell me more about why you believe that and I'll decide for myself whether what you believe is truly warranted from what you know, or if there's some other insight to be gleaned.

Summary

In short, the person who has the burden of proof is always, always the answerer. If you can't support or are unwilling to support your claim with information about why you believe the claim to be true, it's not worth leaving the answer.

It wastes everyone's time: the question asker who doesn't get a useful answer, visitors who have to read through the answers, and yours when you have to justify it to people who question the claims made in your answer. To respect people's time, and make the internet a better place for everyone, tell us something we don't know and explain what it's all about.

Answers are like show-and-tell: we're all dying to hear the story behind why you believe PHP is the devil's work, or how you came up with the idea to use recursion to save little Timmy from the well. Don't hold out on us.

Answer 5

It's easy to agree with "yes, significant claims should be backed up", but it's a vain statement as long as there's no common agreement of what "significant" means.

The problem is: what is a significant claim? What's significant to you may not be significant to someone else, and vice versa. Is it reasonably possible to back up everything that someone out there may consider significant? If not, then who decides what's significant and what's not?

Are the following claims significant or not? Or is some of them significant, but so obvious that it doesn't need to be backed up because of that?

• Developers are notoriously anti-management.
• Managers love processes.
• Projects tend to go overtime.
• Startups can beat big, established corporations.
• Big, established corporations can beat startups.
• Stackexchange sites are popular.
• Beginners tend to make mistakes.
• Big corporations tend to have heavy processes.
• Lighter processes have become more and more popular during the last 10 years.
• Formal processes become a requirement when company grows to > 10 people.
• Climate change is a major issue.
• Java is one of the most used languages out there.
• Smoking kills.
• I feel that a couple of cups coffee does good to my programming efficiency.
• Too much coffee is bad.
• Lines Of Code Per Day (LOCPD) is a lousy measure of developer productivity.

Do all of these need to be backed up? If not, which ones, and why? If yes, how would you back up e.g. the first one ("Developers are notoriously anti-management.")?

Personally I think that since the definition of significance and obviousness depends totally on who you ask, these sites should be kept quite lax - much more lax than some extremists suggest. Backing one's claims should of course be encouraged, but it shouldn't be required. Attempting to suppress noise may improve signal-noise ratio, but it will also more or less distort the signal itself. The fallacy is that one starts to consider his own (or his community's) judgement as the only valid measure of significance or obviousness, and dismisses anything that deviates from it. How is that better than opinions without evidence? Questions with tens of upvotes getting closed is a sad example of this. Lots of people would be interested in the question (that's why they upvoted it), lots of interesting answers have already been written, and then... some police closes the question as "Not a real question" or something. It's really sad.

Answer 6

Let's go back to the original question and my answer that started this discussion. The question was:

Is there a point at which the process gets in the way and becomes an end unto itself?

To which I answered (emphasis on what I meant to say):

Heavy processes are common, unfortunately. Some people - especially management - religiously imagine that processes produce products. So they overdo the processes and forget that it's really a handful of hard-working, smart people who actually create the products. For upper management, it's frightening to even think that their business is in the hands of few geeks, and so the close their eyes from the reality and think of their dear "process" instead, which gives them the illusion of control.

In other words: Since it's really a handful of hard-working, smart people who actually create the products, obviously the process can get in way if it's too heavy. While I (and quite a few colleagues) like to emphasize the people, some (not-so-good) managers tend to think too much in terms of the process. That's the reason for overly heavy processes. The reason that (some not-so-good) managers prefer to overemphasize the process is that it is (or at least seems to be) controllable. Real people are way more difficult to control than some abstract process. Regardless, it's the real people that produce products, not the process.

All the above is said in my original answer, even if somewhat colorfully worded. I don't think there's any particularly significant or outrageous statement. It's mostly common sense written out.

Some folks seem to read it like this:

Heavy processes are common, unfortunately. Some people - especially management - religiously imagine that processes produce products. So they overdo the processes and forget that it's really a handful of hard-working, smart people who actually create the products. For upper management, it's frightening to even think that their business is in the hands of few geeks, and so the close their eyes from the reality and think of their dear "process" instead, which gives them the illusion of control.

If you take only the "anti-managerial tone" from the post, then of course it looks like a stupid rant. Maybe, just maybe, a more neutral tone should be preferred. But on the other hand, I prefer reading strongly written, concentrated statements instead of shy ones. People shouldn't be ashamed of their own biases.

My answer is not very good; it's certainly below my average answer quality (but some answers have to be below the average!), and its lack of quality is emphasized by the number of upvotes it got, which may be seem to be undeserved (but it's not certainly my fault). Still, calling it "mindless, populist drivel" is clearly a result of reading only the colorful part of it and missing the main point.

Answer 7

Thanks for bringing this discussion here. I honestly think it's important (and picking one person/post is only good; it provides something tangible to discuss). Let me present my opinions:

As much as I hate to pick on one person/post, this isn't information, it's entertainment. It's mindless, populist drivel clearly written with one purpose: to provoke an emotional response and farm upvotes.

But these sites aren't strictly about information. If you want strict information, you can go to Wikipedia, IEEE magazines, scientific papers. The world is full of official and dull information. This site is, like the guideline tells, largely about the stuff between objective and subjective. About stuff that one thinks is correct, but can't prove. Such stuff would go to /dev/null without sites like this.

My answer, surely, leans towards the subjective side. As for being "mindless, populist drivel", that's your personal opinion, not information. I certainly did not craft it to farm upvotes. I wrote my opinion, that's all.

But who is correct, him or me? Do we/should we have no standards whatsoever for evidence, or are answers here supposed to be more than just statements of opinion?

I think this largely depends on the question. Take these questions:

1. Is there a point at which the process gets in the way and becomes an end unto itself? Is that even engineering?
2. How to create an edit profile page like Google+?

The second one is about technology how-to. The first one is about opinion. Opinions grow from experiences and they vary with time. Answers to such questions are unavoidably quite subjective. How much evidence did you seen in the other answers to the same question?

My answer obviously does provoke emotional responses, and even if it were completely incorrect, wrong, false, at least it reveals something about the question itself: the right answer, whatever it is, is not clear. Therefore, there is no objective answer. Therefore, the answer must be subjective. And subjective answers are based on personal experiences - something that nobody can call incorrect.

Answer 8

Questions should support their premises where necessary. We've seen a lot of posts along the lines of "Why do experts say not to use OOP" (a made-up example for illustration purposes). Questions like that really should back up their assertions that 'experts' really do say such a thing. A false premise makes the question meaningless. I routinely vote down questions like that.

Answers, on the other hand, really dont have that responsibility. If someone makes a strong case, vote'm up. If they dont, vote'm down. Not every answer needs to be a treatise on a particular subject. Often experience or opinion is exactly whats called for in an answer. If you dont like that 'the unwashed public' votes up answers you personally find insufficient, well, too bad. Thats freedom for ya.