Are you guys turning P.SE into a 404 black hole?

Question

I have been hitting quite a few 404's on P.SE lately. I was wondering how great the idea of perma deleting questions that have been around a long time and indexed and linked to (not spam or such) really is.

For example, the lastest blog post by Yannis Rizos contains a link to a deleted question (391 upvotes FCS!) in the 2nd paragraph, resulting in a 404 error for basically all(?) the audience of the blog.

Not saying that these questions are a good fit to the site, but removing them completely, leaving dangling links all over the internet (am I exaggerating? :-) certainly isn't my idea of Making The Internet a Better Place.

Note that don't mind closing; I wouldn't even mind a trash.stackexchange.com; but in my very humble opinion, deleting valuable content from SE sites (not just P.SE) that has been lying there for weeks and months is just dead stupid.

---

Follow up question on Meta Stack Overflow: Are you guys turning SE into a 404 black hole?

Answer 1

The general policy on deletions

From our FAQ:

Why are some questions or answers removed?

Questions that are extremely off topic, or of very low quality, may be removed at the discretion of the community and moderators.

Over time, closed questions that are not useful as signpoints to other questions may also be removed, as well as questions which have no significant activity over a very long period after being asked. For additional guidance, see How to Ask.

Answers that do not fundamentally answer the question may be removed. This includes answers that are …

• commentary on the question or other answers
• asking another, different question
• “thanks!” or “me too!” responses
• exact duplicates of other answers
• barely more than a link to an external site
• not even a partial answer to the actual question

If you wish to improve an existing answer, click edit. For additional guidance, see How to Answer.

That's the policy, I'm sure you are aware of it, I'm only including it for completeness.

Questions with historical significance

We have a historical lock that has been introduced (fairly recently) as a counter measure to link rot:

What is the purpose of a Historical Lock?

A historical lock preserves older content that was very popular when it was originally posted, but is now off-topic or otherwise out of scope for the site it is posted on. Historically locking a post ends the debate over whether a question should be kept on the site or deleted, and is often the final state of a question that has been deleted and undeleted more than once.

How are questions affected by historical locking?

In addition to the post notice being prominently displayed, posts which are historically locked are "frozen in time;" they cannot be voted on, answered, edited or commented. Historically-locked questions are omitted from normal question lists (those on the home page, /questions, and the various per-tag lists), but can still be found by searching for words in the post or title (via either site-search or Google, etc). The visual appearance of the entire post is altered by removal of the voting arrows from the question and all answers.

Note that historical locked questions on meta sites are not hidden in this manner, though their answers are still frozen in the same fashion.

When is it appropriate to lock a question for historical reasons?

Questions can be historically locked when:

• The post is Off-Topic or Not Constructive, and
• The post is stellar, in spite of its off-topic nature, and
• There are a large number of views, upvotes and inbound links on the post, and
• The post is contentious; i.e. it has been closed and reopened at least once, or deleted and undeleted at least once

When is it not appropriate to historically-lock a question?

Questions should not be historically locked if they:

• Are being actively maintained, or
• Have little or no redeeming value.
• A good rule of thumb: If the question does not minimally meet Jeff Atwood's 3rd rule in the "We Hate Fun Here" blog post, it's probably not a good candidate for historical locking. The third rule is:

Does this question teach me anything that could make me better at my job? Can I learn something from it?

How do I request a historical lock on a question?

Flag the question for moderator attention, with the "custom description" option. In the flag description, explain why you think the post should be historically locked. A moderator will evaluate the question using the criteria outlined above, and will either lock the question, or decline your flag with an explanation.

Alternatively, if you want to try and gain more community support for the question, or contest the moderator decision, you can post a question here on Meta.

Since you are currently under 10K, your only option to request a historical lock is to post a question about it on Meta. I would advice you to use a far more appropriate tone that in this question, and:

• Clearly explain why you think the question has redeeming value,
• Politely request from the community to review it and decide whether it should be historically locked or not.

Not every deleted question qualifies for the historical lock, please do not waste everyone's time by bringing up questions on Meta that clearly don't qualify.

Clarification:

The historical lock is only meant for deleted questions (or questions that are about to be deleted), locking closed questions prevents them from being edited and voted on. Furthermore the community cannot reverse a lock, while it can (and often does) reverse a closure.

Changing the policy

It appears you disagree with it. Fair enough. Change it. Stop complaining about it, and change it.

Personally I'd support any constructive attempt to solve the problem, however I must point out that there have been tons of discussions on Stack Overflow about deletions, and although the "link rot" problem is true for popular Stack Overflow questions it's almost completely irrelevant for lower traffic sites like Programmers.

I can understand that clicking on a link, only to find a 404 page stinks, but should also understand that keeping crap around stinks more. And although there might be some gems amongst our deleted questions, most of them are plain crap. This is not Stack Overflow, if you are interested in changing Stack Overflow's or the network's policy, please post your question on Meta Stack Overflow.

Meta Programmers is a place were we discuss Programmers specific issues, and from a Programmers perspective the deleted popular questions that were linked all over the place were not ours to begin with. Unfortunately tons and tons of crap were migrated here from Stack Overflow in the early days of the site. Most of these questions were closed (at some point), which send them back to Stack Overflow. What was deleted here was their (locked) rejected migration stubs. What was linked all over the place was their Stack Overflow URL, and not (so much) their Programmers one. If you want to make a case for them, please do it on Stack Overflow, not here.

Generally speaking, closed but popular questions are not being deleted on Programmers, at least not systematically. I've deleted more posts than all my fellow moderators combined, if popular questions were being deleted, trust me, I'd know it.

Are we making the internet better?

Well, let's what our most voted deleted questions are:

1. What's your favorite “programmer ignorance” pet peeve? (+410),
2. What's your most controversial programming opinion? (+391),
3. What real life bad habits has programming given you? (+368),
4. As a programmer what single discovery has given you the greatest boost in productivity? (+315),
5. How to hide Thinking at Work so that the Non-Programmers don't suspect Slacking? (+257)
6. What are the best RSS feeds for programmers/developers? (238)
7. Best Java book you have read so far (+211)
8. How did you get your first programming job? (+140)
9. Does the D programming language have a future? (+138)
10. What's wrong with cplusplus.com? (+128)

The "Best Java book you have read so far" one is the one (from the list) with the most views, around 33K. And zero answers. Yeap, zero. It's a question that got all of its answers on Stack Overflow, and 99% of it's activity, including views, there. Gone. Again, if you want to make a case for it, go to Meta Stack Overflow please, not here.

I honestly don't know if we are making the internet better. However having (partially) reviewed some of the above questions, I don't see any indication that we are making the internet worse by deleting them. If they were originally asked on Programmers and were part of our community's history, we would have a whole different discussion, but right now all I can say about them and their eventual fate is: meh.

The blog post

It was the worst possible example you could bring forth:

1. The question was deleted long time ago,
2. I could only find five links to the Programmers version of it,
3. The blog post clearly indicated that it would lead to a 404 for most people,
4. The URL was nofollow, so there wasn't an issue with indexing,
5. The only reason the URL was included was for our 10K users to be able to review it.

Furthermore, the completely inappropriate tone you chose for your question made me re-think the whole effort to salvege old and long forgotten deleted questions by converting them to blog posts. Although I do not have any expectations for appreciation for my efforts to give back to the community, I am not a moron and will not freely volunteer my time to promote the community through an activity that has absolutely nothing to do with my diamond status when this is what I get for it.

At some point you got to say enough is enough. I've removed the url from the blog post, so users who would deliberatily click on a link that says "hey, this is deleted" won't have a bad experience.

The Trash Stack Exchange solution (and its many variants)

This has been proposed countless times, and every time it has been denied by Stack Exchange. Personally I find it a good enough solution, but Stack Exchange doesn't seem to care enough for hosting content that has been deemed unsuitable.

Oh well.

Stargazer712's proposal

Stargazer712 made what I must admit is an interesting suggestion (carefully hidden under tons of borderline insulting crap). However it's based on a misunderstanding of the scale of the effort to salvage questions by converting them to blog posts. We are certainly not talking about converting every deleted question to a blog post, I have found 5 so far that I'd like to salvage, plus another one from Stack Overflow and no one else has shown any interest in this. So, for all intents and purposes when we are talking about the blog as a way to salvage questions, we are talking about a total of six questions. That's it.

Conclusion

Given that 2 days after you posted this you still haven't brought forth any other evidence of systematic deletion of popular content with at least some redeeming value on Programmers, I'm sorry but I'll have to dismiss this as a non issue.

That said, we had numerous organized clean up efforts in the past that included deletions. I'm honestly hoping those are not the deletions you have in mind, as all the organized clean up efforts were discussed thoroughly on Meta before they began, and only began when it was obvious that the community supported them.

If you do update your question, please either ping me in chat or post a comment here so I can review your updates. Apologies for the tone in my earlier answer.

Answer 2

I agree with you. 100%. This link should not go to a 404 error.

Replacing the link with a 404 is stupid. It is godawful stupid. It's, "I'm going to fire the person who did this," stupid. Intuitive behavior would be to respond with 301 redirect. But it doesn't do that. It responds with a 404.

All 400 level errors imply that the user made an error. But in this case, the user did not make an error. The user navigated to a URL that has been valid for years, and it is completely reasonable to expect that the link should continue to work.

Be careful to make the distinction: I said NOT say that the behavior of the link should remain unchanged. I said that the link should still WORK, and a 301 redirect to the blog will cause the desired effect while still allowing the link to work.

Stop telling users to go f*** themselves. Change it to a 301 redirect.

---

Apparently what is obvious to most people (+18/-5 at the time of my writing this) is not good enough, so here it goes with the long winded explanation:

Principle #1 - The User Comes First

I find it amazing that I have to say this, but I guess if everyone knew this, the software world would be a different place.

The attitude that I (and those I work with) have about programming revolves around the user. We're not writing code for ourselves. We're writing code for the user. The North Star of programming is that our code should be for the betterment of the world (or in this case, the internet).

It is here that we agree--we're trying to make the internet a better place. The point that you have consistently ignored is for who are we trying to make the internet a better place for. We're not doing it for ourselves. We're doing it for those who use the site.

You spent a lot of time typing that response, but what is amazing is how little you mentioned the user and how often you mentioned yourself. In fact, it takes little effort to find regions where you talk almost exclusively about yourself, only taking a break to talk negatively about the user:

Although I do not have any expectations for appreciation for my efforts to give back to the community, I am not a moron and will not freely volunteer my time to promote the community through an activity that has absolutely nothing to do with my diamond status when this is what I get for it.

At some point you got to say enough is enough. I've removed the url from the blog post, so users who would deliberatily click on a link that says "hey, this is deleted" won't have a bad experience.

In other words, "Me Me Me Me Me. If the user clicks on a dead link, its their fault."

That doesn't cut it. The user comes first. Always.

Your actions were not made with the user in mind. Quite the opposite in fact. Your actions were (and are) irresponsible.

Principle #2 - Error Messages Are Inherently Rude

Software developers are terrible at this. We just throw up an error and expect the user to deal with it. The problem with error messages is that they rarely do enough to remedy the problem.

Imagine if I walked into your office and said, "Your code has a bug in it." You'd likely ask, "Where did you find the bug? What did it do? Can you recreate it on my machine?" These are normal questions to ask. Now imagine that I simply repeated myself and said, "Your code has a bug in it." Imagine that no matter what, I constantly repeated that. You'd become incredibly annoyed with me and would consider me to be arrogant, because I am not telling you all of the information I know.

That is the problem with error messages. They rarely (if ever) do anything to remedy the problem they report. It is very common that error code will know an extensive amount about the problem, and many times, will have the ability to fix it (or at least get around it). But programmers can be lazy, and that is the result. They take the easy way at the expense of the user.

Its not always obvious what is meant when someone says, "The code could have fixed (or gotten around) the problem," so here's an example:

Lets say I'm writing Microsoft Word, I'm opening a word document, and I encounter an error. Here's an example of a error message:

ERROR: Could not open document.

This is a terrible error message. It tells the user absolutely nothing (except, perhaps, that the programmer needs to be fired). Here's a slightly better example:

ERROR: Could not open document. The file is locked for editing.

This is better, but is still terrible. Rather than saying, "I'm a sucky piece of software," this error message says, "I'm a sucky piece of software, and here's why." Continuing to improve:

ERROR: Could not open document. The file is locked for editing by user 'riwalk'.

We're slowly getting better. Now the error says, "I'm a sucky piece of software, but if you go bug the user 'riwalk', things might get better." Lets keep improving:

ERROR: Could not open document. The file is locked for editing by user 'riwalk'. 

             Would you like to open a readonly copy? (Y/N)

Now, for the first time, we've offered a solution. Someone else already has the file open, so obviously its a popular file. Maybe I just want to read it? Maybe opening a readonly solution and doing a "Save As" is an acceptable solution.

But now the question is, why should I even have to ask the user? Will the user ever say, "No"? Why don't I just open the file in readonly mode and tell the user what I did to fix things? Now, instead of an error message, we simply display an inline message between the document and the toolbar that says:

NOTICE: This document is currently locked for editing by user 'riwalk' and has been
                           opened in readonly mode.

This is a user friendly message. It acknowledges that while there are technical limitations that stop the user from completing the request, we can still get around it.

Now what does this have to do with P.SE? You're actions caused an error message that lies somewhere between the first and second one.

The 404 error page basically says, "Well I know what you were trying to do, but I'm a sucky piece of software, so I'm just going to show you an error message instead."

Are you making the web a better place?

Who knows? Everyone is trying. Not everyone succeeds. Let the masses judge our efforts.

The bigger question is:

Are you making the web a WORSE place?

This one has an answer:

YES!!!

If the goal of software developers is to help the user with meaningful error messages, then I can confidently say that you are making the web a worse place. You're creating 404 messages despite the fact that the intent of the user was obvious.

Don't say that no one notices. This thread exists because people noticed. You're making the web a worse place.

So, what should be done?

Nothing.

Well, almost nothing. Change it back to what it was.

You need to stop worrying so much about the past. So there are old questions out there that we wouldn't consider appropriate today. Fine. Just put a message on them and stop worrying about it.

Paradoxically, you're spending so much time worrying about making the web a better place, that you are confidently making it a worse place. When you're in a hole, stop digging.

So what can you do? Focus on what is being created now. Quickly recognize and close questions that don't meet our current standards. Rather than investing large amounts of time doing crap like this, why don't you invest your time in the Stack Overflow competition that is trying to prevent questions like the ones we're discussing now.

If you want to keep obsessing over these questions, fine. Do whatever the hell you want. Just do me a favor--stop making things worse.

Answer 3

For old questions (like the one linked to), there is a lock for "historical significance":

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: [FAQ].

Unfortunately, I can't speak as to why this question was deleted - there's no user name attached. But this is something that we should look into - undeleting this (and any other legacy questions) and appropriately locking them for historical reasons. I'd only limit this to questions from before the changes to site scope.

Answer 4

Insofar the blog post has been viewed over 120,000 times and has been referenced in at least 20 different places that I know off, including Reddit and other blogs, there are even a couple of Chinese translations out there. No one complained about the dead link, and no one linked to the deleted question.

Only people with > 10k rep on programmers can see the question. How many of those 120k "outsiders" have 10k+ rep in our happy little garden? There is a lot of link rot in the blogosphere and most of them probably would think that the link changed, not that some delete-happy person here deleted the post on "our" end.

My contention is that people have become far too delete-happy, and that if this site originally had the same criteria for locking & deletion that it has now, I'd never have gotten 10k rep.

Answer 5

What if we returned a 404 error message, but the custom error page showed the original content of the page (at least until the page isn't getting hit from search engines any more)? This solves the problem with the user interaction, because I've run into 404's for programming issues on SO myself, and it's always just a little irritating when I'm in a rush.

Showing the original content helps the user, even if the original content had a big "closed as dupe" message, linking to another question. Proceeding with the 404 helps search engines avoid sending people to that link, and will help clear the way for more relevant content to be linked to.

After the page no longer appears in search engines (a few weeks after deleted?, not sure time to propagate), stop displaying the original content, and go to a standard dead link page, which will force other sites with outdated links to update. Other sites should have had ample time to see that their external link was returning a 404 and update it at that point.