What should we do with "Should UTF-16 be considered harmful?"

Question

Today we stumbled across an old question:

Should UTF-16 be considered harmful?

I'm going to ask what is probably quite a controversial question: "Should one of the most popular encodings, UTF-16, be considered harmful?"

The long and short of the question is that UTF-8 is unambiguous and compact, while UTF-32 is unambiguous and fast. UTF-16 is ambiguous (at times) and wins neither on speed nor on size. Since it is non-optimal in every case and there are known defects in operating systems and popular language implementations, why use it?

The answers and all eleventy bajillion comments are excessive, at times inaccurate or opinion-based, but nonetheless do contain a lot of useful information. They make compelling arguments for and against using UTF-16 in various situations. As someone who has been programming for many years and using Unicode for most of them, I learned a few things myself.

We discussed it in chat for a bit, with the following options:

• Delete the whole question.

• Clean up the comments and answers, then lock it.

• Just lock it.

Currently, the question is locked. Should we do anything further with this question? Delete it? Unlock, clean it up, lock it again?

Keep in mind this question does have over 70k views and four questions link to it.

Answer 1

I think the contents of the question, answers, and comments has some problems. If this were asked today it would be aggressively curated, if not closed as "primarily opinion-based."

However, this is a historical question asked near the beginning of the site when the focus and on-topicness was very different. The key question is this: does the value provided outweigh the cruft that collected over the years?

I believe it does, and I am in favor of keeping it locked. Reading through it I found a lot of information and specific details about various implementations and how they deal with UTF-whatever. There are many things that can go wrong when dealing with Unicode, and this question covered quite a few of them.

This particular question is trickier than many other historical questions. Specifically, this one has a ton of comments and the comments add substantially to the knowledge in each answer. Simply deleting all of the comments would lose quite a bit of useful information while also clearing out a lot of useless junk. Given the large amount of work required to curate this question after the fact with little benefit, it is not worth it to spend the time. Just keep it locked.