1

Per my understanding, for a while there was a sort of informal agreement to protect questions that are going to be featured at Ars Technica. Recently this approach has been challenged, so I would like to have a dedicated meta discussion to sort this out.

Does it make sense to protect such questions as soon as we learn about coming exposure at Ars Technica or it is better to stick with approach designed in 2010 for Stack Overflow and Super User - that is, protect when there is "a lot of drive-by noise answers"?

Related:

Those interested in more details about questions we discuss can find links to them in this feed or in edits history of SE community evangelist. Some details of 40 questions featured so far are provided in the 1st revision of this question.

8
  • 4
    My understanding is you protect questions which are gathering a large number of low quality answers. That is to say, protect them after they prove to be a problem, not before. If we're getting a lot of new traffic to a specific question, my guess is there's someone in there that could write an excellent answer, and we wouldn't want to prevent them from doing so just because the question might attract a some bad answers.
    – Rachel
    Apr 13, 2014 at 13:16
  • Also, here's a related MSO post from Shog9 regarding this subject. To quote him: "proactively protecting questions is a stupid idea, and you should never do it even if the system does allow you to do it."
    – Rachel
    Apr 13, 2014 at 13:20
  • @Rachel 2-3 months ago I would argue that reasoning like yours does more harm than good, and comments under Shog's answer you refer provide some insight why that could be. But recent changes in handling widely exposed content (specifically, auto-protection and removal of automatic CW-triggering) made me feel this isn't as clear cut now - that's why I want to learn what is our take on this currently
    – gnat
    Apr 13, 2014 at 13:24
  • 3
    My biggest problem is that we, as a community, have no say in featuring these questions - we can't reject questions for featuring or suggest really good questions that should be featured. Unless someone is watching the account for editing, we have no warning to ensure the question is appropriate and representative of our site (and therefore something that should be featured), actively prune old or unnecessary comments, and down vote or delete low quality answers as appropriate. It would make it easier if people coming here through these features had a good idea of what we're all about.
    – Thomas Owens Mod
    Apr 13, 2014 at 13:26
  • 1
    Also, Programmers is the only site ever featured. There are quite a few that would be suitable for Ars - SO/SU/SF, Programmers, The Workplace, Database Administrators, Information Security, Physics, Electrical Engineering, Android, Ask Different, Unix & Linux, User Experience, Ask Ubuntu, TeX, Webmasters, Web Applications. Those are just the "released" sites. I'm not aware of any high traffic site that features one or more SE sites on a regular basis, so we're a single data point to work with. As far as I know, we're on our own to decide how to handle this particular type of incoming traffic.
    – Thomas Owens Mod
    Apr 13, 2014 at 13:31
  • 2
    A third point to consider is that SO has (or at least had) a "welcome Redditors, this is what we're about" mod notice for questions that were featured there, especially on a high-traffic subreddit. I think that a generic "this question has been featured on a high-volume site, here's some pro tips for our visitors to read before stomping over our community" with links to important help center pages would be nice across the network. I've asked about this before, but it hasn't happened yet.
    – Thomas Owens Mod
    Apr 13, 2014 at 13:33
  • @ThomasOwens are you referring notice discussed in this MSO question: What is the meaning of “Notice added Redditted by …”?
    – gnat
    Apr 13, 2014 at 15:35
  • 1
    @gnat Yes. I asked about making that more useful (add help center links, etc) and bringing it to Programmers for Ars. Not sure if I did that on Meta SO or in more private channels.
    – Thomas Owens Mod
    Apr 13, 2014 at 15:37

3 Answers 3

Reset to default
3

Yes, there was an informal “protection racket” in place to protect questions that were to be featured in Ars Technica. The racket came to be after several of the regulars within The Whiteboard noticed the effects of being featured. Those effects included lots of views and votes, but also frequently included low-quality forumesque answers that pushed the question into community wiki status.

Protecting questions was a purely preventative measure. Most questions that are featured already have a high number of answers (where high ~= 10) and it didn’t take much to convert to CW status. There were two additional factors that encouraged pro-active protection. First, it was difficult to predict when the views from the Ars publicity would start rolling in. Second, many of the questions that have been selected are of marginal quality. Not necessarily close-worthy but definitely ripe for bike shed, forumesque answers.

When I have had time and advance warning of a question to be featured (ie. seeing the edit requests from Sam), I have gone through the question to flag questionable comments and vote on answers if I haven’t already.

And I’m going to echo MichaelT’s answer here for the sake of brevity. I agree with his thoughts regarding protection being a poor tool. But lacking better tools, you do what you can in order to help things along.

All that said, I don’t believe a protection racket is required anymore for questions that receive a lot of publicity. The biggest issue was questions converting to Community Wiki status which unfairly punishes participants in a question. Without the sword of Community Wiki dangling over a question, community moderation has the time it needs in order to clean up from any potential aftermath.

2
  • 1
    auto protection seems to be worth taking into account as well. It's "on the watch" 24x7 and it helps to contain damage of publicity side effects to essentially 3 answers from newcomers (answers from bandwagoning regulars are less of an issue, as they are more motivated to react to downvotes, due to having reputation /here, non-CW also plays a role, people may care about -2 from downvote as opposed to empty toothless "score" in CW posts/)
    – gnat
    Apr 15, 2014 at 12:55
  • two prior answers are very insightful but this one seems to hit the nail on the head
    – gnat
    Apr 16, 2014 at 19:20
6

Protection racket

Your own list illustrates how pointless this sort of proactive activity is: in most cases, few or no new answers were posted regardless of whether the questions were protected. Indeed, the most-answered question of the lot was immediately protected.

It's not the raw traffic that creates the problem. It's traffic coupled with a question that begs for participation. Ars has its own forum attached to these, there's no pressing need for readers to come here to kibitz - unless the question itself is crying out for yet another answer...

Sometimes this is unavoidable. Questions that, within the rules of the community can only be answered in a narrow, controlled fashion may inadvertently imply a much broader scope to outsiders who don't know or don't care about the community norms. Protect is useful in these situations, where every n views will tend to generate m poor/spam/non- answers.

But these tend to be the exceptions, particularly on a site like this which doesn't really go in so much for discussions of tools or hardware or other spam-friendly topics. Protection here should be reserved for those cases where a good, answered question is attracting a large volume of noise / meta commentary from new users.

If you want to pre-emptively do something useful to forestall noise on questions, then edit them. Get rid of bits that implicitly invite any and all opinions, tighten up the scope, focus on making the question reflect its best answers rather than aiming for a big tent that welcomes all participants. Not only can this reduce confusion for outsiders, it can also benefit the folks already on the site who wouldn't be affected by Protection anyway.

Effects of Ars

I'm counting roughly 192 questions that've been linked to from ArsTechnica over the past 2 years that Sam's been doing this. Could be a few more or less; there's a lot of cross-linking, and I might have missed a few related questions - but this should be the vast majority of all questions that've been linked to. You can view a list here if you're interested...

First, some basic stats on how these questions have aged. Out of those 192, 31 are currently closed and 61 are currently protected. The average number of (non-deleted) answers is 12, the average # of deleted answers is 2.

Now... How they fared after being shared. I'm going to look at this from two different angles: a simple look at the number of answers attracted that would have been blocked by Protected status (had it been applied) and a comparison of the answer distribution before and after.

Protect-worthy answers

If all of the questions in that list had been protected prior to being shared, a total of 55 answers would have been blocked.

Of those 55 answers, 18 scored less than 0 and 16 are currently deleted. 23 currently score > 0.

Needless to say, these were posted to the 131 questions that weren't Protected.

Answer distribution

Of the 2578 answers posted before these questions were shared, 2019 are currently scored > 0 while 211 are scoring < 0. 219 are deleted.

Of the 124 answers posted after these questions were shared, 60 are currently scored > 0 while 33 are scoring < 0. 23 are deleted.

Conclusions

As I noted in the first section, it's clear that Protection has relatively little effect on these. This jibes with my observations elsewhere on the network: trying to predict which questions will attract crap answers based purely on traffic is exceedingly error-prone; other heuristics are needed. Which is why we implemented better heuristics, based on observing which questions actually benefited from Protected status.

The % of not-useless answers posted after sharing is considerably lower than the % posted before; this shouldn't be too surprising, given none of them were new or unanswered. The % of not-useless new-user answers posted is lower than the % from existing users, but the volume is considerably lower.

Recommendations

Reserve Protected status for questions that are attracting noise or abuse from new users. Protect is not a prophylactic.

11
  • I just re-checked, of 40 questions in my list 13 were protected long before exposure, 5 were "protected by closure", 17 were protected in the same or next day and 5 weren't protected. Given that only 5 of 40 questions weren't protected, I wonder what regardless means here
    – gnat
    Apr 13, 2014 at 16:40
  • Same thing it usually means. As in, "vampire attacks were rare during the past year, regardless of whether the townsfolk remembered to wear their garlic togas".
    – Shog9
    Apr 13, 2014 at 16:42
  • In order to pre-emptively do things, wouldn't it be better if we were told in advance (without needing to stalk a particular account for edits) what questions would be featured so we could ensure they are of the highest quality (and stop the posting of any question that isn't appropriate at all, should it come up)? Or, better yet, if the questions that were chosen to promote this community were chosen by this community?
    – Thomas Owens Mod
    Apr 13, 2014 at 17:09
  • Anyone here can promote anything they want, whenever they want, @Thomas. If you have something you particularly want to show off, don't hesitate to send us a link - we've tried soliciting these in the past without much response, but don't let that stop you from reaching out if you find something exemplary.
    – Shog9
    Apr 13, 2014 at 17:15
  • 1
    Do people actually realize that they can suggest really good questions/answers for sharing (not just here, but across the network)? I didn't until just now, aside from individuals sharing links using their own social media accounts or posting to link-sharing services like Reddit. And I'm sure that people who aren't reading the comments here on this particular answer wouldn't either. I'm thinking especially in light of recent events as some questions/answers on InfoSec or SF are really good explanations of Heartbleed, yet didn't seem them promoted on any major sites that I follow.
    – Thomas Owens Mod
    Apr 13, 2014 at 17:18
  • 1
    You might want to read this blog post, @Thomas - it's a little bit dated, but Sam talks about some of the things he learned about trying to jump on topical topics. BTW, you may have noticed a link to Information Security in the latest Ars post...
    – Shog9
    Apr 13, 2014 at 17:23
  • hmm my calculations tell these 40 questions have got 509 answers, 320 of which are low score, sounds a bit like an attack to me. Maybe at some other site, with different culture, this would be considered okay
    – gnat
    Apr 13, 2014 at 19:39
  • Your data is crap, @gnat. Hang on, I'll compile an exhaustive list.
    – Shog9
    Apr 13, 2014 at 19:53
  • "edit them" -- really? taking into account that Ars features questions that already have good answers, answers based on what was already presented in the question, this may be a cure worse than decease
    – gnat
    Apr 13, 2014 at 20:04
  • I know, I know... Why fix and maintain things when you can click a few buttons and feel like you've done something, right? Why edit when you can protect, why earn delete privileges when you can flag, why contribute answers when you can criticize others', why look for solutions when you can just pass the buck?
    – Shog9
    Apr 13, 2014 at 20:11
  • "Effects of Ars" is a pretty compelling reasoning, thanks! I'm going to give the question a few days prior to accepting this, only for the case if (quite unlikely) someone provides more compelling answer. (side note if your comment about passing the buck is at me, you better take into account that I am top editor at Programmers and #3 at MSO and Workplace)
    – gnat
    Apr 14, 2014 at 5:21
2

There are two types of questions, the young ones and the old ones.

The young ones are still malleable. We can edit them. And before people say "you can always edit something" there is the thing that must be considered - it gets more and more difficult to edit a question to be a good one the more answers it gets. When a question has half a dozen answers, its probably too late to edit it - the 20ks can only delete answers with a negative score and unless an answer is truly awful it rarely gets there.

Once a question gets half a dozen answers (especially if one is accepted) it becomes very difficult to edit it into a better question without causing some of the answers to become invalid. The culture of the frequent editors appears to be "don't invalidate an answer with an edit", and given that we can't clean up meh but up voted answers on an question that leaves us with fewer tools.

Note that half a dozen answers can happen that first day when we can't protect the question.

The tool to address an older question is one of two: close and protection. Closing is an option, though we (frequent closers) do get some grief for closing borderline 'primarily opinion' or 'too broad' questions that are popular. The other tool is protection. Consider also that sometimes, in the past there have already been attempts at closing a question. Its not uncommon for me to hit a recently bumped historically hot question and see on there "you voted to close on December 14th, 2013"... and, well, I can't kick it into the review queue again. Asking around chat sometimes finds the frequent close voters also may have expired votes on the question making it very difficult to use that tool to address the question.

Protection works poorly as a tool to quiet down a question as noted elsewhere. That said, for experienced answers, protection may give them pause before writing a short, pithy answer. If that is the case, the psychological aspect of protection still does serve a purpose to those experienced editors.

Beyond protection, the only remaining tool that we (non-mods) have at this point is the flag for a mod to close or lock it.

If you ask any of the mods, I'm fairly sure you'll find that I am a fairly frequent historical lock flagger.

A bit more on editing, specially with respect to Ars'ed questions. Doing a significant transformation on the question would make it inconsistent with the Ars post. For the appearance of Stack Exchange on Ars, I'd much rather have a protected question than a closed question and for that question (and answers) to match what is on Ars. Bait and switch isn't a great thing.

Things to think about:

  • Can we expire expired close votes after some period of time? I mean sure, we can flag for a mod to close it, but I'd rather have the community close it than an evil mod (less flack for the mods).

  • What tools / process is appropriate for turning some of those questions into community lock questions. I suspect it would be something like 'Do a community lockable answer, flag for a mod to community lock & delete everything else, edit the resulting question into something good'? Though some of those steps might require some reordering.

  • Are there better tools that could be designed for 10k+ users to be able to better moderate questions and answers? The big thing being after a little bit of time, the only tool that we have that makes any sense is protection.

  • Proactive protection isn't ideal. Part of our considerations is that weekends we have light moderation (from moderators, 20ks with delete answer votes, and down voters to get an answer low enough to do a delete answer vote, and 10ks who will do a 'recommend delete' rather than 'edit' in the low quality review queue). Would it be possible to get some community managers to help with some eyeballs on the Ars'ed questions who could help guide people to give better answers when there's something poor, or help with deleting in a timely manner to not have examples of other poor answers that encourage more?

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .