8

Could we please make a trial run of modified "hotness formula" for Programmers questions?

  • Modification details are described in this MSO post as follows:

    As far as I can tell, substantial part of Qanswers in current formula is fake.

    (log(Qviews)*4) + ((Qanswers * Qscore)/5) + sum(Ascores)

    About 1/3 of the answers studied here (83 of total 254) have score less than 1/100 of top voted post in respective question.

    Given that questions checked were ones with tens thousands views, insultingly low score indicates that assuming these answers to be popular wouldn't even be in the ballpark. Still, the formula pumps these into Qanswers value, as if it is something everyone would be happy to read (hint: it isn't).

    Consider tuning the formula to make it deviate less from voting results. Ignore answers with non-positive score. Or better yet, ignore answers scored less than some reasonable fraction (eg 1/10) of the top one.

Given that current formula appears to give an unjustified value to crappy answers in highly upvoted questions (Qscore/5, no matter how much is answer downvoted), I would like to perform test run to find out if suggested change would make an impact to issues outlined in Programmers meta posts:

  • What can we do to help users understand our site better?

  • Answers quality in hot questions
     
    Note: results of the trial run to be analyzed using study analogous to one performed in Answers quality in hot questions. Current evaluation shows about 101-118 low quality of 218 answers sampled.
     
    Upon completion of trial run, similar evaluation is to be done in order to estimate whether there was a substantial impact, positive or negative.

Complementary information to this request is provided in comments below, marked with "for the record..."

Note I expect modified formula to be competitive to current one at "moderately hot" questions, following the reasoning outlined here:

For questions that are not too hot (likely 2-3 clicks away from top of the list) it's natural to see things working exactly as intended. Answers and comments quality is mostly maintained by site / tag regulars (business as usual), collider brings moderate amount of interested newcomers from other sites with their views, votes and fresh perspective, everything is nice and cool...

Recent hot question about automated testing gives an example of why I would want to try a modified formula. In 17 hours, question collected 17 answers, merely 5-7 of which provide useful original content.

Overheat of fake hotness impacted question like a Black Saturday fire.

http://cache.boston.com/universal/site_graphics/blogs/bigpicture/ausfire_02_09/a01_17890753.jpg

After the fire settled, what is left for future visitors of the question looks like a wasteland of low quality garbage (note that due to high views, it will score high in web searches).

http://cache.boston.com/universal/site_graphics/blogs/bigpicture/ausfire_02_09/a04_17906927.jpg

In the light of above, community wiki status stamped on the question looks especially despicable. It kind of suggests that after damage has done, community members are invited to go over the crap brought in by braindead formula, carefully analyze it and try to tame the pain by downvotes. As if they don't have anything better to do!

Update

Related feature request has been posted at MSO:

In hotness formula, discard answers when voting evidence indicates that these are not good data points

Please stop counting proven low score answers in hotness formula. Please roll the dice fairly, let user voting and time decay contribute to hotness score as intended. Please promote to collider audience less brain-damaging content to learn from.

21

2 Answers 2

Reset to default
2

As a suggestion to your proposed change in the algorithm:

((Qanswers * Qscore)/5) this section should ignore counting any answers that have a total negative value. (ie. -1 or less)

However, sum(Ascores) should still accrue the negative votes.

The net effect is that poor quality answers (aka negatively scored or with many down votes) will drag down the total hotness value.

I don't have a data set to run the proposed change against. Actually, I'm a bit of a n00b in that regard and don't even know where to get said data from...

6
  • I've been using data provided in this MSO question: Don't let questions stick to the top of the hot questions list forever. There are 9 examples of "sticky questions" there, having total 254 non-deleted answers. By the way none of these 254 answers have negative score, which makes me suspect that testing ignoring only negatives will show very little difference compared to current approach: in all 9 example questions Qanswers will remain the same as now
    – gnat
    Jan 25, 2013 at 17:51
  • Not a very good data set then, is it? very good == representative of P.SE hot questions
    – user53019
    Jan 25, 2013 at 18:06
  • well, if we define P.SE hot by number of views (on open questions), 62K will currently cut same number of questions (9) as in original data: programmers.stackexchange.com/… - okay, let's see... About 750 non-deleted answers total, 10 with negative score. Or, if we drop out two "historically locked" questions that were apparently cleaned up, there will be 10 with negative score of total about 250 "votable" answers
    – gnat
    Jan 25, 2013 at 18:37
  • 1
    ...by the way my original idea was just like yours, to drop only neg-score answers (I thought community regulars DVs would suffice to make a difference). It's only after I studied data a little and re-checked my own observations that I changed my mind
    – gnat
    Jan 25, 2013 at 18:41
  • counting zero-score answers assumes that downvotes are needed to make a difference, right? It's like, you know, downvote if you want it to go lower -versus- upvote if you want it to go higher. I wouldn't mind trying either of these approaches but I'd rather prefer to first test one that feels less conflicting. Downvotes are painful, for both posters and voters. Does that make sense?
    – gnat
    Jan 25, 2013 at 22:19
  • 2
    (upon further thinking) one benefit of having 0-score answers count is, it's more sensitive to potentially-hot questions at early stage, when there's not yet enough votes to judge better. Frankly, this is the part I really like in current formula. As far as I can tell, modification that suggests cut off at "1/10 of top-voted answer" behaves about like that in the beginning; it only drops after first upvotes received by answers - if this is too early, one tweak to consider is to cut at top/10 - 1, this will keep zero-voted answers in until one of the answers reaches score more than 10.
    – gnat
    Jan 29, 2013 at 12:08
1

This request has been warp driven to the following requests at MSO:

Side note - this very answer has been used as a "buffer" to draft the latter request: details in revisions history.

5
  • 1
    Hey gnat, I am not going to directly edit because it would be quite significant, but here are my comments. (A) I think this should focus more on 'What questions we want in the hot list' followed by 'How we can accomplish that', (B) Worrying about technical limitations is a side-note at best and shouldn't even be in the same league as the goals of (A), (C) I recommend adding weight to close votes to allow questions that may not be on topic to have a tougher time getting on the list with a ton of interest prior to getting closed and giving a poor impression of on-topic to the SE network
    – jmac
    Jan 14, 2014 at 0:49
  • @jmac thanks, I will try to edit to your points (A) and (B). Point (C) looks worth adding, too. By the way, please feel free to directly edit - as opposed to some, I am huge fan of your aggressive edits :)
    – gnat
    Jan 14, 2014 at 3:28
  • @jmac I think I fully integrated points (A) and (C), would you mind taking a look (edits are welcome, aggressive or not - I trust you there:). As for (B), it's a bit trickier: I moved it visually into the "basement" by separator, but it sort of stays in top league, because it just addresses the main reason for decline of the prior request. It even has been stressed in rev 4 per discussion at Whiteboard: chat.stackexchange.com/transcript/message/13166653#13166653
    – gnat
    Jan 14, 2014 at 7:51
  • 2
    looking great! Thanks for all the hard work, the entire community of small SE sites is grateful for your vigilance on this topic (it really is a bane that should be a boon in many cases).
    – jmac
    Jan 14, 2014 at 23:53
  • @jmac Glen said that version based on your edit was much better than original one. :) To me, your points (A), (B) and (C) were even more valuable than that (Glen couldn't tell because these points were integrated in both versions I asked him to compare)
    – gnat
    Jan 15, 2014 at 9:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .