What changes we should make to the site-specific hotness formula in order to make open-ended questions less 'hot' and make possibly-older-unsolved-but-high-rep questions more 'hot'?

Problem as I perceive it:

  • The current 'hotness' formula rewards questions which get lots of answers and get them quickly.

  • This has the effect of rewarding questions which are open-ended, broadly defined, and can be easily answered without a lot of thought.

  • Those questions are (in my subjective opinion) not as 'good' (for puzzling.SE) as other questions where the answer needs thinking about and there is a single correct answer.

The current 'hotness' formula is here: What formula should be used to determine "hot" questions? "This algorithm will heavily favor questions with LOTS of answers"

What changes - if any - should we make to it for puzzling.SE?

  • 3
    $\begingroup$ One thing is for sure: the number of answers must be made less significant and/or the reputation and activeness of the answerers must be made more significant. I believe it is particularly the first of these that causes Puzzling questions to be featured so often. Not that I have a problem with that by the way. $\endgroup$
    – d'alar'cop
    Nov 17 '14 at 12:13
  • 2
    $\begingroup$ We've had a similar issue at PPCG, and I believe other sites have voiced similar concerns (about selected question "quality"). I'm not sure that the formula can be changed per site, so that might be a good thing to ask as well. $\endgroup$
    – Set Big O
    Nov 17 '14 at 13:46
  • 1
    $\begingroup$ I'm told by a mod that it can (be changed per site). What did ppcg do? $\endgroup$
    – A E
    Nov 17 '14 at 13:53
  • 1
    $\begingroup$ @Geobits It was changed for Stack Overflow, so there is at least one option for changing the hotness formula. Don't know if there are others, though. $\endgroup$
    – user20
    Nov 17 '14 at 17:13
  • $\begingroup$ @Emrakul Ah, didn't know that. thanks! $\endgroup$
    – Set Big O
    Nov 17 '14 at 17:22
  • 1
    $\begingroup$ @AE Nothing directly. The really bad ones were mainly just a surge that accompanied an uptick in traffic. We had to nuke a tag/challenge type, but it mostly evened itself out. We still have some bad ones that get into HNQ, but I think that goes for every site. $\endgroup$
    – Set Big O
    Nov 17 '14 at 17:23
  • $\begingroup$ This critique of the SO points system addresses the issue of easy answers getting more well-rewarded than difficult answers. It was evidently written some time ago - I don't know whether the criticisms remain valid, perhaps the system has changed. $\endgroup$
    – A E
    Nov 18 '14 at 15:50
  • $\begingroup$ @AE I think it's notable that for the two questions he linked in that section, the "simple yet better rewarded" answer has since become the lesser-scoring one. The easy answers get more points at first in my experience, but in the long run better answers usually pass them up. I can't say if the same will happen here, since it seems like a very different dynamic than SO. $\endgroup$
    – Set Big O
    Nov 19 '14 at 2:21
  • $\begingroup$ @Geobits, that's really interesting and encouraging. $\endgroup$
    – A E
    Nov 19 '14 at 9:36

As many others have said, having the hottest questions be those with lots of low quality answers is problematic. I wager the network wide aggregation of hotness (the calculation used to make the actual hot questions list) normalizes all the site specific values and performs its own calculations. Likely this includes number of answers already, so we should factor this out of our calculation. I could be wrong about the way this works, but I'm too lazy to dig through MSE to figure it out, so I'm just going to assume it.

The variables we have to work with, from the linked MSE post, to which I've added abbreviations:

  • QS - Votes [aka Score]
  • V - Views
  • N - [number of] Answers
  • AS - Answer Votes [aka Score]
  • A - Whether the question has an accepted answer or not
  • To - Time question was originally asked
  • Tr - Time of last activity on question
  • Rq - Reputation of asker
  • Ra - Reputation(s) of answerers

Factors we want:

  • QS - Question score
  • AS - Answer scores
  • V - Views
  • Tr - Time of recent activity

Factors we don't want:

  • Answer count. Maybe this can be useful indirectly, but questions experiencing willy-nilly answer accretion aren't what we want to show.
  • Reputation. Some of the best content I've seen (admittedly I haven't been active on here long, so this is going to be biased in a few ways) has come from new or low rep users. Whether or not someone can write a great post has nothing to do with their rep. And high visibility for low rep but high quality contributors will create a positive feedback loop (and similarly a negative one for low quality contributors), resulting in the creation of more good content.

I'm undecided on whether or not to include "has an accepted answer" in the calculation. If so, weight the formula towards not having one. A puzzle which is hot but unsolved is an interesting one.

Calculated metrics I think are valuable:

  • AAS - Average answer score. This is a good way to correct for lots of low quality answers. But, if there are one or two really great answers among the low quality ones, then this will redeem the hotness score somewhat. Which is good, because it means that the puzzle was really tricky, or someone came up with a really great answer; these are the sorts of things that make a question interesting.
  • TA - Length of time active: Tr - To. This needs to be normalized somehow, since it's an interval of time.
  • VD - View density: V / TA. A measure of baseline interest in the question (or at least its title and tags). This corrects for question age, since older questions will naturally have a lot more views.
  • SF - Score frequency: QS / TA. Questions which get more upvotes quickly are objectively hotter.
  • SD - Score density: QS / V. A question with 5 upvotes and 100 views is probably a better question than one with score 10 and 10000 views.

I'm intentionally not supplying an actual formula. I'll leave that discussion (and the inevitable bikeshedding) to others. But personally, I think something that is dominated by Tr and the four meaningfully qualitative calculated metrics (TA doesn't tell us much about the actual content) is the right way to go.


Not the answer you're looking for? Browse other questions tagged .