13
$\begingroup$

There's obviously been some great discussion recently around possible platform changes that could help bring more attention to this site's best content, and we already have the quarterly "Best of" series... However, I've been trying to think of other ways to highlight worthy content, especially content that may have gone by unnoticed for a lot of people.

Issues:

The problems I'm aiming to overcome are:

  1. Quality content slips off the front page too quickly, and while well received, don't get the views they deserve and end up getting missed by people who would otherwise be interested
  2. Due to the way the Hot Network Questions algorithm works, lower quality questions often get vastly more views*, and thus get more votes (purely from this additional exposure), meaning that raw votes are unfortunately not particularly representative of the quality of a post, which makes things like the Hottest Questions This Month tab fairly useless (case in point, 2 of the top 4 posts in that list are on hold/closed)
  3. The "Best of" posts, while awesome, are necessarily limited by the fact that they're only run quarterly and because it requires individual users both remembering specific puzzles and choosing to participate, by nominating them, meaning we're only highlighting a handful of posts each year

Proposal:

Broadly, what I'm proposing is monthly posts here on meta listing collections of high quality content, in a similar way to the existing "Best of" posts, except making them:

  • Fully automated
  • More frequent
  • Highlight significantly more content

With that in mind, I've been playing around with the puzzling data on SEDE trying find ways of identifying posts worthy of bringing additional attention to. As noted above, looking at pure vote or answer counts is not a good indicator, but I've observed that looking at "votes per view" starts to not only filter out some of the chaff, but also uncover some pearls with low view counts.

Obviously just directly sorting posts by votes/views doesn't work particularly well, since you end up with outliers having very low views and one or two votes overwhelming the results. However, using that as the basis, I think it should be possible to find an algorithm to automatically list out "high quality" content on a monthly basis in a reasonably fair and accurate manner.

I've come up with a couple of potential algorithms that appear to be suitable, but have different strengths/weaknesses. I've posted more detail on each as individual answers below, so as not to further clog this post and so that they can be discussed/voted on in their own right.

Discussion Points:

Firstly, let me be clear that I see this as a complementary series of posts to the "Best of" quarterlies, with those being the "people's choice" awards, so to speak, and these being purely automated. With that in mind, some topics for discussion:

  1. Is this generally something worth pursuing?
  2. Which of the algorithms below are most useful/suitable? Or is there an alternative that's better? (noting that I'm far from a statistician, so it's entirely possible that there's issues with my algorithms)
  3. How frequent should these posts be? (I think monthly is probably a good balance)
  4. How many posts should we "highlight"? Top 10? Top 25? All above a given threshold? Top N posts from multiple algorithms?
  5. How should the posts be structured?
    1. Single mega post with a new "answer" each month, containing that month's list
    2. Post per month with a single "answer", containing that month's list
    3. Post per month with several "answers", each containing a list from a different algorithm
    4. Post per month with individual "answers" for each puzzle in the list (so that they can be voted on and "sorted" by the community)

I've posted a bunch of answers showing a few example algorithms and their results, purely for the benefit of gathering vote based feedback, but feel free to post other answers just adding to the discussion, or proposing alternative algorithms.

* The issue with HNQs from PSE's perspective is that the algorithm is based off a question receiving many upvoted answers in a short time, so a low-ish quality, broadly worded riddle/lateral-thinking puzzle will naturally be ranked as "hotter" than a puzzle which sits unsolved for a while, or gets a single correct response and is "done". Then, once in the HNQ list, it gets more views, some proportion of which naturally being upvoters and more answerers, further boosting the "hotness", and skewing the validity of raw vote scores as a measure of quality.

$\endgroup$
8
  • $\begingroup$ Boyoboy, would love to see this for all puzzles of yore too. Can the ongoing treasure hunts really be fully autonomous? Also seems like interleaved sorting could eliminate any need to choose just one algorithm or a cutoff count. $\endgroup$
    – humn
    Commented Jun 10, 2016 at 8:14
  • 2
    $\begingroup$ @humn - on automation: it'd be unofficial, but yes, I'd just write a script to extract the data through the SEDE data API, and have it post here once a month. On historical puzzles: I was thinking of doing retroactive ones too (maybe just 3 monthly to prevent a flood), but in the short term, all the algorithm links in the posts below allow you to change the month/year and do your own queries. $\endgroup$
    – Alconja
    Commented Jun 10, 2016 at 8:55
  • 1
    $\begingroup$ First of all, just having the different search links here is already a great step, thanks for that. Now, from a purely personal note, none of the searches seems to have picked up on my recent recent question Maybe, it just isn't a good one? ( Arrrr.... - Pirate's quote) ;c) $\endgroup$
    – BmyGuest
    Commented Jun 10, 2016 at 13:13
  • 1
    $\begingroup$ Is there some merit in adding in "# favourite votes" into your algorithm? This, in theory, should weighten up questions which in one way or the other were deemed "worthy of revisiting". $\endgroup$
    – BmyGuest
    Commented Jun 10, 2016 at 13:15
  • 4
    $\begingroup$ I second BmyGuest's first point. Maybe the best implementation is just a meta question "What are some SEDE queries to find great puzzles?", with different algorithms as answers. $\endgroup$
    – Julian Rosen
    Commented Jun 10, 2016 at 13:17
  • 1
    $\begingroup$ @BmyGuest The SEDE data is only updated periodically (I think it's weekly, but not sure of the exact time frame), so recent questions won't be there yet. $\endgroup$
    – GentlePurpleRain
    Commented Jun 10, 2016 at 14:45
  • $\begingroup$ @BmyGuest You should note that a lot of people use "favourites" as bookmarks, just to be able to find a post they want to come back to. I do this often, and once I've revisited the post, I often remove the star. Since this behaviour seems fairly common, I don't think using favourite points as a measure of post quality is a good idea. $\endgroup$
    – GentlePurpleRain
    Commented Jun 10, 2016 at 14:47
  • $\begingroup$ @GentlePurpleRain While I still use the favs in a similar manner, the fact remains that questions which (still) have a lot of favs have generally 'something' speaking for them. I'm not suggesting that it should be a prime criteria, but I think it might add to the total "is-potentially-intersting" score in a search. $\endgroup$
    – BmyGuest
    Commented Jun 10, 2016 at 16:40

3 Answers 3

Reset to default
4
$\begingroup$

Algorithm 2: Puzzles You May Have Missed

This algorithm is effectively identical to "Popular Puzzles" except that it filters out anything with over 500 views. So basically it is attempting to highlight very well received posts that didn't get many views before disappearing off the front page.

Pros:

  • Helps to bring attention to the hidden gems each month rather than posts which already got lots of views

Cons:

  • Possibly ends up listing content which isn't worthy of further attention
  • The best post in the world with 501 views gets ignored, meaning there's a big gap of awesome puzzles not represented between this and the current "best of" lists
  • The hard-coded cutoff at 500 might become a problem as the site grows

Sample data - top 10 results for January 2016 (votes / views):

$\endgroup$
3
$\begingroup$

Algorithm 1: Popular Puzzles

This algorithm still gives some weighting to the overall score of the post, but uses the votes per view as a multiplier. This means that a highly scored post will still get represented, but lower vote posts that have a high vote:view ratio will bump out lower quality but high view count posts regardless of their overall score (eg. content that has been artificially inflated by HNQs).

Pros:

  • Because it will still generally include the highest voted questions for the month, this "feels" like a reasonably accurate list of the "best" content

Cons:

  • Users are more likely to already be aware of any content with higher view counts, so this may be bringing attention to posts which don't really need it (of course it could be argued that they deserve it, making this a pro)
  • Arguably doing a similar sort of thing to the quarterly "best of" posts

Sample data - top 10 results for January 2016 (votes / views):

$\endgroup$
2
$\begingroup$

Algorith 3: Well Received Puzzles

Unlike the other two, this algorithm focuses purely on the vote:view ratio, but sets the bar at a minimum of 10 upvotes to filter out the outliers.

Pros:

  • Assuming the bulk of views are from logged in users, this is arguably the "fairest" measure of how well received a post is regardless of how many people see it

Cons:

  • Likely to automatically rule out anything that hits the HNQ, since that will generally get lots more non-voting views, lowering the ratio
  • The highest voted questions will get shared/viewed more by non-logged in users making it harder for them to maintain the ratio compared to lower voted posts only viewed by regulars

Sample data - top 10 results for January 2016 (votes / views):

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .