6
$\begingroup$

There is much furor around the perceived of and others. All such reactions and discussions that I have seen have relied on anecdotes and gut feelings, rather than hard data.

Of course, a regular puzzler's gut feeling might be a good estimate, but confirmation bias is a thing, so let's ask just in case.

What does the data say?
How bad is this problem?
Is this really -specific?

| |
$\endgroup$
  • $\begingroup$ In order to discuss what the data says, we first need to determine what data should be collected, and examine whether (and to what extent) we think that data is representative of the metrics we're trying to collect. This has to happen before data analysis occurs. $\endgroup$ – user20 Sep 9 '16 at 3:03
6
$\begingroup$

I made a chart.

I used this SE Data Explorer query to get up/down votes and question counts for all of our tags, and plotted them with D3.js to get this:

All our tags, by vote ratio (Click for a full view.)

How to read it

Tags appear as circles, which is proportional to the total votes on questions which have that tag. They are arranged vertically by upvote/downvote ratio. Horizontal position is just for visual separation.

The red line shows an average ratio of votes for a tag. This was calculated by dividing all tags' total upvotes by all tags' total downvotes.

Cautionary notes, because I am not a statistician:

  • 35 tags that had no downvotes were excluded, to dodge division-by-zero. They applied to 139 questions at most total, which is a little smaller than the bubble on the left.
  • Overlap caused by questions having multiple tags is unaccounted for here. Every question counts as +1 for all of its tags, regardless of number of tags.
  • This is an overall view, of all questions across all time, not of recent events.

Interpretation

Ideally, we'd want to see as much orange as possible as high up as possible. Our tags are mostly clustered with up/down ratios between 20 and 40.

I would conclude that if has a quality problem, so does almost everything else. Riddles are perhaps getting all the eyes because there are lots of them, both good and bad. Our next two most popular tags and are doing worse on the up/down ratio front, and in fact worse than tags are doing on average.

Conclusions

For our solutions against low-quality content to be long-term, they must generalise beyond a single tag.


I'll put the code on Github sometime soonish in case anyone wants to improve or repurpose it.

| |
$\endgroup$
  • 6
    $\begingroup$ It's worth noting that upvotes and downvotes are extremely suspect in this quality discussion, and may not be a viable metric of question quality. As a result, any discussion based around upvote/downvote ratios is highly suspect, at least to me. That's one of the issues raised. (Also note that failure to consistently downvote low-quality questions is central to the issue at hand, and if we were doing so, this issue wouldn't have been raised at all.) $\endgroup$ – user20 Sep 9 '16 at 3:00
  • $\begingroup$ Strong vote of approval for kicking off this kind of analysis. I hope this turns out to be easy enough to experiment with variations. Strong note of caution: users who have filtered out a tag are virtually unrepresented by votes, and filtering out cannot simply be assumed to be a negative vote because it might even be for lack of knowledge in some areas as much as for disdain of puzzle quality in those areas. (users who habitually skip certain tags have also in effect "filtered out" those tags, even if not automated.) $\endgroup$ – humn Sep 9 '16 at 7:15
  • $\begingroup$ @Emrakul I couldn't think of a better metric. What would you suggest? Perhaps we need a "average-votes-per-question by tag" chart next, to check. $\endgroup$ – Anko Sep 9 '16 at 9:35
  • 1
    $\begingroup$ Previous ideas on quality metrics at Should we implement an automated way of highlighting quality content? $\endgroup$ – humn Sep 9 '16 at 12:14
  • 2
    $\begingroup$ @Anko, perhaps another metric that might provide some insight is the number of questions being closed for being "too broad" or "off topic", or the number of answers per riddle question. It seems like a big part of the problem with riddles has been that poor riddles are open to a lot of interpretation, and often invite a lot of speculative answers. On other sites, a multitude of answers is often welcomed, but here it is often an indication that the question is poorly-defined. $\endgroup$ – GentlePurpleRain Sep 9 '16 at 14:59
  • $\begingroup$ @Anko This is a difficult question, and I'm still thinking about it. I like GPR's idea, though, to start with. $\endgroup$ – user20 Sep 9 '16 at 16:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .