17
$\begingroup$

This is a follow-up to Why do we have review audits?, in which I commented on the weirdness of having review audits on Puzzling when they're otherwise reserved for the largest SE sites. In that post, I enquired as to why they were introduced in the first place, and Emrakul♦ said:

We had enabled audits as an experiment waaaaay back in beta in response to a long, long string of terrible reviews and a handful of review bans. It was a test [...]

if you guys want it disabled, I'm not opposed. It just comes down to whether it annoys, and how much.

This post is intended to discuss that issue: whether review audits on Puzzling are actually doing any good, or whether they're obsolete (or were unnecessary in the first place) and should be abolished.

Should we continue having review audits here?

If so, why? If not, why not?

$\endgroup$
7
$\begingroup$

Normally, audits aren't enabled on sites of this size; there just aren't enough reviews (or enough posts that meet the criteria for becoming audits) to justify it.

And that's true here as well. So, enabling them here was probably a bad idea at the time; the person who prompted the change should've been either banned for a longer period of time (something we only made possible recently) or suspended from the site entirely (something that eventually happened anyway).

I've disabled audits here until such a time as this site reaches a scale to both need and support them.

$\endgroup$
  • 5
    $\begingroup$ There was only one person who prompted the change? Seems like using a sledgehammer to crack a nut - they should have dealt with that person individually rather than introducing a system-wide change affecting everyone to tackle the problem. Anyway, thanks for this! +1 and accepted. $\endgroup$ – Rand al'Thor Mar 14 '17 at 2:17
  • 7
    $\begingroup$ Yeah, this was supposed to be a very temporary "stop the bleeding" kinda thing; then we forgot about it. Beware of "temporary" measures... $\endgroup$ – Shog9 Mar 14 '17 at 2:21
  • $\begingroup$ Nothing is so permanent as a temporary measure. $\endgroup$ – Glorfindel Mar 20 '17 at 16:47
19
$\begingroup$

No.

There are at least a couple of reasons why review audits are, in my view, useless at best.

  1. Nobody's been banned due to them, so they're either not needed or not working.

    Emrakul said he doesn't think anyone has ever been review-banned due to failing too many audits. So either there are no reviewers sufficiently bad to deserve banning (in which case, there's no point in having audits at all) or there are bad reviewers but the audit system is failing to catch them (in which case, it isn't working as it should).

    The whole point of the review audit system is to catch bad reviewers and ban them without taking up mods' time with manual bans. If it isn't doing that, then there's no point in having it.

  2. Puzzling is small enough that many reviewers will recognise every post.

    Often when I come across a review audit, I recognise the post as one I've already seen and flagged/upvoted/whatever. Audits might work well on massive sites like SO where no one person sees more than a fraction of all the posts, but here on Puzzling I've probably seen, if not voted on, a majority of all the questions ever posted, and I certainly watch out for new ones as they come in. Sometimes I even recognise the supposed "first poster" as a high-rep user!

    And of course if you can recognise an audit as an audit, then the system isn't working properly. Audits should be indistinguishable from ordinary reviews, so that people will pass or fail them according to their normal reviewing behaviour, rather than pressing the right button just because they know they should.

The above arguments support "useless at best", which IMO is enough of a reason to get rid of it. But at worst, the audit system is actively bad:

  • What should I do with this audit ? Is it a bug? If you've already flagged a post when it first appeared (perhaps as a real review item), then you can't flag it again when you meet it as an audit, so you fail the audit. This relates to point 2 above: on SO, the chances of coming across the same post twice would be vanishingly small, but here it's quite likely.

  • There have also been cases when the review audit demonstrates a double standard in evaluating quality: a question which should have been closed as off-topic for being a maths problem, but wasn't because it was posted by a high-rep user with enough supporters to vote to reopen/leave open, came up as an audit and the right review action was actually 'wrong'.

$\endgroup$
  • 6
    $\begingroup$ I've emailed the community team about disabling audits on Puzzling. $\endgroup$ – user20 Dec 23 '16 at 4:01
  • 1
    $\begingroup$ @Emrakul Thanks! :-) I assume they'll post here on meta if/when they disable them? $\endgroup$ – Rand al'Thor Dec 26 '16 at 22:50
  • 1
    $\begingroup$ Removing review audits seems like the right decision, but I'd like to take issue with your point 1. Banning really bad reviewers isn't the only purpose of review audits, is it? It looks to me like the other purpose is to give people a nudge if they're reviewing badly, and thus improve the overall quality of reviews. That's helpful even if no one is bad enough to ban. $\endgroup$ – Gareth McCaughan♦ Mar 14 '17 at 12:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .