This main meta post tells us that the only Stack Exchange sites with a review audit system are:

  • Stack Overflow
  • Server Fault
  • Super User
  • Programmers
  • Ask Ubuntu
  • Mathematics
  • Puzzling

These are six of the largest and most active sites on the network, plus this one small site only just out of beta (which already had review audits even while still in beta). Introducing review audits on a small site is a remarkable decision, and apparently unprecedented on Stack Exchange. And yet, in searching our meta I couldn't find any discussion of this decision: not only no community consensus in favour of it, but not even an announcement of its implementation. This begs the question:

why do we have review audits on Puzzling?

(I can't really see the point of them, but I'd be interested to hear what the rationale was behind introducing them. Once we know the reasoning, we can then start to debate whether it's still relevant at this stage or whether review audits should be abolished here.)

| |
  • 2
    $\begingroup$ Huh, weird! I didn't realize it wasn't network-wide. $\endgroup$ – Deusovi Aug 31 '16 at 13:52
  • 5
    $\begingroup$ I just realized that this is actually quite puzzling. $\endgroup$ – Mithical Aug 31 '16 at 16:17
  • 5
    $\begingroup$ @Mithrandir Ba dum tish. groan $\endgroup$ – Rand al'Thor Aug 31 '16 at 16:18

Interesting question, actually. We had enabled audits as an experiment waaaaay back in beta in response to a long, long string of terrible reviews and a handful of review bans. It was a test that... really neither failed nor succeeded. It's hard to say.

We asked for them without prior discussion on meta primarily because it was a pretty inconsequential, low-overhead change that largely was supposed to inform the way we moderate. If they flat-out weren't working, actively harming review, etc. people would come to meta and ask about it, and we'd disable it again, no harm done.

It's been two years with review audits, and I can count the number of times people have been annoyed on two fingers. (In binary.) Data also suggests people aren't having a problem with audits, at over a 90% pass rate. There's definitely something to the "false failures are annoying" line of thought, though.

I'm reluctant to disable it because it might be doing some unmeasured good, but if you guys want it disabled, I'm not opposed. It just comes down to whether it annoys, and how much.

| |
  • $\begingroup$ "Data also suggests people aren't having a problem with audits" ... has the good Lt. Cmdr. been transferred from the Enterprise to Stack Exchange? $\endgroup$ – Rand al'Thor Dec 13 '16 at 11:51

The test (known as a review audit) is designed to help new reviewers hone their moderation skills, while nudging more experienced users that don't seem to be paying close attention to what they're reviewing. Some people were in the habit of up-voting everything they reviewed without regard to quality; extremely low quality posts with a high score are very problematic in a system where the best information naturally rises to the top, so "audit" tasks are mixed in to make sure folks are paying attention.

So apparently we were having a lot of trouble with robo-reviewers, even in beta. I wasn't around then though, so...

See this comment on Meta.SE.

| |
  • 1
    $\begingroup$ Hmm, I never noticed any robo-reviewers. And I strongly doubt there were significantly more robo-reviewers here on Puzzling than on any other SE site except the Big Six which have audits. Still seems strange to me. Maybe Emrakul or someone will chip in with some more info. $\endgroup$ – Rand al'Thor Aug 31 '16 at 14:46
  • 1
    $\begingroup$ Hm, that seems like a very generic statement. I don't think it's meant specifically for Puzzling. $\endgroup$ – Deusovi Aug 31 '16 at 14:47
  • $\begingroup$ @Deusovi But i still think it's relevant. $\endgroup$ – Mithical Aug 31 '16 at 14:48
  • 1
    $\begingroup$ Also, even accepting that there were too many robo-reviewers here in the past, is this still a problem? I suspect not, and will probably post a FR asking if review audits can be removed here. $\endgroup$ – Rand al'Thor Aug 31 '16 at 14:48
  • $\begingroup$ @randal'thor Why should they be removed? They're not doing any harm; they might even do some good by catching any robo-reviewers. (Also, does anyone know why that question about sheep is in the FAQ? $\endgroup$ – Mithical Aug 31 '16 at 14:49
  • 2
    $\begingroup$ @Mithrandir: Honestly, they're kind of annoying. And if you click "close" (say, to see why other people are voting to close) then you suddenly fail. $\endgroup$ – Deusovi Aug 31 '16 at 14:51
  • $\begingroup$ @Deusovi I seem to recall reading that you shouldn't do that; you should VTC based on the post, not on what other people are doing. $\endgroup$ – Mithical Aug 31 '16 at 14:52
  • 3
    $\begingroup$ @Mithrandir Deusovi isn't saying he'd actually VTC the post just because other people have, but it can still be interesting to see why other people have VTCed, and the only way to do that is by clicking the Close button and looking at the blue numbers. $\endgroup$ – Rand al'Thor Aug 31 '16 at 14:53
  • $\begingroup$ @rand I refer you to yourself: meta.puzzling.stackexchange.com/questions/5317/… $\endgroup$ – Mithical Aug 31 '16 at 14:54
  • $\begingroup$ @Mithrandir: I wouldn't close just based on other people. But if, say, they noticed a duplicate I didn't, or there was some other thing that I didn't notice, then I'd like to know. $\endgroup$ – Deusovi Aug 31 '16 at 14:54
  • $\begingroup$ @Mithrandir Sure, and what I said in that comment makes review audits kinda useless as well as annoying. It's also bad to auto-ban people from the review queues just for not agreeing with everyone else - not all highly-voted open questions should be open, and not all closed questions should be closed. (Example.) Also, this comment thread is getting long - shall we take it to chat? $\endgroup$ – Rand al'Thor Aug 31 '16 at 14:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .