Since very early in Puzzling's development, we've adopted a very strong principle that's become virtually universal: "if there is a hint of a notion that a puzzle might be good, we have to act as though it is." This is also not a principle that I think has been explicitly or clearly discussed. (It's also not universally true - but functionally speaking, it's pretty dang close. The only consistent exception I've seen is when someone posts a really bad code puzzle.)

However, it most certainly has come up, and often. We very frequently vote this way - but beyond that, it very frequently stops us from spotting and potentially handling low quality content. This goes further, though: it directly blends with another critical notion: "you can't know the quality of a question until it is answered."

But here's the problem: If you can't know the quality of a question until it's answered, then we always have to act as though there's a chance the question might be good. But if we assume there's a hint that the question is good, we have to act as though it is. So in all cases, we have to act as though the question is good... unless it's obviously not, at which point we downvote (or more frequently, don't vote at all), and feel super uncomfortable about our dissonant selves.

This deadlock leads to a voting culture where bad questions are frequently upvoted because they might be good, rather than because they are; it leads to a set of close reasons that must be applied retroactively; it leads to a system where we are optimizing for sand, not pearls. We're stuck trying to resolve the difference between two incompatible ideals: the principle above, and site quality.

The tools we're given on Stack Exchange work best when they're applied before a question is answered. There's a window of opportunity between when a question is posted and when it starts to generate responses, and past that window, what we do has extremely diminishing returns.

I've been pondering this issue very frequently recently, because it seems that at every turn, when anyone proposes something to improve site quality on Puzzling, we run facefirst into this deadlock.

I want to open a discussion on the way we talk about site standards, because I think the way we're doing it right now is ultimately self-defeating. The reality is, we have a lot of information on what makes a good question. We're graduated - and besides, take a look at our active users page.

So let's be honest: often, we really can read a question, and know if it's a good one, without knowing the answer. We should, at this point, be able to say "sure, this question might be good, but nearly all similar cases I remember have been bad." I want these cases to feel actionable, either through votes, or close votes. And in my honest opinion, that can't happen without changing the way we think and speak about the quality of puzzles we see here.

I'm already rambling on too long, but this is pretty important. I'd love for this to be open to discussion - including, and especially, disagreement - because at minimum, I think the discussion will be helpful going forward.

  • 3
    $\begingroup$ Extremely well written, as usual. Though I'm temped to close as "unclear what you're asking"... $\endgroup$ – Deusovi Aug 3 '16 at 4:43
  • $\begingroup$ I'm new to both Puzzling and Stack Exchange as a whole. Is there some sort of mechanism that would let us set aside the work from users who are proven to produce quality content? A mini-forum where we knew the quality was good, that sort of thing. Would we want to do this even if we could? $\endgroup$ – Topple Aug 3 '16 at 4:43
  • $\begingroup$ (Moved conversation with @Topple to chat.) $\endgroup$ – Aza Aug 3 '16 at 5:01
  • 2
    $\begingroup$ I remember that in past (rarely), a question would be quickly downvoted/closed, and the author would post a comment saying something along the lines of, "I know this looks like a bad question, but wait till you see the answer!" In some cases the question was restored, and in some cases it turned out to be true. But I agree that the burden of proof should be on the puzzle author, instead of requiring the community to "prove" that a puzzle is bad. $\endgroup$ – GentlePurpleRain Aug 3 '16 at 13:44
  • 2
    $\begingroup$ I've been reluctant to use any moderation tools here because I'm not sure what the community is looking for - it strikes me that there's some agreement on meta about the fact that the voting on the main site doesn't necessarily line up with what we think puzzle quality should be, but I've yet to see any clear articulation of our quality standards. Thus, I end up thinking, "Well, maybe this puzzle is just not directed to me - I'll leave it be for others" on basically anything that I might otherwise close/downvote. This behavior might be related to the post - it causes the same symptoms. $\endgroup$ – Milo Brandt Aug 8 '16 at 5:37

I think there's another part to answering this that you haven't mentioned. Let's call it how "Seasoned" any given user is, meaning how long they've been around the site, how frequently they check in, how much they understand the differences between this site and other SE sites, how much Puzzling Meta they've read, how many riddles have eventually outraged them, etc.

My guess—and maybe the mods actually know—is that most of the upvoting Emrakul is referring to is NOT done by Well-Seasoned users. It's probably done by relatively Unseasoned users who aren't thinking (or don't care) about overall site quality.

As I've become more Seasoned, I've realized that downvoting is an important way I can contribute to making this a better site. The numbers @LordOfDark put together during the mod elections were initially a little shocking, but now I actually feel good about the fact that two of our mods—Emrakul and Deusovi—have collectively cast almost 3000 downvotes. They care enough to try to push down the bad stuff which helps the good stuff stand out. And Meta posts like this and this have helped convince me that downvoting is important. Continuing to promote downvoting among Seasoned users seems like a good strategy.

Prescription 1

Encourage Seasoned users to downvote more frequently

But at the same time there are many Unseasoned users, and this number (and percentage) may grow as the site grows. They haven't yet seen the Meta posts, they haven't realized that a lot of stuff is low-effort and poor-quality, and they're reluctant to downvote because that seems mean.

We need to: discourage Unseasoned users from (1) asking low-quality questions AND/OR (2) upvoting low-quality questions. But we can't count on every new user seasoning him/herself before voting. So what actually can we do about this? I see a few practical possibilities.

Prescription 2

Modify the tour to include Puzzling's real differences

I think the tour is one point where we might be able to better inform Unseasoned users about what's going on here. Almost 2000 Informed badges have been awarded, but all those users got was a SE tour, not a real Puzzling tour. If we could include Puzzling-specific warnings/admonitions here, it could make a difference. For example, making this strategy of using Favorites to follow a question while postponing your vote an explicit suggestion could help encourage new users to refrain from upvoting immediately.

Prescription 3

Include site-specific instructions to users posting their first questions

Like "Since you're asking your first question, you may be interested in these three Meta posts about what makes a good contribution to Puzzling SE."

Prescriptions 2 & 3 require (a bit of) site-specific code, and I know that may not be an option, though it seems to me that SE management does care about site quality, so perhaps they could be convinced it's worthwhile.


Prescription 4

Try to facilitate the Seasoning of new users whenever possible

I.e., downvote, but explain. Point them to relevant Meta posts. Encouarge them to look at highly-rated puzzles. Shake salt on them. These are things we Seasoned users can and should do and which will potentially lead to a better-quality site.

And finally—and again this requires site-specific code:

Prescription 5

Get the formula for Hot Network Questions changed for our site

Because the number of answers to a question is an important factor in what makes it to HNQs, and because low-quality puzzles often have many possible and posted answers, we end up with a lot of low-quality puzzles in the HNQ, and many high-quality puzzles (which may not have an answer for days) miss out entirely. Since the HNQ certainly drive traffic to (and introduce users to) the site, it's a problem that their first impression is often our poorest content.

Since we're a "different kind of SE site" perhaps we can make a convincing case that we need a different kind of HNQ algorithm. And if the first entries new users see on Puzzling are really the best we have to offer, it's likely they'll hold themselves to a higher standard.

Overall, I don't think this will be solved/improved unless we can find a way to influence the Unseasoned users somehow. Those of us who are a bit more Seasonsed are already on board, but our votes count as much as the votes of a person who's on the site for the first time on any given day.

  • $\begingroup$ Huh. There's a lot in here I haven't thought about. It seems like, of these solutions, the first and fourth (asking for comments) seems like the most liable to have a serious impact with the least work involved. We have some edit control over the tour as well, which is easy, but possibly not as impactful, depending. $\endgroup$ – Aza Aug 4 '16 at 7:15
  • 1
    $\begingroup$ @Emrakul Yeah, I don't think changing the tour will solve anything by itself, but it might make some % of new users think twice or consider things more carefully, and every little bit will help! $\endgroup$ – Dan Russell Aug 4 '16 at 13:55

I wanted to bring up one suggestion which is connected:

The tools we're given on Stack Exchange work best when they're applied before a question is answered.

Yes, and that is a problem for the reasons you've stated. However, I was wondering if we could do a regular "re-visit" tour of puzzles that have an accepted answer.

How could this work? I'm not sure, but maybe along the lines:

  • In regular intervals - maybe announced on meta - completed puzzles (i.e. questions with an accepted answer) of a certain time-period are put on "display" for reevaluation.
  • A mod (?) 'tags' those questions with an (intermediate) tag, f.e. such that - due to the edit - people can change their votes again. Ideally, we would get these into a "review queue", but I'm affraid the current system does not easily allow for that, does it? But we can filter the (temporarily) meta-tagged questions.
  • After some re-visit period is over, the tag is removed again. (Potentially replaced by a .)

There might be better ways to do this (suggestions?) but essentially we want to introduce a non-typical review process.

I should add, that I'm generally very opposed to meta-tags, but that I do think they are a last-option-out if some StackExchange mechanism can not be otherwise bent into service. i.e. If we need a feature and can't get it, mimicing it by meta-tags is ok for me.

(In a similar manner as I still very much like this idea.)

  • $\begingroup$ I like this in spirit, but it worries me, because I think it might turn into a lynch-mob-esque area where people would only post questions they think should have been downvoted, so that everyone can dogpile downvote them. $\endgroup$ – Aza Aug 9 '16 at 15:59
  • $\begingroup$ Hm, I didn't think that there should be a pre-selection. All ("complete") puzzles of a timer period should receive the tag and the re-evaluation. $\endgroup$ – BmyGuest Aug 9 '16 at 16:08
  • $\begingroup$ Oh! Oh, jeez, that's a lot of questions. Maybe a review queue would be better for that sort of thing, but it'd take a lot of people reviewing... hm. $\endgroup$ – Aza Aug 9 '16 at 16:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .