The benefit of the doubt: should we shift the way we talk about puzzle quality?

Question

Since very early in Puzzling's development, we've adopted a very strong principle that's become virtually universal: "if there is a hint of a notion that a puzzle might be good, we have to act as though it is." This is also not a principle that I think has been explicitly or clearly discussed. (It's also not universally true - but functionally speaking, it's pretty dang close. The only consistent exception I've seen is when someone posts a really bad code puzzle.)

However, it most certainly has come up, and often. We very frequently vote this way - but beyond that, it very frequently stops us from spotting and potentially handling low quality content. This goes further, though: it directly blends with another critical notion: "you can't know the quality of a question until it is answered."

But here's the problem: If you can't know the quality of a question until it's answered, then we always have to act as though there's a chance the question might be good. But if we assume there's a hint that the question is good, we have to act as though it is. So in all cases, we have to act as though the question is good... unless it's obviously not, at which point we downvote (or more frequently, don't vote at all), and feel super uncomfortable about our dissonant selves.

This deadlock leads to a voting culture where bad questions are frequently upvoted because they might be good, rather than because they are; it leads to a set of close reasons that must be applied retroactively; it leads to a system where we are optimizing for sand, not pearls. We're stuck trying to resolve the difference between two incompatible ideals: the principle above, and site quality.

The tools we're given on Stack Exchange work best when they're applied before a question is answered. There's a window of opportunity between when a question is posted and when it starts to generate responses, and past that window, what we do has extremely diminishing returns.

I've been pondering this issue very frequently recently, because it seems that at every turn, when anyone proposes something to improve site quality on Puzzling, we run facefirst into this deadlock.

---

I want to open a discussion on the way we talk about site standards, because I think the way we're doing it right now is ultimately self-defeating. The reality is, we have a lot of information on what makes a good question. We're graduated - and besides, take a look at our active users page.

So let's be honest: often, we really can read a question, and know if it's a good one, without knowing the answer. We should, at this point, be able to say "sure, this question might be good, but nearly all similar cases I remember have been bad." I want these cases to feel actionable, either through votes, or close votes. And in my honest opinion, that can't happen without changing the way we think and speak about the quality of puzzles we see here.

I'm already rambling on too long, but this is pretty important. I'd love for this to be open to discussion - including, and especially, disagreement - because at minimum, I think the discussion will be helpful going forward.

Answer 1

I think there's another part to answering this that you haven't mentioned. Let's call it how "Seasoned" any given user is, meaning how long they've been around the site, how frequently they check in, how much they understand the differences between this site and other SE sites, how much Puzzling Meta they've read, how many riddles have eventually outraged them, etc.

My guess—and maybe the mods actually know—is that most of the upvoting Emrakul is referring to is NOT done by Well-Seasoned users. It's probably done by relatively Unseasoned users who aren't thinking (or don't care) about overall site quality.

As I've become more Seasoned, I've realized that downvoting is an important way I can contribute to making this a better site. The numbers @LordOfDark put together during the mod elections were initially a little shocking, but now I actually feel good about the fact that two of our mods—Emrakul and Deusovi—have collectively cast almost 3000 downvotes. They care enough to try to push down the bad stuff which helps the good stuff stand out. And Meta posts like this and this have helped convince me that downvoting is important. Continuing to promote downvoting among Seasoned users seems like a good strategy.

Prescription 1

Encourage Seasoned users to downvote more frequently

---

But at the same time there are many Unseasoned users, and this number (and percentage) may grow as the site grows. They haven't yet seen the Meta posts, they haven't realized that a lot of stuff is low-effort and poor-quality, and they're reluctant to downvote because that seems mean.

We need to: discourage Unseasoned users from (1) asking low-quality questions AND/OR (2) upvoting low-quality questions. But we can't count on every new user seasoning him/herself before voting. So what actually can we do about this? I see a few practical possibilities.

Prescription 2

Modify the tour to include Puzzling's real differences

I think the tour is one point where we might be able to better inform Unseasoned users about what's going on here. Almost 2000 Informed badges have been awarded, but all those users got was a SE tour, not a real Puzzling tour. If we could include Puzzling-specific warnings/admonitions here, it could make a difference. For example, making this strategy of using Favorites to follow a question while postponing your vote an explicit suggestion could help encourage new users to refrain from upvoting immediately.

Prescription 3

Include site-specific instructions to users posting their first questions

Like "Since you're asking your first question, you may be interested in these three Meta posts about what makes a good contribution to Puzzling SE."

Prescriptions 2 & 3 require (a bit of) site-specific code, and I know that may not be an option, though it seems to me that SE management does care about site quality, so perhaps they could be convinced it's worthwhile.

Still:

Prescription 4

Try to facilitate the Seasoning of new users whenever possible

I.e., downvote, but explain. Point them to relevant Meta posts. Encouarge them to look at highly-rated puzzles. Shake salt on them. These are things we Seasoned users can and should do and which will potentially lead to a better-quality site.

And finally—and again this requires site-specific code:

Prescription 5

Get the formula for Hot Network Questions changed for our site

Because the number of answers to a question is an important factor in what makes it to HNQs, and because low-quality puzzles often have many possible and posted answers, we end up with a lot of low-quality puzzles in the HNQ, and many high-quality puzzles (which may not have an answer for days) miss out entirely. Since the HNQ certainly drive traffic to (and introduce users to) the site, it's a problem that their first impression is often our poorest content.

Since we're a "different kind of SE site" perhaps we can make a convincing case that we need a different kind of HNQ algorithm. And if the first entries new users see on Puzzling are really the best we have to offer, it's likely they'll hold themselves to a higher standard.

---

Overall, I don't think this will be solved/improved unless we can find a way to influence the Unseasoned users somehow. Those of us who are a bit more Seasonsed are already on board, but our votes count as much as the votes of a person who's on the site for the first time on any given day.

Answer 2

I wanted to bring up one suggestion which is connected:

The tools we're given on Stack Exchange work best when they're applied before a question is answered.

Yes, and that is a problem for the reasons you've stated. However, I was wondering if we could do a regular "re-visit" tour of puzzles that have an accepted answer.

How could this work? I'm not sure, but maybe along the lines:

• In regular intervals - maybe announced on meta - completed puzzles (i.e. questions with an accepted answer) of a certain time-period are put on "display" for reevaluation.
• A mod (?) 'tags' those questions with an (intermediate) tag, f.e. reevaluate-request such that - due to the edit - people can change their votes again. Ideally, we would get these into a "review queue", but I'm affraid the current system does not easily allow for that, does it? But we can filter the (temporarily) meta-tagged questions.
• After some re-visit period is over, the reevaluate-request tag is removed again. (Potentially replaced by a reevaluated.)

There might be better ways to do this (suggestions?) but essentially we want to introduce a non-typical review process.

---

I should add, that I'm generally very opposed to meta-tags, but that I do think they are a last-option-out if some StackExchange mechanism can not be otherwise bent into service. i.e. If we need a feature and can't get it, mimicing it by meta-tags is ok for me.

_{(In a similar manner as I still very much like this idea.)}