Okay, the sandbox didn't work. We still have a quality problem; let's figure out how to address it

Question

As of today, the riddle sandbox is no longer mandatory. The close reason related to it has been deactivated, and the requirement has been removed from the sandbox text. It was an experiment to see if it would be sustainable and effective going forward; for a variety of reasons many of you have pointed out, it didn't work. (We also screwed up at judging the amount of support an idea needs before it has consensus, and offering enough time for a thorough discussion of the merits of ideas.)

But that's fine; everything is reversible, and it gives us all a better sense of the capabilities of the system going forward.

The sandbox itself is probably still worth keeping around, though without the mandatory requirement, because as a tool it could still prove to be valuable to anyone who decides to use it. Even if it's never touched again, it does no harm just sitting there. Unless someone thinks of a good reason it shouldn't be there at all, this is probably what's going to happen.

---

However, this still leaves us with a quality problem. The idea of a sandbox requirement came about in response to a growing sense that riddle quality is dropping on the site, and that it's becoming more heavily populated with low quality content. (Number sequence and cipher puzzles were even tacked onto the proposal's discussion for similar problems.)

Most of what I said regarding quality in the riddle sandbox proposal I still stand by:

Riddle quality is dropping. I think most of us have seen it lately: there’s been a slow slide in effort and energy put into riddles, and it’s starting to seriously hurt the site. On Stack Exchange, our goal is to optimize for pearls, not sand, and right now, we’re very much not doing this. If we were, it would not only push the quality of the site up, but also drive us to advance the state of the art.

Nowhere else that I know of on the internet do people collaboratively come together to develop new puzzles - including riddles - and that’s not something we want to stop. However, we need to do something to sort out what makes a riddle high quality for this site, and set better quality standards.

So it’s time for us to set aside some energy and effort to sort this out, and start over with a better structure in place to support riddles.

This is the discussion that I want to see continue. I think most of us recognize that there is a problem, and Hugh Meyers even offered insight into why the proposal might have gotten the sort of initial support that it did, even the idea wasn't ultimately very good:

The massive support the proposal had in its first day shows the widespread recognition of the situation and a strong desire for some sort of a solution.

---

I don't want to belabor the point, rehashing stuff you've probably already read and read again. Instead, it's time for you all to drive the quality discussion on meta.

What we've been doing, bringing these proposals to meta, is partially intended to try and drive discussion. This is where these problems are solved, and there's definitely a solution out there; we just need to find it. (And yes, perhaps more carefully consider its implementation and subsequent effects before diving in head-first.)

So please, please propose and discuss ideas on meta. Don't let this issue stagnate; mods are three of hundreds of community members, and we're all going to get the site that we fight for together.

Answer 1

Totally agree with both of Rand's answers, however, I think we need to hone in a little on what the new close reason should be, so I thought I'd add my thoughts...

New close reason

See Rand's answer for reasoning/precedent, however, I believe the text should be a little different. The primary aim should be to catch/close low quality content, which can be redirected to a sandbox where it can potentially be improved, so I propose the following close reason:

Low Quality¹ - In it's current form, this puzzle does not meet Puzzling's [community quality standards]² and needs refinement. Please take the time to read the guidelines and improve your puzzle. If you would like assistance, you can post your puzzle to the [Puzzling Sandbox]³ to get guidance from the community.

^{1 - Could possibly be renamed to Needs Refinement or something equivalent, to make it slightly less inflammatory/accusatory - we don't want to scare off innocent newbies.
2 - Would link to the meta post described in point 3, below.
3 - Would link to the either a new general purpose sandbox for all puzzle types, or could possibly be reworded to say "...post your puzzle to the [Riddle Sandbox], [Cipher Sandbox] or [General Puzzle Sandbox]...", if we wanted things split up into chunks.}

Community Quality Guides

To keep the close criteria as objective and consistent as possible, we could create an faq tagged meta post, with individual answers per "major" tag (i.e. the big ones, that are currently problematic/controversial like riddle, mathematics, cipher, etc), plus a "general puzzle" catch-all, to help define exactly what constitutes the quality standard minimum, in as clear and unambiguous terms as possible.

We would obviously need individual discussion posts to gather consensus on criteria for each puzzle type above, but there's already some great content here on meta that we could draw from (eg. for ciphers, mathematics, etc). To give you an idea though, I imagine each "answer" would look something like the example below.

---

_{Indicative example of a Community Quality Guide, take with a grain of salt... the important stuff of this post is above.}

---

Riddle Quality Criteria

To meet minimum quality standards for Puzzling, a riddle must:

• Have a single "obviously correct" answer
• Be more than just a straight description of the solution's features
• Describe a common everyday object/concept (or be tagged with an appropriate secondary tag, such as trivia, movies, literature, etc to identify that specific domain knowledge is required)

Additionally, your riddle must meet at least three of the following criteria:

• Use well structured meter and rhyme
• Be concise and well written
• Use a creative/unique structure or presentation
• Employ letter/wordplay
• Make use of metaphor/polysemy/turns of phrase

Answer 2

This may not make a huge difference, but since new users are often high on the blame list for the poor-quality stuff, we should

change the Puzzling Tour to actually reflect this site.

I think this is probably a low-effort but some-reward step we could take, so why not do it. If you don't think it needs to be changed, go take the Puzzling Tour right now and try not to furrow your brow.

At least some of the recent poor quality questions (example, example, example) come from users who do have the Informed badge. That means at least there's one potential point of intervention to communicate something to new users. Right now, nothing Puzzling-specific is on that Tour, and in fact you could argue some of the stuff is Puzzling-detrimental.

And I think Emrakul indicated (as a comment on this post) that we do have some control over this. So...why not?

Answer 3

A chatroom for detecting and discussing low-quality puzzles.

Whether or not we implement a new custom close reason for 'bad' puzzles as proposed in my other answer here, we need people to be aware of it and of any new puzzles coming in that should be closed using it, otherwise there's no point. Emrakul mentioned in comments on another meta post that part of the reason why moderators have been closing too many questions unilaterally is because the community haven't been active enough in closing questions which should be closed.

I propose creating a dedicated chatroom for crap-catching. It would have a feed to post new questions into the room, and in it people would discuss the possible closure of particular questions. This would:

• be easier to keep a constant eye on than the Close Votes review queue, since people could hang out and chat idly in between discussing questions to be closed
• encourage active community discussion rather than just hitting the "Close" or "Leave Open" button, which would be helpful in shaping policies for the future
• draw people's attention to questions in need of closing quicker than the review queue does, since no initial VTC would be needed - all new questions would appear on the room feed.

For the first while, when not many people frequent the chatroom, we might find that more or less the same bunch of users are closing many of the questions, but as we attract more and more participants to the room and the project, that should change. Note that users with <3k rep would also be welcome to join in: even though they can't actually vote to close questions, they can still flag them for closure and take part in the discussions around them.

Of course we'd have to publicise this room as much as possible to make people aware of its existence. Ways to do this might include a featured meta post linking to it and superpings from any chat mods involved to bring active community members into the room.

Answer 4

A custom close reason for 'bad' riddles/puzzles.

Normally, the way to deal with 'bad' questions on Stack Exchange is simply to downvote them, while closing is reserved for questions which are off-topic, opinion-based, or otherwise inappropriate for the site. However, certain sites which have a consistent problem with floods of low-quality questions have developed a custom close reason to use for such questions, in order to get them shut down quickly before they can gather multiple answers and reach HNQs.

• Movies & TV is plagued with low-quality ID questions: the OP remembers some details about a film they once saw and asks if anyone can identify it. While ID questions as a whole are on-topic, the M&TV community have implemented a close reason which reads as follows:

  Identification questions must contain sufficient detail to meet the site's quality standards and should not be about a commercial or music video. For help writing a good identification question, see: Identify-This-X Questions.

  Thus, 'good' ID questions which include sufficient details about the film in question are fine, but 'bad' ones which say little more than "I saw a film when I was younger, it had an alien and a dog in it, please help me find it" can be quickly closed by the community.

• English Language & Usage is plagued with questions that are easily answerable by simply using a dictionary. In order to discourage such 'trivial' questions and focus on questions which require some expertise to answer, the ELU community have implemented the following close reason:

  Please include the research you've done, or consider if your question suits our English Language Learners site better. Questions that can be answered using commonly-available references are off-topic.

  Again, questions closed with this reason would be considered on-topic by network-wide standards: they're about English language and usage, they're objectively answerable, but they're just 'bad'. (For an example of the kind of question which would be shut down instantly today, see this historically locked post.)

We can do something like this at Puzzling. Technically, since our site scope covers puzzles posted as questions, we shouldn't close anything as off-topic if it's a puzzle? Not necessarily! If we get enough community consensus to do so, it will be possible to create a new custom close reason for 'bad' or low-quality riddles, provided we can define some sort of criteria for when it should be used.

My initial proposal for a criterion for a 'bad' puzzle would be that the intended solution fits the puzzle no better than one or more other possible solutions. This would cover "guess what I'm thinking" type puzzles, but I'm aware that there may be other categories of 'bad' puzzles which don't come under this header. Everyone, please do feel free to make amendments to this criterion, either by leaving comments here or by coming to discuss it in chat.

Before anyone points out that a couple of existing close reasons could be used for this: yes, I realise that puzzles of the type I'm talking about would also fall under "too broad", or under this existing close reason. The advantages of making a new one are:

• we can try to pinpoint as accurately as possible exactly what the problem is with the puzzles we consider 'bad', less generically than the existing close reasons
• we can make our close reasons more up-to-date, according to what the Puzzling site is today rather than what it was 2 years ago at the time of the meta post I linked above for the current close reason
• we can and should include a link to the sandbox in the close-reason text, so that people whose puzzles are closed for being 'bad' can go directly to a place where they can get helpful feedback, improve their puzzle, and eventually perhaps get it reopened.

Of course, we still need plenty of discussion before deciding exactly what such a close reason should say, and then we need time to get community consensus before actually implementing it. But in the long run, I think this may be the best solution.

Answer 5

I mentioned this in a comment but I think it ought to go in as an answer. If we have requirements and standards (as in the some of the suggestions for the tutorial and help center) they really ought to go in a prominent place on the "Ask Question" page right above where the Title box is. The yellow rectangle on the right is not good enough. That's where annoying banner advertising goes on most sites and my eyes automatically screen that out most of the time.

Here's what we say (in the yellow box):

How to Ask

Is your question about creation and solving of puzzles?

We prefer questions that can be answered, not just discussed.

Provide details. Share your research.

If your question is about this website, ask it on meta instead.

visit the help center » asking help »

Our top problem (I believe) is not that people are trying to start discussions, or that they are posting on topics other than the creation and solving of puzzles. We are concerned about puzzle quality. We should say so. If we have rules concerning the quality or nature of riddles, we should say so and we should say so here. This is the last stop before the submission goes onto the site. After it is posted then it needs to be read, discussed, voted on, and closed. Surely it is much better to do what we can to make sure that unacceptable content is not posted in the first place.

If the problem is over-eager newcomers posting unacceptable content, the help center and tutorial will reach only a small fraction of this segment. Our best chance is to put our message in big letters where they can't help but see it before they post.

RULES FOR RIDDLES

or

MINIMUM POSTING STANDARDS

followed by a short, readable list with links. In short we should do all we can to make the poster the crap filter rather than the already burdened mods and the community at large.

I there are a lot of other good suggestions here. My point is just that if the problem is a lot of crap on the site we should at a minimum be telling people in a loud voice not to post crap and exactly what we consider to be crap.

Answer 6

Text building blocks

How about making a common list of responses that can be used to improve posted riddles? Mostly the riddles aren't bad, just need some more work. Instead of closing them immediately or discouraging to post, let's help the users to improve the question posting a comment. The text building blocks will be there to help with it.

I'm thinking of a list of customisable comments similar to this one in TeX StackExchange.

This doesn't exclude the proposal in Hugh Meyers' answer and is even complementary to it, engaging the problem in similar positive manner.

Answer 7

I've kind of brought this up before, but speaking as someone who is a content creator in the puzzle world, I am actively disincentivized when it comes to posting here. I think it must be some larger site policy for stack exchange, but if I post something here it legally means I give up ownership of it. I'm trying to start a puzzle magazine but I can't even show you guys samples here because it actually hurts me to do so. Now I don't claim to be the best puzzler of all time, but I try to make unique games and hire artists to flesh everything out and bring them on par with newspaper puzzles so it's not just some chunk of text. In my opinion, this is the sort of stuff that should be making its way here to you guys. If you want to create an influx of decent to good quality stuff, you need to create an environment conducive to bringing in people who are trying to make a living off of puzzling.

Answer 8

Riddle Restrictions

This is just an idea and I'm not sure if it will be possible or will work, but its worth bringing up the idea.

Users on Stack Exchange can only use 40 votes in a day, after that it restricts you.

So can we do the same for riddles? Once a user has posted a question tagged riddle, they can't post another one for 24 hours.

This may require code changes to the site so may not be possible.

New or Edited Help Center Page

This will probably work better. The help center contains the question

• What types of questions should I avoid asking?

Maybe it would be a good idea to add there something about riddle and give a few guidelines.

Alternatively we could create a new help center page specifically for riddles. The question could be something along the lines of:

• Should I ask riddles?

• How do I ask riddles?

• How can I write a good riddle?

• Should I post riddles here?

• What sort of riddles can I post?

The page can give advice on riddles, this will hopefully raise riddle quality to an acceptable level

Answer 9

I had a crazy idea that, I hope, might inspire someone to propose a less crazy idea. I propose a "workshopping" tag (or perhaps a "sandbox" tag).

If a puzzle is tagged "workshopping", it means that it is a work in progress. People won't (or shouldn't) get mad if the question changes. I have seen users post questions that have problems and then get flayed for "totally changing" (i.e. attempting to fix) the question in response to criticism. To me, this feels much more friendly than closing or not allowing people to post unless certain criteria are met. I know that I have a few puzzles (I'm thinking of Alice In Cryptic in particular) that I would like to fix but I don't know how to do it without partially plagiarizing myself or changing a question with an accepted answer.

This tag would be automatically applied under appropriate conditions. Maybe it is applied to every question by default. Maybe it is applied by default if your reputation is below a certain level. I believe some sort of automatic application is important because a lot of problem content is posted by people who are not familiar with the way the site works, the available tags, and posting standards. My suggestion would be to let users remove it if they want to but to include it by default to get them to stop and think.

If people don't want to see "the chaff" then they can exclude that tag. Maybe it is excluded from common searches by default.

A question with the "workshopping" tag would be automatically deleted after some time period under the assumption that the poster must have given up on it. I believe there is a lot of "fire and forget" low-quality content so an automatic deletion process has a lot of appeal. I don't believe we can actually stop bad content from happening but perhaps we can make it ignorable and transient.

Maybe it would be nice if there was an option to reset votes to zero (and hence erase rep loss) when the tag comes off. (Probably not practical and I'm not even 100% sure it's a good idea.)

How is this different from the sandbox? Well, chiefly, it feels different to me. We are not saying "your content is not welcome on our site." We are giving people a chance to post their content in a safe way. You aren't trying to earn your way onto the main site. There is no "dual posting" with the problems associated with users seeing the question on meta before it goes to the main site. My hope is that when good questions are posted, the users will take the tag off fairly quickly and that bad questions will be abandoned and deleted. Along the way, maybe we can salvage some questions that had problems and alienate fewer novice puzzle makers.

There are a number of problems with this idea. We would somehow have to let new users know what this tag signifies. We would need to communicate to existing users the etiquette regarding "workshopping". I don't have anything clever to say about what happens when people answer "workshopping" questions. If you accept an answer, does that mean that the question is no longer in workshop mode? It is also a meta-tag and, as such, explicitly discouraged for good reason. I would say in defense that puzzling se is for the creation and solving of puzzles. This tag says that the poster is (for the moment anyway) interested in puzzle creation and is asking for feedback.

Again, I am mostly hoping that this might spark someone to have a better idea.

Answer 10

Any time I read "standards are dropping", I hear "in my day, we had to walk ten miles to school, uphill both ways...". If you've heard most classical riddles and become accustomed to riddler thinking, then it's very likely that you will feel the standards have dropped from when you were new, and the riddles seemed fresh, new and exciting.

Simply put, I don't believe your claims. You should not be doing anything at all (you should not even have made this post!) until you can prove that something needs to be done. Opinion is simply not sufficient grounds for the formation of new policies.

What statistics have been gathered to support the argument that riddle quality is dropping?

What metrics could be used?

As an example metric, has the average ratio of views to upvotes risen or fallen over time?

TL;DR: You seem to be tilting at a windmills. Show us the data.

Answer 11

My suggestion:

Allow a user to post only one riddle per fortnight^*, unless their previous riddle has been solved and received a vote score of at least 15^* (in which case they can post them earlier). If the rule is violated, place the new puzzle(s) on hold for the rest of the period.
(* Exact period and score values TBD.)

I think a major source of the problem is new enthusiastic users who get carried away.

1. This approach does not directly stop them from posting, nor does it directly describe their content as not being good enough for the site. I think it more objectively reflects the intentions of the community and hints at the problems that are being caused by casual puzzle creation.
2. The rule encourages questioners to prefer quality over quantity. A person with 3 half-baked ideas will now be inclined to stop and think about which their best idea is first, and try to make it better so that they have more of a chance to post their next one soon. In the process, they automatically start thinking about how to polish their act.
3. If they fail, they can get feedback in comments and/or opt for the sandbox before their next attempt. Alternatively, they can start thinking of posting a different type of puzzle.
4. A low-quality riddle can still be posted every fortnight, but the perception (hopefully) changes from "I'm not being given a chance" to "I need to can do better than this".
5. This can be applied to any dominating / mass-producible type of puzzle that seems to be overrunning the site at any time, and not necessarily just to riddles.

This doesn't require site changes or moderator involvement to work: community members can VTC errant puzzles and that too in a completely objective manner. The subjective feedback is left to the general PSE community and is not restricted to the Meta community or specific reviewers. Proactive community members can give a heads-up and advice to new users in the comments of their first puzzle as soon as it is posted.

I know this still leaves open holes for a few things like sock-puppets, but I feel that a large portion of the users causing this problem are really just honest and enthusiastic but casual puzzle creators - who are either really not good enough and will get restrained, or those that can do much better once they realize that they ought to.

Answer 12

Can't you form a committee of riddle experts on Puzzling StackExchange - maybe chosen 5-10-20 people, who review every riddle and vote/discuss whether it is good enough for the website? You may argue that the opinion of so few users may not be representative for the mass, but isn't that the main issue here - the mass is often voting up simple, lower quality puzzles.

The riddles which do not pass the selection process can be still kept in the sandbox (or other similar place), along with explanation what is wrong and what can be improved on them.