# Getting riddle quality back on track: a proposal for the path forward

Riddle quality is dropping. I think most of us have seen it lately: there’s been a slow slide in effort and energy put into riddles, and it’s starting to seriously hurt the site. On Stack Exchange, our goal is to optimize for pearls, not sand, and right now, we’re very much not doing this. If we were, it would not only push the quality of the site up, but also drive us to advance the state of the art.

Nowhere else that I know of on the internet do people collaboratively come together to develop new puzzles - including riddles - and that’s not something we want to stop. However, we need to do something to sort out what makes a riddle high quality for this site, and set better quality standards.

So it’s time for us to set aside some energy and effort to sort this out, and start over with a better structure in place to support riddles. Here’s what we’re proposing:

1. We set up a permanent riddle feedback sandbox. Anyone interested in posting a riddle posts it in the sandbox (optionally along with its solution), where it can be critiqued and voted upon by others.

The intent is to allowing people to receive feedback on structure and quality. This discussion is also intended to drive a broader discussion of what makes a riddle a good fit, and on-topic, for this site.

2. We set up a temporary moratorium on non-sandboxed riddles until we come to a consensus about handling quality standards going forward. During this time, unless a riddle received a score of +10 or more in the riddle sandbox, it would be closed with the following reason:

There is currently a temporary moratorium on posting riddles on Puzzling Stack Exchange. This will last until more complete quality standards are developed, and during that time, no riddles may be posted on the main site unless they are [sandboxed first](link to sandbox). For more information, see [this meta post](link to here).

This will both allow us to discuss which questions should be on the site, as well as figure out which types of riddles we actually miss seeing on the site. I strongly believe some types of riddles will pass unmissed, and these are the types we want to watch out for.

Right now, on Puzzling, we’re stuck in a mindset of “good until it can be shown to be low-quality.” This moratorium is intended to turn that on its head, and change it into something closer to “riddles must prove their value to the site.” The first approach is often not effective when trying to curate high-quality content.

This also isn’t intended to stop riddles from being posted on the main site long-term. Riddles are a legitimate category of puzzle, and there is high value in keeping them around. It is, however, intended to drive stronger restrictions about which riddles we’re okay with having here, and hopefully improve site quality overall.

My goal and hope for this process is that we will come through it with a clear understanding of what makes a high-quality riddle, and of which riddles are not appropriate for this site. That will allow us, the entire Puzzling community, to more effectively moderate the riddles that are posted, and truly start optimizing for pearls I know we can create.

• I think this has great potential for improving the quality of riddles on this site.
– GentlePurpleRain Mod
Aug 21 '16 at 17:25
• I think this would be a good start. However, it may have unintended consequences. So be prepared for that. In general, though, the riddle quality issue is a subset of the overall tug-of-war between what the site is meant to be and how it is used by a large number (majority?) of users who simply visit the site to post and answer puzzles of pretty much any quality. I personally think it will be a monumental task to steer the community back to focusing on puzzle quality over the simple desire to try and solve every puzzle posted. Aug 21 '16 at 19:46
• Anything. Please. Also, I suggest that giving a high profile to some positive guidelines on what a good riddle looks like, with links to examples of riddles which are good in different ways, would be helpful in showing examples of good practice to new users.
– A E
Aug 21 '16 at 21:49
• Several times within the last (almost) 2 years I've stated the following opinion and I'm still meaning it as nothing was done to prevent it ever: The site's rep system leads people to temptation too much. On other SE-sites this may be no problem since they're not opinion based as much as it is here. But PSE seriously suffers from 'rep-whorery' and this won't change with the soft guidance. So I strongly approve your scheme! Aug 22 '16 at 5:41
• Man, I was hoping for some general support, but I was not expecting +13/-0.
– user20
Aug 22 '16 at 6:55
• Also, what about ciphers? Those are arguably worse than the bad riddles. Aug 22 '16 at 9:31
• To get an idea how the sandbox would look and/or feel like you can check out the PCG sandbox and their FAQs. Aug 22 '16 at 12:59
• excellent idea. And I agree with @Mithrandir on ciphers. They usually are much worse than riddles and I would say to put even numerical sequences on sandbox as well.
– Sid
Aug 22 '16 at 14:01
• Is there firm support for adding cipher questions to this as well? If we're setting this up anyway, which it definitely looks like we're doing, it's real easy to add another category of question. (Or maybe adding cipher is a suggestion that could go in an answer here?)
– user20
Aug 22 '16 at 14:48
• For me it is baffling why so many simple riddles get so high rating on this website. For example, "I have 3 fingers" may be an interesting trivia, but I don't think it deserves 50+ upvotes. In my opinion average riddles are generally easy to make, but nevertheless they are the ones which get most upvotes for some reason. Aug 22 '16 at 14:53
• Nitpick: A +10 barrier seems too high - after some initial activity, I'd imagine that the sandbox would be voted on mostly by users with higher investment in the community (since checking on a single thread repeatedly is not typical SE behavior) and getting 10 such votes could be extremely rare. A lower threshold (like +5 or +3) and some minimum time duration (like 48 hours) might help. (Even just "your riddle must be in the sandbox for 48 hours and have non-negative score" might help - it lowers the incentive for writing lots of riddles quickly) Aug 22 '16 at 17:16
• Any time I read "standards are dropping", I hear "in my day, we had to walk ten miles to school, uphill both ways...". If you've heard most classical riddles and become accustomed to riddler thinking, then it's very likely that you will feel the standards have dropped from when you were new, and the riddles seemed fresh, new and exciting. What statistics have been gathered to support the argument that riddle quality is dropping? What metrics are used? Has the average ratio of views to upvotes risen or fallen over time? Aug 23 '16 at 4:28
• I strongly disagree with this idea. It restricts people when there are other measures that can be taken like close votes, downvotes, flags, deleting the post if something is very low quality. Aug 23 '16 at 12:12
• @Marius: We've tried for months to advocate for downvoting - it hasn't worked. Flags, deletion, and close votes are only for things that are actual moderation issues (not just low-quality posts).
– Deusovi Mod
Aug 23 '16 at 14:19
• @Marius These tools only work when there are criteria in place for how they should be used, and we have quite literally none of these. Part of that is because we've all been so reluctant to question a puzzle's quality unless it can be proven beyond a shadow of a doubt to be low-quality. This is designed to reverse this pattern so we can build these tools, and is only temporary until that's done.
– user20
Aug 23 '16 at 14:42

Requiring riddles to pass through a sandbox on meta before being permitted on the main site is going to encourage cliquishness and in-crowdery and alienate new users. Here are some of the biggest problems (in my view) with the new scheme:

1. It prevents new users from participating. Essentially, it does. If you're a new riddle enthusiast joining PSE for the first time and finding yourself blocked from posting riddles, what are you going to do? Earn enough rep to post on meta by posting about something you're not interested in or by editing other people's posts - or walk straight out again and take your talent to Reddit or other riddle sites? If I was a newbie, I certainly wouldn't bother jumping through unnecessary hoops on a site I didn't know just so that I could do them the favour of sharing my creativity.

2. It allows people to abuse the system for rep. Some of the riddles posted on meta include solutions; even for those that don't, any discussion in comments may give hints towards the solution. An unscrupulous person could read all of this without participating, wait for the riddle to be posted on main, and then immediately answer it without having to do any work. The 'honour system' is all very well in theory, but we all know there are people who'll do anything for rep.

3. The 'in crowd' will have an unfair advantage. Requiring a certain score in the sandbox in order to be released on the main site is a terrible idea. Again, in theory people should vote based on quality, but in practice new or unpopular people are going to find it much harder to get upvotes than, say, Alconja will. Ad hominem voting is a thing, however much we try to discourage it.

4. A determined clique can rig the system. Related to the last point: 5 or 6 users who all know each other will have a very good chance of getting their riddles through the sandbox no matter how crappy they are. People will have more motivation than ever to set up sockpuppet accounts - which is, of course, against SE rules, but sadly not all sockpuppets get caught.

5. It can be hard to judge a riddle until it's solved. Not everyone will want to include their riddle's solution in the sandbox (e.g. because of point 2 above), and sometimes people's opinions of a riddle change dramatically after they've seen the solution. E.g. this awesome puzzle which seemed silly until it was solved, or this now-deleted puzzle which seemed fine until a $-22$-score answer was accepted. Without solutions, there will be false positives and false negatives.

6. People will lose interest and momentum. A lot of the fun in solving lies in seeing a new puzzle posted and immediately getting down to work on it, eking out the solution, and posting it up as quickly as possible. All this excitement disappears with a sandbox: people will work out the solution and then forget it, or they'll post it 2 days later when they're not excited about it any more, or someone else will beat them to it when the riddle goes on the main site.

7. People may have to choose between participating in the sandbox and on the main site. If a new riddle gets posted on the sandbox with solution, do I want to look at it and vote on it to help it towards appearing on the main site, or do I want to wait and actually be able to solve it myself? Active users will have to choose between meta participation and main-site participation, which will have a net negative effect on both.

8. There aren't enough meta-active people. I may be wrong on this one, but I believe that on a relatively small SE like Puzzling, the meta community is fairly small. It'll be hard for any riddle to get as many as 7 upvotes in the sandbox, and those that do will all tend to get them from the same group of people. It amounts to handing over control of the tag to one small clique.

9. People are already leaving the site over this. Hugh Meyers, a very talented riddler, has opted to take his talent elsewhere. As he put it in a now-deleted post:

I do understand the reasons for this place, and I'm not saying it's entirely a bad idea. But I gave it a try and I find it hard to accept being made to sit in a sandbox. Maybe it's an age thing. :-) Anyway, I did enjoy my six months as a puzzle maker. No hard feelings. [...] So long, and thanks for the t-shirt.

I myself am seriously considering whether or not to continue here. I'm on this site for the riddles; I've always been primarily a riddler, despite also participating in other tags; and if riddle activity on this site is going to be so heavily monitored and channelled through a sandbox, it will be easier for me to post my riddles elsewhere.

Don't get me wrong: I appreciate there's a problem with riddle quality, but this is the wrong way to deal with it. We should simply start coming down harder on bad riddles, being quicker to close them, rather than doing what amounts to killing off the tag completely (or at least killing both the fun in it and the possibility of recruiting new blood to the site). I have some other suggestions as to how we can impose some quality control on riddles without destroying them completely, including based on my experience from other sites ... but this answer is long enough already, so I'll post them later.

• Speaking of voting rings; meta.askubuntu.com/questions/15317/exchange-votes-or-favours, this one was running for years and totally distorted an entire stack. Aug 28 '16 at 20:56
• @Richard Yep, seen that. And I have a very good guess as to who on this site would lose out the most if the same query Shog ran on AU was run here. Aug 28 '16 at 21:44
• [1]. No, it prevents new users from participating by posting riddles. For the past few months, we've had dozens and dozens of new users coming in, posting low-quality riddles, and still getting a ton of upvotes. Now, they can still post other puzzles, or even answer other ones to get rep so they can make a meta post. It's an extremely low bar, and it's only for one specific type of post. [3,4,8]. Riddles are already mostly getting upvotes. That's still true on meta, but to a slightly lesser extent. The sandbox filters out the worst, since more veteran users visit and vote those down.
– Deusovi Mod
Aug 28 '16 at 21:49
• ...Currently, I don't think we have to worry about the sandbox only letting specific people though - nearly everyone's getting through now, especially since we've lowered the threshold to +7 rather than +10. [2, 7] Most riddles are being posted without answers anyway. Only one has been posted pre-main with solution, and that was a misunderstanding on the part of the poster.
– Deusovi Mod
Aug 28 '16 at 21:52
• [5]. Sure. Our goal here is mostly to eliminate the ones we can clearly identify as bad, though - everything else seems to be passing through. We can deal with the "bad after solution is seen" ones once they get to main. The negative rate is so low that I don't think we have to worry about false negatives. [6] Haven't seen that happen so far. [9] Yeah, that is an issue. That's my biggest regret from the whole sandbox thing - Hugh Meyers was one of my favorite puzzlers, and I am deeply going to miss him.
– Deusovi Mod
Aug 28 '16 at 21:57
• Back when this was originally posted though, there were no objections. On this thread, we originally got a score of +25/-0 - completely unanimous, which we had never seen before. The objections started pouring in after it was implemented (exactly as promised, in this thread). In any case, it's temporary - we may lift the moratorium soon.
– Deusovi Mod
Aug 28 '16 at 21:59
• (Also, I'd love to hear your other suggestions. Many of these bad riddles cannot be closed, since they are not objectively off-topic, just bad.)
– Deusovi Mod
Aug 28 '16 at 22:02
• +1, you raise some good points. Though, mirroring @Deusovi's last comment, I'd like to hear your suggestions. When the sandbox proposal was first made, I +1'd it, because I felt that something had to be done. But after having seen the result I'm not sure it's the right something. Aug 28 '16 at 23:23
• Off topic: Though, re, Deusovi's "bad [puzzles] cannot be closed, since they are not objectively off-topic, just bad" comment, I think I secretly just wish there was a close reason for crappy content. It's kind of depressing seeing rubbish and knowing there's literally nothing you can do about it beyond giving it -1. Aug 28 '16 at 23:25
• @Deusovi I could respond to your comments here, but that might end up getting a bit too long for a comment chain. Is this the right chatroom for discussing these issues? Aug 28 '16 at 23:34
• Yep, sounds good to me!
– Deusovi Mod
Aug 28 '16 at 23:35
• @Alconja 1) See above comment to Deusovi - I notice you're already lurking in that chatroom :-) 2) Given enough community support, we can create an off-topic close reason for 'crappy riddle', which would (IMHO) be a far far better solution than this sandbox thing. Aug 28 '16 at 23:37
• Thank goodness someone finally posted an "anti" answer to this post. While I'm sure behind the origins of this idea are good intentions, there are obvious objections to it that should've been an obvious part of the conversation before any site-wide rules were adopted. Aug 29 '16 at 3:45
• @randal'thor I'm waiting to see your proposal on meta.SE for a "crappy riddle" close reason. It might reach +25/0 faster than this one did!
– GentlePurpleRain Mod
Aug 29 '16 at 4:26

I will now try to formulate a loose summary for the riddle tag since I'm usually committing crimes under its flag. I hope I can restore its reputation and atone for my sins:
(I will try to improve this summary now and then)

Reasons for writing a riddle:

• I enjoy the process of writing riddles
• I have a good idea what my riddles will be about
• I've already created the first parts easily
• I finished the riddle and even after 1 day it still sounds great to me

Reasons that should make you think twice before releasing a riddle:

• I already posted 2 other riddles today
• I can't wait to collect the reputation
• I will realease another one right after this one
• The more riddles I post the more reputation I will get
• I already expect the riddle to be closed as "too broad" but doesn't matter - why not give it a try

Other questions to consider:

• When was the last time I browsed through other riddles to see what others are doing?
• Can I really create that many good riddles in such short time?
• Why was one of my latest riddles not so well received?
• Did I change anything to improve quality?
• What tags should I use?
• Is the riddle really solvable as is?
• Did I create this riddle only because I saw a good riddle and wanted to do something similar?

The next point really is the most painful one and hard to follow. I want to encourage people to, at least, think about it for a moment:

Reasons to dismiss the current idea (and some spiritual guidance):

• I had an idea for the riddle and made everything work somehow
(Do you want to waste the good idea on a bad riddle?)
• It feels like I forced everything to work
(Save your good ideas! There will be a better time to make it work)
• I must finish this riddle now because ...
(Don't push yourself. There are many other riddles. We won't be bored. My best ideas came to my head out of nowhere while I was doing something else.)
• Many of the clues are very far-fetched
(Looking at the riddle after some time helps to estimate difficulty)
• A lot of specific knowledge or thorough research is needed to solve the riddle
(Do you want people to read your riddle or to look around other sites for hours? Maybe the clues are too hard to understand? Step back and think it over)
• I've created the riddle within 5 minutes and I'm too annoyed to finish it properly
(Again, don't waste the idea. Maybe some of your other ideas are easier to handle at the moment? You don't want people to eat something half-baked)

Items that have influence on the fun of solving:

• The clues are well designed and solvers can find them
• Solvers can confirm partial solutions along the way and don't have to collect all the data until everything falls to pieces eventually
• Focusing on a certain style/theme always makes a riddle appear more 'real' or 'authentic'
• Too many different techniques can be confusing. Solvers may have a hard time to find out what they have to look for in the first place
• Riddles based on one very small idea tend to be low quality over all. Even if the small idea was an awesome catch in the first place, don't just 'fill up' the remaining lines to wrap your idea in a low-quality riddle
• Try use rhyming. This is not a must and sometimes very hard. But have a try sometimes. It will make you deal with your riddle for quite some time and in the process will improve it
• Don't just line up clues and encrypt them somehow. Make sure to have a certain flow in the riddle and nice wording (as long the letter play allows for it)
• Be aware of you own language skills. Feel free to address another user who is, preferably, of any 'English-speaking descent'(you could compare this to reading someone elses code which can be a pain if not done correctly)
• One may disagree with one point or the other. Like art in general this is my opinion based on my own experience. Feel free to add any valid points. Aug 22 '16 at 6:44
• I would suggest adding the suggestion that people proofread their riddles for English spelling and grammar, or if their own English isn't great they ask a native speaker to do so (the 'sandbox' will help with this). Otherwise it's sometimes hard to tell what's a clue and what's just poor English.
– A E
Aug 22 '16 at 8:13
• (Not a problem with your own riddles Avi, but with some other people's).
– A E
Aug 22 '16 at 8:16
• @AE I'm glad my English creates only the intended amount of confusion! Aug 22 '16 at 8:24
• You might also want to look at the tag wiki for the riddle tag. It has a lot of good suggestions.
– GentlePurpleRain Mod
Aug 22 '16 at 14:31
• There's a lot of interesting thoughts in this post, but I can't tell what your eventual conclusion is. Do you approve of the mandatory sandbox or not? Or is this just a bunch of random ramblings? :-) Aug 30 '16 at 14:37
• @randal'thor I think Emrakul is talking a lot about riddles and their quality in his post and so I felt this would be the appropriate place to tell everyone how it's done ;) I can see your answer is kinda really referring to the actual matter: the sand box Aug 30 '16 at 14:40

Let's include ciphers.

Bad ciphers have long been the bane of this community. Therefore, I propose that ciphers also be sandboxed first.

• Does this include every question that uses the "cipher" tag? What if the puzzle employs a cipher as a small part of the overall puzzle? Riddles generally stand alone. Ciphers are trickier, because they are often part of a larger puzzle.
– GentlePurpleRain Mod
Aug 22 '16 at 16:56
• @GentlePurpleRain This was meant basically only for stand-alone ciphers. Aug 22 '16 at 19:48

I really like this idea, but I'm unclear what happens when a riddle clears the sandbox:

• Does it automatically get posted or is a manual repost by the OP required?

• Do any votes from the sandbox get transferred?

• What's to stop people who are more interested in gaining reputation than in the quality of the site from watching the sandbox items and then pouncing on the riddles when they appear on the full site with the correct answer?

• Will the search engine check the sandbox as well as the full site?

• My answers, as far as I can figure it out (not having done this before): (1) Manual repost. (2) No. (3) Nothing, unfortunately. I would hope that people who have reviewed the riddle would have the decency not to immediately answer when it appears on main. (4) If you mean the Puzzling search box, the main site search won't search the sandbox. The meta site search will. Google will search both, I'm sure.
– GentlePurpleRain Mod
Aug 22 '16 at 12:10
• That term for people who love rep was retired... See this. Aug 22 '16 at 12:57
• @Mithrandir I've rephrased the answer. Apologies for any offence caused. Aug 22 '16 at 13:02
• Regarding (3), we'd probably ask that people who discovered the answer while the riddle was sandboxed refrain from answering on the main site, and base it on an honor system. But, ultimately, if someone really wants to, there's very little to stop them.
– user20
Aug 22 '16 at 14:40
• For (4), there was a close reason. Now, because the Sandbox is just a toy, there isn't any close reason for it anymore. Sep 6 '16 at 8:57

I'm wary of going too far with an idea that we haven't seen work in practice yet, but it seems like if there's this much support we could safely add another couple of our consistently lower quality categories, to see how it works for different types of puzzle.

Should we add this to the sandboxing moratorium?

• Looks like Mithrandir beat me to suggesting ciphers ;) Edited that one out.
– user20
Aug 22 '16 at 15:20

I think this is a great idea, and agree with everything you've said so far. However there are a few issues I'd like to bring up:

1. Should everyone be allowed to vote for riddles in this sandbox? New users may vote for riddles which shouldn't be voted for as they don't have enough experience with riddles to know which are good and which are bad. So I propose that maybe only users with 2000/3000 rep can vote.

2. Alternatively, we could allow these riddles to kept on being posted but have a new Review in the Review list for anything tagged as a riddle. That way we can keep an eye on riddles without having to make drastic changes and slow the question asking on this site which I'd say is currently at a good pace.

On a side note for the review list, maybe a riddle needs 3 users to review before it disappears off the list, and 2 users to either recommend deletion for it to be deleted or 2 users to say it looks Ok for it to be cleared as so

• Note that the sandbox would be on meta, which is not as frequented by new users. That probably addresses your first point. Your second point is really an entirely different proposal, and is not something that could be enacted by the Puzzling mods, since it would require code changes to the website (to enact the "review" process you describe, and to change the numbers for deletion, etc.).
– GentlePurpleRain Mod
Aug 22 '16 at 11:44
• Note that 2k/3k rep is a really high boundary!! Sep 6 '16 at 8:58