I'm not sure that the possibility of multiple equally likely answers is enough to declare a question definitively off-topic. There are sometimes answers that are very clearly correct -- in questions that do satisfy this, there's no reason to declare them off-topic a priori.
But... I do see some problems with them that make me believe: yes, we should declare IQ-test-style "find the next entry in this sequence" questions as off-topic, at least those from other sources.
Do new IQ-test-style questions add value?
Let's start with the obvious. I do think that they are consistently some of the lowest-quality questions we get, even when they are technically on-topic. In our site's chat, there are almost always groans whenever one of them appears in the question feed.
Not allowing future questions would significantly boost question quality overall.
But there's another issue that I believe we haven't considered:
Do old IQ-test-style questions add value?
Part of our site's general ethos -- the way I see it, at least, is to be an archive of both puzzles and questions about puzzles, in hopes that they will be helpful to other people.
The puzzle questions "help" in a different way than much of the SE network: puzzle questions serve more as examples for puzzle design and explanations of how solvers approach things, rather than direct answers to questions.
But I still believe that old puzzles like this can be useful for puzzle-crafters looking to learn how to design more effectively, or solvers looking to learn the ins and outs of specific puzzle types. I've recommended old grid-deduction and cryptic-crossword puzzles here to novice solvers before, and I've even used questions here to practice certain genres myself.
Plus, the site has been useful for people looking to understand the explanations to puzzles they found somewhere else. We have lots of old, well-known questions that get a lot of visitors looking to understand answers - particularly the detailed explanations of the Blue Eyes problem and the "64=65" rectangle dissection.
IQ test questions do not serve this purpose:
- They frequently cannot be answered with certainty, making them useless for solvers.
- The thought process required is almost always "guess what random
manipulations the author is thinking of", making them useless for
creators.
- They are practically impossible to search for, making them useless for people trying to find answers. (I have seen multiple instances of IQ-test-type questions that were definitely duplicates, but I had absolutely no luck finding the originals. Even if the source is given, you still have to search through every single post with that source, because there's no nice way to search for the images used.)
Because of this, these questions have no lasting value after they are answered.
Learning from other sites
So, I looked on other sites to see if there was any precedent for something like this. And it turns out there is: several sites based around fiction have had problems with "identify this book/movie/game" questions. These seem to me to be a very good parallel to our IQ test questions: they are common, often low-quality, and provide very little lasting value after they're answered.
So, how did other sites on the network deal with these?
Arqade has strict rules for game ID questions. In particular, they need an "artifact" - some piece of tangible media that would let them confirm, without any doubt, that the answer they give is correct. Otherwise, it is off-topic.
Anime & Manga has also had to implement tight restrictions on ID questions.
Movies & TV has banned ID questions entirely, after a lot of debate. This appears to be a controversial decision, but several of the answers to the original discussion post detail some of the reasons they have hurt the site -- in particular, Napoleon Wilson's answer goes into great detail on this harm, with some helpful quotes from SE employees.¹
Literature seems to be allowing them. Rand al'Thor explains that this is (in part) because book ID questions let the askers provide lots of details. He also notes that Sci-Fi & Fantasy allows ID questions, and they get a lot of comments from people looking for the same thing. In my memory, we have never had someone comment that they were looking for the same answer to an IQ test question; I did a brief search modifying Rand's search mentioned in that post, and it gives no results.
We cannot get 100% correct verification on whether reasoning for an IQ test question from another source is correct, so the 'escape hatch' Arqade allowed would be impossible for us. Because of this precedent and the lack of usefulness to later visitors, I believe that we should close all pattern questions from other sources as off-topic.
Footnotes:
[1] Some quotes from SE employees given in this answer:
Identification [IQ test] questions are like... The cabbage, radishes and shredded carrots that some sushi shops put on the platter before stacking up the meat & rice. They make the plate look nice and full, and they're something to chew on once you've eaten all the meat... But you still gotta have plenty of meat ...If this site gets a tiny handful of ID [IQ test] questions, just enough to fill in the gaps between more interesting and useful questions, they'll probably be fine. But if a majority of questions fall into this category, it becomes quite unlikely that the site will survive. And, let's be honest: if the site takes off, y'all will want to get rid of ID [IQ test] questions anyway simply because when there's plenty of meat they're a distraction, keeping them around doesn't help anyone, and the folks who ask them don't stick around to help anyone else.
-Shog9, lightly edited by me
The problem is they are easy to ask, but they ultimately help exactly one person, and then they're useless. It gets tiresome, and drives away avid users who drive this site. They will continue to pervade the question space. And then they wear down a community. ...It's really hard to articulate as to why these questions are not good for this site; but suffice it to say that, once they permeate the front page, a few users will work long and hard to try and get rid of them. But the masses will keep them coming, inexorably...In a not-too-distant future, most of those questions will be asked by hit-and-run users who will never return to this site. And you'll get bored having to tease out a decent question and provide answers to a post that will not add one lick of value or interest to this site.
-Robert Cartaino
