6
$\begingroup$

I pose the question of how can we objectively determine how well a beta site is doing, in particular Puzzling.SE? I genuinely want this site to be a good one, but I have yet to be convinced that the site is of poor quality, despite many people saying so, but perhaps if we can objectively define quality, and beta success, we will all have a common framework we can use to assess how well the site is performing.

A beta site can be the most popular site in the stack exchange network and still be considered a failure because site popularity in no way reflects how healthy a beta site is. Currently there are users who claim this site is of poor quality (and providing examples), and other users who claim the site is producing questions and answers of great quality (also providing examples).

Quality seems to be a very subjective property of a beta site, and I wonder if we can flesh out how to objectively assess the quality of a site and assess how "successful" a beta site is, rather than by anecdotally picking out examples of high and low quality questions.

$\endgroup$
2
  • 1
    $\begingroup$ +1 this is very important $\endgroup$
    – d'alar'cop
    Commented Nov 20, 2014 at 2:55
  • 1
    $\begingroup$ This is important. I feel like there are a lot of high-quality questions and answers; at the same time, there are probably just as many low-quality questions, if not more. I don't know how we can examine this objectively, though. $\endgroup$
    – mdc32
    Commented Nov 20, 2014 at 2:59

3 Answers 3

Reset to default
4
$\begingroup$

Nobody knows what a site needs to do to graduate or: define "how well a beta site is doing"

I've been a member at Programming Puzzles and Code Golf (1392 days in beta and counting...) for some time now, and nobody there (mods included) has any clue if/when we will ever graduate. We had a Community Manager post back in March about the site's progress, only to find out there was a divide on what "progress" even meant.

If we don't know the goal, we don't know what to measure

There's nothing spelled out in official terms anywhere that I've ever seen, yet the common refrain is "stats aren't everything". Which is fine, but without that other key piece of information, we have no way of knowing what to measure. This is of course assuming that graduation is the goal (which seems like a reasonable goal for anything labelled 'beta').

Let's just try to have a mission or purpose, the rest will follow

While I run the risk of over-representing PPCG here, I'm using them as an example because I'm active there and they are an "atypical" SE beta site, much like this one in some ways. They asked a question early on that I think is key:

Where does Programming Puzzles & Code Golf fit in the SE mission?

The overwhelming answer is that while it may seem like it's "just fun", you can actually learn something. Personally, I've learned more about the intricacies of my usual languages than I would elsewhere. I've tried new languages, learned optimization techniques, the list goes on.

I feel I can learn something here, too, but I don't really think that finding the pun in the third line of a riddle helps much with that. I've seen plenty of teachable questions, but plenty of nothing-but-fun ones, too. I'm not getting into specific examples (others already have), but if having a repository of useful knowledge is the goal, then some types of questions seem better than others.

In addition, PPCG has a rather strict policy of questions being meticulously thorough. If you forget to specify something, you can count on being called out in the comments, and most likely downvoted and/or closed until fixed. I don't see that happening much here. Maybe it does and I just haven't seen it yet, I don't know.

But all in all, that's what makes quality IMO. Having a mission/purpose and the discipline to stick with it. Whatever that mission ends up being is up to the people here. While that will be the source of some pain while deciding, I think it's an infinitely better use of time than trying to objectively measure something that seems inherently subjective.

$\endgroup$
3
  • 1
    $\begingroup$ 1392 days? Man, I've been there for almost half of the SE's lifespan now. $\endgroup$
    – user88
    Commented Nov 20, 2014 at 3:59
  • 1
    $\begingroup$ As much as I like this answer, I don't feel like it actually answers Mew's question, even though Mew accepted it. Could you point to some objective statistics that exemplify the aspects of PPCG you're talking about? $\endgroup$
    – xnor
    Commented Nov 20, 2014 at 6:45
  • $\begingroup$ @xnor No I can't, because the main point of my answer is that trying to find objective statistics for quality is not a good use of time (beside the useful-to-a-point Area 51 stats). I understand it doesn't answer the OP's question as asked, but IMO it's an X/Y problem. $\endgroup$
    – Set Big O
    Commented Nov 20, 2014 at 12:20
3
$\begingroup$

First of all, props to Geobits for the excellent answer. Only thing I want to add is that it's not only up to 'us' to just decide whatever we want to, but also think within the scope of StackExchange as a whole.

So, how is StackExchange different from all other sites? Well, when you open its homepage you get the following three points:

  • Expert communities
  • The right answer. Right on top.
  • Share knowledge. Earn trust.

These are definitely things we should keep in mind and differentiating this site from 'just another y! answer site' as was recently brought up is absolutely crucial. A SE network site is not about having fun, it's about helping each other, learning along the way and likely having fun whilst doing it. And yes, within that context CGPP.SE is definitely an odd one out and I have heard it being suggested by more than one person that that might have a lot to do with the reason why the site still is in beta. Are they right? No idea. Still wouldn't surprise me if it were true.

Either way, back on topic, I do think that point 1 and 3 allow some subjective objective determining of how we are doing. Are we an expert community? Are we sharing knowledge or 'just random puzzles'? Is reputation here a measure of skill and trust or just popularity? I know that in wording these questions in this way I am already showing a lot of bias, but I do think that those are the things that SE is all about.

$\endgroup$
5
  • 2
    $\begingroup$ Good points are raised here, but I don't think this answer is of any use in trying to settle disputes about the quality of this site in an objective way. Your three points can not be assessed for an entire site objectively at all. The three points are quite vague, e.g. earn trust, share knowledge, and one could easily say knowledge is being shared or not shared and it is hard to argue either way. So I like what you've written, but I think we can do better in trying to find a way to measure how good our site is. $\endgroup$
    – Kenshin
    Commented Nov 20, 2014 at 4:50
  • 1
    $\begingroup$ @Mew: That's why I said that I agree with Geobits answer, there isn't any way to measure such a thing objectively and he explains perfectly why that is so. I was mostly expanding and commenting on what he did write and at the same time trying to address your question. $\endgroup$
    – David Mulder
    Commented Nov 20, 2014 at 4:52
  • $\begingroup$ yes I understand that, thank you for your contribution. $\endgroup$
    – Kenshin
    Commented Nov 20, 2014 at 4:56
  • $\begingroup$ I think "Expert community" and "share knowledge" do not contradict sharing and solving puzzles, if they are puzzles written by experts and targeted at experts. PPCG is an example of this. $\endgroup$
    – xnor
    Commented Nov 20, 2014 at 11:48
  • $\begingroup$ @xnor: That's why I put the 'random' in front of 'puzzles' ;-) . It's definitely not impossible, just impossible for us at the current state we are in. But more seriously, we should really stop modelling ourselves after a site that already is an odd one out. $\endgroup$
    – David Mulder
    Commented Nov 20, 2014 at 16:41
3
$\begingroup$

Only 2 (well 3) answer postings and comments of only 7 different people for such an important question? (By 1.12.2014)

I think this in itself might be an indicator about the site. A lot of people enjoy it, but how many of them care about where the site is going?

I have to admit, that I'm on this site for a very short period only myself by now. And I was not lured by any mission statement, but by an interesting looking puzzle popping up on the right-hand side list of hot-network-questions. Once on the site, I was caught though. Can't resist a good puzzle (although I'm not the best in solving them.), and absolutely love to have a site where I can bounce off my ideas - good and bad, I've produced examples of both (I hope). So until I was brought to this meta-posting by a another of these "right-hand-side-links", I didn't think too much about the site and it aims itself. And I think that might be the problem most users here have.

So this leaves one with the main question, I guess:

Where do we want the site to go?

And maybe more importantly:

Who is we?

  • Is it the majority ? In this case, "having fun" might as well be the uppermost goal and forever beta wouldn't hurt.

  • Is it the founders of the site? In this case, it is their responsibility to define the goals of the site. (And I shouldn't be listened to.)

  • Is it the majority of those who care? In this case, there should be a regular attempt of directly guiding users towards a meta-discussion (like this post) and then a vote on goals.

From the answers given already, I assume some sort of mission-statement/goal is generally wanted. What could it be? Is this site about

  • Having great puzzles (i.e. finding great puzzles on site)
  • Teaching/Learning facts about puzzles
  • Teaching/Learning about solving puzzles
  • Teaching/learning about building puzzles

At first glance the site seems to be of the first category, but I think people on this posting are generally of the opinion That's not enough. - and I agree.

The second category seems the best defined, but at the same time least interesting/appealing one. Yeah, it's great if I have an 'expert' place I can ask my odd question, but can such a site hold enough interest to keep together a community of sufficient size? I somehow doubt it. Personally, I'd likely Google-browse the site, but never participate.

This leaves the last two categories, which I personally think is the life and soul of this site. There are people who enjoy solving puzzles and want to improve their skills by ever increasing difficulty/complexity of puzzles, and there are those (like me) who like inventing puzzles but can only improve their skills by getting feedback of 'serious' puzzlers. And guess what, if those two categories are catered for, the first category evolves out of it on its own!

I think this is what helps making this site great, so the question possibly is: How do we best support and steer this development?

If I were to devise a mission statement for this site, it potentially would be:

"We help puzzlers to become better. Better at building puzzles, better at understanding puzzles, and better at solving puzzles."

I think we should not be so focused on the quality of the puzzles.

Instead, we should be far more focused on the quality of the comments and potentially the quality of the edits.

Yes, there is potentially a large amount of poor-quality puzzles. Lets vote them down (not necessarily close them), but don't judge them with the vote. Learning is a process of failure and repeat. Use negative votes but only in combination with a comment explaining that&why you down-voted the puzzle.

I wouldn't even "forbid" bad puzzles, just repeating the same mistake should be discouraged. There will (hopefully) be new users of this site all the time - they will make similar mistakes. But let's build on the experienced of the old users, i.e. the comments & edits.

I think a really poor-quality puzzle/riddle with constructive feedback on how to improve or what went wrong can teach follow-up readers a lot! And the vote-score will tell potential puzzle-seekers which ones are well-built puzzles, and which potentially aren't.

Similarly, we should judge the quality of solutions by how well do they teach how the puzzle was solved? How much reasoning was given? Don't up-vote the "fastest" or "most correct" answer, but the one which teaches most. Again, use comments to promote editing towards this goal.

Finally, I think with a mission statement like (or similar to) the one above, we can come up with guidance-rules we agree (and/or vote?) on which can be summarized in a meta-posting as a reference. Measuring against those rules would potentially give us a measure of "quality" of the site in accordance with the SO philosophy. In particular, if one of the policies is that

Votes are used to indicate quality of the posting in accordance with the guidelines.

Votes should not be used to indicate a voters like/dislike of the puzzle.

Votes should not be used to indicate the quality of the puzzle itself.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .