16
$\begingroup$

In a few weeks, we will have our approximately bi-annual site self-evaluation, during which the site self-eval review queue will be open.

Last time, however, we found that the criteria don't really apply as much to us. There are three categories we can triage questions into:

Excellent if this question is well-written and has a clear, comprehensive answer that is far better than other available resources. This question shows up early in search results.

Satisfactory if this question has a clear, comprehensive answer that is comparable or better than the information found elsewhere. This question shows up later in the search results or requires oddly specific search terms to find.

Needs Improvement if this question is poorly written, not generally useful, or has an answer that is no better than what can be found elsewhere. Alternatively, this question does not turn up on the first page or two in Google despite repeated search attempts.

However, a lot of this is based around the principle that a good question is easily searchable and contains accessible information someone is likely to search for.

This advice doesn't seem very helpful, given that Puzzling's questions are mostly not of the variety that are intended to be searched.

So the hard question comes up once again. How should we sort questions in the self-evaluation? Much of this, I think, will come down to a discussion on what an excellent question actually is, but hey, it's a discussion. If you have other thoughts, feel free to tack those on here, too.

I'm asking this a bit early to give us ample time to mull it over and be prepared, but the self-eval will start later this month.

$\endgroup$
  • $\begingroup$ Thanks for picking up the hot potato well in advance. Hopefully, we can sail through this time without roughing too many feathers... $\endgroup$ – BmyGuest May 5 '15 at 7:47
  • $\begingroup$ Good question - I don't know the answer. Perhaps it would help for us to ask: what is the purpose of the self-evaluation? $\endgroup$ – A E May 6 '15 at 8:05
  • $\begingroup$ @AE That's a question most people will give you different answers to. The gist of it, though, is that it's a reality check for the health of the site. It's a gentle reminder that we're not here to dawdle, and that we do need to ultimately find a clear purpose and evaluate how well we can fulfill it. That's a broad question, and it's not expected that we tackle it at once, but we do need to get there eventually. $\endgroup$ – Aza May 6 '15 at 8:30
  • 4
    $\begingroup$ This might be worth you or one of the other mods liasing with some of the SE team about...? $\endgroup$ – starsplusplus May 6 '15 at 13:58
8
$\begingroup$

Self evaluations serve two basic purposes:

  1. They give a community a chance to reflect on their progress and
  2. They give community managers a sense of the how the site is progressing.

Of the two, the first is far and away the most important. The difficulty we've had as a team is determining what the results of the evaluation mean. In the past, we've found that some sites are generous in evaluating their own content and many others are entirely too critical. So we can't look at the raw results and determine whether the content on the site is of high quality and easily findable. There's very little correlation between the numbers reported and our evaluation of how a site is doing.

From the very earliest moment of a site's life on Area 51, the general vector of a successful Stack Exchange site points from individual needs to collective needs. A proposal begins with a single person pitching an idea. If the proposal has legs, other people will begin defining the scope of the proposed site with example questions and later committing to participate in the private beta. During beta, the conversation moves to meta and begins to focus on how the community should be organized and how it should present itself to the outside world. When a site nears graduation, a healthy community has gone beyond answering it's own questions to strategizing about how to answer questions for people who never intend to participate on the site at all.

As Joel explains:

For every person who asks a question and gets an answer on Stack Overflow, hundreds or thousands of people will come read that conversation later. Even if the original asker got a decent answer and moved on, the question lives on and may continue to be useful for decades.

This is fundamentally different from Usenet or any of the web-based forums. It means that Stack Overflow is not just a historical record of questions and answers. It’s a lot more than that: it’s actually a community-edited wiki of narrow, “long-tail” questions — questions that aren’t quite important enough to deserve a page on Wikipedia, but which come up over and over again.

Our criteria obsessively focuses on search results because historically search is the only way for people to find the sort of longtail questions that our narrow Q&A format excels in. As the network has grown and thanks to putting Hot Questions in the sidebar, more and more sites are getting traffic from within the network. Puzzling is definitely in that category. Since the beginning of the year, 49% of traffic has come from search engines (mostly Google) and 45% of traffic has come from referring sites (mostly Stack Overflow and other sites on the network). A big reason this site has such strong Area 51 statistics is that its questions are often featured on the sidebar of other sites.

The situation reminds me a bit of the history of mining towns. Initially people show up strictly to extract precious metal from the ground and ship it off to the people who can make something of it. By analogy, Stack Overflow, Unix, Mathematics, and so on are the network's miners. As the town grows, more people move in to provide services to the miners themselves. These folks might offer tool repair, entertainment, food, laundry, and other auxiliary services. They are important for quality of life, but they don't accomplish the primary mission of the town. So far, Puzzling has provided entertainment for people in need of a break from more productive sites.

As auxiliary sites mature, I think they begin to extend their reach beyond the Stack Exchange network. The Workplace, for instance, had a very similar profile as Puzzling does. But it has matured to the point where more than 80% of its traffic came from search. We don't really require this of graduated sites, but I do think this becomes a natural impulse for communities as they grow. Everything about the way our Q&A engine works points people toward writing for people beyond the existing users of the site.

In any case, great titles that often help a question successfully reach people searching for answers also serve as the only real way they are discovered in the Hot Questions list. So I'd take special notice of questions with excellent titles. Google tends to be an objective evaluator of usefulness of a page, so I think attempting the exercise of searching for each question in the sample is useful. Finally, the self-evaluation begins a process of the community examining its own content. I recommend taking a moment or two to write up your conclusions after you have evaluated all the questions.

$\endgroup$
  • 1
    $\begingroup$ I'd be interested in exactly which questions those 45% of search results are looking for. $\endgroup$ – Joe Z. May 23 '15 at 3:45
1
$\begingroup$

I'm not sure what the point of the self evaluation is, given that only 10 questions were selected out of the approximately 1700-1800 questions that were posted in the past 6 months. That is an extremely tiny sample size, so I don't see how the ratings for those questions (which appear to be all over the map) could be representative of anything about the site in general.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .