Self evaluations serve two basic purposes:
- They give a community a chance to reflect on their progress and
- They give community managers a sense of the how the site is progressing.
Of the two, the first is far and away the most important. The difficulty we've had as a team is determining what the results of the evaluation mean. In the past, we've found that some sites are generous in evaluating their own content and many others are entirely too critical. So we can't look at the raw results and determine whether the content on the site is of high quality and easily findable. There's very little correlation between the numbers reported and our evaluation of how a site is doing.
From the very earliest moment of a site's life on Area 51, the general vector of a successful Stack Exchange site points from individual needs to collective needs. A proposal begins with a single person pitching an idea. If the proposal has legs, other people will begin defining the scope of the proposed site with example questions and later committing to participate in the private beta. During beta, the conversation moves to meta and begins to focus on how the community should be organized and how it should present itself to the outside world. When a site nears graduation, a healthy community has gone beyond answering it's own questions to strategizing about how to answer questions for people who never intend to participate on the site at all.
As Joel explains:
For every person who asks a question and gets an answer on Stack Overflow, hundreds or thousands of people will come read that conversation later. Even if the original asker got a decent answer and moved on, the question lives on and may continue to be useful for decades.
This is fundamentally different from Usenet or any of the web-based forums. It means that Stack Overflow is not just a historical record of questions and answers. It’s a lot more than that: it’s actually a community-edited wiki of narrow, “long-tail” questions — questions that aren’t quite important enough to deserve a page on Wikipedia, but which come up over and over again.
Our criteria obsessively focuses on search results because historically search is the only way for people to find the sort of longtail questions that our narrow Q&A format excels in. As the network has grown and thanks to putting Hot Questions in the sidebar, more and more sites are getting traffic from within the network. Puzzling is definitely in that category. Since the beginning of the year, 49% of traffic has come from search engines (mostly Google) and 45% of traffic has come from referring sites (mostly Stack Overflow and other sites on the network). A big reason this site has such strong Area 51 statistics is that its questions are often featured on the sidebar of other sites.
The situation reminds me a bit of the history of mining towns. Initially people show up strictly to extract precious metal from the ground and ship it off to the people who can make something of it. By analogy, Stack Overflow, Unix, Mathematics, and so on are the network's miners. As the town grows, more people move in to provide services to the miners themselves. These folks might offer tool repair, entertainment, food, laundry, and other auxiliary services. They are important for quality of life, but they don't accomplish the primary mission of the town. So far, Puzzling has provided entertainment for people in need of a break from more productive sites.
As auxiliary sites mature, I think they begin to extend their reach beyond the Stack Exchange network. The Workplace, for instance, had a very similar profile as Puzzling does. But it has matured to the point where more than 80% of its traffic came from search. We don't really require this of graduated sites, but I do think this becomes a natural impulse for communities as they grow. Everything about the way our Q&A engine works points people toward writing for people beyond the existing users of the site.
In any case, great titles that often help a question successfully reach people searching for answers also serve as the only real way they are discovered in the Hot Questions list. So I'd take special notice of questions with excellent titles. Google tends to be an objective evaluator of usefulness of a page, so I think attempting the exercise of searching for each question in the sample is useful. Finally, the self-evaluation begins a process of the community examining its own content. I recommend taking a moment or two to write up your conclusions after you have evaluated all the questions.
