Stats
We don’t track our progress as a project very well. We have relatively few stats that we look at over time to see how we’re growing/changing, and those we do have are largely cobbled together with various combinations of manual labor and scripting. One of the things I want to do this year is get us going in the direction of collecting stats on our work and participation levels, and to make as much of it as possible an automated process. I recognize that this stuff is non-trivial. That said, I can’t create an overall wishlist for Otto to shoot down until we figure out what stats would be good to have. 
What stats would be useful/helpful/just plain cool to know around your team? You guys are way ahead of all the other teams put together, with your “weekly stats: trends” page. This is brainstorming… don’t think about APIs or if/how it could be collected, just throw out ideas in the comments of what information you think it would great to start seeing, say on a monthly basis. List any and all ideas, including stats you are already collecting. I’ll collate all the teams’ ideas and see what the Meta team says we can do.
@chipbennett: As team rep, can you try to rally your group to make suggestions over the coming week? Thanks!
Emil Uzelac 12:44 am on December 27, 2012 Permalink |
Maybe we can extend this:
There are 86 new tickets waiting for review. 70 themes were reviewed in the last 7 days.
To stats which will include:
Emil
Jose Castaneda 1:07 am on December 28, 2012 Permalink |
I like the idea of Approved and Not-approved stats and wonder if if would be possible to do how many were updated as well.
Chip Bennett 2:48 pm on December 28, 2012 Permalink |
I like the idea of tracking spam submissions. We would have to add a keyword, or other way of identifying such tickets, though.
Also, we would need to track “insincere” submissions: non-spam tickets submitted for which the developer obviously made no effort to conform to the guidelines.
Interestingly, the ratio of tickets closed as Approved to tickets closed as Not-Approved (excluding newer-version-uploaded) has held steady at around 20-25%. I would be quite interested in the following:
Ideally, for non-spam submissions, we should be shooting for no abandoned submissions, and Approval status within 2 tickets for “sincere” submissions.
Jane Wells 12:48 am on December 27, 2012 Permalink |
Some ideas:
Emil Uzelac 12:53 am on December 27, 2012 Permalink |
I like it!
mrwweb 1:03 am on December 27, 2012 Permalink |
This data would require some time before it becomes meaningful, but seeing the changes in theme tags used over time could be cool.
Jane Wells 1:14 am on December 27, 2012 Permalink |
Ooh, that’s a good one!
Almost all stats require time to be meaningful. We can think long-term here.
Jacopo Tarantino 5:36 am on December 27, 2012 Permalink |
Could we do stats on theme ratings? Like a mean rating and/or weighted median rating? To that effect, it’d be really awesome if we did something that no other company does well: weighting ratings according to number of ratings. When a theme has a rating of 5 stars but from only 1 review it’s hard for me to know if it’s good.
nofearinc 9:15 am on December 27, 2012 Permalink |
Let’s not forget theme authors too. Number of authors, number of themes, regularity of submitting a new theme and so on (in addition to Jane’s list which was in my queue).
@mercime 3:19 pm on December 27, 2012 Permalink |
A. We could go back and retrieve the raw data of the following since WPTRT was created (more than 3 years isn’t it).
Just to start off. We can get averages, set critical points, determine and evaluate decision points, set up graphs and charts, and find correlations which are relevant to the theme review process. That way, as we move forward in 2013, decisions in improving theme review process http://make.wordpress.org/themes/2012/11/01/how-can-we-improve-the-theme-review-process/ for example, can be measured and later adjusted if necessary.
B. Number of themes from WP repo which are used at WordPress.com
(outside the scope but would be interesting to know)
C. Number of themes submitted to WP repo with links to PRO versions. Plus info from theme authors re increase in sales of a PRO version after submission/acceptance of free/lite theme in WP repo
Chip Bennett 2:54 pm on December 28, 2012 Permalink |
Tracking the up-sell Themes is an interesting idea. I wonder if we can do that via Trac ticket keyword, or an auto/forced Directory filter tag?
Konstantin Kovshenin 4:06 pm on December 27, 2012 Permalink |
How about average wait times in each queue? These will tell theme developers how fast they can ship a new theme or update an existing one.
Jane Wells 4:15 pm on December 27, 2012 Permalink |
Good idea.
Chip Bennett 2:56 pm on December 28, 2012 Permalink |
Yep, this was one of my original stats, though not segregated by queue.
The idea/goal is:
But, we struggle to keep up with that – which is the reason for the current Priority #2 queue (any ticket older than 2 weeks). If we could get to the point where that queue is always empty, then we would be in pretty good shape.
Chip Bennett 2:41 pm on December 28, 2012 Permalink |
During the time that I attempted to publish periodic review stats, I made a few observations:
If those observations still hold, then we’ll want to be careful about what data we gather, and how we analyze those data. Theme Review is a rigorous, time-consuming, volunteer activity; and every little bit helps. But in trying to analyze the data, I found that it was going to be more meaningful/useful to segregate the stats according to activity level (i.e., separate out the top 10-20% from the rest), to get an idea of the data for “active” reviewers as separate from the data for “casual” reviewers.
(Here, “active” would be defined as anyone who has reviewed, say, 5 Themes in a week, or more than 50 total Themes.)
In fact, one of the reasons that I stopped publishing stats was because I thought he focus on stats (and especially any effort to segregate/differentiate based on activity level) might discourage people from wanting to contribute what little time they had to offer.
(For the same reason, the idea of “quotas”, even as little as one ticket per week, failed miserably.)
Here is an example of the stats that I used to compile (manually) each week:
http://www.chipbennett.net/themes/theme-review/weekly-stats/
Note that, with another full year’s worth of data, the trends never varied significantly. (Which was another reason that I abandoned the effort: tracking/trending wasn’t helping to change anything, and the time spent compiling the data was better-spent reviewing a few more tickets.)
All that said, stats are, and can be, incredibly useful. Generally speaking, we generate enough data for monthly and even weekly stats to be meaningful/useful, so assume a “per week” or “per month” for all of the following suggestions.
Bearing in mind that one of our primary goals is to make the process as painless as possible for Theme developers, the following data are especially useful:
From the perspective of making the process as efficient as possible, the following data are useful:
From the perspective of reviewer workload, the following data are useful:
Juliet Sallette 4:56 pm on December 30, 2012 Permalink |
WOW… This is impressive information… You really are a gem to take the time to tabulate and digest this information.
I am new to the theme review group and hope to be useful in reviewing themes.
It is nice to know that the community is filled with people that are dedicated and committed to making WP stronger.
Thanks for the inspiration… Your recap was Awesome!!!
The WordPress Weekend Roundup - WP Daily 5:00 pm on January 5, 2013 Permalink |
[...] WordPress Themes [...]