As I write this post, Theme-Trac indicates the following:
Welcome to WordPress Themes Trac
There are 161 new tickets waiting for review. 24 themes were reviewed in the last 7 days.
Although right now is the holiday season here in the US, and most of us have been busy getting ready for the just-released WordPress 3.3, that review queue number is extraordinarily high. That number represents a lot of developers awaiting feedback on their Themes, and a lot of end users not benefiting from new and/or updated Themes. Let’s see what we can do to address it!
Revising the Trainee Reviewer Experiment
First things first: the Theme Review Team has been experimenting with a training program for new reviewers. After a few months, it appears that this experiment isn’t working out as intended. New reviewers aren’t getting assigned tickets as quickly as possible, and full reviewers don’t seem to have enough time to follow up on those reviews, provide feedback, and resolve/close the tickets. While we want to provide training for new reviewers, so that we equip our volunteers with the tools and skills necessary to complete Theme reviews, we can’t let that training get in the way of completing Theme reviews.
So, effective immediately, we’re going to try something else: anyone who requests a ticket to review, and then completes their assigned review, will be given full “reviewer” status in Theme-Trac. That means that, once a new reviewer has requested and completed their first ticket, they will then be able to assign themselves tickets, and will be able to resolve/close tickets on their own. An admin reviewer will double-check tickets resolved as “approved”, but otherwise, once you’ve done one review, you’ll be free to review without the burden of any additional oversight.
Where previously we have been slow to “promote” to full reviewer status, and exceedingly slow to remove reviewer privileges, now we will look to be very quick in granting reviewer privileges, and perhaps somewhat quicker than we were previously with removing reviewer privileges, either due to inactivity or poor reviews (mainly, approving Themes that should not be approved).
Revising the Queue Priorities
Second: for a considerably longer period, the Theme Review Team has used a three-tiered priority approach to the review queue:
- Priority #1: Currently Approved Themes
- Priority #2: Previously Reviewed, Not-Approved Themes
- Priority #3: Never-Reviewed Themes
This three-tiered approach has worked fairly well, especially for Themes that have successfully passed the review process, and are currently approved. However, the Theme Review Team has been unable to keep the Priority #2 queue cleared, meaning that new Themes end up waiting weeks (or longer) to get even an initial review.
So, effective immediately, we’re adding a fourth tier to the prioritization, which will become the new #2 priority: tickets that have been in the review queue for longer than two weeks, regardless of previous review/approval status. The new prioritization will be as follows:
- Priority #1: Currently Approved Themes
- Priority #2: Tickets Older Than 2 Weeks
- Priority #3: Previously Reviewed, Not-Approved Themes
- Priority #4: Never-Reviewed Themes
Hopefully with this change, the oldest tickets will be reviewed in a more timely manner. Our long-term goal will be that this new Priority #2 queue will be – and stay – empty; but for now, it will help ensure that tickets don’t stay in the queue for weeks on end.
Revising Handling of Review-Based Theme Revisions
Currently, once a Theme is reviewed, if the developer revises the Theme to address issues from the review, and then re-submits the Theme, the re-submitted Theme goes to the end of the Previously-Reviewed Themes queue. This process does not encourage or facilitate such review-based Theme revisions. Understandably – especially with the state of the review queue – Theme developers may be discouraged by the process, and may choose never to submit a revision. This outcome helps no one: end users don’t benefit from an approved Theme becoming available, Theme developers don’t benefit from having their Theme hosted in the repository, and the Theme Review Team expends time and effort reviewing a Theme that never ultimately gets approved.
For some time, the Theme Review Team has had an informal policy, subject to the discretion of each reviewer, of allowing a review to be continued on a subsequent ticket, if a revision is submitted in a timely manner. This informal policy has facilitated prompt Theme revision submissions, and has led to more Themes eventually passing Theme review, and being approved.
So, effective immediately, we’re formalizing this policy: any review-based Theme revision that is submitted within two days of the previous review will be assigned to the previous-ticket reviewer, and the review continued on the new ticket. Bear in mind: exercise of this policy will still be at the discretion of each reviewer, and should be considered to be a privilege, based on a good-faith effort on the part of the Theme developer. While most Theme developers will exercise such good-faith effort, I want to make clear that this policy is not a license to use the review process as quality control.
Revising Review Emphasis
The Theme Review Guidelines currently include a requirement for W3C HTML/CSS validation. It has become clear that this requirement is largely a distraction, both to developers and to reviewers. From the beginning, the W3C Validation requirement was intended to be a holistic tool used in the review process, rather than a rigid requirement; however, all too often, reviews have emphasized W3C Validation results – even at the expense of far more important guideline requirements.
So, effective immediately, W3C Validation criticality is being reduced from REQUIRED to RECOMMENDED, and Theme reviewers will no longer make review comments regarding W3C Validation. Theme developers can/should use W3C Validation as a development tool, but it will no longer be part of the review process.
Also, the Theme Review Team has made an effort to ensure that every review is thorough and complete, in order to reduce the number of times any given Theme must go through the submission/review process prior to approval. However, it is clear that this emphasis on thorough reviews has acted to the detriment of performing reviews expeditiously. What was intended to be a service to Theme developers has actually led to more developer frustration, as the review queue continues to grow.
So, effective immediately, the Theme Review Team will no longer emphasize complete and thorough reviews, and will instead close tickets upon observation of any non-trivial issues. Theme developers have always been responsible for knowing and following the Theme Review guidelines. The key take-away on this point: if you have a question, ASK. As much as we would like to walk every developer through the review/approval process, we simply don’t have enough volunteers to spare the time. If you are unsure about a review guideline, or need clarification on a review comment, ASK. Ask on the theme-reviewers mail-list before you submit your Theme. Ask in the review ticket after you submit. We are here to help developers, but will need to rely more on developers to communicate your questions and concerns, so that we can focus more on reviewing/closing tickets.
Summarizing The Changes
Again, here’s what we’re going to be doing differently, effective immediately, to help facilitate Theme reviews:
- Anyone who requests a ticket to review, and then completes their assigned review, will be given full “reviewer” status in Theme-Trac
- We’re adding a fourth tier to the prioritization, which will become the new #2 priority: tickets that have been in the review queue for longer than two weeks, regardless of previous review/approval status
- Any review-based Theme revision that is submitted within two days of the previous review will be assigned to the previous-ticket reviewer, and the review continued on the new ticket
- W3C Validation criticality is being reduced from REQUIRED to RECOMMENDED, and Theme reviewers will no longer make review comments regarding W3C Validation
- The Theme Review Team will no longer emphasize complete and thorough reviews, and will instead close tickets upon observation of any non-trivial issues
I would also like to add a gentle reminder that the Theme Review Team is a 100% volunteer effort. None of the Theme reviewers are paid for our efforts. For almost all of us, Theme reviews take time away from other development work (whether paid or free) – time that we are happy to contribute. That said: while we understand your frustration in waiting for your Theme review to be completed, I would like to make a request: before you email the theme-reviewers mail-list asking when your Theme will be reviewed, post a comment to the current Trac Ticket Review Request Queue, and ask to be assigned a ticket of your own to review. This request is not a pay-for-play scheme, and no tickets or developers are given special treatment for participating in Theme reviews; rather, the more people we have reviewing Themes, the faster the review queue will get processed. (And as a bonus: you’ll learn more than you might otherwise imagine about what is required to pass the review process successfully.)
Remember: in the end, our entire effort is all about ensuring that WordPress end users have the best-possible Themes available for use.
If you have any other suggestions for how we can improve the process, please let us know – either in the comments, or via the theme-reviewers mail list.
Greg Priday 2:09 pm on March 4, 2013 Permalink |
Great work everyone! It was amazing to see the #2 queue just melt away. We should make these month-end pushes a regular thing.
Nisha Singh 6:59 am on March 5, 2013 Permalink |
Thanks Chip for appreciation and your Support.