Should Priority #1 Queue be Reserved for Admin Review?
I’d like to throw out an idea for discussion/feedback. If we get a consensus, we’ll implement it; otherwise, we’ll maintain our current process.
Currently, there is a delay between tickets being closed as “Approved”, and Themes being made live in Extend (“Extend” is no more) the Theme directory. This delay exists for two reasons: one, because the synchronization between Theme-Trac and the Theme directory is manual, and two, because Admins do a final audit of “Approved” tickets before making the Themes live. (Because we have so many volunteer reviewers, and don’t have the time or resources to implement any kind of formal training process, we basically give anyone who wants to review Themes, and shows the initiative to get involved, the privileges necessary to assign/resolve/close Theme-Trac tickets. So, the compromise is that, for tickets resolved as “Approved”, the Admins add in an extra, quality-control step before making those Themes live in the Theme directory.)
Because of that delay, the Admins have asked for up to a week to complete the synchronization to make Themes approved in Theme-Trac live in the Theme directory. While this delay is (unfortunately) not a huge additional amount of time to wait for most Themes in the queue, for currently approved Themes, the delay prevents end users from getting bugfixes or security updates for Themes they currently use.
Because it is especially important for updates for currently approved (and currently in-use by end users) Themes to be made live in the Theme directory as quickly as possible, I would like to propose a change to the Review process: to reserve the Priority #1 Queue (i.e. currently approved Themes) for review by Admins, who can review, approve, and synchronize those Themes immediately, rather than sorting through the queue of approved Themes waiting to be synchronized.
I’ve added a poll. Please leave any additional comments or feedback in the post comments.
Andrew Nacin 12:10 am on May 21, 2013 Permalink |
I’m confused by your proposal — the priority #1 queue as it exists now is great. If we implemented this plan as-is, I’d hope that we’d just have priority #2 be the old #1, etc.
I think we can solve this with better tools. What about this?
I can have this working in a few hours, after dinner.
Chip Bennett 12:57 am on May 21, 2013 Permalink |
Right now, any reviewer can review any ticket in any queue: basically, take the first Theme in the highest priority. If #1 is empty, check #2, and so on. The only change here would be that only Admins (who can review, approve, and then go immediately to the Theme directory and synchronize as Live) would review Priority #1 queue tickets, and everyone else would start with Priority #2.
The reasoning is that the vast majority of Priority #1 Queue tickets are resolved as “Approved”, and represent the greatest workload of “Approved” Themes that the Admins then have to audit and synchronize. This proposal would thus short-circuit some of the workflow, and get more “Approved” Themes into the hands of end users sooner.
We have experimented with the Theme-Trac workflow (previously, via keywords), and we found that it only created more work. The ideal solution would have Admins interacting with Theme-Trac tickets as little as possible during the synchronization. Besides: Otto has already given us ways to filter through the “Approved” Themes waiting to be synchronized (in the Theme directory admin).
Ultimately, the only thing this proposal changes is that it helps get Priority #1 Queue tickets processed and synchronized faster.
Andrew Nacin 3:04 am on May 21, 2013 Permalink |
Isn’t Priority #1 for updates to themes that are previously approved, but where the updates still need to be reviewed? If Priority #1 becomes updates that are approved and ready to go live, then surely you need for a second Priority 1A which is the current priority 1?
Andrew Nacin 3:10 am on May 21, 2013 Permalink |
Here is what we are envisioning:
Additionally:
And, some potential presentational changes. Rather than everything being a “theme”, it can be a:
We already calculate this data in the reports, but this will now be much easier to query and look at. It will also make this reports much faster, because right now they are really slow (and probably are making the server unhappy).
Tickets will also be able to be “unassigned” (by default), “assigned” (to others, by theme admins only), and “reviewing” (by a reviewer). Regular reviewers will also be unable to “resolve” a theme as approved or otherwise prior to saying they are reviewing it, and they won’t be burdened with additional UI such as the ability to assign to others or resolve a theme, without accepting it. End goal: It will be more streamlined and easier to approved themes.
This will take up to a few weeks to get to everything, but the first part (the top four bullets) are going to be implemented now.
Chip Bennett 3:07 pm on May 21, 2013 Permalink |
I like all of these changes – though I do think they’re still separate from the proposed idea in the post.
Josh Pollock 12:12 am on May 21, 2013 Permalink |
I think that removing the one week window for approval would greatly benefit end users. Concentrating the efforts of non-admin reviewers on new themes will help cut down on that backlog. As someone who just had to wait 6 weeks to get a new theme reviewed that is a very enticing prospect. The only downside I see to this is it could create too much work for the admins.
As I proposed in the list, I think we should implement this system on a trial basis and if this creates too much work for the admins, thus leading to a backlog of previously approved themes needing reviewed, we can designate 1 or 2 highly experienced reviewers to help out on that queue when needed.
Srikanth 12:13 am on May 21, 2013 Permalink |
Not all updates to already approved themes are security/bug fixes, mostly they are code cleanup, new feature/skin additions. We are already allowed to request a quick update/review and not wait for 7 days if its security related.
If admins can dedicate a few minutes every day, then its a good idea. They can also use these few minutes to push the already approved themes live too.
Emil Uzelac 12:23 am on May 21, 2013 Permalink |
My #1 priorities are pretty much always pushed live after the review and I do agree with you that this should be an admin only.
Let’s place everything else aside, what happens if there was a “security” fix for any given Theme, newbie won’t know the difference whether or not the fix was done properly.
Srikanth 12:33 am on May 21, 2013 Permalink |
I don’t assign myself the ticket if i don’t understand whats going on, kind of pre screening, I assume everyone does that?
Andrew Nacin 12:29 am on May 21, 2013 Permalink |
I am talking with @Otto42, we are going to overhaul how themes.trac is set up tonight. No need for your workflow to be so manual — we can make it better! Stay tuned.
Jose Castaneda 12:30 am on May 21, 2013 Permalink |
Oooh, can’t wait to hear about this.
@mercime 1:43 am on May 21, 2013 Permalink |
+ 1
Frumph 2:00 am on May 21, 2013 Permalink |
Has anyone fathomed any thought to the abundance of regulations and requirements being the reason the queue time is so long? Or is it just the lack of time …. read on twitter that it’s now at 2 months wait time.
Chip Bennett 2:17 am on May 21, 2013 Permalink |
I honestly don’t think so. Half of the Guidelines are covered by the uploader script and Theme Check. The two biggest issues:
1) The overall quality of Themes submitted. The quality Themes submitted by developers making a sincere effort to conform to the Guidelines are still overwhelmed by the number of Themes that exhibit little or no effort to conform to the Guidelines.
2) Participation is wide, but shallow. I just did a very rough estimate. Of over 100 people with reviewer privileges, less than 25 have closed at least 100 tickets (over an almost 3-year period). Now, that number has improved significantly; in fact, it’s doubled in the past six months or so. But, it’s still a classic Pareto: 20% of the reviewers do 80% of the work. In a volunteer activity, that is simply the nature of things. We can’t demand more of anyone’s time; we can only be grateful for the involvement we get.
wpweaver 2:33 am on May 21, 2013 Permalink |
Sometimes, it would be great to get a priority 1 theme approved and pushed live – especially if there are important bug fixes. But really, sometimes it doesn’t matter. Many themes are well-established, and updates are thus often feature related, not bug related. So it doesn’t matter if it is a week or 10 days.
But sometimes there are critical bugs, and it would be great to have these turned around asap. So perhaps there could be some way to indicate urgent, bug related updates that affect end users, and those that are not really critical. Maybe even a new 0 level for urgent fixes?
Justin Tadlock 2:59 am on May 21, 2013 Permalink |
I don’t like this proposal because I don’t think it’s the best solution to the problem and like the current workflow as it is. I definitely don’t want to add another layer of complexity (review these themes but not those).
I’ve never had much problem with the wait times after approval and don’t see it as an issue (from a theme author’s perspective). However, better tools are always welcome. So, if Nacin and Otto come up with something that would make this less manual for the admins, I’d love to see that put into place.
Abhik 10:28 am on May 21, 2013 Permalink |
What about giving some additional permissions to veteran reviewers so that they can approve themes and make it live? That’ll off load little pressure from admins.
But let’s see ehat Andrew and Otto come up with.
Edward Caissie 2:46 pm on May 21, 2013 Permalink |
I’m all for a better workflow and more streamlined processes. Although some of these “changes” may require a bit of thought in how they are fully implemented I’m all for finding a better way to manage the “bug-fix” and “security issues” scenarios. Otherwise, most of the wait times are within reason for approved themes given the volunteer powered workforce doing all of the heavy lifting.