Chip Bennett 12:06 am on May 21, 2013

Should Priority #1 Queue be Reserved for Admin Review?

I’d like to throw out an idea for discussion/feedback. If we get a consensus, we’ll implement it; otherwise, we’ll maintain our current process.

Currently, there is a delay between tickets being closed as “Approved”, and Themes being made live in ~~Extend~~ (“Extend” is no more) the Theme directory. This delay exists for two reasons: one, because the synchronization between Theme-TracTrac Trac is the place where contributors create issues for bugs or feature requests much like GitHub.https://core.trac.wordpress.org/. and the Theme directory is manual, and two, because Admins do a final audit of “Approved” tickets before making the Themes live. (Because we have so many volunteer reviewers, and don’t have the time or resources to implement any kind of formal training process, we basically give anyone who wants to review Themes, and shows the initiative to get involved, the privileges necessary to assign/resolve/close Theme-Trac tickets. So, the compromise is that, for tickets resolved as “Approved”, the Admins add in an extra, quality-control step before making those Themes live in the Theme directory.)

Because of that delay, the Admins have asked for up to a week to complete the synchronization to make Themes approved in Theme-Trac live in the Theme directory. While this delay is (unfortunately) not a huge additional amount of time to wait for most Themes in the queue, for currently approved Themes, the delay prevents end users from getting bugfixes or security updates for Themes they currently use.

Because it is especially important for updates for currently approved (and currently in-use by end users) Themes to be made live in the Theme directory as quickly as possible, I would like to propose a change to the Review process: to reserve the Priority #1 Queue (i.e. currently approved Themes) for review by Admins, who can review, approve, and synchronize those Themes immediately, rather than sorting through the queue of approved Themes waiting to be synchronized.

I’ve added a poll. Please leave any additional comments or feedback in the post comments.

Andrew Nacin 12:10 am on May 21, 2013
I’m confused by your proposal — the priority #1 queue as it exists now is great. If we implemented this plan as-is, I’d hope that we’d just have priority #2 be the old #1, etc.

I think we can solve this with better tools. What about this?
- Create a new status of “Live”
- Allow theme admins to change a ticket that is closed as “Approved” to “Live”
- Create a separate report that theme admins can look at for “Approved” tickets that are not “Live”
I can have this working in a few hours, after dinner.
- Chip Bennett 12:57 am on May 21, 2013
  
  Right now, any reviewer can review any ticket in any queue: basically, take the first Theme in the highest priority. If #1 is empty, check #2, and so on. The only change here would be that only Admins (who can review, approve, and then go immediately to the Theme directory and synchronize as Live) would review Priority #1 queue tickets, and everyone else would start with Priority #2.
  
  The reasoning is that the vast majority of Priority #1 Queue tickets are resolved as “Approved”, and represent the greatest workload of “Approved” Themes that the Admins then have to audit and synchronize. This proposal would thus short-circuit some of the workflow, and get more “Approved” Themes into the hands of end users sooner.
  
  We have experimented with the Theme-TracTrac Trac is the place where contributors create issues for bugs or feature requests much like GitHub.https://core.trac.wordpress.org/. workflow (previously, via keywords), and we found that it only created more work. The ideal solution would have Admins interacting with Theme-Trac tickets as little as possible during the synchronization. Besides: Otto has already given us ways to filterFilter Filters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output. through the “Approved” Themes waiting to be synchronized (in the Theme directory admin).
  
  Ultimately, the only thing this proposal changes is that it helps get Priority #1 Queue tickets processed and synchronized faster.
  - Andrew Nacin 3:04 am on May 21, 2013
    
    Isn’t Priority #1 for updates to themes that are previously approved, but where the updates still need to be reviewed? If Priority #1 becomes updates that are approved and ready to go live, then surely you need for a second Priority 1A which is the current priority 1?
- Andrew Nacin 3:10 am on May 21, 2013
  Here is what we are envisioning:
  - A theme that is approved by a reviewer causes the ticket to be closed.
  - These themes end up in a new report, called something like “Approved themes that must be pushed live by a theme admin”.
  - When the resolution of the ticket is changed to “Live” by an admin, it is automatically approved in the theme directory. There will be no need to go to wordpress.orgWordPress.org The community site where WordPress code is created and shared by the users. This is where you can download the source code for WordPress core, plugins and themes as well as the central location for community conversations and organization. https://wordpress.org//themes/admin in this scenario.
  - A theme admin can directly mark a ticket as “Live”, rather than a two-step of marking it as “Approved” and then “Live”.
  Additionally:
  - When a new version is uploaded, with an old version that is still pending review, rather than this being a manual process where reviewers look for new versions, we can automatically (on upload of the new version) update the old version’s existing ticket. This way, things get cross-referenced automatically, and that theme doesn’t lose its place in the queue. We’ll be able to actually adjust the diff link to account for the need to review changes across multiple themes.
  - We’re going to actually *list* other tickets of that theme in the ticket body, rather than linking to this list. It will be a dynamic list that will update on older tickets even as newer tickets are submitted, automatically creating full, easy cross-references.
  - If the theme is an update of a previously approved theme, we may actually be able to show the *entire diff* directly in the ticket. No more needing to open a new window.
  And, some potential presentational changes. Rather than everything being a “theme”, it can be a:
  - previously approved theme
  - new theme
  - new theme (previous versions not approved)
  We already calculate this data in the reports, but this will now be much easier to query and look at. It will also make this reports much faster, because right now they are really slow (and probably are making the server unhappy).
  
  Tickets will also be able to be “unassigned” (by default), “assigned” (to others, by theme admins only), and “reviewing” (by a reviewer). Regular reviewers will also be unable to “resolve” a theme as approved or otherwise prior to saying they are reviewing it, and they won’t be burdened with additional UIUI UI is an acronym for User Interface - the layout of the page the user interacts with. Think ‘how are they doing that’ and less about what they are doing. such as the ability to assign to others or resolve a theme, without accepting it. End goal: It will be more streamlined and easier to approved themes.
  
  This will take up to a few weeks to get to everything, but the first part (the top four bullets) are going to be implemented now.
  - Chip Bennett 3:07 pm on May 21, 2013
    
    I like all of these changes – though I do think they’re still separate from the proposed idea in the post.
Josh Pollock 12:12 am on May 21, 2013

I think that removing the one week window for approval would greatly benefit end users. Concentrating the efforts of non-admin reviewers on new themes will help cut down on that backlog. As someone who just had to wait 6 weeks to get a new theme reviewed that is a very enticing prospect. The only downside I see to this is it could create too much work for the admins.

As I proposed in the list, I think we should implement this system on a trial basis and if this creates too much work for the admins, thus leading to a backlog of previously approved themes needing reviewed, we can designate 1 or 2 highly experienced reviewers to help out on that queue when needed.
Srikanth 12:13 am on May 21, 2013

Not all updates to already approved themes are security/bug fixes, mostly they are code cleanup, new feature/skin additions. We are already allowed to request a quick update/review and not wait for 7 days if its security related.
If admins can dedicate a few minutes every day, then its a good idea. They can also use these few minutes to push the already approved themes live too.
Emil Uzelac 12:23 am on May 21, 2013

My #1 priorities are pretty much always pushed live after the review and I do agree with you that this should be an admin only.

Let’s place everything else aside, what happens if there was a “security” fix for any given Theme, newbie won’t know the difference whether or not the fix was done properly.
- Srikanth 12:33 am on May 21, 2013
  
  I don’t assign myself the ticket if i don’t understand whats going on, kind of pre screening, I assume everyone does that?
Andrew Nacin 12:29 am on May 21, 2013

I am talking with @Otto42, we are going to overhaul how themes.tracTrac Trac is the place where contributors create issues for bugs or feature requests much like GitHub.https://core.trac.wordpress.org/. is set up tonight. No need for your workflow to be so manual — we can make it better! Stay tuned.
- Jose Castaneda 12:30 am on May 21, 2013
  
  Oooh, can’t wait to hear about this. 🙂
  - @mercime 1:43 am on May 21, 2013
    
    + 1
Frumph 2:00 am on May 21, 2013

Has anyone fathomed any thought to the abundance of regulations and requirements being the reason the queue time is so long? Or is it just the lack of time …. read on twitter that it’s now at 2 months wait time.
- Chip Bennett 2:17 am on May 21, 2013
  
  I honestly don’t think so. Half of the Guidelines are covered by the uploader script and Theme Check. The two biggest issues:
  
  1) The overall quality of Themes submitted. The quality Themes submitted by developers making a sincere effort to conform to the Guidelines are still overwhelmed by the number of Themes that exhibit little or no effort to conform to the Guidelines.
  
  2) Participation is wide, but shallow. I just did a very rough estimate. Of over 100 people with reviewer privileges, less than 25 have closed at least 100 tickets (over an almost 3-year period). Now, that number has improved significantly; in fact, it’s doubled in the past six months or so. But, it’s still a classic Pareto: 20% of the reviewers do 80% of the work. In a volunteer activity, that is simply the nature of things. We can’t demand more of anyone’s time; we can only be grateful for the involvement we get.
wpweaver 2:33 am on May 21, 2013

Sometimes, it would be great to get a priority 1 theme approved and pushed live – especially if there are important bug fixes. But really, sometimes it doesn’t matter. Many themes are well-established, and updates are thus often feature related, not bug related. So it doesn’t matter if it is a week or 10 days.

But sometimes there are critical bugs, and it would be great to have these turned around asap. So perhaps there could be some way to indicate urgent, bug related updates that affect end users, and those that are not really critical. Maybe even a new 0 level for urgent fixes?
Justin Tadlock 2:59 am on May 21, 2013

I don’t like this proposal because I don’t think it’s the best solution to the problem and like the current workflow as it is. I definitely don’t want to add another layer of complexity (review these themes but not those).

I’ve never had much problem with the wait times after approval and don’t see it as an issue (from a theme author’s perspective). However, better tools are always welcome. So, if Nacin and Otto come up with something that would make this less manual for the admins, I’d love to see that put into place.
Abhik 10:28 am on May 21, 2013

What about giving some additional permissions to veteran reviewers so that they can approve themes and make it live? That’ll off load little pressure from admins.
But let’s see ehat Andrew and Otto come up with.
Edward Caissie 2:46 pm on May 21, 2013

I’m all for a better workflow and more streamlined processes. Although some of these “changes” may require a bit of thought in how they are fully implemented I’m all for finding a better way to manage the “bug-fix” and “security issues” scenarios. Otherwise, most of the wait times are within reason for approved themes given the volunteer powered workforce doing all of the heavy lifting.
Rohit Tripathi 4:12 pm on October 29, 2013

A solution to this problem would be creating a class of reputed reviewers, who are experienced at reviewing themes like tskk or zGani, and along with admins, they have the right to review those themes.