Tag Archives: review queue

Chip Bennett 5:14 pm on April 8, 2014
Tags: , , review queue   

Selective Ticket Assignment and Ticket Hogging

Based on recent comments, I would like to clarify a couple points about ticket assignment.

Selective Ticket Assignment

First, when you assign a ticket to yourself, assign the first ticket in the queue. That means assign from tickets in Priority #1 queue, before assigning tickets from Priority queues 2, 3, or 4. Also, and more importantly: assign the first-available ticket in the queue. Do not cherry-pick tickets based on ease of review, reporter (i.e. Developer), etc.

Selectively assigning tickets is unfair to developers who are required to wait their turn in the review queue, and is a means of gaming the incentive program. Several reviewers have alleged that this practice takes place. We have not monitored it, but we will do so – something that will take even more time, and result in further delays in getting Themes approved and live. Anyone found to be selectively assigning tickets will risk being disqualified for the incentive program.

Ticket Hogging

Until now, we have not set limits on the number of open/assigned tickets a reviewer can have at any one time. Several reviewers have complained that often there are no tickets available, while some reviewers have several open/assigned tickets. So, going forward: reviewers may not have more than 5 open/assigned tickets at any one time.

In order to ensure that reviewers don’t get stuck with developer-abandoned tickets, admins will close tickets older than one week with no activity or developer response. For this reason, it is even more important not to “bump” tickets by posting comments requesting status updates.

Also, while this will be much more difficult to monitor: reviewers should only assign themselves one ticket at a time. Assign a ticket, conduct a full review, post comments, and only then assign another ticket.

Also, this clarification isn’t intended to introduce delays in developers getting their Themes reviewed. Any ticket that appears in the Priority #3 queue (tickets older than 2 weeks) are fair game to anyone. (But if the priority queue system is followed properly – see above – there should never be any tickets in the Priority #3 queue.)

Please discuss in the comments below.

#review-incentive, #review-process, #review-queue

Chip Bennett 10:39 pm on July 16, 2013
Tags: review queue   

Review Incentive Trial Program

With the recent changes to the Theme Review infrastructure, we are now much more easily able to track Theme review and approval of new Themes, previously reviewed Themes, and updates to currently approved Themes.  While we have a much higher volume of tickets being closed than we have had in the past, we end up mired in a cycle of Themes being not-approved, re-submitted, and then not-approved again. If we are ever to gain long-term control of the review queue, we need to break that cycle – and the only real way to do that is to ensure that, as often as possible, all Themes submitted eventually get through the review process, and approved.

In an effort not only to bring the Theme Review queue under control but also to further our ultimate objective: to ensure Themes are approved and made available in the Theme directory, we are going to try an incentive program of sorts. As a side benefit, this incentive program will partially address another area of frequent discussion: the Featured Themes list.

This Trac Report tracks new and previously reviewed Themes that have been approved and made live in the past 30 days. It will be the basis for our experimental incentive program.

Every month, we will determine the three reviewers with the most tickets in this report. Each of those reviewers will get to select a Theme to be included in the Featured Themes list for the next month.

Details:

  • “Winners” will be selected on the first of the month
  • To count, tickets must be for never-before-approved Themes (i.e. new and previously-reviewed, but not theme-update)
  • To count, tickets must not only be “approved” by the reviewer, but closed and made “live” by an admin
  • Selected Featured Themes must be currently approved in the Theme directory, and must be current
    • (Yes, you can choose one of your own Themes)
  • Final decisions are at the sole discretion of the TRT admins
  • The incentive program is intended to be positive; if it becomes contentious or causes problems, it will be discontinued
  • Additional details/clarification will be added, as needed

Again, the goals of this program are to help bring the review queue under long-term control, to facilitate more Themes completing the review process through approval, and to provide some community input into the Featured Themes list, while providing a bit of incentive to Theme reviewers.

Please discuss in the comments. Let us know what you think of the idea, or if you have any questions.

The first “winners” will be selected on August 1st. Good luck!

#review-queue

Chip Bennett 1:56 pm on March 4, 2013
Tags: review queue   

Follow-Up on End-of-February Review Queue Push

Last week, I mentioned that the review queue had reached about 150, and suggested that we get together for an end-of-week/end-of-February push to clear the queue.

So, how did we do?

As it currently stands, the review queue is at approximately 50 tickets, with 260 tickets closed in the past 7 days, by 28 reviewers. Special thanks to @gpriday, @nishasingh, and @life.object, who each closed more than 10 tickets each; and @slobodanmanic, @mercime, @labor4it, @jcastaneda, @Frank_Klein, and @Fingli, who each closed at least 5 tickets each.

(Note: you can now see real-time stats for the past 7 days, here.)

As it now stands, we have completely cleared the Priority #2 queue, which means that we have no unassigned tickets older than two weeks. Fantastic job on that front as well, since any tickets that wind up in that queue represent developers who have been waiting at least 2 weeks just to get feedback on their Theme. (Ideally, we will never have any tickets fall into the Priority #2 queue.)

But even better: we now only have 15 unassigned tickets older than one week. While I see the Priority #2 queue as an indication that we’re really not getting the job done, I would love to make it a goal to keep the number of Themes older than one week at zero as well. Getting to the point where the team can turn around all tickets in a week or less, on a regular basis, would be a huge improvement, and would, I hope, keep the developers happy who submit Themes.

So, let’s see if we can get that queue of 50 Themes knocked out (we last saw review queue “Inbox zero” over a year ago, IIRC), and then see if, from a fresh start, we can keep the number of tickets open longer than a week at zero.

As always, thanks everyone for your contributions. We know that this is a volunteer activity, and that contributing to the Theme Review Team takes away from valuable time spent elsewhere.

#review-queue

Chip Bennett 9:31 pm on February 26, 2013
Tags: review queue   

End-of-February Queue Push

Who’s up for a push to clear out the queue?

We’ll call it the “end of February” push, but we can use the weekend, too.

There are almost 150 tickets awaiting review.

We have about 90 people with close-ticket privileges.

That’s less than one and a half tickets per reviewer, by the end of the weekend.

Who will commit to reviewing 1-2 tickets between now and Sunday?

#review-queue

Lance Willett 11:59 pm on August 24, 2012
Tags: review queue,   

Hi everyone Are you ready for a new…

Hi everyone. Are you ready for a new default theme? I am. And, now it’s almost ready.

I submitted a .9 release of Twenty Twelve today—see https://themes.trac.wordpress.org/ticket/9199. Theme Check had a few warnings, I noted the reasoning for those in the Trac ticket notes.

If you have some time this weekend could you go through it? We’ve been cranking on it in coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. a ton and now it’s time for spit and polish, tightening up documentation, and making sure we covered all the bases.

Note for themes TracTrac Trac is the place where contributors create issues for bugs or feature requests much like GitHub.https://core.trac.wordpress.org/. moderators: This theme should not be pushed live after it’s approved, per instructions from the core development team.

#review-queue, #twentytwelve