Chip Bennett 5:14 pm on April 8, 2014
Tags: review incentive ( 10 ), review process ( 3 ), review queue ( 7 )

Selective Ticket Assignment and Ticket Hogging

Based on recent comments, I would like to clarify a couple points about ticket assignment.

Selective Ticket Assignment

First, when you assign a ticket to yourself, assign the first ticket in the queue. That means assign from tickets in Priority #1 queue, before assigning tickets from Priority queues 2, 3, or 4. Also, and more importantly: assign the first-available ticket in the queue. Do not cherry-pick tickets based on ease of review, reporter (i.e. Developer), etc.

Selectively assigning tickets is unfair to developers who are required to wait their turn in the review queue, and is a means of gaming the incentive program. Several reviewers have alleged that this practice takes place. We have not monitored it, but we will do so – something that will take even more time, and result in further delays in getting Themes approved and live. Anyone found to be selectively assigning tickets will risk being disqualified for the incentive program.

Ticket Hogging

Until now, we have not set limits on the number of open/assigned tickets a reviewer can have at any one time. Several reviewers have complained that often there are no tickets available, while some reviewers have several open/assigned tickets. So, going forward: reviewers may not have more than 5 open/assigned tickets at any one time.

In order to ensure that reviewers don’t get stuck with developer-abandoned tickets, admins will close tickets older than one week with no activity or developer response. For this reason, it is even more important not to “bump” tickets by posting comments requesting status updates.

Also, while this will be much more difficult to monitor: reviewers should only assign themselves one ticket at a time. Assign a ticket, conduct a full review, post comments, and only then assign another ticket.

Also, this clarification isn’t intended to introduce delays in developers getting their Themes reviewed. Any ticket that appears in the Priority #3 queue (tickets older than 2 weeks) are fair game to anyone. (But if the priority queue system is followed properly – see above – there should never be any tickets in the Priority #3 queue.)

Please discuss in the comments below.

#review-incentive, #review-process, #review-queue

Rohit Tripathi 5:30 pm on April 8, 2014

Although, I don’t disagree with any of the above, there is one thing which I would like to say in regard to Selective Assignment.

When, I review a ticket from a Developer say XYZ. It took me an hour, to perform a complete review and point out the mistakes. Now, another theme from the same developer is towards the bottom of the queue. I am obviously inclined to pick it up, as I have already reviewed the code. And chances are that 90% of the code is same in 2 different themes from the same developer. This will save my time, and others’ too as they won’t have to put in the same amount of effort which I had to put in on the first theme of XYZ.

Is this still an unhealthy practice?
- Chip Bennett 5:34 pm on April 8, 2014
  
  Yes, because all those tickets in between represent developers who have also patiently waited their turn to have their Theme reviewed.
  
  Also, it is healthy for more reviewers to be exposed to more Theme code from more Theme developers, because it helps make us all better at reviewing.
  - Rohit Tripathi 5:39 pm on April 8, 2014
    
    One more thing. Is it necessary for us to pick us from #1 first? There some reviewers dedicated towards Theme Updates like @tskk and @robin90 who aim for the incentive by reviewing updates, while others go for the new themes.
    - Emil Uzelac 6:20 pm on April 8, 2014
      
      Either one should work.
tskk 5:41 pm on April 8, 2014

If a ticket at top is way over my head, i have to wait till its picked up, tickets below have to wait unnecessarily.

Tickets are flying off the list, there won’t be much unjust to if we skip the one’s over my head.
- Chip Bennett 6:12 pm on April 8, 2014
  
  If you always skip tickets for more difficult Themes, one, you force other developers to review the Themes that take more effort; and two, you never benefit from learning from those more difficult Themes. If you need help with a ticket, ask. Everyone on the mail-list should be willing to jump in and help another reviewer who needs help with a complex Theme (especially when tickets are getting assigned so quickly that we have none in the queue). We are a team, and the expectation is that we will act like a team, and help one another like a team.
  - tskk 6:17 pm on April 8, 2014
    
    true, i did learn a lot reviewing themes. next step in learning it is….
- Frank Klein 6:17 pm on April 8, 2014
  
  I agree with Chip.
  
  If you go into the queue and look for small updates, you can do a huge number of tickets in a short amount of time. It’s simply not cool for the theme developers and for the other reviewers if you go in and only take the low hanging fruit, and leave the large and difficult to dissect reviews for others.
  
  At the same time, we should encurage developers to do smaller commits, instead of mashing together different features and bug fixes in one gigantic kitchen sinkKitchen Sink When using the WYSIWYG (What You See Is What You Get) editor in WordPress, you can expand the capabilities to allow more options. This expanded area is called the "Kitchen Sink." commit.
Emil Uzelac 5:43 pm on April 8, 2014

Additionally: (very much related)

Current program is to help community and in return TRT features your Theme. This is not a race or competition.

At the same time our program was not designed as a promotional tool from i.e. Theme Name to Theme Name Pro, but it has been used as such and we have no problems there.
robin90 5:48 pm on April 8, 2014

Thanks Chip for outlining all those. All the points that you have outlined above were already there and has been discussed many times except this one I think 5 open/assigned tickets. But reviewers seemed to neglect that and grab more tickets simultaneously. It’s sad that we have to outline each and every point. If reviewers would have been little more sensible than we wouldn’t have needed all this.
- Emil Uzelac 6:16 pm on April 8, 2014
  
  It will get better 🙂
  - robin90 7:51 pm on April 8, 2014
    
    let’s hope so 🙂
    - Catch Themes 10:52 am on April 10, 2014
      
      Yes lots of complain and we should remember that this is reward and we should appreciate the Admins efforts. I hope it will be better soon 🙂
ThinkUpThemes 7:29 pm on April 8, 2014

Thanks Chip for this! I totally agree that limiting tickets to 5 per reviewer really helps to give everyone a better chance at getting hold of a ticket and getting that valuable learning experience gained from reviewing. Quick question, if a theme is set to “approved”. Does it still count towards the 5 limit until it’s either made live / reopened / etc… by an admin?
- Chip Bennett 7:49 pm on April 8, 2014
  
  Once a ticket is “approved”, it is no longer open/assigned, it is closed/approved.
CyberChimps 8:23 pm on April 8, 2014

We are glad we’re addressing this now, but as long as the reopen ticket rule exists, the system can still easily be gamed.

A blog post on this topic doesn’t negate the fact that this practice lead to several of this months winners winning the contest, when completely legitimate reviews were discredited for erroneous reasons.

You guys rewarded those who cheated this month and punished legitimate reviews which disqualified two winners.

A blog post pointing out that people cheated while still honoring those who cheated and punishing legit reviews doesn’t undue the damage caused. This isn’t a fair contest.
- Chip Bennett 11:06 pm on April 8, 2014
  
  We are glad we’re addressing this now, but as long as the reopen ticket rule exists, the system can still easily be gamed.
  
  How so? Unless you’re suggesting that the Admins are attempting to game the system?
  
  A blog post on this topic doesn’t negate the fact that this practice lead to several of this months winners winning the contest, when completely legitimate reviews were discredited for erroneous reasons.
  
  Untrue. And for the third time: if you have issues with tickets that you believe were unopened unfairly, the place to raise that issue is in the specific ticket.
  
  You guys rewarded those who cheated this month and punished legitimate reviews which disqualified two winners.
  
  Again untrue. There is no evidence of “cheating”, and no reviews were “punished”.
  
  A blog post pointing out that people cheated while still honoring those who cheated and punishing legit reviews doesn’t undue the damage caused.
  
  Still untrue. There is no evidence of “cheating”, and no reviews were “punished”.
  
  This isn’t a fair contest.
  
  If you think the contest isn’t fair, you’re welcome not to participate.
  - CyberChimps 12:47 am on April 9, 2014
    
    If reviewers are selectively reviewing themes, and more complicated tickets are being left to the remaining reviewers the odds of the tickets being reopen is higher. Therefore the reviewers not selectively reviewing are at a greater risk of taking on tickets that may get reopened negating all of their review work. Which is exactly what just happened, and unfair.
    
    The tickets we had weren’t reopened unfairly, the rule of disqualifying those reviews is unfair.
    
    We’ve sent a list of the theme reviewers who were selectively reviewing to you personally to review as we would prefer not to present this data publicly, although it is clear upon reviewing Theme TracTrac Trac is the place where contributors create issues for bugs or feature requests much like GitHub.https://core.trac.wordpress.org/. who the offenders are.
    
    We consider disqualifying legitimate reviews as being punished because it lost us the contest this monthly unfairly, where as reviewers who selectively reviewed were rewarded and won the contest. This what we consider to be unfair.
    
    CyberChimps has been providing Theme Reviews for 3 years now long before the contest and will continue to contribute no matter how unfair the circumstances. We’ve sponsored many WordCamps, the Community Summit, and continue to do what we can to support the community. Our reputation in this community alone, as well as having the most popular theme on WordPress.orgWordPress.org The community site where WordPress code is created and shared by the users. This is where you can download the source code for WordPress core, plugins and themes as well as the central location for community conversations and organization. https://wordpress.org/ and in the world should be more then enough to back up our credibility on such matters.
    - tskk 6:58 am on April 9, 2014
      
      So you are saying people should be punished for not following rules of future while you yourself are breaking the rules of the present? you have 13 open tickets right now and your employee’s are still assigning tickets to them self, while the current limit is 5. You should be disqualified from this month’s competition as you have more tickets than the limit.
      - CyberChimps 1:07 am on April 10, 2014
        
        We took on those tickets before the new rule.
        
        We’re going to review all of those tickets, we could easily be doing 15-20 reviews a month. We have several new people being trained to do reviews.
        
        Like CyberChimps, you have been one of the few reviewers not selectively reviewing. Thank you for that.
    - tskk 9:20 am on April 9, 2014
      
      I personally don’t think all these rules are unnecessary, program was working for months and certain company will complain no matter what, as long as they don’t get their theme featured.
      
      Not assigning next ticket without posting the full review is enough.
      
      All this is too much work for admins and will only result in program getting cancelled.
      - tskk 9:29 am on April 9, 2014
        
        *necessary
      - Catch Themes 10:50 am on April 10, 2014
        
        +1 to @tskk. Thanks.
ThinkUpThemes 8:54 am on April 9, 2014

Hi All,

I wanted to share some thought on the cap on open/allocated tickets and hear the thoughts of others. Although the need of a cap, of sorts, is a great idea. I worry that this new cap although will reduce the risk of ticket hogging (which affected reviewers), it will however introduce a new and greater risk of ticket churning (which will affect developers).

Consider a scenario where a reviewer has 5 allocated tickets, all of which have been reviewed, and on which a developer response is required. Reviewers might:

1. Rush a review to the approval stage simply to free up a slot to allocate a new review.
2. Rush to disapprove a theme purely because it may take longer to review or require more support to be provided to the developer.

In addition, inexperienced developers with a passion to learn may not get the support they need as reviewers simply may not be inclined to keep tickets open.

Also the risk of selective reviews will increase, however the point at which the risk arises will simply shift to a later stage of the review cycle. Rather than being at the ticket allocation stage (which personally I don’t think happens) the selective review risk will shift to when a ticket is now allocated. Essentially reviewers will ‘cherry pick’ by choosing to decline complicated themes that may take a while to review in the hope that the next theme they choose will be more simple.

Another worry is that the move of closing tickets after 1 week is a move to keep reviewers happy so that 1 of the valuable 5 allocations isn’t unnecessarily filled by an unresponsive developer. I feel a little uneasy about changes which shift the focus of the review process from benefiting developers to benefiting reviewers. The previous rule of admins following up on a ticket after 1 week, and them closing after 2 (if no response is received) is great for developers as it gives them a chance to take action. Disapproving tickets for lack of response after 1 week, especially if the developer hasn’t been warned, might lead to them bring disheartened with our process and discouraged from resubmitting in future.

Just an initial thought, could there be a way to impose a cap for example, reviewers cannot allocate more than 5 tickets unless all allocated tickets have been fully reviewed. I know this makes it harder to monitor and more time for admins. So to discourage poor reviewer behaviour, I was thinking that a penalty can be imposed whereby any tickets in addition to this 5 will not count towards the incentive program if an admin feels a “good” attempt has not been made in reviewing their existing tickets. Obviously the definition of “good” here is subjective and would be purely be based on the admins view of what is considered “good”.

Anyway, thanks so much to anyone who’s taken the time to read this. I’d love to hear your thoughts.

Afzaal
- Chip Bennett 1:02 pm on April 9, 2014
  
  Note that both of these are counter-productive (and the latter is contrary to our current workflow):
  
  1. Rush a review to the approval stage simply to free up a slot to allocate a new review.
  2. Rush to disapprove a theme purely because it may take longer to review or require more support to be provided to the developer.
  
  Neither of these would result in a ticket being made approved and live, and so wouldn’t count toward the incentive. And premature ticket closure is contrary to our workflow.
  
  Another worry is that the move of closing tickets after 1 week is a move to keep reviewers happy so that 1 of the valuable 5 allocations isn’t unnecessarily filled by an unresponsive developer. I feel a little uneasy about changes which shift the focus of the review process from benefiting developers to benefiting reviewers. The previous rule of admins following up on a ticket after 1 week, and them closing after 2 (if no response is received) is great for developers as it gives them a chance to take action. Disapproving tickets for lack of response after 1 week, especially if the developer hasn’t been warned, might lead to them bring disheartened with our process and discouraged from resubmitting in future.
  
  This is a valid concern. I would be fine with saying “no more than 5 open/assigned tickets less than a week old“, and then Admin-closing after two weeks of no Developer response. But, I think that a week is more than enough time for a developer to provide some kind of response in-ticket. Let’s see how it goes as-is, and we’ll adjust as necessary.
  
  Just an initial thought, could there be a way to impose a cap for example, reviewers cannot allocate more than 5 tickets unless all allocated tickets have been fully reviewed. I know this makes it harder to monitor and more time for admins. So to discourage poor reviewer behaviour, I was thinking that a penalty can be imposed whereby any tickets in addition to this 5 will not count towards the incentive program if an admin feels a “good” attempt has not been made in reviewing their existing tickets. Obviously the definition of “good” here is subjective and would be purely be based on the admins view of what is considered “good”.
  
  My bigger concern – and the concern that could be the downfall of the entire incentive program – is that reviewers are performing every action and basing every decision on how those actions/decisions impact their ability to win the incentive program. If we have reached that point, then the program has, in fact, failed. If reviewers are expending more effort on gaming the incentive program than they are expending effort on contributing to WordPress via Theme reviews, then we’ve failed. If Admins are expending more effort running and managing the review incentive program than we are expending effort on final-approving Themes/pushing Themes live, finding ways to make the review process easier for developers and reviewers, and helping to educate about Theme development best practices, then we’ve failed.
  
  That we’re getting this much into the weeds with the incentive program is a pretty major indication that it’s becoming more of a detriment than a benefit – which is a shame, because it’s been instrumental in more than doubling the rate of new Themes being made live in the Theme Directory.
  - ThinkUpThemes 1:19 pm on April 9, 2014
    
    Thanks Chip! I really appreciate your comments! 🙂
PageLines 9:40 am on April 10, 2014

Hey Chip! Thank you for the comments and additional guidelines, we’ll do our best to stick with them.

We do have one question, though, related to the cap:

What if we have 5 reviews, all completed, but waiting for dev answer? This means that we cannot assign any more reviews until those have been finalised, but devs often take a few days to come back and upload the new version.

We should be able to assign further tickets as long as the review is started and posted immediately after the ticket assignment, and *before* any further tickets are assigned, right?

Thank you for your time.
- Chip Bennett 12:14 pm on April 10, 2014
  
  What if we have 5 reviews, all completed, but waiting for dev answer? This means that we cannot assign any more reviews until those have been finalised, but devs often take a few days to come back and upload the new version.
  
  I’ve been informally monitoring the number of assigned tickets per reviewer for a while. It is very rare for more than a handful of reviewers to have more than 5 open/assigned tickets at any one time. It is also very rare for any reviewers to have more than 20 closed/live tickets in any one month. So, any real impact here is going to be on the extreme edges, based on current review activity.
  
  If the 5-ticket cap starts causing issues with the queue, we’ll revisit it.
  
  We should be able to assign further tickets as long as the review is started and posted immediately after the ticket assignment, and *before* any further tickets are assigned, right?
  
  Of course. Just make sure the first review is thorough and complete, leave review comments, and then feel free to move on to the next ticket.
  - tskk 12:25 pm on April 10, 2014
    
    “Of course. Just make sure the first review is thorough and complete, leave review comments, and then feel free to move on to the next ticket.”
    
    He is talking about further tickets beyond the cap of 5.
    Can you consider increasing the limit to 10?
    - Chip Bennett 1:56 pm on April 10, 2014
      
      Can you consider increasing the limit to 10?
      
      Not at this time. We’ll reevaluate as we move forward, but as I said: based on informal monitoring of assigned tickets, it is extremely rare for any one reviewer to have more than 5 tickets open/assigned at any one time.
      
      Also: the more tickets open/assigned, the more work it is for a given reviewer to perform follow-up actions when developers respond: actions such as responding to questions, providing guidance for resolving required issues, and reviewing submitted updates.
      
      I’d rather see a reviewer put more time into a single review, and do one review a day, than to race through multiple reviews a day. And the current throughput and reviewer activity level doesn’t really indicate a need for any single reviewer to have more than 5 open/assigned tickets at one time.
Catch Themes 10:49 am on April 10, 2014

Thanks Chip and the Admin Team for highlighting the points so that we have clear winner 🙂
ThinkUpThemes 1:09 pm on April 11, 2014

Hi Chip,

In relation to the cap I just wanted to seek clarification on one of your points:

I would be fine with saying “no more than 5 open/assigned tickets less than a week old“

So am I correct in thinking that if a ticket has been open for more than 5 days then a reviewer can assign a new ticket? So essentially the 5 cap applies only to themes opened less than 7 days ago, thereby eliminating the issue of hogging?

Thanks as always!
- Chip Bennett 1:36 pm on April 11, 2014
  
  For now, please stick to the 5-ticket cap. Admins will close tickets that have no response for longer than a week.
tskk 1:26 pm on April 15, 2014

@chipbennett how do we notify selective assignment if spotted?
Or is that something you will take care of yourself?
- Chip Bennett 1:28 pm on April 15, 2014
  
  If you find someone selectively assigning tickets, email me directly with the details, and I’ll address it.
Ulrich 3:55 pm on April 18, 2014

This is my suggestion to the admins.

Reward the top 10 reviewers by letting them choose a theme that will be featured. This should help reduce the amount of competition. We have ten spaces then why not use them all.

Every theme should be reviewed by second theme reviewer. We can use the workflow keyword “2nd opinion” to mark the themes are ready to be approved but need to be reviewed by another reviewer.

The report can be seen here.
https://themes.trac.wordpress.org/query?keywords=~2nd-opinion&col=id&col=summary&col=status&col=owner&col=type&col=priority&col=time&col=changetime&report=5&order=priority

This should reduce the work for the admins and cultivate team work. This will reduce the risk of a theme not being reopened due to human error or not knowing something. This should also help standardize the theme reviews.
Emil Uzelac 9:53 pm on April 18, 2014

FYI: https://make.wordpress.org/themes/2014/04/18/theme-review-incentive-program/