WordPress.org

Ready to get started?Download WordPress

Review WordPress Themes

Tagged: review queue Toggle Comment Threads | Keyboard Shortcuts

  • Chip Bennett 5:14 pm on April 8, 2014 Permalink
    Tags: , , review queue   

    Selective Ticket Assignment and Ticket Hogging 

    Based on recent comments, I would like to clarify a couple points about ticket assignment.

    Selective Ticket Assignment

    First, when you assign a ticket to yourself, assign the first ticket in the queue. That means assign from tickets in Priority #1 queue, before assigning tickets from Priority queues 2, 3, or 4. Also, and more importantly: assign the first-available ticket in the queue. Do not cherry-pick tickets based on ease of review, reporter (i.e. Developer), etc.

    Selectively assigning tickets is unfair to developers who are required to wait their turn in the review queue, and is a means of gaming the incentive program. Several reviewers have alleged that this practice takes place. We have not monitored it, but we will do so – something that will take even more time, and result in further delays in getting Themes approved and live. Anyone found to be selectively assigning tickets will risk being disqualified for the incentive program.

    Ticket Hogging

    Until now, we have not set limits on the number of open/assigned tickets a reviewer can have at any one time. Several reviewers have complained that often there are no tickets available, while some reviewers have several open/assigned tickets. So, going forward: reviewers may not have more than 5 open/assigned tickets at any one time.

    In order to ensure that reviewers don’t get stuck with developer-abandoned tickets, admins will close tickets older than one week with no activity or developer response. For this reason, it is even more important not to “bump” tickets by posting comments requesting status updates.

    Also, while this will be much more difficult to monitor: reviewers should only assign themselves one ticket at a time. Assign a ticket, conduct a full review, post comments, and only then assign another ticket.

    Also, this clarification isn’t intended to introduce delays in developers getting their Themes reviewed. Any ticket that appears in the Priority #3 queue (tickets older than 2 weeks) are fair game to anyone. (But if the priority queue system is followed properly – see above – there should never be any tickets in the Priority #3 queue.)

    Please discuss in the comments below.

     
    • Rohit Tripathi 5:30 pm on April 8, 2014 Permalink

      Although, I don’t disagree with any of the above, there is one thing which I would like to say in regard to Selective Assignment.

      When, I review a ticket from a Developer say XYZ. It took me an hour, to perform a complete review and point out the mistakes. Now, another theme from the same developer is towards the bottom of the queue. I am obviously inclined to pick it up, as I have already reviewed the code. And chances are that 90% of the code is same in 2 different themes from the same developer. This will save my time, and others’ too as they won’t have to put in the same amount of effort which I had to put in on the first theme of XYZ.

      Is this still an unhealthy practice?

      • Chip Bennett 5:34 pm on April 8, 2014 Permalink

        Yes, because all those tickets in between represent developers who have also patiently waited their turn to have their Theme reviewed.

        Also, it is healthy for more reviewers to be exposed to more Theme code from more Theme developers, because it helps make us all better at reviewing.

        • Rohit Tripathi 5:39 pm on April 8, 2014 Permalink

          One more thing. Is it necessary for us to pick us from #1 first? There some reviewers dedicated towards Theme Updates like @tskk and @robin90 who aim for the incentive by reviewing updates, while others go for the new themes.

    • tskk 5:41 pm on April 8, 2014 Permalink

      If a ticket at top is way over my head, i have to wait till its picked up, tickets below have to wait unnecessarily.

      Tickets are flying off the list, there won’t be much unjust to if we skip the one’s over my head.

      • Chip Bennett 6:12 pm on April 8, 2014 Permalink

        If you always skip tickets for more difficult Themes, one, you force other developers to review the Themes that take more effort; and two, you never benefit from learning from those more difficult Themes. If you need help with a ticket, ask. Everyone on the mail-list should be willing to jump in and help another reviewer who needs help with a complex Theme (especially when tickets are getting assigned so quickly that we have none in the queue). We are a team, and the expectation is that we will act like a team, and help one another like a team.

        • tskk 6:17 pm on April 8, 2014 Permalink

          true, i did learn a lot reviewing themes. next step in learning it is….

      • Frank Klein 6:17 pm on April 8, 2014 Permalink

        I agree with Chip.

        If you go into the queue and look for small updates, you can do a huge number of tickets in a short amount of time. It’s simply not cool for the theme developers and for the other reviewers if you go in and only take the low hanging fruit, and leave the large and difficult to dissect reviews for others.

        At the same time, we should encurage developers to do smaller commits, instead of mashing together different features and bug fixes in one gigantic kitchen sink commit.

    • Emil Uzelac 5:43 pm on April 8, 2014 Permalink

      Additionally: (very much related)

      Current program is to help community and in return TRT features your Theme. This is not a race or competition.

      At the same time our program was not designed as a promotional tool from i.e. Theme Name to Theme Name Pro, but it has been used as such and we have no problems there.

    • robin90 5:48 pm on April 8, 2014 Permalink

      Thanks Chip for outlining all those. All the points that you have outlined above were already there and has been discussed many times except this one I think 5 open/assigned tickets. But reviewers seemed to neglect that and grab more tickets simultaneously. It’s sad that we have to outline each and every point. If reviewers would have been little more sensible than we wouldn’t have needed all this.

      • Emil Uzelac 6:16 pm on April 8, 2014 Permalink

        It will get better :)

        • robin90 7:51 pm on April 8, 2014 Permalink

          let’s hope so :)

          • Catch Themes 10:52 am on April 10, 2014 Permalink

            Yes lots of complain and we should remember that this is reward and we should appreciate the Admins efforts. I hope it will be better soon :)

    • ThinkUpThemes 7:29 pm on April 8, 2014 Permalink

      Thanks Chip for this! I totally agree that limiting tickets to 5 per reviewer really helps to give everyone a better chance at getting hold of a ticket and getting that valuable learning experience gained from reviewing. Quick question, if a theme is set to “approved”. Does it still count towards the 5 limit until it’s either made live / reopened / etc… by an admin?

      • Chip Bennett 7:49 pm on April 8, 2014 Permalink

        Once a ticket is “approved”, it is no longer open/assigned, it is closed/approved.

    • CyberChimps 8:23 pm on April 8, 2014 Permalink

      We are glad we’re addressing this now, but as long as the reopen ticket rule exists, the system can still easily be gamed.

      A blog post on this topic doesn’t negate the fact that this practice lead to several of this months winners winning the contest, when completely legitimate reviews were discredited for erroneous reasons.

      You guys rewarded those who cheated this month and punished legitimate reviews which disqualified two winners.

      A blog post pointing out that people cheated while still honoring those who cheated and punishing legit reviews doesn’t undue the damage caused. This isn’t a fair contest.

      • Chip Bennett 11:06 pm on April 8, 2014 Permalink

        We are glad we’re addressing this now, but as long as the reopen ticket rule exists, the system can still easily be gamed.

        How so? Unless you’re suggesting that the Admins are attempting to game the system?

        A blog post on this topic doesn’t negate the fact that this practice lead to several of this months winners winning the contest, when completely legitimate reviews were discredited for erroneous reasons.

        Untrue. And for the third time: if you have issues with tickets that you believe were unopened unfairly, the place to raise that issue is in the specific ticket.

        You guys rewarded those who cheated this month and punished legitimate reviews which disqualified two winners.

        Again untrue. There is no evidence of “cheating”, and no reviews were “punished”.

        A blog post pointing out that people cheated while still honoring those who cheated and punishing legit reviews doesn’t undue the damage caused.

        Still untrue. There is no evidence of “cheating”, and no reviews were “punished”.

        This isn’t a fair contest.

        If you think the contest isn’t fair, you’re welcome not to participate.

        • CyberChimps 12:47 am on April 9, 2014 Permalink

          If reviewers are selectively reviewing themes, and more complicated tickets are being left to the remaining reviewers the odds of the tickets being reopen is higher. Therefore the reviewers not selectively reviewing are at a greater risk of taking on tickets that may get reopened negating all of their review work. Which is exactly what just happened, and unfair.

          The tickets we had weren’t reopened unfairly, the rule of disqualifying those reviews is unfair.

          We’ve sent a list of the theme reviewers who were selectively reviewing to you personally to review as we would prefer not to present this data publicly, although it is clear upon reviewing Theme Trac who the offenders are.

          We consider disqualifying legitimate reviews as being punished because it lost us the contest this monthly unfairly, where as reviewers who selectively reviewed were rewarded and won the contest. This what we consider to be unfair.

          CyberChimps has been providing Theme Reviews for 3 years now long before the contest and will continue to contribute no matter how unfair the circumstances. We’ve sponsored many WordCamps, the Community Summit, and continue to do what we can to support the community. Our reputation in this community alone, as well as having the most popular theme on WordPress.org and in the world should be more then enough to back up our credibility on such matters.

          • tskk 6:58 am on April 9, 2014 Permalink

            So you are saying people should be punished for not following rules of future while you yourself are breaking the rules of the present? you have 13 open tickets right now and your employee’s are still assigning tickets to them self, while the current limit is 5. You should be disqualified from this month’s competition as you have more tickets than the limit.

            • CyberChimps 1:07 am on April 10, 2014 Permalink

              We took on those tickets before the new rule.

              We’re going to review all of those tickets, we could easily be doing 15-20 reviews a month. We have several new people being trained to do reviews.

              Like CyberChimps, you have been one of the few reviewers not selectively reviewing. Thank you for that.

          • tskk 9:20 am on April 9, 2014 Permalink

            I personally don’t think all these rules are unnecessary, program was working for months and certain company will complain no matter what, as long as they don’t get their theme featured.

            Not assigning next ticket without posting the full review is enough.

            All this is too much work for admins and will only result in program getting cancelled.

    • ThinkUpThemes 8:54 am on April 9, 2014 Permalink

      Hi All,

      I wanted to share some thought on the cap on open/allocated tickets and hear the thoughts of others. Although the need of a cap, of sorts, is a great idea. I worry that this new cap although will reduce the risk of ticket hogging (which affected reviewers), it will however introduce a new and greater risk of ticket churning (which will affect developers).

      Consider a scenario where a reviewer has 5 allocated tickets, all of which have been reviewed, and on which a developer response is required. Reviewers might:

      1. Rush a review to the approval stage simply to free up a slot to allocate a new review.
      2. Rush to disapprove a theme purely because it may take longer to review or require more support to be provided to the developer.

      In addition, inexperienced developers with a passion to learn may not get the support they need as reviewers simply may not be inclined to keep tickets open.

      Also the risk of selective reviews will increase, however the point at which the risk arises will simply shift to a later stage of the review cycle. Rather than being at the ticket allocation stage (which personally I don’t think happens) the selective review risk will shift to when a ticket is now allocated. Essentially reviewers will ‘cherry pick’ by choosing to decline complicated themes that may take a while to review in the hope that the next theme they choose will be more simple.

      Another worry is that the move of closing tickets after 1 week is a move to keep reviewers happy so that 1 of the valuable 5 allocations isn’t unnecessarily filled by an unresponsive developer. I feel a little uneasy about changes which shift the focus of the review process from benefiting developers to benefiting reviewers. The previous rule of admins following up on a ticket after 1 week, and them closing after 2 (if no response is received) is great for developers as it gives them a chance to take action. Disapproving tickets for lack of response after 1 week, especially if the developer hasn’t been warned, might lead to them bring disheartened with our process and discouraged from resubmitting in future.

      Just an initial thought, could there be a way to impose a cap for example, reviewers cannot allocate more than 5 tickets unless all allocated tickets have been fully reviewed. I know this makes it harder to monitor and more time for admins. So to discourage poor reviewer behaviour, I was thinking that a penalty can be imposed whereby any tickets in addition to this 5 will not count towards the incentive program if an admin feels a “good” attempt has not been made in reviewing their existing tickets. Obviously the definition of “good” here is subjective and would be purely be based on the admins view of what is considered “good”.

      Anyway, thanks so much to anyone who’s taken the time to read this. I’d love to hear your thoughts.

      Afzaal

      • Chip Bennett 1:02 pm on April 9, 2014 Permalink

        Note that both of these are counter-productive (and the latter is contrary to our current workflow):

        1. Rush a review to the approval stage simply to free up a slot to allocate a new review.
        2. Rush to disapprove a theme purely because it may take longer to review or require more support to be provided to the developer.

        Neither of these would result in a ticket being made approved and live, and so wouldn’t count toward the incentive. And premature ticket closure is contrary to our workflow.

        Another worry is that the move of closing tickets after 1 week is a move to keep reviewers happy so that 1 of the valuable 5 allocations isn’t unnecessarily filled by an unresponsive developer. I feel a little uneasy about changes which shift the focus of the review process from benefiting developers to benefiting reviewers. The previous rule of admins following up on a ticket after 1 week, and them closing after 2 (if no response is received) is great for developers as it gives them a chance to take action. Disapproving tickets for lack of response after 1 week, especially if the developer hasn’t been warned, might lead to them bring disheartened with our process and discouraged from resubmitting in future.

        This is a valid concern. I would be fine with saying “no more than 5 open/assigned tickets less than a week old“, and then Admin-closing after two weeks of no Developer response. But, I think that a week is more than enough time for a developer to provide some kind of response in-ticket. Let’s see how it goes as-is, and we’ll adjust as necessary.

        Just an initial thought, could there be a way to impose a cap for example, reviewers cannot allocate more than 5 tickets unless all allocated tickets have been fully reviewed. I know this makes it harder to monitor and more time for admins. So to discourage poor reviewer behaviour, I was thinking that a penalty can be imposed whereby any tickets in addition to this 5 will not count towards the incentive program if an admin feels a “good” attempt has not been made in reviewing their existing tickets. Obviously the definition of “good” here is subjective and would be purely be based on the admins view of what is considered “good”.

        My bigger concern – and the concern that could be the downfall of the entire incentive program – is that reviewers are performing every action and basing every decision on how those actions/decisions impact their ability to win the incentive program. If we have reached that point, then the program has, in fact, failed. If reviewers are expending more effort on gaming the incentive program than they are expending effort on contributing to WordPress via Theme reviews, then we’ve failed. If Admins are expending more effort running and managing the review incentive program than we are expending effort on final-approving Themes/pushing Themes live, finding ways to make the review process easier for developers and reviewers, and helping to educate about Theme development best practices, then we’ve failed.

        That we’re getting this much into the weeds with the incentive program is a pretty major indication that it’s becoming more of a detriment than a benefit – which is a shame, because it’s been instrumental in more than doubling the rate of new Themes being made live in the Theme Directory.

    • PageLines 9:40 am on April 10, 2014 Permalink

      Hey Chip! Thank you for the comments and additional guidelines, we’ll do our best to stick with them.

      We do have one question, though, related to the cap:

      What if we have 5 reviews, all completed, but waiting for dev answer? This means that we cannot assign any more reviews until those have been finalised, but devs often take a few days to come back and upload the new version.

      We should be able to assign further tickets as long as the review is started and posted immediately after the ticket assignment, and *before* any further tickets are assigned, right?

      Thank you for your time.

      • Chip Bennett 12:14 pm on April 10, 2014 Permalink

        What if we have 5 reviews, all completed, but waiting for dev answer? This means that we cannot assign any more reviews until those have been finalised, but devs often take a few days to come back and upload the new version.

        I’ve been informally monitoring the number of assigned tickets per reviewer for a while. It is very rare for more than a handful of reviewers to have more than 5 open/assigned tickets at any one time. It is also very rare for any reviewers to have more than 20 closed/live tickets in any one month. So, any real impact here is going to be on the extreme edges, based on current review activity.

        If the 5-ticket cap starts causing issues with the queue, we’ll revisit it.

        We should be able to assign further tickets as long as the review is started and posted immediately after the ticket assignment, and *before* any further tickets are assigned, right?

        Of course. Just make sure the first review is thorough and complete, leave review comments, and then feel free to move on to the next ticket.

        • tskk 12:25 pm on April 10, 2014 Permalink

          “Of course. Just make sure the first review is thorough and complete, leave review comments, and then feel free to move on to the next ticket.”

          He is talking about further tickets beyond the cap of 5.
          Can you consider increasing the limit to 10?

          • Chip Bennett 1:56 pm on April 10, 2014 Permalink

            Can you consider increasing the limit to 10?

            Not at this time. We’ll reevaluate as we move forward, but as I said: based on informal monitoring of assigned tickets, it is extremely rare for any one reviewer to have more than 5 tickets open/assigned at any one time.

            Also: the more tickets open/assigned, the more work it is for a given reviewer to perform follow-up actions when developers respond: actions such as responding to questions, providing guidance for resolving required issues, and reviewing submitted updates.

            I’d rather see a reviewer put more time into a single review, and do one review a day, than to race through multiple reviews a day. And the current throughput and reviewer activity level doesn’t really indicate a need for any single reviewer to have more than 5 open/assigned tickets at one time.

    • Catch Themes 10:49 am on April 10, 2014 Permalink

      Thanks Chip and the Admin Team for highlighting the points so that we have clear winner :)

    • ThinkUpThemes 1:09 pm on April 11, 2014 Permalink

      Hi Chip,

      In relation to the cap I just wanted to seek clarification on one of your points:

      I would be fine with saying “no more than 5 open/assigned tickets less than a week old“

      So am I correct in thinking that if a ticket has been open for more than 5 days then a reviewer can assign a new ticket? So essentially the 5 cap applies only to themes opened less than 7 days ago, thereby eliminating the issue of hogging?

      Thanks as always!

      • Chip Bennett 1:36 pm on April 11, 2014 Permalink

        For now, please stick to the 5-ticket cap. Admins will close tickets that have no response for longer than a week.

    • tskk 1:26 pm on April 15, 2014 Permalink

      @chipbennett how do we notify selective assignment if spotted?
      Or is that something you will take care of yourself?

      • Chip Bennett 1:28 pm on April 15, 2014 Permalink

        If you find someone selectively assigning tickets, email me directly with the details, and I’ll address it.

    • Ulrich 3:55 pm on April 18, 2014 Permalink

      This is my suggestion to the admins.

      Reward the top 10 reviewers by letting them choose a theme that will be featured. This should help reduce the amount of competition. We have ten spaces then why not use them all.

      Every theme should be reviewed by second theme reviewer. We can use the workflow keyword “2nd opinion” to mark the themes are ready to be approved but need to be reviewed by another reviewer.

      The report can be seen here.
      https://themes.trac.wordpress.org/query?keywords=~2nd-opinion&col=id&col=summary&col=status&col=owner&col=type&col=priority&col=time&col=changetime&report=5&order=priority

      This should reduce the work for the admins and cultivate team work. This will reduce the risk of a theme not being reopened due to human error or not knowing something. This should also help standardize the theme reviews.

    • Emil Uzelac 9:53 pm on April 18, 2014 Permalink

  • Chip Bennett 10:39 pm on July 16, 2013 Permalink
    Tags: review queue   

    Review Incentive Trial Program 

    With the recent changes to the Theme Review infrastructure, we are now much more easily able to track Theme review and approval of new Themes, previously reviewed Themes, and updates to currently approved Themes.  While we have a much higher volume of tickets being closed than we have had in the past, we end up mired in a cycle of Themes being not-approved, re-submitted, and then not-approved again. If we are ever to gain long-term control of the review queue, we need to break that cycle – and the only real way to do that is to ensure that, as often as possible, all Themes submitted eventually get through the review process, and approved.

    In an effort not only to bring the Theme Review queue under control but also to further our ultimate objective: to ensure Themes are approved and made available in the Theme directory, we are going to try an incentive program of sorts. As a side benefit, this incentive program will partially address another area of frequent discussion: the Featured Themes list.

    This Trac Report tracks new and previously reviewed Themes that have been approved and made live in the past 30 days. It will be the basis for our experimental incentive program.

    Every month, we will determine the three reviewers with the most tickets in this report. Each of those reviewers will get to select a Theme to be included in the Featured Themes list for the next month.

    Details:

    • “Winners” will be selected on the first of the month
    • To count, tickets must be for never-before-approved Themes (i.e. new and previously-reviewed, but not theme-update)
    • To count, tickets must not only be “approved” by the reviewer, but closed and made “live” by an admin
    • Selected Featured Themes must be currently approved in the Theme directory, and must be current
      • (Yes, you can choose one of your own Themes)
    • Final decisions are at the sole discretion of the TRT admins
    • The incentive program is intended to be positive; if it becomes contentious or causes problems, it will be discontinued
    • Additional details/clarification will be added, as needed

    Again, the goals of this program are to help bring the review queue under long-term control, to facilitate more Themes completing the review process through approval, and to provide some community input into the Featured Themes list, while providing a bit of incentive to Theme reviewers.

    Please discuss in the comments. Let us know what you think of the idea, or if you have any questions.

    The first “winners” will be selected on August 1st. Good luck!

     
    • Emil Uzelac 10:42 pm on July 16, 2013 Permalink

      Great incentive! Good luck to all.

    • tskk 10:43 pm on July 16, 2013 Permalink

      Maybe add already approved tickets too but at maybe 1/3rd value of new tickets?

    • tskk 10:49 pm on July 16, 2013 Permalink

      Will the 3 new featured themes be in addition to 11 or will some of the current themes be removed?

      • Chip Bennett 10:52 pm on July 16, 2013 Permalink

        Not sure yet. We’re not limited to 11, but we may not want almost 15. We’ll probably play that one by ear.

    • Harish Chouhan 10:51 pm on July 16, 2013 Permalink

      This is awesome. Great for those who have their own themes and also volunteer on a regular basis.

      • Konstantin Obenland 10:54 pm on July 16, 2013 Permalink

        Ha! I didn’t even think of that. Not sure that is intended though.

        • Chip Bennett 11:03 pm on July 16, 2013 Permalink

          I kind of assumed that people would want to choose one of their own Themes. ;)

    • Bryan Hadaway 10:57 pm on July 16, 2013 Permalink

      Very smart idea. It’s always good to add a little incentive for volunteer-based activities. Some friendly competition is always great to get things fired up too.

      Could go a long way to appreciate those reviewers that help out most, that may sometimes go under-appreciated. I know @tskk is a reviewing machine. :)

      • tskk 11:03 pm on July 16, 2013 Permalink

        But my fav queue is not eligible :)
        I better make friends with Zulfikar Nore :)

        • Zulfikar Nore 11:38 pm on July 16, 2013 Permalink

          lol, at the rate I’m going if I keep up I’ll probably run out of my own themes to put forward for consideration – so yeah come on over and lets be friends :)

          At the same token I think there are a 101 themes to pick from and suggest for inclusion on the featured list, I’ll end up being spoiled for choice.

          • tskk 11:44 pm on July 16, 2013 Permalink

            Ah already made a list :)

      • Emil Uzelac 4:39 am on July 17, 2013 Permalink

        @tskk was already rewarded last month ;) And this will make him even better now.

        • tskk 4:49 am on July 17, 2013 Permalink

          Thank you for featuring Alexandria :)

    • Stephen 10:58 pm on July 16, 2013 Permalink

      “only approved”? It take same effort to “approve” or “disapprove” theme. What I disapprove a theme and author never submit a new version? I am worried about we are encourage wrong behavior. At the end, we add more responsibility for admins.

      • Bryan Hadaway 10:59 pm on July 16, 2013 Permalink

        I think the key focus though is that reviewers work with theme designers more to help them get it approved.

        • Zulfikar Nore 11:40 pm on July 16, 2013 Permalink

          +1 that idea. I see to many themes being shot down at first opportunity to Not Approve when a little guiding hand could have solved the issue(s) and the theme approved.

      • Chip Bennett 11:03 pm on July 16, 2013 Permalink

        Yes, this incentive is specifically to encourage getting Themes all the way through the review process, to approval. It was really the only way to balance quantity with quality.

        • Stephen 11:10 pm on July 16, 2013 Permalink

          Understand the intention. Admins will have more responsibilities as gate keepers.

        • tskk 11:42 pm on July 16, 2013 Permalink

          But if the theme author doesn’t follow up, its not our fault and that ticket should be counted too. It sounds fair to me in my head :)

          • Chip Bennett 11:50 pm on July 16, 2013 Permalink

            There’s simply no fair way to include not-approved Themes. How would we reasonably be able to differentiate between a sincere, full review that results in “not-approved”, and a “find-a-single-problem-and-close-as-not-approved” review? It’s just not practical, unfortunately.

            Also, this particular objective isn’t just about reducing the queue, it is about getting Themes approved – and that objective is intentional.

            I would suggest not closing tickets right away. Leave comments – including comments asking the developer to respond, leave the ticket open, and move on. Follow up on it a week later. Take advantage of the recent infrastructure changes, since by leaving the ticket open, any subsequent submissions will be appended to the open ticket. Let developers know that, as long as they are working with you and the ticket stays open, submitted updates will be appended to the open ticket, and the Theme will retain its place in the queue.

            I suspect that you’ll quickly get a feel for differentiating between the tickets submitted by developers who will follow up and work with you, and tickets submitted by developers who will abandon the effort.

            • tskk 11:59 pm on July 16, 2013 Permalink

              Yes, I meant in case they simply don’t respond with updates after doing all the above you suggested i.e. putting a good faith effort to get the theme approved, but if there are too many of those, i guess you will do right by us anyway….

            • Zulfikar Nore 12:00 am on July 17, 2013 Permalink

              Agreed.

              I tend to note the ticket with something like “Keeping ticket open for the next 48 or 2days” and more often than note I get a response by either a new version uploaded or a simple “Working on that”.

              That way I know the author is dealing with the issue and move on to the next ticket while checking back so often to see which author has made the effort and continue with their review.

              I try not to close a theme unless it has major required issues that I know will take time to put right.

            • tskk 4:25 am on July 17, 2013 Permalink

              Also quick partial reviews was a policy to try and get rid of spammers etc. We tried that and now we try this :)

      • Bryan Hadaway 11:10 pm on July 16, 2013 Permalink

        There are only 1,828 approved themes in the repo. There must be countless others abandoned or who decided just to host themselves. I’d love to see the ratio of submitted : approved themes.

        A few years back I had a pretty tough time getting themes approved. Was pretty intimidated by the whole process and frankly, only made it through with reviewers who really did a lot of hand-holding to get me there. If I didn’t have such a desire to get in the repo, I would have been gone.

        If I would have known about the Theme-Check plugin at the time, would have saved me a lot of grief. A great way to get themes approved would probably be if you can tell they’re new to the review process because of the amount of commonly overlooked issues their theme presents, referring them to http://wordpress.org/plugins/theme-check/ will go along way.

        • Stephen 11:13 pm on July 16, 2013 Permalink

          First theme is always tough. On other hand, this process do help reviewer differentiate “required” and “recommended”. Personally, I never ask author to do things that are “recommended” only. Most authors do not even know the difference.

          • Bryan Hadaway 11:20 pm on July 16, 2013 Permalink

            Very good point. I still run into that problem today, overzealous reviewers who fail to differentiate Recommended and Required and simply list everything which implies it’s all required.

            If I didn’t know any better, if I were a new theme designer trying to get approved, I would spend a lot of time on all issues presented assuming they’re all required.

            It’s good to try and push best practices, but we all need to remember too that “you just have to get 1.0 out there”.

            I appreciate reviewers who make suggestions for the next version as apposed to going straight to DEFCON 1 over a small issue.

            • poena 2:28 am on July 17, 2013 Permalink

              It takes time to get there, I hadnt reviewed anything for a year when I started again and its only by reading alot of other reviews and also reviewing themes made by reviewers that I hopefully will find a middle way. Its not as easy as required or recommended.

    • Zulfikar Nore 12:12 am on July 17, 2013 Permalink

      Awesome idea Chip :)

      “approved” by the reviewer, but closed and made “live” – Approved and Live are the keywords! Makes the whole idea and system that much fair as a “Live” theme is a true measure that it was a good (Meet requirements) Approval.

      Having had 2 of my reviews and “Approved” themes re-opened goes to show that just the “Approval” alone as the measuring factor is not enough.

      Thumbs up alround.

    • JoshPollock 1:23 am on July 17, 2013 Permalink

      Big 5 thumbs up for this idea. I hope this will cut down on the number of quick partial reviews, with no offer to re-review. I also hope it will motivate me to pick up my pace with reviews a bit. I’ve been slacking since non-admins stopped reviewing Priority #1 queue, as the ration of “interesting interactions with talented developers” to “irritating interactions with people that have no understanding of the GPL and are just here to game the system with their totally unoriginal theme” has taken a fairly unhealthy swing towards the latter.

      • Chip Bennett 1:27 am on July 17, 2013 Permalink

        By the way, the Priority #1 queue thing was never actually made policy. We discussed it, but the feedback was mixed, so we never made a decision. Feel free to continue to review Priority #1 tickets if you want. :)

    • Kakoma 4:30 am on July 17, 2013 Permalink

      I wish you knew just how much I approve of this incentive. I’ve submitted a theme since the beginning of this year and each time it gets rejected for (and this is very subjective) very small reasons that I actually fix in minutes….but I have to wait 5 more weeks only to be rejected for something even smaller. 7 months and still counting.
      I’ll keep reviewing other submitted themes; thing is you grow with each review. The sad thing is sometimes your eye for detail is blurred when reviewing your own theme prior to submission

    • stuartwider 7:26 am on July 17, 2013 Permalink

      This new Theme nomination superpower sounds excellent.

      It could also be an opportunity to maybe add some extra fun / excitement into the mix for both reviewers and theme designers. :)

      I’d suggest some additional theme nominations be added, which would be then carefully considered for inclusion by TRT admins when compiling their featured theme list for the month.

      • The ‘Hey Theme Designer – Job well done!’ nomination

      Top reviewers nominate a theme from the list of themes they have reviewed that month (because they want to publicly acknowledge the excellent job the theme designer has done)

      • The ‘Undiscovered Gem’ nomination

      Top reviewers nominate a theme which has never made it onto the featured list before (maybe to help unearth some excellent but as yet little known themes).

  • Chip Bennett 1:56 pm on March 4, 2013 Permalink
    Tags: review queue   

    Follow-Up on End-of-February Review Queue Push 

    Last week, I mentioned that the review queue had reached about 150, and suggested that we get together for an end-of-week/end-of-February push to clear the queue.

    So, how did we do?

    As it currently stands, the review queue is at approximately 50 tickets, with 260 tickets closed in the past 7 days, by 28 reviewers. Special thanks to @gpriday, @nishasingh, and @life.object, who each closed more than 10 tickets each; and @slobodanmanic, @mercime, @labor4it, @jcastaneda, @Frank_Klein, and @Fingli, who each closed at least 5 tickets each.

    (Note: you can now see real-time stats for the past 7 days, here.)

    As it now stands, we have completely cleared the Priority #2 queue, which means that we have no unassigned tickets older than two weeks. Fantastic job on that front as well, since any tickets that wind up in that queue represent developers who have been waiting at least 2 weeks just to get feedback on their Theme. (Ideally, we will never have any tickets fall into the Priority #2 queue.)

    But even better: we now only have 15 unassigned tickets older than one week. While I see the Priority #2 queue as an indication that we’re really not getting the job done, I would love to make it a goal to keep the number of Themes older than one week at zero as well. Getting to the point where the team can turn around all tickets in a week or less, on a regular basis, would be a huge improvement, and would, I hope, keep the developers happy who submit Themes.

    So, let’s see if we can get that queue of 50 Themes knocked out (we last saw review queue “Inbox zero” over a year ago, IIRC), and then see if, from a fresh start, we can keep the number of tickets open longer than a week at zero.

    As always, thanks everyone for your contributions. We know that this is a volunteer activity, and that contributing to the Theme Review Team takes away from valuable time spent elsewhere.

     
    • Greg Priday 2:09 pm on March 4, 2013 Permalink

      Great work everyone! It was amazing to see the #2 queue just melt away. We should make these month-end pushes a regular thing.

    • Nisha Singh 6:59 am on March 5, 2013 Permalink

      Thanks Chip for appreciation and your Support.

  • Chip Bennett 9:31 pm on February 26, 2013 Permalink
    Tags: review queue   

    Who’s up for a push to clear out the queue?

    We’ll call it the “end of February” push, but we can use the weekend, too.

    There are almost 150 tickets awaiting review.

    We have about 90 people with close-ticket privileges.

    That’s less than one and a half tickets per reviewer, by the end of the weekend.

    Who will commit to reviewing 1-2 tickets between now and Sunday?

     
    • Chip Bennett 9:32 pm on February 26, 2013 Permalink

      I’m in, for at least one per day, between today and Saturday.

    • Greg Priday 9:36 pm on February 26, 2013 Permalink

      I need a bit of a break after my push this weekend. Hoping to be back on it by this weekend. It’d be great to get the queue down to ~1 week… and keep it there.

      I’m in.

    • jr.duboc 9:39 pm on February 26, 2013 Permalink

      Hi all,
      I haven’t been active here for a long time due to studies and things, but I can do one, maybe two, before the WE.
      Will get going on it tomorrow night.

    • Frank Klein 10:07 pm on February 26, 2013 Permalink

      I’m in, I got some time this Saturday. When is the precise deadline?

      • Chip Bennett 11:18 pm on February 26, 2013 Permalink

        There is no “precise deadline”. Theme review is an entirely volunteer-led activity. Sometimes, we will try to organize a concentrated group effort, if the queue starts getting too long in the tooth.

        Any time you’re able to contribute, we welcome the help. :)

    • Stephen 10:13 pm on February 26, 2013 Permalink

      Pickup one just now.

    • fingli 10:22 pm on February 26, 2013 Permalink

      Count me in

    • JSallette 12:21 am on February 27, 2013 Permalink

      I would like to review one this weekend. This will be my first review. I look forward to helping make it happen.
      J-

      • Chip Bennett 12:33 am on February 27, 2013 Permalink

        Great! We’ll be glad to have you. Just post a comment on the review queue request, with your Trac username, and we’ll assign a ticket to you!

    • Rizqy Hidayat 12:57 am on February 27, 2013 Permalink

      just picking up one. maybe one or two this week :)

    • SriniG 4:02 am on February 27, 2013 Permalink

      I’m in. Let’s clear this up.

    • Slobodan Manic 6:04 am on February 27, 2013 Permalink

      I’ll try to do a few by the end of the week as well.

    • Tikendra Maitry 6:07 am on February 27, 2013 Permalink

      I am also in. :)

    • Qamar 6:19 am on February 27, 2013 Permalink

      I will review 5-8 tickets from now to Sunday and let count me in.

    • Sanjiv 6:38 am on February 27, 2013 Permalink

      I am in. Nice to see all people joining in. :)

    • Jose Castaneda 2:41 pm on February 27, 2013 Permalink

      Count me in. I’ll try to get one or two done before the weekend.

    • priyanshu mittal 5:56 am on February 28, 2013 Permalink

      Hey love to review one or two username priyanshu.mittal

    • Emil Uzelac 11:35 am on February 28, 2013 Permalink

      you got it brother!

  • Lance Willett 11:59 pm on August 24, 2012 Permalink | Log in to leave a Comment
    Tags: review queue,   

    Hi everyone. Are you ready for a new default theme? I am. And, now it’s almost ready.

    I submitted a .9 release of Twenty Twelve today—see http://themes.trac.wordpress.org/ticket/9199. Theme Check had a few warnings, I noted the reasoning for those in the Trac ticket notes.

    If you have some time this weekend could you go through it? We’ve been cranking on it in core a ton and now it’s time for spit and polish, tightening up documentation, and making sure we covered all the bases.

    Note for themes Trac moderators: This theme should not be pushed live after it’s approved, per instructions from the core development team.

     
    • mercime 12:10 am on August 25, 2012 Permalink | Log in to Reply

      So we should be testing Twenty Twelve on WP 3.4.1 and/or WP 3.5 trunk?

    • Japh 2:17 am on August 25, 2012 Permalink | Log in to Reply

      Brilliant, Lance! Very much looking forward to this new theme :)

    • Chantal Coolsma 7:29 am on August 27, 2012 Permalink | Log in to Reply

      I love it. Already found an issue.

    • Lance Willett 2:55 pm on August 28, 2012 Permalink | Log in to Reply

      Has everyone had a chance to take a look and test?

      Today we’re pushing the theme live on WordPress.com and in the announcement it’d be nice to be able to link to Extend for any self-hosted folks who want to try it out before 3.5 officially comes out.

    • Lance Willett 3:25 pm on August 28, 2012 Permalink | Log in to Reply

      Update: after discussion with Nacin and Matt some more here’s the game plan for releasing to WP.org Extend, soon-ish (say 2-3 weeks).

      1. WPTRT continues to review it and test it, then an admin there approves the theme without pushing it live so it has gone through a round of theme review. Keeping it version .9 as we find bugs and fix them.
      2. Come up with a RC version, say .9.x — Lance will keep submitting to Themes Trac with new test candidates
      3. Nacin will handle letting the core contributor group know, via http://make.wordpress.org/core/ site that we’d like to do a formal launch very soon
      4. Then dot the “i”s and cross the “t”s and make sure it is ready for a final WP.org release.

      At that point we’ll submit a new ZIP with the 1.0 final and that one can be pushed live for everyone, with a possible announcement on WP.org news blog at that point (exact details TBD on how to announce).

  • Edward Caissie 8:26 pm on February 29, 2012 Permalink | Log in to leave a Comment
    Tags: review queue,   

    Another Review-In: March 3, 2012 

    The community clamors for more and who are we to say no. That being said, we will be having another review-in on Saturday March 3, 2012.

    Brings your coffee, your bacon, and your wits because as they say “two outta three ain’t bad” … so lets have some fun and take a run at the Theme Review Queues.

    … and don’t forget to log into the IRC channel #wordpress-themes, too.

     
  • Chip Bennett 1:48 am on December 15, 2011 Permalink | Log in to leave a Comment
    Tags: , review queue   

    Getting the Review Queue Back on Track 

    As I write this post, Theme-Trac indicates the following:

    Welcome to WordPress Themes Trac

    There are 161 new tickets waiting for review. 24 themes were reviewed in the last 7 days.

    Although right now is the holiday season here in the US, and most of us have been busy getting ready for the just-released WordPress 3.3, that review queue number is extraordinarily high. That number represents a lot of developers awaiting feedback on their Themes, and a lot of end users not benefiting from new and/or updated Themes. Let’s see what we can do to address it!

    Revising the Trainee Reviewer Experiment

    First things first: the Theme Review Team has been experimenting with a training program for new reviewers. After a few months, it appears that this experiment isn’t working out as intended. New reviewers aren’t getting assigned tickets as quickly as possible, and full reviewers don’t seem to have enough time to follow up on those reviews, provide feedback, and resolve/close the tickets. While we want to provide training for new reviewers, so that we equip our volunteers with the tools and skills necessary to complete Theme reviews, we can’t let that training get in the way of completing Theme reviews.

    So, effective immediately, we’re going to try something else: anyone who requests a ticket to review, and then completes their assigned review, will be given full “reviewer” status in Theme-Trac. That means that, once a new reviewer has requested and completed their first ticket, they will then be able to assign themselves tickets, and will be able to resolve/close tickets on their own. An admin reviewer will double-check tickets resolved as “approved”, but otherwise, once you’ve done one review, you’ll be free to review without the burden of any additional oversight.

    Where previously we have been slow to “promote” to full reviewer status, and exceedingly slow to remove reviewer privileges, now we will look to be very quick in granting reviewer privileges, and perhaps somewhat quicker than we were previously with removing reviewer privileges, either due to inactivity or poor reviews (mainly, approving Themes that should not be approved).

    Revising the Queue Priorities

    Second: for a considerably longer period, the Theme Review Team has used a three-tiered priority approach to the review queue:

    1. Priority #1: Currently Approved Themes
    2. Priority #2: Previously Reviewed, Not-Approved Themes
    3. Priority #3: Never-Reviewed Themes

    This three-tiered approach has worked fairly well, especially for Themes that have successfully passed the review process, and are currently approved. However, the Theme Review Team has been unable to keep the Priority #2 queue cleared, meaning that new Themes end up waiting weeks (or longer) to get even an initial review.

    So, effective immediately, we’re adding a fourth tier to the prioritization, which will become the new #2 priority: tickets that have been in the review queue for longer than two weeks, regardless of previous review/approval status. The new prioritization will be as follows:

    1. Priority #1: Currently Approved Themes
    2. Priority #2: Tickets Older Than 2 Weeks
    3. Priority #3: Previously Reviewed, Not-Approved Themes
    4. Priority #4: Never-Reviewed Themes

    Hopefully with this change, the oldest tickets will be reviewed in a more timely manner. Our long-term goal will be that this new Priority #2 queue will be – and stay – empty; but for now, it will help ensure that tickets don’t stay in the queue for weeks on end.

    Revising Handling of Review-Based Theme Revisions

    Currently, once a Theme is reviewed, if the developer revises the Theme to address issues from the review, and then re-submits the Theme, the re-submitted Theme goes to the end of the Previously-Reviewed Themes queue. This process does not encourage or facilitate such review-based Theme revisions. Understandably – especially with the state of the review queue – Theme developers may be discouraged by the process, and may choose never to submit a revision. This outcome helps no one: end users don’t benefit from an approved Theme becoming available, Theme developers don’t benefit from having their Theme hosted in the repository, and the Theme Review Team expends time and effort reviewing a Theme that never ultimately gets approved.

    For some time, the Theme Review Team has had an informal policy, subject to the discretion of each reviewer, of allowing a review to be continued on a subsequent ticket, if a revision is submitted in a timely manner. This informal policy has facilitated prompt Theme revision submissions, and has led to more Themes eventually passing Theme review, and being approved.

    So, effective immediately, we’re formalizing this policy: any review-based Theme revision that is submitted within two days of the previous review will be assigned to the previous-ticket reviewer, and the review continued on the new ticket. Bear in mind: exercise of this policy will still be at the discretion of each reviewer, and should be considered to be a privilege, based on a good-faith effort on the part of the Theme developer. While most Theme developers will exercise such good-faith effort, I want to make clear that this policy is not a license to use the review process as quality control.

    Revising Review Emphasis

    The Theme Review Guidelines currently include a requirement for W3C HTML/CSS validation. It has become clear that this requirement is largely a distraction, both to developers and to reviewers. From the beginning, the W3C Validation requirement was intended to be a holistic tool used in the review process, rather than a rigid requirement; however, all too often, reviews have emphasized W3C Validation results – even at the expense of far more important guideline requirements.

    So, effective immediately, W3C Validation criticality is being reduced from REQUIRED to RECOMMENDED, and Theme reviewers will no longer make review comments regarding W3C Validation. Theme developers can/should use W3C Validation as a development tool, but it will no longer be part of the review process.

    Also, the Theme Review Team has made an effort to ensure that every review is thorough and complete, in order to reduce the number of times any given Theme must go through the submission/review process prior to approval. However, it is clear that this emphasis on thorough reviews has acted to the detriment of performing reviews expeditiously. What was intended to be a service to Theme developers has actually led to more developer frustration, as the review queue continues to grow.

    So, effective immediately, the Theme Review Team will no longer emphasize complete and thorough reviews, and will instead close tickets upon observation of any non-trivial issues. Theme developers have always been responsible for knowing and following the Theme Review guidelines. The key take-away on this point: if you have a question, ASK. As much as we would like to walk every developer through the review/approval process, we simply don’t have enough volunteers to spare the time. If you are unsure about a review guideline, or need clarification on a review comment, ASK. Ask on the theme-reviewers mail-list before you submit your Theme. Ask in the review ticket after you submit. We are here to help developers, but will need to rely more on developers to communicate your questions and concerns, so that we can focus more on reviewing/closing tickets.

    Summarizing The Changes

    Again, here’s what we’re going to be doing differently, effective immediately, to help facilitate Theme reviews:

    • Anyone who requests a ticket to review, and then completes their assigned review, will be given full “reviewer” status in Theme-Trac
    • We’re adding a fourth tier to the prioritization, which will become the new #2 priority: tickets that have been in the review queue for longer than two weeks, regardless of previous review/approval status
    • Any review-based Theme revision that is submitted within two days of the previous review will be assigned to the previous-ticket reviewer, and the review continued on the new ticket
    • W3C Validation criticality is being reduced from REQUIRED to RECOMMENDED, and Theme reviewers will no longer make review comments regarding W3C Validation
    • The Theme Review Team will no longer emphasize complete and thorough reviews, and will instead close tickets upon observation of any non-trivial issues

    I would also like to add a gentle reminder that the Theme Review Team is a 100% volunteer effort. None of the Theme reviewers are paid for our efforts. For almost all of us, Theme reviews take time away from other development work (whether paid or free) – time that we are happy to contribute. That said: while we understand your frustration in waiting for your Theme review to be completed, I would like to make a request: before you email the theme-reviewers mail-list asking when your Theme will be reviewed, post a comment to the current Trac Ticket Review Request Queue, and ask to be assigned a ticket of your own to review. This request is not a pay-for-play scheme, and no tickets or developers are given special treatment for participating in Theme reviews; rather, the more people we have reviewing Themes, the faster the review queue will get processed. (And as a bonus: you’ll learn more than you might otherwise imagine about what is required to pass the review process successfully.)

    Remember: in the end, our entire effort is all about ensuring that WordPress end users have the best-possible Themes available for use.

    If you have any other suggestions for how we can improve the process, please let us know – either in the comments, or via the theme-reviewers mail list.

     
c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel