WordPress.org

Ready to get started?Download WordPress

Review WordPress Themes

Tagged: review process Toggle Comment Threads | Keyboard Shortcuts

  • Chip Bennett 5:14 pm on April 8, 2014 Permalink | Log in to leave a Comment
    Tags: , review process,   

    Selective Ticket Assignment and Ticket Hogging 

    Based on recent comments, I would like to clarify a couple points about ticket assignment.

    Selective Ticket Assignment

    First, when you assign a ticket to yourself, assign the first ticket in the queue. That means assign from tickets in Priority #1 queue, before assigning tickets from Priority queues 2, 3, or 4. Also, and more importantly: assign the first-available ticket in the queue. Do not cherry-pick tickets based on ease of review, reporter (i.e. Developer), etc.

    Selectively assigning tickets is unfair to developers who are required to wait their turn in the review queue, and is a means of gaming the incentive program. Several reviewers have alleged that this practice takes place. We have not monitored it, but we will do so – something that will take even more time, and result in further delays in getting Themes approved and live. Anyone found to be selectively assigning tickets will risk being disqualified for the incentive program.

    Ticket Hogging

    Until now, we have not set limits on the number of open/assigned tickets a reviewer can have at any one time. Several reviewers have complained that often there are no tickets available, while some reviewers have several open/assigned tickets. So, going forward: reviewers may not have more than 5 open/assigned tickets at any one time.

    In order to ensure that reviewers don’t get stuck with developer-abandoned tickets, admins will close tickets older than one week with no activity or developer response. For this reason, it is even more important not to “bump” tickets by posting comments requesting status updates.

    Also, while this will be much more difficult to monitor: reviewers should only assign themselves one ticket at a time. Assign a ticket, conduct a full review, post comments, and only then assign another ticket.

    Also, this clarification isn’t intended to introduce delays in developers getting their Themes reviewed. Any ticket that appears in the Priority #3 queue (tickets older than 2 weeks) are fair game to anyone. (But if the priority queue system is followed properly – see above – there should never be any tickets in the Priority #3 queue.)

    Please discuss in the comments below.

     
    • Rohit Tripathi 5:30 pm on April 8, 2014 Permalink | Log in to Reply

      Although, I don’t disagree with any of the above, there is one thing which I would like to say in regard to Selective Assignment.

      When, I review a ticket from a Developer say XYZ. It took me an hour, to perform a complete review and point out the mistakes. Now, another theme from the same developer is towards the bottom of the queue. I am obviously inclined to pick it up, as I have already reviewed the code. And chances are that 90% of the code is same in 2 different themes from the same developer. This will save my time, and others’ too as they won’t have to put in the same amount of effort which I had to put in on the first theme of XYZ.

      Is this still an unhealthy practice?

      • Chip Bennett 5:34 pm on April 8, 2014 Permalink | Log in to Reply

        Yes, because all those tickets in between represent developers who have also patiently waited their turn to have their Theme reviewed.

        Also, it is healthy for more reviewers to be exposed to more Theme code from more Theme developers, because it helps make us all better at reviewing.

    • tskk 5:41 pm on April 8, 2014 Permalink | Log in to Reply

      If a ticket at top is way over my head, i have to wait till its picked up, tickets below have to wait unnecessarily.

      Tickets are flying off the list, there won’t be much unjust to if we skip the one’s over my head.

      • Chip Bennett 6:12 pm on April 8, 2014 Permalink | Log in to Reply

        If you always skip tickets for more difficult Themes, one, you force other developers to review the Themes that take more effort; and two, you never benefit from learning from those more difficult Themes. If you need help with a ticket, ask. Everyone on the mail-list should be willing to jump in and help another reviewer who needs help with a complex Theme (especially when tickets are getting assigned so quickly that we have none in the queue). We are a team, and the expectation is that we will act like a team, and help one another like a team.

      • Frank Klein 6:17 pm on April 8, 2014 Permalink | Log in to Reply

        I agree with Chip.

        If you go into the queue and look for small updates, you can do a huge number of tickets in a short amount of time. It’s simply not cool for the theme developers and for the other reviewers if you go in and only take the low hanging fruit, and leave the large and difficult to dissect reviews for others.

        At the same time, we should encurage developers to do smaller commits, instead of mashing together different features and bug fixes in one gigantic kitchen sink commit.

    • Emil Uzelac 5:43 pm on April 8, 2014 Permalink | Log in to Reply

      Additionally: (very much related)

      Current program is to help community and in return TRT features your Theme. This is not a race or competition.

      At the same time our program was not designed as a promotional tool from i.e. Theme Name to Theme Name Pro, but it has been used as such and we have no problems there.

    • robin90 5:48 pm on April 8, 2014 Permalink | Log in to Reply

      Thanks Chip for outlining all those. All the points that you have outlined above were already there and has been discussed many times except this one I think 5 open/assigned tickets. But reviewers seemed to neglect that and grab more tickets simultaneously. It’s sad that we have to outline each and every point. If reviewers would have been little more sensible than we wouldn’t have needed all this.

    • ThinkUpThemes 7:29 pm on April 8, 2014 Permalink | Log in to Reply

      Thanks Chip for this! I totally agree that limiting tickets to 5 per reviewer really helps to give everyone a better chance at getting hold of a ticket and getting that valuable learning experience gained from reviewing. Quick question, if a theme is set to “approved”. Does it still count towards the 5 limit until it’s either made live / reopened / etc… by an admin?

    • CyberChimps 8:23 pm on April 8, 2014 Permalink | Log in to Reply

      We are glad we’re addressing this now, but as long as the reopen ticket rule exists, the system can still easily be gamed.

      A blog post on this topic doesn’t negate the fact that this practice lead to several of this months winners winning the contest, when completely legitimate reviews were discredited for erroneous reasons.

      You guys rewarded those who cheated this month and punished legitimate reviews which disqualified two winners.

      A blog post pointing out that people cheated while still honoring those who cheated and punishing legit reviews doesn’t undue the damage caused. This isn’t a fair contest.

      • Chip Bennett 11:06 pm on April 8, 2014 Permalink | Log in to Reply

        We are glad we’re addressing this now, but as long as the reopen ticket rule exists, the system can still easily be gamed.

        How so? Unless you’re suggesting that the Admins are attempting to game the system?

        A blog post on this topic doesn’t negate the fact that this practice lead to several of this months winners winning the contest, when completely legitimate reviews were discredited for erroneous reasons.

        Untrue. And for the third time: if you have issues with tickets that you believe were unopened unfairly, the place to raise that issue is in the specific ticket.

        You guys rewarded those who cheated this month and punished legitimate reviews which disqualified two winners.

        Again untrue. There is no evidence of “cheating”, and no reviews were “punished”.

        A blog post pointing out that people cheated while still honoring those who cheated and punishing legit reviews doesn’t undue the damage caused.

        Still untrue. There is no evidence of “cheating”, and no reviews were “punished”.

        This isn’t a fair contest.

        If you think the contest isn’t fair, you’re welcome not to participate.

        • CyberChimps 12:47 am on April 9, 2014 Permalink | Log in to Reply

          If reviewers are selectively reviewing themes, and more complicated tickets are being left to the remaining reviewers the odds of the tickets being reopen is higher. Therefore the reviewers not selectively reviewing are at a greater risk of taking on tickets that may get reopened negating all of their review work. Which is exactly what just happened, and unfair.

          The tickets we had weren’t reopened unfairly, the rule of disqualifying those reviews is unfair.

          We’ve sent a list of the theme reviewers who were selectively reviewing to you personally to review as we would prefer not to present this data publicly, although it is clear upon reviewing Theme Trac who the offenders are.

          We consider disqualifying legitimate reviews as being punished because it lost us the contest this monthly unfairly, where as reviewers who selectively reviewed were rewarded and won the contest. This what we consider to be unfair.

          CyberChimps has been providing Theme Reviews for 3 years now long before the contest and will continue to contribute no matter how unfair the circumstances. We’ve sponsored many WordCamps, the Community Summit, and continue to do what we can to support the community. Our reputation in this community alone, as well as having the most popular theme on WordPress.org and in the world should be more then enough to back up our credibility on such matters.

          • tskk 6:58 am on April 9, 2014 Permalink | Log in to Reply

            So you are saying people should be punished for not following rules of future while you yourself are breaking the rules of the present? you have 13 open tickets right now and your employee’s are still assigning tickets to them self, while the current limit is 5. You should be disqualified from this month’s competition as you have more tickets than the limit.

            • CyberChimps 1:07 am on April 10, 2014 Permalink

              We took on those tickets before the new rule.

              We’re going to review all of those tickets, we could easily be doing 15-20 reviews a month. We have several new people being trained to do reviews.

              Like CyberChimps, you have been one of the few reviewers not selectively reviewing. Thank you for that.

          • tskk 9:20 am on April 9, 2014 Permalink | Log in to Reply

            I personally don’t think all these rules are unnecessary, program was working for months and certain company will complain no matter what, as long as they don’t get their theme featured.

            Not assigning next ticket without posting the full review is enough.

            All this is too much work for admins and will only result in program getting cancelled.

    • ThinkUpThemes 8:54 am on April 9, 2014 Permalink | Log in to Reply

      Hi All,

      I wanted to share some thought on the cap on open/allocated tickets and hear the thoughts of others. Although the need of a cap, of sorts, is a great idea. I worry that this new cap although will reduce the risk of ticket hogging (which affected reviewers), it will however introduce a new and greater risk of ticket churning (which will affect developers).

      Consider a scenario where a reviewer has 5 allocated tickets, all of which have been reviewed, and on which a developer response is required. Reviewers might:

      1. Rush a review to the approval stage simply to free up a slot to allocate a new review.
      2. Rush to disapprove a theme purely because it may take longer to review or require more support to be provided to the developer.

      In addition, inexperienced developers with a passion to learn may not get the support they need as reviewers simply may not be inclined to keep tickets open.

      Also the risk of selective reviews will increase, however the point at which the risk arises will simply shift to a later stage of the review cycle. Rather than being at the ticket allocation stage (which personally I don’t think happens) the selective review risk will shift to when a ticket is now allocated. Essentially reviewers will ‘cherry pick’ by choosing to decline complicated themes that may take a while to review in the hope that the next theme they choose will be more simple.

      Another worry is that the move of closing tickets after 1 week is a move to keep reviewers happy so that 1 of the valuable 5 allocations isn’t unnecessarily filled by an unresponsive developer. I feel a little uneasy about changes which shift the focus of the review process from benefiting developers to benefiting reviewers. The previous rule of admins following up on a ticket after 1 week, and them closing after 2 (if no response is received) is great for developers as it gives them a chance to take action. Disapproving tickets for lack of response after 1 week, especially if the developer hasn’t been warned, might lead to them bring disheartened with our process and discouraged from resubmitting in future.

      Just an initial thought, could there be a way to impose a cap for example, reviewers cannot allocate more than 5 tickets unless all allocated tickets have been fully reviewed. I know this makes it harder to monitor and more time for admins. So to discourage poor reviewer behaviour, I was thinking that a penalty can be imposed whereby any tickets in addition to this 5 will not count towards the incentive program if an admin feels a “good” attempt has not been made in reviewing their existing tickets. Obviously the definition of “good” here is subjective and would be purely be based on the admins view of what is considered “good”.

      Anyway, thanks so much to anyone who’s taken the time to read this. I’d love to hear your thoughts.

      Afzaal

      • Chip Bennett 1:02 pm on April 9, 2014 Permalink | Log in to Reply

        Note that both of these are counter-productive (and the latter is contrary to our current workflow):

        1. Rush a review to the approval stage simply to free up a slot to allocate a new review.
        2. Rush to disapprove a theme purely because it may take longer to review or require more support to be provided to the developer.

        Neither of these would result in a ticket being made approved and live, and so wouldn’t count toward the incentive. And premature ticket closure is contrary to our workflow.

        Another worry is that the move of closing tickets after 1 week is a move to keep reviewers happy so that 1 of the valuable 5 allocations isn’t unnecessarily filled by an unresponsive developer. I feel a little uneasy about changes which shift the focus of the review process from benefiting developers to benefiting reviewers. The previous rule of admins following up on a ticket after 1 week, and them closing after 2 (if no response is received) is great for developers as it gives them a chance to take action. Disapproving tickets for lack of response after 1 week, especially if the developer hasn’t been warned, might lead to them bring disheartened with our process and discouraged from resubmitting in future.

        This is a valid concern. I would be fine with saying “no more than 5 open/assigned tickets less than a week old“, and then Admin-closing after two weeks of no Developer response. But, I think that a week is more than enough time for a developer to provide some kind of response in-ticket. Let’s see how it goes as-is, and we’ll adjust as necessary.

        Just an initial thought, could there be a way to impose a cap for example, reviewers cannot allocate more than 5 tickets unless all allocated tickets have been fully reviewed. I know this makes it harder to monitor and more time for admins. So to discourage poor reviewer behaviour, I was thinking that a penalty can be imposed whereby any tickets in addition to this 5 will not count towards the incentive program if an admin feels a “good” attempt has not been made in reviewing their existing tickets. Obviously the definition of “good” here is subjective and would be purely be based on the admins view of what is considered “good”.

        My bigger concern – and the concern that could be the downfall of the entire incentive program – is that reviewers are performing every action and basing every decision on how those actions/decisions impact their ability to win the incentive program. If we have reached that point, then the program has, in fact, failed. If reviewers are expending more effort on gaming the incentive program than they are expending effort on contributing to WordPress via Theme reviews, then we’ve failed. If Admins are expending more effort running and managing the review incentive program than we are expending effort on final-approving Themes/pushing Themes live, finding ways to make the review process easier for developers and reviewers, and helping to educate about Theme development best practices, then we’ve failed.

        That we’re getting this much into the weeds with the incentive program is a pretty major indication that it’s becoming more of a detriment than a benefit – which is a shame, because it’s been instrumental in more than doubling the rate of new Themes being made live in the Theme Directory.

    • PageLines 9:40 am on April 10, 2014 Permalink | Log in to Reply

      Hey Chip! Thank you for the comments and additional guidelines, we’ll do our best to stick with them.

      We do have one question, though, related to the cap:

      What if we have 5 reviews, all completed, but waiting for dev answer? This means that we cannot assign any more reviews until those have been finalised, but devs often take a few days to come back and upload the new version.

      We should be able to assign further tickets as long as the review is started and posted immediately after the ticket assignment, and *before* any further tickets are assigned, right?

      Thank you for your time.

      • Chip Bennett 12:14 pm on April 10, 2014 Permalink | Log in to Reply

        What if we have 5 reviews, all completed, but waiting for dev answer? This means that we cannot assign any more reviews until those have been finalised, but devs often take a few days to come back and upload the new version.

        I’ve been informally monitoring the number of assigned tickets per reviewer for a while. It is very rare for more than a handful of reviewers to have more than 5 open/assigned tickets at any one time. It is also very rare for any reviewers to have more than 20 closed/live tickets in any one month. So, any real impact here is going to be on the extreme edges, based on current review activity.

        If the 5-ticket cap starts causing issues with the queue, we’ll revisit it.

        We should be able to assign further tickets as long as the review is started and posted immediately after the ticket assignment, and *before* any further tickets are assigned, right?

        Of course. Just make sure the first review is thorough and complete, leave review comments, and then feel free to move on to the next ticket.

        • tskk 12:25 pm on April 10, 2014 Permalink | Log in to Reply

          “Of course. Just make sure the first review is thorough and complete, leave review comments, and then feel free to move on to the next ticket.”

          He is talking about further tickets beyond the cap of 5.
          Can you consider increasing the limit to 10?

          • Chip Bennett 1:56 pm on April 10, 2014 Permalink | Log in to Reply

            Can you consider increasing the limit to 10?

            Not at this time. We’ll reevaluate as we move forward, but as I said: based on informal monitoring of assigned tickets, it is extremely rare for any one reviewer to have more than 5 tickets open/assigned at any one time.

            Also: the more tickets open/assigned, the more work it is for a given reviewer to perform follow-up actions when developers respond: actions such as responding to questions, providing guidance for resolving required issues, and reviewing submitted updates.

            I’d rather see a reviewer put more time into a single review, and do one review a day, than to race through multiple reviews a day. And the current throughput and reviewer activity level doesn’t really indicate a need for any single reviewer to have more than 5 open/assigned tickets at one time.

    • Catch Themes 10:49 am on April 10, 2014 Permalink | Log in to Reply

      Thanks Chip and the Admin Team for highlighting the points so that we have clear winner :)

    • ThinkUpThemes 1:09 pm on April 11, 2014 Permalink | Log in to Reply

      Hi Chip,

      In relation to the cap I just wanted to seek clarification on one of your points:

      I would be fine with saying “no more than 5 open/assigned tickets less than a week old“

      So am I correct in thinking that if a ticket has been open for more than 5 days then a reviewer can assign a new ticket? So essentially the 5 cap applies only to themes opened less than 7 days ago, thereby eliminating the issue of hogging?

      Thanks as always!

      • Chip Bennett 1:36 pm on April 11, 2014 Permalink | Log in to Reply

        For now, please stick to the 5-ticket cap. Admins will close tickets that have no response for longer than a week.

    • tskk 1:26 pm on April 15, 2014 Permalink | Log in to Reply

      @chipbennett how do we notify selective assignment if spotted?
      Or is that something you will take care of yourself?

  • Chip Bennett 8:05 pm on April 7, 2014 Permalink | Log in to leave a Comment
    Tags: , , review process   

    How To Ensure Your Ticket Passes Final Approval Audit 

    Over the past couple of months, the number of approved tickets that have been reopened due to issues found during final-approval audit has declined, but many still get reopened. As a team, we want to ensure that tickets get approved (so that new Themes get added to the directory, and into the hands of end users), and we want reviewers to be able to take advantage of the incentive program.

    So I want to step through the things I check when performing a final review audit. We’re looking for some high-level and/or high-impact things that would cause problems for end users:

    Overall File Structure

    • Does the Theme look like it is derived from a common Theme (Underscores, Twenty Ten-Fourteen, etc.)? Are there included functional files that I’ll need to check. Are there asset folders (fonts, scripts, etc.) that I’ll need to check?

    style.css

    • Check ThemeURI and AuthorURI. Are they appropriate?
    • If ThemeURI or AuthorURI reference commercial Themes, are those Themes sold as GPL-compatible?
    • If the Theme appears to be derived, does it include a proper derivative-work copyright/license attribution?

    readme.txt

    • If the Theme has bundled resources/assets, are they listed in the readme with copyright/license attribution, or will I need to check file headers?
    • Is license for all bundled resources GPL or GPL-compatible?

    header.php

    • Are Theme options properly escaped on output?
    • Is favicon, if used, disabled by default?
    • Does the TITLE tag include anything other than the call to wp_title()?
    • Does wp_nav_menu() reference theme_location, and not menu?
    • Are any stylesheet or script links output instead of being properly enqueued

    footer.php

    • Does the Theme only use one credit link? Is that credit link exactly ThemeURI or AuthorURI, with no SEO-seeding of link text, title attribute, etc.?
    • Are any footer scripts output instead of being enqueued properly?

    index.php

    • Does template markup look generally appropriate?

    Front/Home Page Templates

    • Does the Theme have front-page.php? If so, is it used properly? Does it account for both a static page and the blog posts index as site front page?
    • Does the Theme have home.php? If so, is it used properly as the default blog posts index template?
    • Does the Theme have any custom page templates intended to be used as either front-page.php or home.php?

    comments.php

    • Does the Theme properly output wp_list_comments() for the comments list?
    • Does the Theme properly output comment_form() for the comment reply form, rather than hard-coding the form?

    page.php

    • Does the page template properly call comments_template()?

    functions.php

    • Are all functions and other things in the public namespace properly prefixed?
    • Is all functional output properly wrapped in callbacks and hooked into appropriate actions?
    • Is any of the functionality Plugin territory?
    • Does any of the functionality replicate core functionality?
    • Does the Theme use Theme options? Are they handled properly (single DB entry, proper settings page, sanitized on input, etc.)?
    • Do any of the Theme options replicate core options?

    This list comprises 99% of what I look for in an audit, and accounts for the vast majority of issues encountered that require reopening tickets. So, if you verify these things before resolving the ticket as “approved”, the chances that your ticket will get reopened will go down considerably.

    (Note: @emiluzelac may have other things to add to the list, for things he checks during an audit.)

     
    • ruphel 9:16 pm on April 7, 2014 Permalink | Log in to Reply

      Thanks for the checklist Chip

    • Emil Uzelac 11:47 pm on April 7, 2014 Permalink | Log in to Reply

      My process is very similar to @chipbennett and there is nothing more to add.

      I would like to say that rushing through reviews will most definitely resolve getting your ticket reopened. Take your time and you will be golden.

      Your time and efforts are greatly appreciated!

      For some weird reason, my post got published 4 times, sorry about that.

    • ThemeZee 7:28 am on April 8, 2014 Permalink | Log in to Reply

      “Does the Theme have any custom page templates intended to be used as either front-page.php or home.php?”

      Does this mean we are not allowed to use for example template-homepage.php for a custom front page template, but have to use front-page.php instead ?

      • Justin Tadlock 3:10 pm on April 9, 2014 Permalink | Log in to Reply

        home.php and front-page.php have very specific meanings in WP. They should only be used when they’re meant to be used. template-example.php, even if intended to be used on the front page, can also be used if the template can be used with any page.

        Personally, I prefer that devs make a custom page template that’s usable on any page. It offers more flexibility for users.

        • Chip Bennett 3:22 pm on April 9, 2014 Permalink | Log in to Reply

          Intent is important here. If it’s obviously intended as the site front page template, then it should be `front-page.php`. If the developer doesn’t necessarily intend for it to be the site front page, then call it a “featured content” template (or something similar), and name it `template-featured.php` (or whatever).

    • Jose Castaneda 11:25 pm on April 8, 2014 Permalink | Log in to Reply

      One of the things I often look for are hooks and filters. I’ve seen a few themes that register meta boxes and some that add post types. It just depends on how they are being used ( though post types should really be avoided in themes ).

      One thing I’ve learned is using the command line to my advantage and using grep to find all the add_action/add_filter calls a theme uses.

    • PageLines 12:30 pm on April 9, 2014 Permalink | Log in to Reply

      Thanks for the checklist guys!

  • Emil Uzelac 2:23 am on November 1, 2012 Permalink | Log in to leave a Comment
    Tags: review process   

    How can we improve the Theme Review process? Please share your thoughts.

     
    • JarretC 5:01 am on November 1, 2012 Permalink | Log in to Reply

      It has been awhile since I last did a theme review but I remember that though links are provided check what to clean there isn’t a single page that lists everything that needs to be checked.

      I had to jump around between a few pages to make sure that everything was checked off and accounted for. Would make it a lot smoother if it was all on one page.

    • Himanshu 5:47 am on November 1, 2012 Permalink | Log in to Reply

      Hi Emil,

      I agree with Jarret. I was also theme reviewer in past and I always try to review themes now below are some key points I found while working in theme review process.

      1. if somebody submitting theme for review after 1-2 attempt it should get priority instead of going back to the beginning.

      2. Don’t de-prioritize existing themes in the queue upon newer submissions.

      3. we can initiate a user rating system where if somebody has very good reputation in theme repository we can approve their theme quickly by giving them priority.

      Thanks,
      Himanshu

      • Emil Uzelac 6:04 am on November 1, 2012 Permalink | Log in to Reply

        Hi Himanshu,

        Thanks for the comment.

        When Themes are submitted they will be in Priority #4 (New Themes, Never Reviewed) and if is not approved and upon second submission Theme goes to Priority #3 (Previously Reviewed, Not-Approved Themes), so they are not sent back.
        As far as I know, there’s no de-prioritization, do you mind giving us some examples?

        Reputation -vs newer submission is something I personally see as an unfair “rule”.
        Just because “author A” is more reputable than “author B” should not give “author A” more “rights” in the trac :)

        Emil

      • Chip Bennett 12:30 pm on November 1, 2012 Permalink | Log in to Reply

        Actually, if Theme developers leave a comment pointing to a newer ticket, then the new ticket will take the old ticket’s place in the queue. That’s always been in place. That said, here are some ideas I’ve been considering to propose:

        1) Have a reviewer follow a given Theme through the entire review process, from ticket to ticket.

        2) Have some way of establishing *trust-worthiness*, so that developers who earn the privilege would have their tickets auto-closed as approved (leaving only the final, admin review upon pushing live in Extend).

        • Konstantin Obenland 2:55 pm on November 1, 2012 Permalink | Log in to Reply

          I don’t know if I like having reviewers being dedicated to a theme. Sometimes a fresh pair of eyes can do wonders and I think various reviewers with their individual review-style improve the overall quality of the submitted theme.

          I like 2). Though I fear it will increase the Admin-bottleneck as you will have to spend more time on reviewing those themes.

          • Emil Uzelac 5:46 pm on November 1, 2012 Permalink | Log in to Reply

            I always like when other people to go over my codes, just to make sure that I have not missed something, same applies for Themes as well. That’s bit harder when solo reviewer does it, don’t you agree?

            “trust-worthiness” is definitely yes from me :)

        • jcastaneda 9:01 am on November 2, 2012 Permalink | Log in to Reply

          I do like the idea of one reviewer throughout the entire process because then it can make seeing what was fixed that much easier.
          As far as “trustworthiness” goes what would be the pre-requisites? Must have minimum three themes in the repository? Must be active within the community? Don’t get me wrong I love that idea but wonder what it would take to get that status as well as maintaining it.

        • kwight 4:07 pm on November 16, 2012 Permalink | Log in to Reply

          I’d suggest not having one reviewer per ticket; I think it would result in 1) creating bottlenecks of volunteer availability, and 2) less active reviewers. When I decide to review a theme, it’s because I have a bit of time that day, for that one review. If I was “on the hook” for a full review process without knowing what that process would entail time-wise, I’d be a lot more reticent to grab tickets.

    • Himanshu Parashar 6:55 am on November 1, 2012 Permalink | Log in to Reply

      Hi Emil.

      Thanks for your reply I agree with you on first point but some what disagree on reputation. It is not giving rights it is giving them credit for contribution to wordpress. we can process their submission in separate queue.

      Meanwhile I found one 404 error while visiting following page

      http://make.wordpress.org/themes/about/trac-ticket-request-queue/

      the most recent queue hyperlink is pointing aug 2012 queue which doesn’t exist. I think we should replace that link with oct link.

      Thanks,
      Himanshu

      • Emil Uzelac 5:40 pm on November 1, 2012 Permalink | Log in to Reply

        I completely understand that. What I meant was not “rights” literally, however credits do sound better for sure.
        We’re open here and also a team where decisions should be made collectively :)

      • Mark Bain 10:14 am on November 6, 2012 Permalink | Log in to Reply

        I’m not fond of having a new thread every month, for that reason. And it must be more difficult to admin, as it means dealing with two threads at month.end.

    • Frumph 7:42 am on November 1, 2012 Permalink | Log in to Reply

      SVN access to theme creators with stipulations (rules) or they lose access and have to return to the theme review process.
      Theme Review team privately chooses who can get SVN access to their themes and ‘suggests’ to Otto who gets the deciding factor.

      There are a couple dozen theme creators who I would suggest right away as they are upstanding members of the WordPress theme development community.

      • Emil Uzelac 7:47 am on November 1, 2012 Permalink | Log in to Reply

        Hi Phil,

        I guess this should be discussed with Otto, the subject is beyond TRT’s reach I’m afraid :)

        Thanks,
        Emil

      • Chip Bennett 12:34 pm on November 1, 2012 Permalink | Log in to Reply

        Direct SVN-commit access is possible, but due to the infrastructure between SVN and Extend, it wouldn’t actually work. With Plugins, SVN is scanned automatically on a periodic basis. The Theme directory, however, works differently. All of the synchronization scripts happen *only* when the “Make Theme Live” button is submitted in the Extend admin.

        As an alternative: what if we could come up with a way to establish a “trustworthy developer” privilege, such that when such a developer uploads a Theme, the Trac ticket is created and then auto-closed as approved, leaving only the step of an admin making the Theme live in Extend?

        • mercime 4:37 pm on November 1, 2012 Permalink | Log in to Reply

          D-v-l’s advocate on “trustworthy developer” status/privilege – Nobody’s perfect. Even wordpressdotorg’s theme team (take twenty twelve) expects Theme Reviewers to check for something they have overlooked and I consider them at the top of the “trustworthy developer” totem pole.

          If themes submitted were perfect, it would take a few minutes to review. It’s the theme which has issues that take the most time to review – includes screenshots and code snippets where required.

          It is already to the credit of the WPTRT admins that the review rules no longer require complete reviews before tagging the theme not-approved, since this has facilitated the review process. We could further streamline and expedite the process by prioritizing and setting review stages.

          For example, some companies receive project proposals or feasibility studies by the hundreds each week and they need method/system to screen proposals or studies efficiently and effectively. The proposals/studies submitted should follow the guidelines set by the company in presentation and content which should include contact information. So here’s the process:

          Stage 1

          • Company mail room – no return address on Manila envelop or boxed submission are thrown away
          • WPTRT – Theme Name, Theme URL, Author URL, Screenshot – must pass, otherwise not-approved right away

          Stage 2

          • Company project assistant – checks that management, financial, technical summaries are in second, third and fourth page of proposal, otherwise contact author to submit another proposal with complete information
          • WPTRT – Theme Check, Log Deprecated Notices, Debogger – must pass, otherwise not-approved

          Stage 3

          • Company project ex-o – reviews project components, then either forward the proposal to project manager or contact author about revisions or rejection
          • WPTRT – reviews code and conduct theme unit test – must pass, otherwise not approved.e

          Stage 4 – etc.

      • Edward Caissie 12:47 pm on November 1, 2012 Permalink | Log in to Reply

        “Ease of Access” for “trustworthy developers” has always been a touchstone ideal from the community … this definitely needs to be a take away point from this thread.

    • max2501 8:18 am on November 1, 2012 Permalink | Log in to Reply

      I think it would be a great idea, if we have checkboxes in the trac for required thinks or something like this. We can check them and maybe a full review could be generated in a few minutes. :-)

      • Chip Bennett 12:34 pm on November 1, 2012 Permalink | Log in to Reply

        Funny you should ask! I’m working on just such a checklist (albeit not in Trac), to see if it would be feasible.

    • christine 2:51 pm on November 1, 2012 Permalink | Log in to Reply

      As Frederik mentioned at #wpcs, having a single trac number per theme would make it easier to follow the review and changes, but I don’t know what the logistics of that would involve. It seems like it would make it simpler, but perhaps there are programming limitations, otherwise, I’m sure someone would have done it by now.

      As a theme developer (and a new one with only 2 themes in the repo) I personally would love to have a more thorough review. But again, Theme reviewers are volunteers and there’s only so much one can ask for. :)

      • Emil Uzelac 5:50 pm on November 1, 2012 Permalink | Log in to Reply

        I am a strong believer that single trac would speed up the process for sure, but at the same time new reviewers and old ones as well would need to pay extra attention, if this was new or previously approved Theme.

    • karmatosed 4:56 pm on November 1, 2012 Permalink | Log in to Reply

      I have to admit to the status of lapsed theme reviewer and the reasons for that aren’t just due to a lack of time unfortunately. I mean this all with a positive mindset though and would love to be part of the solution not a nay sayer.

      Aside from time the biggest hurdle for me getting back into reviewing themes was the biggest hurdle stopping themes. That is simply a lack of clear boundaries and vagueness on what should / shouldn’t pass. Yes, a lot of it is self education but there needs to be tightening up so it’s at least accessible to first users or those dipping back in.

      Themes are not rigidly set and I don’t argue they should be so easy to pass / fail as a simple checklist. It’s a guide. But, the point is the guide was a bit too vague. It makes it both hard to start as a reviewer and to come back when you’ve had a period of absence (life and contributions to other areas happens).

      I’d like to see some form of regular theme review meeting if possible. I know that’s not the easiest thing in world but it does help.

      Perhaps also those of us who are lapsed but had review status could get a few mentoring checks on our reviews (or easily request it) when we do come back. I’d like to request that now for myself.

      • Emil Uzelac 5:54 pm on November 1, 2012 Permalink | Log in to Reply

        That is actually the point of our new posts, the idea is to bring reviewers step closer to each-other. And it’s understandable for volunteers to prioritize their day-jobs first :)

      • kwight 4:10 pm on November 16, 2012 Permalink | Log in to Reply

        Actively following the theme-reviewers mailing list is also low-hanging fruit for keeping up with theme review. That’s where I would post questions, ideas or concerns about evolving best practices, etc.

    • Tskk 9:37 pm on November 1, 2012 Permalink | Log in to Reply

      I think the process we have now is fine, just need more volunteers.
      Some sort of incentive to volunteers should speed things up :)

      • Emil Uzelac 2:05 am on November 2, 2012 Permalink | Log in to Reply

        Thanks for the comment, more volunteers are always a big plus!

      • Mark Bain 10:24 am on November 6, 2012 Permalink | Log in to Reply

        I think we all agree that more volunteers would be great, but speaking personally, it was the complexity and seeming vagueness of the review process that put me off getting involved for years. And having now completed a few reviews, I’m still not clear. What’s more, how do I know if things have changed since I last reviewed a theme? It feels to me that I’d need to do a few hours research just to get back up to speed before even requesting a theme (and then there’s a wait to be allocated, and by that time, the window of opportunity/motivation has perhaps passed… couldn’t this be automated?)

    • Frank Klein 11:10 pm on November 1, 2012 Permalink | Log in to Reply

      I’m pretty new to the theme review process, but here are my two cents anyway ;).

      1) “Trustworthy” Developers
      I’m actually not in favor of this, because I think that automatically auto-closing as approved is problematic because of two reasons.

      First of all even veteran developers still can make mistakes like using a deprecated function, writing non valid HTML&CSS or forget to include licensing information for a bundled ressource. So a reviewer should look at the theme, and as they are good developers, these reviews would be done in a couple of minutes.

      Secondly I also fear that this might divide the theme development community into those whose themes get approved on the fast track and those who have to wait in line.

      2) Different review stages
      I like mercime’s idea a lot, because the Stage 1 review of verifying the licensing, credit links, theme name and screenshot could be done really fast and it would help cleaning the theme review queue of spammers and other malignant developers.

      The second step would be also pretty fast since it involves little work besides downloading the theme and doing the checks with the plugins. This would mean that only the “worthy” themes make it to stage 3, in which a reviewer has to dig in and do all the manual checks that are needed.

      3) The learning curve
      I agree with Chip that the learning curve is steep. My main motivation for joining the WPTRT was to learn more about themes, as suggested by Matt in the State of the Word 2012.

      There is a lot of self-education involved which is great for improving your skills as a WordPress developer (which happens to be my main job :)) but it also means that you have to search for information if you are not clear on an issue.

      In all of the cases in which I wasn’t sure if something was permitted I found the answer in the WPTRT mailing list archives. But searching in those archives is a real pain, so some kind of FAQ style ressource with categories, tags and a search function would be a better place to store all this knowledge.

      For beginners it might be good to get coupled with a mentor when they start out reviewing themes. This mentor would be an experienced theme reviewer that could then help out with the first reviews of his protégé. Just like Emil mentioned, coders appreciate peer review, so I think for reviewers this might work as well.

      4) The community
      Building a community is always a good idea! :) Besides Chip and Emil, I don’t even “know” any of the other theme reviewers.

      So maybe we could interact a little more besides the mailing list, maybe by having a theme reviewer’s chat like the core developers do?

      As far as the incentives mentioned by Tskk are concerned, I think the biggest reward might be appreciation from the community. If you are a core contributor, your name is in the release notes, if you are a plugin developer, your name is displayed in the plugin repository, etc. Not sure how this could work out for theme reviewers though.

    • Fingli 11:46 pm on November 1, 2012 Permalink | Log in to Reply

      The guidelines should have to be reduced. In half. Or at least the reviewers should follow more simple rules.
      The current guidelines are very detailed, as such they are suitable for (kind of) WP standards for best practice but make review process slower and tedious. But to stimulate developers to stick to detailed guidelines there might be some ‘reward’ like badge in theme’s page or something else indicating that this theme is compatible with WP standards of good practice and quality.

      How I imagine the process: It should be divided to two parts. The main will check for general basic things like the theme have to contain such and such files, such and such functions, no malicious/obfuscated code, no affiliate links and so… It it fulfilled the requirements make it pass.
      The second part is up to author – if he/she wants may ask for full review according to WP guidelines, and if the theme pass it will receive a mark for quality.

    • Vicky Arulsingam 8:16 am on November 2, 2012 Permalink | Log in to Reply

      More important than reducing the number of guidelines is identifying tiers of approval. Mercime suggestion is a great idea, especially for new theme submissions. At the present with some tickets waiting over a month for a review, I find it’s faster to just do a complete review since the developer’s been waiting for so long.

      The majority of issues that I’ve found are for things readily discovered in the Theme Guidelines and Theme Unit Tests. So is it that developers are not reading the guidelines or are those guidelines confusing?
      This is where a regular reviewer’s meeting would be good so we can discuss the common problems we’re having.

      I’m not in favor of having a “Trustworthy Developer” tag which would allow the developer to auto-approve their themes. It’s always good to have a second or third pair of eyes on your code – everyone makes mistakes from time to time.

    • Tskk 10:55 am on November 2, 2012 Permalink | Log in to Reply

      1) Create a list of guidelines and update it whenever a change is agreed upon in the mailing list.

      2) Encourage reviewers by giving them priority review for 3 of their themes for say 100 themes they review

      3) Theme authors to take a quiz of the guidelines to submit their theme. ( So the incoming themes will be already following the rules )

      • Chip Bennett 6:36 pm on November 2, 2012 Permalink | Log in to Reply

        Number 1) is essentially what we have now: the Guidelines are established, with a defined update mechanism and frequency. (The Guidelines are updated with each major core update, unless security or a similarly major issue necessitates an immediate revision.)

        Number 2) would be very difficult to implement without encouraging quantity-over-quality reviews. It also moves us into the subjective area of favoritism, and would need to be handled very carefully in that regard.

        Number 3) is, for the most part, covered by the Theme-Check Plugin, which the uploader script runs against the Theme before completing the upload and creating a Trac ticket.

        • tskk 7:04 pm on November 2, 2012 Permalink | Log in to Reply

          2) You can decide on boundaries for it, it will be nice to have that privilege :)
          It will definitely encourage more reviewers but logistics has to be worked out.

    • Emil Uzelac 6:26 pm on November 2, 2012 Permalink | Log in to Reply

      Thank you all for comments, I see many great ideas around, should we maybe start getting the list together?

    • kwight 4:21 pm on November 16, 2012 Permalink | Log in to Reply

      It’s interesting reading this thread and all the comments about how vague and confusing it is to do a review. I think it’s heavily tied in to the same frustrations that submitters face when trying to make their code acceptable to pass a theme review: documentation of the guidelines.

      I think a lot of these issues would solve themselves if we could present the guidelines in a clearer, more effective way. Each requirement could have 1) “What”: what is the requirement? 2) “Why”: Why is this a requirement? Best practice? Security? Breaks plugins if not followed? 3) “How”: How do we properly write this code? Actual code examples and, eventually, a demo site and demo theme downloads that show these requirements in action. In my head I even see each requirement displayed with those three tabs, with the “What” tab showing by default.

      My problem with the guidelines is the same as my problem with the Codex in general: a huge, dense, mind-numbing block of text that makes most people run screaming. The guidelines have served well up to this point, but for theme review to grow and sustain itself, an overhaul of how we present the guidelines would be a huge bonus to reviewers and submitters alike.

    • rohitink 2:49 pm on November 19, 2012 Permalink | Log in to Reply

      Is it necessary for a Person to Provide a Theme URL for the theme to be approved ?

      • Chip Bennett 12:00 am on December 2, 2012 Permalink | Log in to Reply

        No. You can optionally omit both Theme URI and Author URI, though it is *recommended* to include at least one or the other.

    • kwight 7:55 pm on November 19, 2012 Permalink | Log in to Reply

      I love how extend/plugins gives “Requires” and “Compatible up to”. Reviewers would have a better idea of when to give an approved theme a closer look (changing guidelines, possible errors missed by previous reviewers), and users would have a better idea of the theme’s compatibility with their own install.

    • kwight 8:01 pm on November 19, 2012 Permalink | Log in to Reply

      A small thing that would help a lot would be to have a link directly to the ticket in the emails we receive from Trac.

c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel