Tag Archives: review incentive

Chip Bennett 5:14 pm on April 8, 2014
Tags: review incentive, ,   

Selective Ticket Assignment and Ticket Hogging

Based on recent comments, I would like to clarify a couple points about ticket assignment.

Selective Ticket Assignment

First, when you assign a ticket to yourself, assign the first ticket in the queue. That means assign from tickets in Priority #1 queue, before assigning tickets from Priority queues 2, 3, or 4. Also, and more importantly: assign the first-available ticket in the queue. Do not cherry-pick tickets based on ease of review, reporter (i.e. Developer), etc.

Selectively assigning tickets is unfair to developers who are required to wait their turn in the review queue, and is a means of gaming the incentive program. Several reviewers have alleged that this practice takes place. We have not monitored it, but we will do so – something that will take even more time, and result in further delays in getting Themes approved and live. Anyone found to be selectively assigning tickets will risk being disqualified for the incentive program.

Ticket Hogging

Until now, we have not set limits on the number of open/assigned tickets a reviewer can have at any one time. Several reviewers have complained that often there are no tickets available, while some reviewers have several open/assigned tickets. So, going forward: reviewers may not have more than 5 open/assigned tickets at any one time.

In order to ensure that reviewers don’t get stuck with developer-abandoned tickets, admins will close tickets older than one week with no activity or developer response. For this reason, it is even more important not to “bump” tickets by posting comments requesting status updates.

Also, while this will be much more difficult to monitor: reviewers should only assign themselves one ticket at a time. Assign a ticket, conduct a full review, post comments, and only then assign another ticket.

Also, this clarification isn’t intended to introduce delays in developers getting their Themes reviewed. Any ticket that appears in the Priority #3 queue (tickets older than 2 weeks) are fair game to anyone. (But if the priority queue system is followed properly – see above – there should never be any tickets in the Priority #3 queue.)

Please discuss in the comments below.

#review-incentive, #review-process, #review-queue

Chip Bennett 8:05 pm on April 7, 2014
Tags: , review incentive,   

How To Ensure Your Ticket Passes Final Approval Audit

Over the past couple of months, the number of approved tickets that have been reopened due to issues found during final-approval audit has declined, but many still get reopened. As a team, we want to ensure that tickets get approved (so that new Themes get added to the directory, and into the hands of end users), and we want reviewers to be able to take advantage of the incentive program.

So I want to step through the things I check when performing a final review audit. We’re looking for some high-level and/or high-impact things that would cause problems for end users:

Overall File Structure

  • Does the Theme look like it is derived from a common Theme (Underscores, Twenty Ten-Fourteen, etc.)? Are there included functional files that I’ll need to check. Are there asset folders (fonts, scripts, etc.) that I’ll need to check?

style.css

  • Check ThemeURI and AuthorURI. Are they appropriate?
  • If ThemeURI or AuthorURI reference commercial Themes, are those Themes sold as GPLGPL GPL is an acronym for GNU Public License. It is the standard license WordPress uses for Open Source licensing https://wordpress.org/about/license/. The GPL is a ‘copyleft’ license https://www.gnu.org/licenses/copyleft.en.html. This means that derivative work can only be distributed under the same license terms. This is in distinction to permissive free software licenses, of which the BSD license and the MIT License are widely used examples.-compatible?
  • If the Theme appears to be derived, does it include a proper derivative-work copyright/license attribution?

readme.txt

  • If the Theme has bundled resources/assets, are they listed in the readme with copyright/license attribution, or will I need to check file headers?
  • Is license for all bundled resources GPL or GPL-compatible?

header.php

  • Are Theme options properly escaped on output?
  • Is favicon, if used, disabled by default?
  • Does the TITLE tag include anything other than the call to wp_title()?
  • Does wp_nav_menu() reference theme_location, and not menu?
  • Are any stylesheet or script links output instead of being properly enqueued

footer.php

  • Does the Theme only use one credit link? Is that credit link exactly ThemeURI or AuthorURI, with no SEO-seeding of link text, title attribute, etc.?
  • Are any footer scripts output instead of being enqueued properly?

index.php

  • Does template markup look generally appropriate?

Front/Home Page Templates

  • Does the Theme have front-page.php? If so, is it used properly? Does it account for both a static page and the blog posts index as site front page?
  • Does the Theme have home.php? If so, is it used properly as the default blog posts index template?
  • Does the Theme have any custom page templates intended to be used as either front-page.php or home.php?

comments.php

  • Does the Theme properly output wp_list_comments() for the comments list?
  • Does the Theme properly output comment_form() for the comment reply form, rather than hard-coding the form?

page.php

  • Does the page template properly call comments_template()?

functions.php

  • Are all functions and other things in the public namespace properly prefixed?
  • Is all functional output properly wrapped in callbacks and hooked into appropriate actions?
  • Is any of the functionality PluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party territory?
  • Does any of the functionality replicate coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. functionality?
  • Does the Theme use Theme options? Are they handled properly (single DB entry, proper settings page, sanitized on input, etc.)?
  • Do any of the Theme options replicate core options?

This list comprises 99% of what I look for in an audit, and accounts for the vast majority of issues encountered that require reopening tickets. So, if you verify these things before resolving the ticket as “approved”, the chances that your ticket will get reopened will go down considerably.

(Note: @emiluzelac may have other things to add to the list, for things he checks during an audit.)

#guidelines, #review-incentive, #review-process

Chip Bennett 12:23 pm on April 2, 2014
Tags: review incentive   

Theme Review Incentive: March 2014 Winners

We are pleased to announce the following winners for March 2014.

New Themes

Theme Updates

Please post a comment below, indicating your Featured Theme selection.

Monthly Stats

  • New Themes:
    • Approved/Live: 97
    • Reviewers: 15
    • Eligibility: 10
    • Reopened: 19
    • Missed Eligibility due to reopened tickets: 2
  • Theme Updates
    • Approved/Live: 468
    • Reviewers: 19
    • Eligibility: 117

 

Featured Theme Rotation

Per the program changes, all Themes but the following will be removed from the list:

  • Twenty Fourteen
  • Twenty Thirteen

Note: review totals are taken as a snapshot at an arbitrary point in time, based on this Theme-Trac report, and varying the filters as appropriate.  Totals are as accurate as reasonably possible.

#review-incentive

Chip Bennett 8:30 pm on March 3, 2014
Tags: review incentive   

Theme Review Incentive: February 2014 Winners

As you may recall, last month we announced changes to the Theme review incentive program, both opening up the incentive to more winners, and adding in a measure of accountability.  Per those changes, we are pleased to announce the following winners for February 2014.

New Themes

Theme Updates

Please post a comment below, indicating your Featured Theme selection.

For the month, there were a total of 88 new Themes added to the directory by 21 reviewers, and 369 Theme updates added by 16 reviewers. The eligibility requirements for this month were 8 New Themes approved, and 92 Theme updates approved.

 

Featured Theme Rotation

Per the program changes, all Themes but the following will be removed from the list:

  • Twenty Fourteen
  • Twenty Thirteen

Note: review totals are taken as a snapshot at an arbitrary point in time, based on this Theme-Trac report, and varying the filters as appropriate.  Totals are as accurate as reasonably possible.

#review-incentive

Chip Bennett 1:25 am on February 5, 2014
Tags: review incentive   

Theme Review Incentive: January 2014 Winners (and Program Updates)

Congratulations to @rohitink, @ZGani, and @alex27; you three have completed the most new Theme approvals for January! (This marks the third time in a row that these three have won the incentive.)

Please post your Featured Theme selections in the comments, with a link to the Theme’s WPORG Theme Directory listing. The Themes that will be cycled out are: Ridizain, Sugar and Spice, and Sixteen.

A total of 78 new Themes were  approved by 17 reviewers during the review period.

While we have not had a formal incentive program for Theme Updates (Priority #1 Themes), we have decided once again to recognize @tskk, who completed 143 Theme update approvals during the review period. (This marks the third time in a row that he has also won the incentive.) @tskk, please also post a Featured Theme selection in the comments, with a link to the Theme’s WPORG Theme Directory listing. The Theme that will be cycled out is: Alexandria.

Program Changes

As mentioned previously, we’re going to be making some changes to the Review Incentive program – changes that are designed both to increase the number of monthly winners, and also to increase the accountability of reviews. We want to be able to reward more reviewers, which a “top three” program limits. And unfortunately, whether due to the increased number of new reviewers, or less-than-thorough reviews resulting in approval, the Admins have had to spend more time than we should on auditing and reopening tickets that need further review. (Part of that, of course, is reviewer training/education. We’re committed to improving that as well – but right now, all my time is spent on approved-Theme audits, leaving no time to work on improving training/education.)

So, the program changes will hopefully address both needs.

More Winners

The first change we’re announcing is that we’re no longer using a “top three”method of choosing winners. Instead, each month, any reviewer who meets the minimum number of new-Theme approvals will get to choose a Featured Theme. That minimum number will vary from month to month, and will be the greater of a fixed minimum (5 Theme approvals) and a maximum of 1/10 of the total number of new Themes approved for the month. (That means that each month can have up to 10 winners.)

So for instance, this month, 78 new Themes were approved and made Live. So, the minimum would be 1/10 of that number, or 8 (7.8). So, any reviewer who had at least 8 new Theme approvals would be eligible to select a Featured Theme for the month. (An additional 2 reviewers would have been eligible.)

We are also formalizing the Theme Update reviews accordingly. For Theme Updates, the minimum number will be 1/4 of all Theme Updates approved and made live. So for instance, this month 306 Theme Updates were approved. So, any reviewer who had at least 76 Theme Updates approved would be eligible to select a Featured Theme for the month.

We have a total of 10 Featured Theme slots to reward, and each reviewer may choose only one Featured Theme (even if eligible via both New Themes and Theme Updates). So in the off chance that we have the maximum number of winners – 14 – with no duplication, the Admins will determine which 10 will win. (I don’t anticipate this ever happening.)

More Accountability

But along with drastically increasing the number of monthly winners, we are making changes to the Theme approval process.

The approved-Theme audits that Admins perform have too often become secondary reviews – to the point that we simply can’t get through the queue fast enough. Generally, I have been reopening 8 or 9 out of every 10 tickets closed as “approved” – and many times, for significant and/or fairly obvious issues. Please: ask questions in-ticket (cc: an admin if you need to) or on the mail-list during your review. Ask for help if you need it (e.g. for overly complex Theme options libraries). We are a team, and are here to help one another. But once it’s closed as “approved”, that means that the reviewer believes the Theme meets all Theme Review Guidelines.

So, starting this month, any ticket that has to be reopened after being closed as “approved” will not count toward the incentive program.

Hopefully, this change will free up some of the Admins’ time, and will enable us to focus more time on training, education, and finding ways to continue to make Theme review easier, especially for new reviewers.

Caveats

Just as a reminder: the Review Incentive program has always been intended to be a fun way to encourage reviews and to reward reviewers. If it becomes burdensome, causes too much team strife, results in too many people attempting to game the system, causes people to be unwilling to work together as a team, or otherwise distracts from the purpose of the Theme Review Team, it will have to be discontinued. Please keep the incentive in its proper perspective. I believe it has been wildly successful up to this point. (Themes are being approved and made live at a rate at least three or four times greater than before the program was instituted.) Let’s all have fun with it, and remember that it is intended to help us fulfill our purpose as a team: to get as many great Themes as possible into the hands of end users.

As always: thank you, all, for your contributions, and for volunteering your time and effort. If there is anything that the Admins can do to improve the review process, please let us know.

Update

Based on feedback in the comments, we have decided that the two reviewers who would be eligible this month under the new incentive rules should also get to select a Featured Theme. So, congratulations, @pagelines and @cyberchimpscode! Please post your Featured Theme selections in the comments, with a link to the Theme’s directory listing.

#review-incentive