Agenda for 2017 April 18

The Theme Review Team holds a meeting weekly and we encourage all members to attend.

Channel: #themereview | Time: Tuesday at 17:00 UTC 17:00 UTC

Recap:

Topics:

This week we want to start defining specific items that we want to prioritise so that we can get back to Matt with an update since out last chat.

  • Prepare a post comparing the theme and plugin review process to see how we can work towards a goal of making both processes very similar so that any improvements made to one will also profit the other. Responsible: @grapplerulrich
  • Put a team together to tackle the theme previews as this is a major issue. Get Matt to put resources towards this.
  • Automation is still a priority to catch common issues and giving us more confidence that themes cannot be shipped that give sites a white screen of death.
  • Define the minimum requirements that need to be manually reviewed for the themes to be hosted on w.org. The basic items I could think of are checking for licencing, anything illegal, dishonest, or morally offensive.
  • Agreeing to try out a way to get more feedback from users on different aspects of themes and using the extra data on the theme to rank them appropriately in the repository.

Once we are agree on these items we can get back to Matt and discuss the next steps.

 

If you have any topics, then please reply to this post and include the topic, along with a brief description of what you would like to discuss.

Restructuring the Theme Review Team

At the meeting on Friday with Matt, it was suggested that we should have Team Reps who could make the final decision as a direct democracy does not really work well in our situation. A few reason for this is the number of people active in a team meeting and voting on the same issue in another place and time may give different results.

The suggested restructure of the theme review team would be:

Team Reps/Team Leads

  • These people would be people who are experienced with theme reviews and willing to give their time to leading the Theme Review Team forward.
  • Responsibilities would include
    • Leading the team forward
    • Listening to the team opinions and making a decision
    • Communicate with other Teams and Team Leads
    • Help moderators with their tasks

Moderators

  • These would be people who are experienced members of the theme review team who help out with different administrative tasks.
  • These tasks could involve:
    • Doing final reviews and setting themes live
    • Returning tickets to the queue that have been abandoned
    • Assigning Trainee Reviewers themes
    • Giving a second opinion in tickets
    • Maintaining the requirements documentation
    • Leading team meetings
    • Writing the agenda
    • Fixing issues with themes not going live
    • Removing themes from the repo
  • Depending on what administratives tasks are done access will be provided.
  • All current Key Reviewers and Trusted Reviewers will become moderators.

Reviewers

  • There is no real change here. They can assign themselves reviews.
  • Are able to lead and work on projects and bring ideas of ways to improve.

Trainee Reviewers

  • This is everyone else who can assign themselves a ticket via the assign ticket button.
  • Are able to lead and work on projects and bring ideas of ways to improve.

Chat with Matt about the future of theme repo

On Friday evening, @jcastaneda, @poena and I met with Matt to discuss the future of the theme repository on WordPress.org. @greenshady had to drop out last minute since Matt wanted to do a voice call and Justin was unable to due to his internet connection. We asked Jose to join so that we would not need to delay the call any longer.

Matt wanted to know more about the Theme Review team. We mentioned our plans with automation, problems with the previews, and how the portability of content affects users. We also mentioned that the common issues in themes are security, code errors and prefixing.
A suggestion that we got back was that we should check if themes could be prevented from being activated if there are PHP fatal errors, like it is done with plugins.

The reason for having the meeting now and not at the community summit is so that we can start working on improvements before the community summit.

Matt’s goal for the theme repository is to make it the main place for users to search and find themes.
Matt is interested in seeing how the repository reacts if we remove the manual review process and switch to a process where user feedback helps rank the themes.
Amazon was an example where search results are influenced by different meta data including reviews and how reviews affect users buying a product or not. On Wix you can search for nearly any term and a related theme is shown. That should be possible on WordPress.org too.

Another comparison was made between Google and Yahoo where Yahoo used to have a directory of links that were reviewed before being added to the list but Google indexes all links but shows the best at the top.

Allowing users to give feedback in different formats like adding tags or giving a tag a thumbs up or down if the tag was correct or not thus allowing bad themes to drop and good themes to rise. Users should be able to decide how much upsell is too much.
An example of this is how Google Maps asks users for feedback after visiting is place.

Matt asked us how the featured themes/homepage functioned and we explained that they were random themes that have been updated in the last two years. As this list of featured themes are not always the best we need to come up with ideas how to improve it.

As we are not sure if the process will function without manual reviews, we will start working on getting better user feedback on themes. Once we have a good infrastructure in place we can experiment with how the repository reacts with no manual reviews.

We discussed the process we would go about making decisions on changes to the theme repository and came to the consensus that a direct democracy is too fragile and representative democracy would be a better solution.

We were given a few tasks to complete and then get back to Matt with the results so that he can see where the resources are needed.

Tasks:

  • Matt has asked that we choose a few people who will represent the team and make the final decisions.
    The representatives are people who can be contacted to get feedback from and provide feedback to.
  • To create a list of blockers or resources that are needed to improve the theme repo.
  • Once we have a list to contact him so that he can assign the necessary resources.

A few things that we discussed as potential places where we can improve on.

  • SVN access
  • Multiple screenshots
  • Improved search to be able find a theme related to a certain topic for example “Thai Restaurant”.
  • Improved previews that display the theme at a basic level and allow theme authors to show their own theme demo.
  • Showing more information about the theme, like the readme and changelogs.
  • Allow a way for users to report bad themes, but also if a theme has a certain feature.
  • If we allow themes to have content then we should allow custom post types and taxonomies as they are better than pseudo custom post types.

Review statistics

These statistics should be taken with a pinch of salt, since the numbers does not assure statistical accuracy for the entire theme directory.

While we can follow the graph and even search live themes, that won’t tell us what the reviews contain, so this was done manually, human errors included.

Out of 531 themes that were closed as not approved between December and February, I looked closer at 100 tickets. Out of 100 tickets, 21 tickets were closed because the themes were copies of a theme that is already in the directory.

  • 14 tickets where closed because the author did not reply within 7 days.
  • 9 tickets where closed on author request.
  • 9 tickets where closed because the author already had an open ticket (the one theme rule).

The reviewer did not complete the review and the ticket had to be returned to the queue on 8 occasions (Not necessarily 8 separate themes).

The most common problems mentioned by the reviewers were:

  • Missing escaping or using the wrong functions: 23 themes
  • Text that is not translation ready: 21 themes
  • Missing prefix: 20 themes
  • Scripts or styles are not enqueued: 18 themes
  • PHP notices, errors or warnings: 12 themes
  • Style tags does not correspond with theme functionality, or are deprecated: 10 themes

You can find the numbers for the not approved themes here.

 

In an attempt to compare the results of the two categories, I also looked at 100 out of 177 new themes that went live between December and February.

The reviewer did not complete the review and the ticket had to be returned to the queue on 45 occasions (Not necessarily 45 separate themes).

37 of the tickets were reopened as a second reviewer found additional problems.

One of  the more alarming results was that in 51 out of 100 tickets, the reviewer pointed out that escaping was either missing, or the wrong functions were used. 

 

The most common problems mentioned by the reviewers were:

  • Missing escaping or using the wrong functions: 51 Themes
  • Text that is not translation ready: 44 Themes
  • Missing prefix: 39 Themes
  • Missing license or copyright information for included assets: 34 Themes
  • Unused code or files: 25 Themes
  • PHP notices, errors or warnings: 20 Themes
  • Missing sanitization, or using the wrong functions: 18 Themes
  • Options in the customizer that are not working: 18 Themes

On 15 occasions, the reviewer asked the author to remove demo content.

On 12 occasions, the reviewer asked the author to remove or reduce content creation.

Compared to the themes that were not approved, only 11 themes had scripts or styles that were not enqueued.

In a couple of tickets, the reviewer asked the author to replace the following custom functionality and use WordPress functions instead:

  • Logo: 11 Themes
  • Custom CSS: 10 Themes
  • Custom excerpt: 8 Themes
  • Custom pagination: 6 Themes

You can find the numbers for the live themes here.

I also wrote down how long it took for the themes to go live:

Live after 13 months: 1
Live after 12 months: 1
Live after 11 months: 4
Live after 10 months: 8
Live after 9 months: 9
Live after 8 months: 17
Live after 7 months: 23
Live after 6 months: 22
Live after 5 months: 11
Live after 4 months: 3
Live after 3 months: 1

 

We will be able to repeat this in a few months, to make sure that these numbers are going down.

Requirements project March 23

Our next project meeting will be on Thursday at 19:00

Channel: #themereview

We have a few more short term fixes to discuss:

  • Child theme names
  • Theme URI and  footer credit links
  • Admin bar menu items
  • Screenshot

There is also one recommendation that perhaps should be removed:
Themes may optionally unregister core Widgets. This is allowed, but is it recommended?

As a reminder, we also have the Trello board.

 

Those who have time after the meeting are welcome to continue,  with the purpose of defining what a “Letting users decide” approach would be for the team. We need to define it before we can continue any productive discussion on the topic.

 

Child themes should not include the name of the parent theme.

Suggestion: Child themes may only use the parent theme name if both themes are made by the same author.

Suggestion 2: Remove it (again).

If the theme adds a footer credit link, there should only be one (link to WordPress.org does not count).

Suggestion: There can only be a single footer credit link, which is either the the Theme URI or Author URI (link to WordPress.org does not count).

Theme URI:

Using WordPress.org as theme URI is reserved for official themes.

This was brought up on slack this weekend:
“The purpose of the Theme URI is to point the user to a page specifically about the theme that we are hosting here on WordPress.org. That means that the majority of the info on that page should about the theme that we’re hosting.”
“Theme URI needs to be a) under a domain you control, not “wordpress.org”, and b) unique to the theme. Not unique to the theme and version number or anything else, just unique to the theme. ”

Themes are not allowed to add a menu item to the WordPress admin bar.

I believe this is used in practice but it is not currently listed on the requirement page. Is this something that we want to add or is it something that we would consider allowing?

The screenshot should be of the actual theme as it appears with default options, not a logo or mockup.

Suggestion: The screenshot should be of the theme, not a logo or mockup. The screenshot may optionally show plugins, settings and templates. The user should be able to make the theme look like the screenshot.