It has been awhile since I last did a theme review but I remember that though links are provided check what to clean there isn’t a single page that lists everything that needs to be checked.
I had to jump around between a few pages to make sure that everything was checked off and accounted for. Would make it a lot smoother if it was all on one page.
It looks like it has been improved quite a bit since I last reviewed a theme. I guess you can disregard my comment as I hadn’t checked it before commenting
I think, for new reviewers, this is still a valid concern. The learning curve is still quite high. I’m beginning the process of creating a step-by-step how-to (and possibly even a review checklist) that will hopefully make the process much easier.
Konstantin Obenland
2:49 pm on November 1, 2012 Permalink
I agree with Jarret. I was also theme reviewer in past and I always try to review themes now below are some key points I found while working in theme review process.
1. if somebody submitting theme for review after 1-2 attempt it should get priority instead of going back to the beginning.
2. Don’t de-prioritize existing themes in the queue upon newer submissions.
3. we can initiate a user rating system where if somebody has very good reputation in theme repository we can approve their theme quickly by giving them priority.
When Themes are submitted they will be in Priority #4 (New Themes, Never Reviewed) and if is not approved and upon second submission Theme goes to Priority #3 (Previously Reviewed, Not-Approved Themes), so they are not sent back.
As far as I know, there’s no de-prioritization, do you mind giving us some examples?
Reputation -vs newer submission is something I personally see as an unfair “rule”.
Just because “author A” is more reputable than “author B” should not give “author A” more “rights” in the trac
Actually, if Theme developers leave a comment pointing to a newer ticket, then the new ticket will take the old ticket’s place in the queue. That’s always been in place. That said, here are some ideas I’ve been considering to propose:
1) Have a reviewer follow a given Theme through the entire review process, from ticket to ticket.
2) Have some way of establishing *trust-worthiness*, so that developers who earn the privilege would have their tickets auto-closed as approved (leaving only the final, admin review upon pushing live in Extend).
Konstantin Obenland
2:55 pm on November 1, 2012 Permalink
| Reply
I don’t know if I like having reviewers being dedicated to a theme. Sometimes a fresh pair of eyes can do wonders and I think various reviewers with their individual review-style improve the overall quality of the submitted theme.
I like 2). Though I fear it will increase the Admin-bottleneck as you will have to spend more time on reviewing those themes.
I always like when other people to go over my codes, just to make sure that I have not missed something, same applies for Themes as well. That’s bit harder when solo reviewer does it, don’t you agree?
I do like the idea of one reviewer throughout the entire process because then it can make seeing what was fixed that much easier.
As far as “trustworthiness” goes what would be the pre-requisites? Must have minimum three themes in the repository? Must be active within the community? Don’t get me wrong I love that idea but wonder what it would take to get that status as well as maintaining it.
I’d suggest not having one reviewer per ticket; I think it would result in 1) creating bottlenecks of volunteer availability, and 2) less active reviewers. When I decide to review a theme, it’s because I have a bit of time that day, for that one review. If I was “on the hook” for a full review process without knowing what that process would entail time-wise, I’d be a lot more reticent to grab tickets.
Thanks for your reply I agree with you on first point but some what disagree on reputation. It is not giving rights it is giving them credit for contribution to wordpress. we can process their submission in separate queue.
Meanwhile I found one 404 error while visiting following page
I completely understand that. What I meant was not “rights” literally, however credits do sound better for sure.
We’re open here and also a team where decisions should be made collectively
I’m not fond of having a new thread every month, for that reason. And it must be more difficult to admin, as it means dealing with two threads at month.end.
SVN access to theme creators with stipulations (rules) or they lose access and have to return to the theme review process.
Theme Review team privately chooses who can get SVN access to their themes and ‘suggests’ to Otto who gets the deciding factor.
There are a couple dozen theme creators who I would suggest right away as they are upstanding members of the WordPress theme development community.
Direct SVN-commit access is possible, but due to the infrastructure between SVN and Extend, it wouldn’t actually work. With Plugins, SVN is scanned automatically on a periodic basis. The Theme directory, however, works differently. All of the synchronization scripts happen *only* when the “Make Theme Live” button is submitted in the Extend admin.
As an alternative: what if we could come up with a way to establish a “trustworthy developer” privilege, such that when such a developer uploads a Theme, the Trac ticket is created and then auto-closed as approved, leaving only the step of an admin making the Theme live in Extend?
D-v-l’s advocate on “trustworthy developer” status/privilege – Nobody’s perfect. Even wordpressdotorg’s theme team (take twenty twelve) expects Theme Reviewers to check for something they have overlooked and I consider them at the top of the “trustworthy developer” totem pole.
If themes submitted were perfect, it would take a few minutes to review. It’s the theme which has issues that take the most time to review – includes screenshots and code snippets where required.
It is already to the credit of the WPTRT admins that the review rules no longer require complete reviews before tagging the theme not-approved, since this has facilitated the review process. We could further streamline and expedite the process by prioritizing and setting review stages.
For example, some companies receive project proposals or feasibility studies by the hundreds each week and they need method/system to screen proposals or studies efficiently and effectively. The proposals/studies submitted should follow the guidelines set by the company in presentation and content which should include contact information. So here’s the process:
Stage 1
Company mail room – no return address on Manila envelop or boxed submission are thrown away
WPTRT – Theme Name, Theme URL, Author URL, Screenshot – must pass, otherwise not-approved right away
Stage 2
Company project assistant – checks that management, financial, technical summaries are in second, third and fourth page of proposal, otherwise contact author to submit another proposal with complete information
“Ease of Access” for “trustworthy developers” has always been a touchstone ideal from the community … this definitely needs to be a take away point from this thread.
I think it would be a great idea, if we have checkboxes in the trac for required thinks or something like this. We can check them and maybe a full review could be generated in a few minutes.
As Frederik mentioned at #wpcs, having a single trac number per theme would make it easier to follow the review and changes, but I don’t know what the logistics of that would involve. It seems like it would make it simpler, but perhaps there are programming limitations, otherwise, I’m sure someone would have done it by now.
As a theme developer (and a new one with only 2 themes in the repo) I personally would love to have a more thorough review. But again, Theme reviewers are volunteers and there’s only so much one can ask for.
I am a strong believer that single trac would speed up the process for sure, but at the same time new reviewers and old ones as well would need to pay extra attention, if this was new or previously approved Theme.
I have to admit to the status of lapsed theme reviewer and the reasons for that aren’t just due to a lack of time unfortunately. I mean this all with a positive mindset though and would love to be part of the solution not a nay sayer.
Aside from time the biggest hurdle for me getting back into reviewing themes was the biggest hurdle stopping themes. That is simply a lack of clear boundaries and vagueness on what should / shouldn’t pass. Yes, a lot of it is self education but there needs to be tightening up so it’s at least accessible to first users or those dipping back in.
Themes are not rigidly set and I don’t argue they should be so easy to pass / fail as a simple checklist. It’s a guide. But, the point is the guide was a bit too vague. It makes it both hard to start as a reviewer and to come back when you’ve had a period of absence (life and contributions to other areas happens).
I’d like to see some form of regular theme review meeting if possible. I know that’s not the easiest thing in world but it does help.
Perhaps also those of us who are lapsed but had review status could get a few mentoring checks on our reviews (or easily request it) when we do come back. I’d like to request that now for myself.
That is actually the point of our new posts, the idea is to bring reviewers step closer to each-other. And it’s understandable for volunteers to prioritize their day-jobs first
Actively following the theme-reviewers mailing list is also low-hanging fruit for keeping up with theme review. That’s where I would post questions, ideas or concerns about evolving best practices, etc.
I think we all agree that more volunteers would be great, but speaking personally, it was the complexity and seeming vagueness of the review process that put me off getting involved for years. And having now completed a few reviews, I’m still not clear. What’s more, how do I know if things have changed since I last reviewed a theme? It feels to me that I’d need to do a few hours research just to get back up to speed before even requesting a theme (and then there’s a wait to be allocated, and by that time, the window of opportunity/motivation has perhaps passed… couldn’t this be automated?)
Frank Klein
11:10 pm on November 1, 2012 Permalink
| Reply
I’m pretty new to the theme review process, but here are my two cents anyway .
1) “Trustworthy” Developers
I’m actually not in favor of this, because I think that automatically auto-closing as approved is problematic because of two reasons.
First of all even veteran developers still can make mistakes like using a deprecated function, writing non valid HTML&CSS or forget to include licensing information for a bundled ressource. So a reviewer should look at the theme, and as they are good developers, these reviews would be done in a couple of minutes.
Secondly I also fear that this might divide the theme development community into those whose themes get approved on the fast track and those who have to wait in line.
2) Different review stages
I like mercime’s idea a lot, because the Stage 1 review of verifying the licensing, credit links, theme name and screenshot could be done really fast and it would help cleaning the theme review queue of spammers and other malignant developers.
The second step would be also pretty fast since it involves little work besides downloading the theme and doing the checks with the plugins. This would mean that only the “worthy” themes make it to stage 3, in which a reviewer has to dig in and do all the manual checks that are needed.
3) The learning curve
I agree with Chip that the learning curve is steep. My main motivation for joining the WPTRT was to learn more about themes, as suggested by Matt in the State of the Word 2012.
There is a lot of self-education involved which is great for improving your skills as a WordPress developer (which happens to be my main job ) but it also means that you have to search for information if you are not clear on an issue.
In all of the cases in which I wasn’t sure if something was permitted I found the answer in the WPTRT mailing list archives. But searching in those archives is a real pain, so some kind of FAQ style ressource with categories, tags and a search function would be a better place to store all this knowledge.
For beginners it might be good to get coupled with a mentor when they start out reviewing themes. This mentor would be an experienced theme reviewer that could then help out with the first reviews of his protégé. Just like Emil mentioned, coders appreciate peer review, so I think for reviewers this might work as well.
4) The community
Building a community is always a good idea! Besides Chip and Emil, I don’t even “know” any of the other theme reviewers.
So maybe we could interact a little more besides the mailing list, maybe by having a theme reviewer’s chat like the core developers do?
As far as the incentives mentioned by Tskk are concerned, I think the biggest reward might be appreciation from the community. If you are a core contributor, your name is in the release notes, if you are a plugin developer, your name is displayed in the plugin repository, etc. Not sure how this could work out for theme reviewers though.
The guidelines should have to be reduced. In half. Or at least the reviewers should follow more simple rules.
The current guidelines are very detailed, as such they are suitable for (kind of) WP standards for best practice but make review process slower and tedious. But to stimulate developers to stick to detailed guidelines there might be some ‘reward’ like badge in theme’s page or something else indicating that this theme is compatible with WP standards of good practice and quality.
How I imagine the process: It should be divided to two parts. The main will check for general basic things like the theme have to contain such and such files, such and such functions, no malicious/obfuscated code, no affiliate links and so… It it fulfilled the requirements make it pass.
The second part is up to author – if he/she wants may ask for full review according to WP guidelines, and if the theme pass it will receive a mark for quality.
More important than reducing the number of guidelines is identifying tiers of approval. Mercime suggestion is a great idea, especially for new theme submissions. At the present with some tickets waiting over a month for a review, I find it’s faster to just do a complete review since the developer’s been waiting for so long.
The majority of issues that I’ve found are for things readily discovered in the Theme Guidelines and Theme Unit Tests. So is it that developers are not reading the guidelines or are those guidelines confusing?
This is where a regular reviewer’s meeting would be good so we can discuss the common problems we’re having.
I’m not in favor of having a “Trustworthy Developer” tag which would allow the developer to auto-approve their themes. It’s always good to have a second or third pair of eyes on your code – everyone makes mistakes from time to time.
Number 1) is essentially what we have now: the Guidelines are established, with a defined update mechanism and frequency. (The Guidelines are updated with each major core update, unless security or a similarly major issue necessitates an immediate revision.)
Number 2) would be very difficult to implement without encouraging quantity-over-quality reviews. It also moves us into the subjective area of favoritism, and would need to be handled very carefully in that regard.
Number 3) is, for the most part, covered by the Theme-Check Plugin, which the uploader script runs against the Theme before completing the upload and creating a Trac ticket.
2) You can decide on boundaries for it, it will be nice to have that privilege
It will definitely encourage more reviewers but logistics has to be worked out.
It’s interesting reading this thread and all the comments about how vague and confusing it is to do a review. I think it’s heavily tied in to the same frustrations that submitters face when trying to make their code acceptable to pass a theme review: documentation of the guidelines.
I think a lot of these issues would solve themselves if we could present the guidelines in a clearer, more effective way. Each requirement could have 1) “What”: what is the requirement? 2) “Why”: Why is this a requirement? Best practice? Security? Breaks plugins if not followed? 3) “How”: How do we properly write this code? Actual code examples and, eventually, a demo site and demo theme downloads that show these requirements in action. In my head I even see each requirement displayed with those three tabs, with the “What” tab showing by default.
My problem with the guidelines is the same as my problem with the Codex in general: a huge, dense, mind-numbing block of text that makes most people run screaming. The guidelines have served well up to this point, but for theme review to grow and sustain itself, an overhaul of how we present the guidelines would be a huge bonus to reviewers and submitters alike.
I love how extend/plugins gives “Requires” and “Compatible up to”. Reviewers would have a better idea of when to give an approved theme a closer look (changing guidelines, possible errors missed by previous reviewers), and users would have a better idea of the theme’s compatibility with their own install.
JarretC 5:01 am on November 1, 2012 Permalink |
It has been awhile since I last did a theme review but I remember that though links are provided check what to clean there isn’t a single page that lists everything that needs to be checked.
I had to jump around between a few pages to make sure that everything was checked off and accounted for. Would make it a lot smoother if it was all on one page.
Emil Uzelac 6:04 am on November 1, 2012 Permalink |
Hi Jarret,
You mean the trac itself?
Thanks for the comment,
Emil
JarretC 6:12 am on November 1, 2012 Permalink |
I just looked at the theme review codex doc – http://codex.wordpress.org/Theme_Review
It looks like it has been improved quite a bit since I last reviewed a theme. I guess you can disregard my comment as I hadn’t checked it before commenting
Emil Uzelac 7:39 am on November 1, 2012 Permalink |
No problem, I am glad that you commented
Chip Bennett 12:26 pm on November 1, 2012 Permalink |
I think, for new reviewers, this is still a valid concern. The learning curve is still quite high. I’m beginning the process of creating a step-by-step how-to (and possibly even a review checklist) that will hopefully make the process much easier.
Konstantin Obenland 2:49 pm on November 1, 2012 Permalink
Where is the like button when you need it?
Emil Uzelac 5:41 pm on November 1, 2012 Permalink
good stuff! and by the way, we can add like button as well
Himanshu 5:47 am on November 1, 2012 Permalink |
Hi Emil,
I agree with Jarret. I was also theme reviewer in past and I always try to review themes now below are some key points I found while working in theme review process.
1. if somebody submitting theme for review after 1-2 attempt it should get priority instead of going back to the beginning.
2. Don’t de-prioritize existing themes in the queue upon newer submissions.
3. we can initiate a user rating system where if somebody has very good reputation in theme repository we can approve their theme quickly by giving them priority.
Thanks,
Himanshu
Emil Uzelac 6:04 am on November 1, 2012 Permalink |
Hi Himanshu,
Thanks for the comment.
When Themes are submitted they will be in Priority #4 (New Themes, Never Reviewed) and if is not approved and upon second submission Theme goes to Priority #3 (Previously Reviewed, Not-Approved Themes), so they are not sent back.
As far as I know, there’s no de-prioritization, do you mind giving us some examples?
Reputation -vs newer submission is something I personally see as an unfair “rule”.
Just because “author A” is more reputable than “author B” should not give “author A” more “rights” in the trac
Emil
Chip Bennett 12:30 pm on November 1, 2012 Permalink |
Actually, if Theme developers leave a comment pointing to a newer ticket, then the new ticket will take the old ticket’s place in the queue. That’s always been in place. That said, here are some ideas I’ve been considering to propose:
1) Have a reviewer follow a given Theme through the entire review process, from ticket to ticket.
2) Have some way of establishing *trust-worthiness*, so that developers who earn the privilege would have their tickets auto-closed as approved (leaving only the final, admin review upon pushing live in Extend).
Konstantin Obenland 2:55 pm on November 1, 2012 Permalink |
I don’t know if I like having reviewers being dedicated to a theme. Sometimes a fresh pair of eyes can do wonders and I think various reviewers with their individual review-style improve the overall quality of the submitted theme.
I like 2). Though I fear it will increase the Admin-bottleneck as you will have to spend more time on reviewing those themes.
Emil Uzelac 5:46 pm on November 1, 2012 Permalink |
I always like when other people to go over my codes, just to make sure that I have not missed something, same applies for Themes as well. That’s bit harder when solo reviewer does it, don’t you agree?
“trust-worthiness” is definitely yes from me
jcastaneda 9:01 am on November 2, 2012 Permalink |
I do like the idea of one reviewer throughout the entire process because then it can make seeing what was fixed that much easier.
As far as “trustworthiness” goes what would be the pre-requisites? Must have minimum three themes in the repository? Must be active within the community? Don’t get me wrong I love that idea but wonder what it would take to get that status as well as maintaining it.
kwight 4:07 pm on November 16, 2012 Permalink |
I’d suggest not having one reviewer per ticket; I think it would result in 1) creating bottlenecks of volunteer availability, and 2) less active reviewers. When I decide to review a theme, it’s because I have a bit of time that day, for that one review. If I was “on the hook” for a full review process without knowing what that process would entail time-wise, I’d be a lot more reticent to grab tickets.
Himanshu Parashar 6:55 am on November 1, 2012 Permalink |
Hi Emil.
Thanks for your reply I agree with you on first point but some what disagree on reputation. It is not giving rights it is giving them credit for contribution to wordpress. we can process their submission in separate queue.
Meanwhile I found one 404 error while visiting following page
http://make.wordpress.org/themes/about/trac-ticket-request-queue/
the most recent queue hyperlink is pointing aug 2012 queue which doesn’t exist. I think we should replace that link with oct link.
Thanks,
Himanshu
Emil Uzelac 5:40 pm on November 1, 2012 Permalink |
I completely understand that. What I meant was not “rights” literally, however credits do sound better for sure.
We’re open here and also a team where decisions should be made collectively
Mark Bain 10:14 am on November 6, 2012 Permalink |
I’m not fond of having a new thread every month, for that reason. And it must be more difficult to admin, as it means dealing with two threads at month.end.
Frumph 7:42 am on November 1, 2012 Permalink |
SVN access to theme creators with stipulations (rules) or they lose access and have to return to the theme review process.
Theme Review team privately chooses who can get SVN access to their themes and ‘suggests’ to Otto who gets the deciding factor.
There are a couple dozen theme creators who I would suggest right away as they are upstanding members of the WordPress theme development community.
Emil Uzelac 7:47 am on November 1, 2012 Permalink |
Hi Phil,
I guess this should be discussed with Otto, the subject is beyond TRT’s reach I’m afraid
Thanks,
Emil
Frumph 8:22 am on November 1, 2012 Permalink |
You asked though
Emil Uzelac 5:26 pm on November 1, 2012 Permalink |
Yes sir I did
Chip Bennett 12:34 pm on November 1, 2012 Permalink |
Direct SVN-commit access is possible, but due to the infrastructure between SVN and Extend, it wouldn’t actually work. With Plugins, SVN is scanned automatically on a periodic basis. The Theme directory, however, works differently. All of the synchronization scripts happen *only* when the “Make Theme Live” button is submitted in the Extend admin.
As an alternative: what if we could come up with a way to establish a “trustworthy developer” privilege, such that when such a developer uploads a Theme, the Trac ticket is created and then auto-closed as approved, leaving only the step of an admin making the Theme live in Extend?
mercime 4:37 pm on November 1, 2012 Permalink |
D-v-l’s advocate on “trustworthy developer” status/privilege – Nobody’s perfect. Even wordpressdotorg’s theme team (take twenty twelve) expects Theme Reviewers to check for something they have overlooked and I consider them at the top of the “trustworthy developer” totem pole.
If themes submitted were perfect, it would take a few minutes to review. It’s the theme which has issues that take the most time to review – includes screenshots and code snippets where required.
It is already to the credit of the WPTRT admins that the review rules no longer require complete reviews before tagging the theme not-approved, since this has facilitated the review process. We could further streamline and expedite the process by prioritizing and setting review stages.
For example, some companies receive project proposals or feasibility studies by the hundreds each week and they need method/system to screen proposals or studies efficiently and effectively. The proposals/studies submitted should follow the guidelines set by the company in presentation and content which should include contact information. So here’s the process:
Stage 1
Stage 2
Stage 3
Stage 4 – etc.
Edward Caissie 12:47 pm on November 1, 2012 Permalink |
“Ease of Access” for “trustworthy developers” has always been a touchstone ideal from the community … this definitely needs to be a take away point from this thread.
Emil Uzelac 5:57 pm on November 1, 2012 Permalink |
so how do we propose that? if everyone else agrees let’s add that our list!
max2501 8:18 am on November 1, 2012 Permalink |
I think it would be a great idea, if we have checkboxes in the trac for required thinks or something like this. We can check them and maybe a full review could be generated in a few minutes.
Chip Bennett 12:34 pm on November 1, 2012 Permalink |
Funny you should ask! I’m working on just such a checklist (albeit not in Trac), to see if it would be feasible.
christine 2:51 pm on November 1, 2012 Permalink |
As Frederik mentioned at #wpcs, having a single trac number per theme would make it easier to follow the review and changes, but I don’t know what the logistics of that would involve. It seems like it would make it simpler, but perhaps there are programming limitations, otherwise, I’m sure someone would have done it by now.
As a theme developer (and a new one with only 2 themes in the repo) I personally would love to have a more thorough review. But again, Theme reviewers are volunteers and there’s only so much one can ask for.
Emil Uzelac 5:50 pm on November 1, 2012 Permalink |
I am a strong believer that single trac would speed up the process for sure, but at the same time new reviewers and old ones as well would need to pay extra attention, if this was new or previously approved Theme.
karmatosed 4:56 pm on November 1, 2012 Permalink |
I have to admit to the status of lapsed theme reviewer and the reasons for that aren’t just due to a lack of time unfortunately. I mean this all with a positive mindset though and would love to be part of the solution not a nay sayer.
Aside from time the biggest hurdle for me getting back into reviewing themes was the biggest hurdle stopping themes. That is simply a lack of clear boundaries and vagueness on what should / shouldn’t pass. Yes, a lot of it is self education but there needs to be tightening up so it’s at least accessible to first users or those dipping back in.
Themes are not rigidly set and I don’t argue they should be so easy to pass / fail as a simple checklist. It’s a guide. But, the point is the guide was a bit too vague. It makes it both hard to start as a reviewer and to come back when you’ve had a period of absence (life and contributions to other areas happens).
I’d like to see some form of regular theme review meeting if possible. I know that’s not the easiest thing in world but it does help.
Perhaps also those of us who are lapsed but had review status could get a few mentoring checks on our reviews (or easily request it) when we do come back. I’d like to request that now for myself.
Emil Uzelac 5:54 pm on November 1, 2012 Permalink |
That is actually the point of our new posts, the idea is to bring reviewers step closer to each-other. And it’s understandable for volunteers to prioritize their day-jobs first
kwight 4:10 pm on November 16, 2012 Permalink |
Actively following the theme-reviewers mailing list is also low-hanging fruit for keeping up with theme review. That’s where I would post questions, ideas or concerns about evolving best practices, etc.
Tskk 9:37 pm on November 1, 2012 Permalink |
I think the process we have now is fine, just need more volunteers.
Some sort of incentive to volunteers should speed things up
Emil Uzelac 2:05 am on November 2, 2012 Permalink |
Thanks for the comment, more volunteers are always a big plus!
Mark Bain 10:24 am on November 6, 2012 Permalink |
I think we all agree that more volunteers would be great, but speaking personally, it was the complexity and seeming vagueness of the review process that put me off getting involved for years. And having now completed a few reviews, I’m still not clear. What’s more, how do I know if things have changed since I last reviewed a theme? It feels to me that I’d need to do a few hours research just to get back up to speed before even requesting a theme (and then there’s a wait to be allocated, and by that time, the window of opportunity/motivation has perhaps passed… couldn’t this be automated?)
Frank Klein 11:10 pm on November 1, 2012 Permalink |
I’m pretty new to the theme review process, but here are my two cents anyway
.
1) “Trustworthy” Developers
I’m actually not in favor of this, because I think that automatically auto-closing as approved is problematic because of two reasons.
First of all even veteran developers still can make mistakes like using a deprecated function, writing non valid HTML&CSS or forget to include licensing information for a bundled ressource. So a reviewer should look at the theme, and as they are good developers, these reviews would be done in a couple of minutes.
Secondly I also fear that this might divide the theme development community into those whose themes get approved on the fast track and those who have to wait in line.
2) Different review stages
I like mercime’s idea a lot, because the Stage 1 review of verifying the licensing, credit links, theme name and screenshot could be done really fast and it would help cleaning the theme review queue of spammers and other malignant developers.
The second step would be also pretty fast since it involves little work besides downloading the theme and doing the checks with the plugins. This would mean that only the “worthy” themes make it to stage 3, in which a reviewer has to dig in and do all the manual checks that are needed.
3) The learning curve
I agree with Chip that the learning curve is steep. My main motivation for joining the WPTRT was to learn more about themes, as suggested by Matt in the State of the Word 2012.
There is a lot of self-education involved which is great for improving your skills as a WordPress developer (which happens to be my main job
) but it also means that you have to search for information if you are not clear on an issue.
In all of the cases in which I wasn’t sure if something was permitted I found the answer in the WPTRT mailing list archives. But searching in those archives is a real pain, so some kind of FAQ style ressource with categories, tags and a search function would be a better place to store all this knowledge.
For beginners it might be good to get coupled with a mentor when they start out reviewing themes. This mentor would be an experienced theme reviewer that could then help out with the first reviews of his protégé. Just like Emil mentioned, coders appreciate peer review, so I think for reviewers this might work as well.
4) The community
Besides Chip and Emil, I don’t even “know” any of the other theme reviewers.
Building a community is always a good idea!
So maybe we could interact a little more besides the mailing list, maybe by having a theme reviewer’s chat like the core developers do?
As far as the incentives mentioned by Tskk are concerned, I think the biggest reward might be appreciation from the community. If you are a core contributor, your name is in the release notes, if you are a plugin developer, your name is displayed in the plugin repository, etc. Not sure how this could work out for theme reviewers though.
Fingli 11:46 pm on November 1, 2012 Permalink |
The guidelines should have to be reduced. In half. Or at least the reviewers should follow more simple rules.
The current guidelines are very detailed, as such they are suitable for (kind of) WP standards for best practice but make review process slower and tedious. But to stimulate developers to stick to detailed guidelines there might be some ‘reward’ like badge in theme’s page or something else indicating that this theme is compatible with WP standards of good practice and quality.
How I imagine the process: It should be divided to two parts. The main will check for general basic things like the theme have to contain such and such files, such and such functions, no malicious/obfuscated code, no affiliate links and so… It it fulfilled the requirements make it pass.
The second part is up to author – if he/she wants may ask for full review according to WP guidelines, and if the theme pass it will receive a mark for quality.
Vicky Arulsingam 8:16 am on November 2, 2012 Permalink |
More important than reducing the number of guidelines is identifying tiers of approval. Mercime suggestion is a great idea, especially for new theme submissions. At the present with some tickets waiting over a month for a review, I find it’s faster to just do a complete review since the developer’s been waiting for so long.
The majority of issues that I’ve found are for things readily discovered in the Theme Guidelines and Theme Unit Tests. So is it that developers are not reading the guidelines or are those guidelines confusing?
This is where a regular reviewer’s meeting would be good so we can discuss the common problems we’re having.
I’m not in favor of having a “Trustworthy Developer” tag which would allow the developer to auto-approve their themes. It’s always good to have a second or third pair of eyes on your code – everyone makes mistakes from time to time.
Tskk 10:55 am on November 2, 2012 Permalink |
1) Create a list of guidelines and update it whenever a change is agreed upon in the mailing list.
2) Encourage reviewers by giving them priority review for 3 of their themes for say 100 themes they review
3) Theme authors to take a quiz of the guidelines to submit their theme. ( So the incoming themes will be already following the rules )
Chip Bennett 6:36 pm on November 2, 2012 Permalink |
Number 1) is essentially what we have now: the Guidelines are established, with a defined update mechanism and frequency. (The Guidelines are updated with each major core update, unless security or a similarly major issue necessitates an immediate revision.)
Number 2) would be very difficult to implement without encouraging quantity-over-quality reviews. It also moves us into the subjective area of favoritism, and would need to be handled very carefully in that regard.
Number 3) is, for the most part, covered by the Theme-Check Plugin, which the uploader script runs against the Theme before completing the upload and creating a Trac ticket.
tskk 7:04 pm on November 2, 2012 Permalink |
2) You can decide on boundaries for it, it will be nice to have that privilege
It will definitely encourage more reviewers but logistics has to be worked out.
Emil Uzelac 6:26 pm on November 2, 2012 Permalink |
Thank you all for comments, I see many great ideas around, should we maybe start getting the list together?
kwight 4:21 pm on November 16, 2012 Permalink |
It’s interesting reading this thread and all the comments about how vague and confusing it is to do a review. I think it’s heavily tied in to the same frustrations that submitters face when trying to make their code acceptable to pass a theme review: documentation of the guidelines.
I think a lot of these issues would solve themselves if we could present the guidelines in a clearer, more effective way. Each requirement could have 1) “What”: what is the requirement? 2) “Why”: Why is this a requirement? Best practice? Security? Breaks plugins if not followed? 3) “How”: How do we properly write this code? Actual code examples and, eventually, a demo site and demo theme downloads that show these requirements in action. In my head I even see each requirement displayed with those three tabs, with the “What” tab showing by default.
My problem with the guidelines is the same as my problem with the Codex in general: a huge, dense, mind-numbing block of text that makes most people run screaming. The guidelines have served well up to this point, but for theme review to grow and sustain itself, an overhaul of how we present the guidelines would be a huge bonus to reviewers and submitters alike.
rohitink 2:49 pm on November 19, 2012 Permalink |
Is it necessary for a Person to Provide a Theme URL for the theme to be approved ?
Chip Bennett 12:00 am on December 2, 2012 Permalink |
No. You can optionally omit both Theme URI and Author URI, though it is *recommended* to include at least one or the other.
kwight 7:55 pm on November 19, 2012 Permalink |
I love how extend/plugins gives “Requires” and “Compatible up to”. Reviewers would have a better idea of when to give an approved theme a closer look (changing guidelines, possible errors missed by previous reviewers), and users would have a better idea of the theme’s compatibility with their own install.
kwight 8:01 pm on November 19, 2012 Permalink |
A small thing that would help a lot would be to have a link directly to the ticket in the emails we receive from Trac.