Search Issues

UPDATE (@dd32): All issues should be resolved as of 2:15AM UTC. The root cause was a change in the behaviour of Jetpack Search which we rely upon causing queries to fail. A network outage had caused issues for some queries earlier in the day, but was completely unrelated.

You may have noticed that search is acting up. Per @dd32:

w.org is experiencing a few network issues at present in the datacenter, it’s likely that connectivity between the API and wp.com’s elastic search is up-and-down, and when it’s down, search will be offline.

Yes, that means search for plugins too.

There’s nothing to do but wait at this point. It may be up and down while the connectivity is being sorted.

Rewrite endpoints API

During the development of WordPress 3.4 I have spent some time working on improving the rewrite API. One of the tickets this involved was #16303: “Improve documentation and usability of WP_Rewrite Endpoint support”. Endpoints are a really cool feature of the rewrite API, but unfortunately also little known and misunderstood. So, with this post my aim is to get more plugin developers to read and understand the new and improved endpoint documentation.

What are endpoints?

Using endpoints allows you to easily create rewrite rules to catch the normal WordPress URLs, but with a little extra at the end. For example, you could use an endpoint to match all post URLs followed by “gallery” and display all of the images used in a post, e.g. http://example.com/my-fantastic-post/gallery/.

A simple case like this is relatively easy to achieve with your own custom rewrite rules. However, the power of endpoints shines for more complex situations. What if you wanted to recognise URLs for posts and pages ending with “gallery”? What if you wanted to be able to catch multiple different archive URLs, e.g. day, month, year and category archives, with “xml” appended in order to output an XML representation of the archive? For these situations endpoints are very useful as they allow you to add a string to the end of multiple rewrite structures with a single function call.

How to use them

There is one function for interacting with endpoints: add_rewrite_endpoint(). It takes two parameters $name and $places.

$name is a string and is, wait for it… the name of the endpoint. $name is what is used in the URL and is the name of the query variable that the endpoint URL will be rewritten to. For example, an endpoint named “print” added to post permalinks would use a URL like http://example.com/my-awesome-post/print/.

$places is an integer value which represents the locations (places) to which the endpoint will be added, e.g. posts, pages or year achives. To understand $places you need to learn about the endpoint mask constants.

In wp-includes/rewrite.php (browse wp-includes/rewrite.php on Trac) a number of constants are defined all with names beginning with “EP_”:

define('EP_NONE', 0);        // 0000000000000
define('EP_PERMALINK', 1);   // 0000000000001
define('EP_ATTACHMENT', 2);  // 0000000000010
define('EP_DATE', 4);        // 0000000000100
define('EP_YEAR', 8);        // 0000000001000
// ...
define('EP_PAGES', 4096);    // 1000000000000
define('EP_ALL', 8191);      // 1111111111111

These are the endpoint masks which describe sets of URLs; post permalinks are described by EP_PERMALINK, year archives are EP_YEAR, etc. They should be thought of in terms of their binary values (see the comment I’ve added to the end of each line). Every EP_* mask, except for EP_ALL, is a different power of two and so has a different bit set to one. This allows us to build up combinations of endpoint masks by using the bitwise OR operator:

// all posts or attachments
EP_PERMALINK | EP_ATTACHMENT // 0000000000011

// all full dates (yyyy/mm/dd), years or pages
EP_DATE | EP_YEAR | EP_PAGES // 1000000001100

$places should also be thought of as a binary number. It should be set to one of the EP_* constants or a combination of them using the bitwise OR operator. If we wanted to add our endpoint to all post permalinks we would use EP_PERMALINK. For both posts and pages: EP_PERMALINK | EP_PAGES. For posts, pages, and categories: EP_PERMALINK | EP_PAGES | EP_CATEGORIES. There is also a special value to add an endpoint to all URLs that support endpoints: EP_ALL.

NB: The values of the EP_* constants are not guaranteed to stay the same which is why you must say $places = EP_PERMALINK and not $places = 2. This is particularly important for EP_ALL which will change every time a new endpoint mask is added.

It’s time to put this information into practise. The running example will be a plugin that adds JSON representations of our content using a new rewrite endpoint called “json”. So, the goal is to get URLs such as http://example.com/about/json/ to return a JSON response that gives information about the “about” page. To add the “json” endpoint to post and page rewrite structures:

add_rewrite_endpoint( 'json', EP_PERMALINK | EP_PAGES );

This is called in a function hooked into the init action:

function makeplugins_add_json_endpoint() {
    add_rewrite_endpoint( 'json', EP_PERMALINK | EP_PAGES );
}
add_action( 'init', 'makeplugins_add_json_endpoint' );

Now we want to act on requests for JSON content. This is done by hooking into template_redirect. We want to detect appropriate requests and include our custom template for serving up posts and pages in JSON format:

function makeplugins_json_template_redirect() {
    global $wp_query;

    // if this is not a request for json or a singular object then bail
    if ( ! isset( $wp_query->query_vars['json'] ) || ! is_singular() )
        return;

    // include custom template
    include dirname( __FILE__ ) . '/json-template.php';
    exit;
}
add_action( 'template_redirect', 'makeplugins_json_template_redirect' );

And we’re done. For a full example plugin see https://gist.github.com/2891111.

How do they work?

The best way to understand how anything works is to take a look at the source, so let’s do that. Endpoints are added with the add_rewrite_endpoint() function in wp-includes/rewrite.php:

/**
 * Add an endpoint, like /trackback/.
 *
 * Adding an endpoint creates extra rewrite rules for each of the matching
 * places specified by the provided bitmask. For example:
 *
 * <code>
 * add_rewrite_endpoint( 'json', EP_PERMALINK | EP_PAGES );
 * </code>
 *
 * will add a new rewrite rule ending with "json(/(.*))?/?$" for every permastruct
 * that describes a permalink (post) or page. This is rewritten to "json=$match"
 * where $match is the part of the URL matched by the endpoint regex (e.g. "foo" in
 * "/json/foo/").
 *
 * A new query var with the same name as the endpoint will also be created.
 *
 * When specifying $places ensure that you are using the EP_* constants (or a
 * combination of them using the bitwise OR operator) as their values are not
 * guaranteed to remain static (especially EP_ALL).
 *
 * Be sure to flush the rewrite rules - flush_rewrite_rules() - when your plugin gets
 * activated and deactivated.
 *
 * @since 2.1.0
 * @see WP_Rewrite::add_endpoint()
 * @global object $wp_rewrite
 *
 * @param string $name Name of the endpoint.
 * @param int $places Endpoint mask describing the places the endpoint should be added.
 */
function add_rewrite_endpoint( $name, $places ) {
      global $wp_rewrite;
      $wp_rewrite->add_endpoint( $name, $places );
}

So this is just a wrapper for the add_endpoint() method of the WP_Rewrite class. Although the (excellent!) documentation gives us some clues as to what it does we’ll have to dig deeper to find the how:

/**
 * Add an endpoint, like /trackback/.
 *
 * See {@link add_rewrite_endpoint()} for full documentation.
 *
 * @see add_rewrite_endpoint()
 * @since 2.1.0
 * @access public
 * @uses WP::add_query_var()
 *
 * @param string $name Name of the endpoint.
 * @param int $places Endpoint mask describing the places the endpoint should be added.
 */
function add_endpoint($name, $places) {
    global $wp;
    $this->endpoints[] = array ( $places, $name );
    $wp->add_query_var($name);
}

Another very short and simple function. All it does is append the two parameters passed to it to the private $endpoints property of the WP_Rewrite class and also add a new query variable using WP::add_query_var().

Okay, so that’s still not useful for a full understanding of endpoints. All we know is that the arguments you pass to add_rewrite_endpoint() are stored in a private array of the $wp_rewrite global. To find out more we’ll have to search wp-includes/rewrite.php for “>endpoints” (i.e. code accessing the WP_Rewrite::$endpoints property). There are only three references to this: WP_Rewrite::add_endpoint() we have seen, WP_Rewrite::init() is boring (initialising the array), and the third is WP_Rewrite::generate_rewrite_rules():

$ep_query_append = array ();
foreach ( (array) $this->endpoints as $endpoint) {
    //match everything after the endpoint name, but allow for nothing to appear there
    $epmatch = $endpoint[1] . '(/(.*))?/?$';
    //this will be appended on to the rest of the query for each dir
    $epquery = '&' . $endpoint[1] . '=';
    $ep_query_append[$epmatch] = array ( $endpoint[0], $epquery );
}

// ... a lot of code removed ...

foreach ( (array) $ep_query_append as $regex => $ep) {
    //add the endpoints on if the mask fits
    if ( $ep[0] & $ep_mask || $ep[0] & $ep_mask_specific )
        $rewrite[$match . $regex] = $index . '?' . $query . $ep[1] . $this->preg_index($num_toks + 2);
}

In the code above the first foreach is looping through the defined endpoints and building a new array called $ep_query_append. This new array uses regular expressions that match a specific endpoint as keys and the values are the endpoint $places and $epquery which is a partial query string to append to a full query. So, for our JSON endpoint example we would get:

$ep_query_append[ 'json(/(.*))?/?$' ] = array( EP_PERMALINK | EP_PAGES, '&json=' );

The second loop generates the final rewrite rules for our endpoint. It loops through $ep_query_append checking if the current permastructure being generated has an endpoint mask, $ep_mask, that matches any of the endpoints. If the bitwise AND produces a non-zero value then there’s a match and the endpoint rewrite rules should be added to this permastructure.

For our JSON example, if WP_Rewrite::generate_rewrites_rules() has been called for the posts permalink structure then $ep_mask = EP_PERMALINK and $ep[0] = EP_PERMALINK | EP_PAGES. The bitwise AND of these values produces 1, therefore a new entry is added to $rewrite. Assuming that the post permalink structure is “/%postname%/” it would look something like:

$rewrite[ '([^/]+)/json(/(.*))?/?$' ] = 'index.php?name=$1&json=$3'

This is the final rewrite rule for our JSON endpoint applied to post permalinks. It matches a request for “post-slug/json/” and sets up the appropriate query variables “name” and “json”. Our template_redirect hook now picks this up and produces the required response.

And you made it to the end, phew! Time for a drink…

Conclusion

I hope that after all of that you understand how to use endpoints and how they work. If you have any questions please don’t hesitate to ask them in the comments. Always remember that the best way to understand a function is to look at the source and follow its execution.

New Directory Status

As everyone knows, the first phase of opening up the directory to more reviewers was getting on the new system. We’re not quite there yet, however a great deal of progress has been made!

So far, we’ve run into a few weird flow issues that are blocking us from being able to invite new people. The biggest issue is that if you know the old system, it’s easy to move tickets through the new one. But it’s set up in a way that is very very easy to make mistakes and put tickets in unrecoverable states. So we need to mitigate that as much as possible before we let new people in. Basically we don’t want to break things for users because we didn’t think about use-cases.

Okay, fine, you say. What can you do to help?

I’m glad you asked!

We have 100 tickets open in Meta Trac. You can install the meta-environment in VVV and help us out with patches. Sadly, the meta env isn’t complete. It’s missing data, so you’ll end up having to add in plugins in order to mess with the state flow.

But if you can’t patch, and I do understand that, remember to come to the Plugin Directory revamp meetings on Wednesday at 2200 UTC in #meta on Slack. And please, test test test everything! The more we break the directory, the better it is 🙂

SVN Status: Seems to be Okay

I know Dion mentioned it in a comment, but here’s the official… We think it’s okay now post (I delayed to be more sure).

The SVN sync stuff SEEMS to be okay. The main issues appear to be sorted out, so 🤞🏾

We’re keeping a close eye on it, but please do remember to be nice to our poor system 🙂

#announcement

SVN Syncing Issues Continued

tl;dr – Yes we know, yes we’re working on it, no you don’t need to email.

I’m really sorry about this issue, but right now literally all I know is that the tool we use to automagically schedule everything that happens after you use SVN to bump your plugins is acting like a truculent child. It’s slow, it’s dragging it’s feet, and it’s taking WAY more than six hours (which is usually the outside norm for this stuff) to finish, if it does at all. It took 36 hours for one plugin, and even then some people got weird results.

And no, we don’t really know why yet.

It’s possible this is related to the new directory. It’s possible it’s from the entire .org slowdown last week or maybe it’s because we released the Beta and everything is slow from that. We literally don’t know.

I apologize for a series of very curt emails, but with the volume of people complaining, we had to resort to an auto-reply of, basically, we know, please be patient. If I have anything else to tell you, I will post, but right now we don’t know why and we can’t magically tell you what we don’t know, so please be patient with us.

Also no, there’s not a ticket because this is actually outside the meta repository. That means it’s not open source, the part that’s busted. The people with the access are aware and yes, I’ve pushed to escalate this. But it’s the weekend and it’s Mothers’ Day in the US, so you’re just going to have to be a little extra patient.

Protocol Relative Enqueues

With the http/2 and https features of WordPress on the future plan, it’s time for a reminder about how to enqueue things. If you haven’t read John’s post about https configurations, please do. We want to make things work well for everyone and future proof your code 🙂

A common method to enqueue fonts is to use the CSS url like this:

wp_enqueue_style( 'my_awesome_css', 'http://somecoolurl.com/my-awesome.min.css' );

The problem with this, as we move to more and more of an https world, is that will cause errors with people who want that beautiful green padlock. In order to make their life easier, please use protocol relative URLs in your enqueues:

wp_enqueue_style( 'my_awesome_css', '//somecoolurl.com/my-awesome.min.css' );

It’s really that simple. The future will thank you.

Edit: Yes, if a url has HTTPS and you can use it, use it. Eventually we’ll all be https and none of this will matter, but hard coding in http is making life difficult 🙂

#schema

Repository Syncing Issues (Updated)

Update from Otto: “The brunt of the plugin-delay problem has been solved now, so plugins should be nice and speedy again. There may be a few stragglers that got rescheduled for later due to the overload, they will clear themselves up in the next couple hours.”

Due to a network issue in the wee hours of May 11, updates to SVN are taking longer than normal to show up on the directory. The issue created a 2 hour backlog, which normally would work itself out pretty quickly. It’s currently being exacerbated by everyone who sees their code NOT showing up right away and pushing it again.

As Otto says:

In short, when it’s being slow, then it’s being slow and nothing you can do will speed it up, so just relax, go outside, walk around, and wait for it.

Do we want to make the process faster? Of course. And if you’re interested in helping that, please check out Meta ticket #1578 to understand how it was all written for the new directory.

To remind you:

  • Changes can take a couple hours to display even on the good days. While it is generally much shorter, 6-12 hours is about when you should start wondering if something’s up, not before.
  • Multiple commits to SVN means multiple builds of your code. The more times you commit, the more builds, and the slower it goes for everyone. Commit once, then go make a sandwich.
  • Zips are built for for every tag’d release in your repository. Fewer tags, faster builds. We advocate removing older builds as they’re generally not necessary for use.
  • Patience, patience, patience. With over 60k plugins (counting the closed ones), this is a HUGE database of things.

#announcement

Server Load Woes

If you got an error like this when using SVN today, it’s not just you:

Committing transaction...:   Error: Commit failed (details follow):  Error: Error running context: The server unexpectedly closed the connection.  Completed!:

Or

Committing transaction...:   Error: Commit failed (details follow):  Error: Failed to start '/home/svn/repos/wp-plugins/hooks/pre-commit' hook  

There was a high server load on WordPress.org on Monday May 8th, which caused the system to have those errors. When it happens to you, just go make a coffee or bake a pie and come back in a bit.

#downtime, #server-load

Handling Bad Reviews

In general, the Plugin Review team is not the go-to recourse for bad reviews.  Instead, we have a totally brilliant forum support team! There’s some overlap of jurisdiction of course, and some of us are on both teams, but the point here is you should go to the right group to get the right help.

I’m also going to put this out there. You will get a bad review. Most of the time, it will not be deleted. So before you get any further in this post, know that the way you chose to respond, in public, to a 1-star review of your plugin is your own choice.

Our goal with the WordPress.org repository is to have a good place for users to get plugins that fulfill their needs. The reviews are an extension of that, and should viewed as a way for users to educate other users on their experiences. Also a review is about an experience. If someone’s experience with your product is poor, that doesn’t make their review invalid. And to go back to that previous statement, the way you react to those poor experiences is going to impact your reputation, and that of your plugin, a heck of a lot more than that review.

Now, that said, we have a few ‘common’ types of problems with reviews. This post is going to help you handle them and explain when you should call for help, as well as from whom. Later on we’ll be adding it to our documentation, once it’s refined as best we can make it. Please remember, we do not want to make a ‘rule’ for everything. That just invites people to play rules-lawyers and tip over everyone’s cornflakes.

Here’s how you do it and when and why.

First off… How to add a tag!

99.999999% of the time, you’re going to be adding ‘tags’ to posts. This is so easy, you may kick yourself for missing it. On a post, look on the right hand side, under About this Topic and you’ll see a section for Tags

Tags are listed on the right hand side of a post

This is a free-form field where you can add any tag you want. Anyone can add any tag. The forum moderators have an easy way to know who added what, though, so keep in mind we do monitor that. If you want to add a tag to a post and reply, add the tag, press the Add button, and THEN come back to reply. It works better.

Tag abuse (that is calling moderators needlessly) is not okay. Be smart. Be thoughtful. Remember that every last member of the forum and plugin teams is a volunteer. We’re not being paid by Automattic to do this.

The spam review

This is easy. Don’t reply, just add the tag modlook to the post and walk away. The forum team will delete it. If you think it may not be obvious spam, add the tag spam as well.

The sockpuppet review

When a person (or group of persons) makes multiple accounts with the sole intention of leaving reviews on their own plugins (or leaving poor reviews on their competitors), this is called being a Sock Puppet.

This behavior is expressly NOT welcome on the WordPress Forums as it is spamming. But it comes in two flavors:

  1. Someone 5-star spamming their own plugin
  2. Someone 1-star spamming their competition

Both are bad behavior. Both will get plugins removed from the repository and a stern email from us. If you’re doing this, stop right away. Contact your team and tell them ‘Don’t do this!’ Also keep in mind, asking everyone in your company to 5-star review your own plugins is gauche. I mean, really. You’re stacking the deck on purpose and that’s not beneficial to anyone.

Again, do not reply! Add the tag modlook AND sockpuppet to the post and walk away.

The attack/troll review

These are the worst. When someone attacks you and the review seems like all it exists for is to make you feel terrible, you’re going to have to take a deep breath and walk away. An attack is a troll, regardless of how the original poster (OP) feels, they’ve basically been a troll. They’re writing something they know will make you mad and hurt and angry, and they’re doing it on purpose. That’s a troll. And you shouldn’t feed the trolls. You won’t win, and you’ll just make yourself look bad.

Again, do not reply! Add the tag modlook to the post and walk away. These are usually pretty self evident after all.

The review that should have been a support post

This includes the sub-genre “People who submit 1-star reviews in order to emotionally blackmail you for support.”

We all get them.

  1. Reply with a link to the support section of your plugin (or directions on how to get support, or even a note that you don’t provide free support) and remind them that next time, they should ask for help before reviewing.
  2. See if you can fix the problem, but give it no more or less priority than you would any other support request.
  3. If you can solve it, ask them to modify their review. If they go back to https://wordpress.org/support/view/plugin-reviews/PLUGINNAME and scroll to the bottom, they can edit their reviews!

You’ll notice we’re not telling you to tag the post? Right now we can’t move a review into the support forums and vice versa, so there’s really no point. The forum moderators won’t do anything about it except say “Well, that does suck.” If we could move them, we would, but right now we technically don’t have that ability.

The review about your premium/pro version

If you upsell your plugin’s pro version in the free one, and someone leaves a bad review because the pro version they bought, on the basis of your free one, is bad, congratulations. The review stays. You opened the door with your upsell, encouraging them to do this, and that experience reflects on your plugin as a whole.

If you do not upsell, and there’s no direct link between the free and pro version, or the plugin having the issue is a premium only add-on, tag it modlook and someone will come take a look.

The review about someone else’s plugin

This one can be fixed! Reply and let them know it’s not your plugin, it’s the other one, and then tag it modlook and then use the tag wrongplugin (all one word) to let the mods know what’s going on.

But I really need a plugin moderator!

Okay. So you think you’re an exception? Use the tag pluginmod and a plugin admin will come take a look. Be prepared, though, as we generally will perform a full review on your plugin and any and all guideline violations will result in your plugin being removed until you fix them. Including using too many tags.

#guidelines, #support

2016 in Review

It’s May. I forgot to post this. Go on and laugh 🙂

Anyway. At the end of last year, I looked at all the posts on Make/Updates and manually crafted out a spreadsheet that listed all the plugins submissions, rejections, approvals and closures. And here’s how the year looked:

image

If you don’t want to calculate it all yourself, I’ve made my Google sheet sharable. I didn’t go back and do 2015, and while I do plan to do one for 2017, we will have a big gap in numbers as I don’t have a way to calculate closed plugins at the moment.

By the way, there was a higher number of closed plugins in 2016 than ever before. That’s because more people than ever have asked us to close plugins, but also because we closed plugins without a committer with a valid email address. And that was a lot of people.

#announcement