New plugin: Background Update Tester

New plugin: Background Update Tester, from @dd32 and me. It does what it says… Most sites are able to apply updates in the background. This plugin checks your site for compatibility and explains any problems.

After activating this plugin, visit the Dashboard → Update Tester screen. (If you are using multisite, visit Updates → Update Tester in the network admin.)

Feedback and enhancement requests welcome. (I think we should add the ability to send a test email to see that your install can email you.) This is an early rough cut.

#3-7, #dev-notes

Tonight’s 3.7 nightly build is marked WordPress 3.7.1…

Tonight’s 3.7 nightly build is marked WordPress 3.7.1-beta1. Here’s the zip or use the Beta Tester plugin set to “point release nightlies.” It fixes #25689, #25690, #25700, #25705, and #25706. (See the report.) We’re also looking into a PCRE UTF-8 issue in #25709.

#25700 is a particularly nasty bug that affects working with image captions in the visual editor. It was caused by a regression in Uglify.js. Please test. (It should also work with 3.7 with SCRIPT_DEBUG set, sans-patch.)

As noted in my auto updates post, update fatigue is much less of an issue now, so our strategy for maintenance releases will surely undergo a shift. In this case, though, this one would have required a quick release no matter what. (We just might have felt a little bad cause of it.) Usually critical bugs are also hard to trigger, thus affecting few users. This one is easy to trigger.

At this point, I’d expect 3.7.1 to drop over the weekend.

#3-7, #3-7-1

The definitive guide to disabling auto updates in WordPress 3.7

There are a lot of ways to adjust automatic background updates. A number of constants and filters offer a range of control, from the fine-grained to the heavy-handed. I will readily admit there are a few compelling reasons to disable auto updates, including:

  • You manage your site using version control
  • You implement your own deployment mechanism (potentially to multiple servers)
  • You are a managed WordPress host and feel confident in pushing timely updates yourself

I’d argue that “I don’t want them” is not a compelling reason for disabling updates. If you don’t keep your site up to date, you are making the web a less safe place for you and everyone who visits your website.

Background updates are incredibly, incredibly safe. Sites already running WordPress 3.7 have attempted more than 110,000 updates without a single critical failure, thanks to a number of verification steps that have made updates that much more reliable. A background update for a minor or security release (which is all they are enabled for, by default) means downloading and copying over just a few files. We’ve gotten really good at that. We’ve also spent years honing our craft of shipping stable and targeted fixes in minor releases — we don’t indiscriminately backport bug fixes. They must be serious bugs, and fixes go through additional levels of review, including at least two lead developers. And, we have the ability to roll out an automatic update over a period of hours or days. For 3.7.1, we’ll likely see how a few hours of user-initiated updates go before telling about 1% of sites to update, then steadily increase that percentage.

Automatic updates also support older branches. If the current version is 3.8.1, and we release a new 3.8.2 security release, we now have the option to release a 3.7.2 package with those fixes so 3.7 and 3.7.1 installs can automatically update to a secure version. (You’ll still be told to update to 3.8.2, of course.) Yes, automatic updates will definitely change how we approach maintenance releases and security fixes. We’d love the opportunity to keep lagging installs secure. I’d also expect more frequent maintenance releases for the current branch of WordPress, since update fatigue is much less of an issue now.

Updates are also really fast. Installs take about 24 seconds to update on average, but that includes downloading and unzipping the package. A core update should put your site into maintenance mode for only a few seconds.

What if you want automatic updates, but your site is telling you it is unable to apply these updates automatically? [Updated:] There is a new plugin, Background Update Tester, that will explain exactly why your site doesn’t support automatic background updates, and if necessary, what to ask your web host. This didn’t ship in core because it’s a pretty complicated flowchart and most options for recourse are complicated and technical. We think about 80 percent of installs support automatic updates. Most common reasons why they don’t work: your install’s file permissions require us to use FTP, and we don’t know your password; you have it disabled (or are using version control, see below); the WordPress cron (which also handles things like scheduled posts) doesn’t work on your server; or your install doesn’t allow for secure communication with WordPress.org. There are also situations (like inconsistent file permissions) where WordPress thinks it can do an update, but when it tries to, it finds out it can’t. (Your install will email you in that case.)

It’s worth noting that the “automatic updater” controls more than just WordPress core. If the updater finds it can’t or shouldn’t update, it’ll still send site administrator an email. (Want to disable only that? It’s also covered in this post.) The automatic updater also supports themes and plugins on an opt-in basis. And by default, translations (for themes, plugins, and eventually core) are updated automatically. At some point in the future, the WordPress.org plugin security team will be able to suggest that installs automatically update malicious or dangerously insecure plugins. That’s a huge win for a safer web.

It’s pretty clear that disabling the entire updater also disables some pretty nice features! Selectively disabling only what you want is going to be best. So, here’s the different ways to disable automatic background updates:

1. Use version control.

If WordPress detects a version control system, it recognizes you know what you are doing and avoids automatic updates of any kind. It looks for Subversion, Git, Mercurial, and Bazaar, and it looks everywhere.

It works by searching two directories (ABSPATH and whatever you are updating, like WP_PLUGINS_DIR, or WP_LANG_DIR) for VCS directories (.svn, .git, .hg, .bz). And it looks a level up, too — and keeps looking until it reaches the root of the drive. So if you are running a single Subversion checkout at / or /var/www/ or /var/www/mysite.com/, the WordPress install at /var/www/mysite.com/public_html/wordpress/ will be blocked from receiving updates. Clearly, it errs on the site of caution.

There is a filter, automatic_updates_is_vcs_checkout. If you’d like to use automatic updates anyway, you can just return false through that filter. The filter is also passed the directory it is searching in addition to ABSPATH, so if you wanted to update languages even though you were running a Subversion checkout, this would work:

function update_languages_vcs( $checkout, $context ) {
	if ( $context == WP_LANG_DIR )
		return false;
	return $checkout;
}
add_filter( 'automatic_updates_is_vcs_checkout', 'update_languages_vcs', 10, 2 );

If WordPress can’t update due to version control, the admin email will still get notified of new minor releases.

One technique that may interest some is to allow automatic updates even when using version control, with the intention of then checking in the changes once you get around to it. For the personal site of a busy developer, it’s an intriguing idea.

2. Disallow all file changes, period.

Most people are not going to want to use this one (honestly, please don’t), but if you are already doing your own deployments or are managing multiple servers, you might already be.

The DISALLOW_FILE_MODS constant blocks any kind of filesystem changes, not just by background updates but by all users as well. So, gone are the file editors; the ability to update core, themes, or plugins; and the ability to install new themes or plugins. This is crazy stupid to use unless you know exactly what you are doing. I am only mentioning it because I wanted to make it clear that automatic updates is smart enough to avoid breaking any custom deployments.

This will also block the update notifications sent via email for minor core releases.

3. Disallow the entire automatic updater.

The constant AUTOMATIC_UPDATER_DISABLED can be used to disable the automatic updater entirely. It’s like DISALLOW_FILE_MODS — no changes allowed at all — but it’s specific to the auto updater.

Don’t use this to block only core updates! You’ll also be blocking a lot of other functionality. You won’t get translation updates (language packs) for core, themes, and plugins. You won’t receive update notifications sent via email to alert you of new WordPress releases. It also disables all opportunity for fine-grained control.

There are very limited use cases for disabling the automatic updater but not disabling all file changes with DISALLOW_FILE_MODS. Just remember it disables everything, not just core updates, which are just one component.

There’s also a filter by the same name, automatic_updater_disabled. (It overrides the constant.)

4. Disable only core updates.

The easiest way to manipulate core updates is with the WP_AUTO_UPDATE_CORE constant:

# Disables all core updates:
define( 'WP_AUTO_UPDATE_CORE', false );

# Enables all core updates, including minor and major:
define( 'WP_AUTO_UPDATE_CORE', true );

# Enables core updates for minor releases (default):
define( 'WP_AUTO_UPDATE_CORE', 'minor' );

There are also some filters you can use for even finer control, which override the constant if used: allow_dev_auto_core_updates (like updating from 3.7-RC to 3.7-RC2), allow_minor_auto_core_updates (updating from 3.7 to 3.7.1), allow_major_auto_core_updates (3.7 to 3.8). Return true through these filters to allow such updates, false to disallow.

5. Manipulate core, plugin, theme, and translation updates as they are prepared to be run.

The previous configuration options are all-or-nothing. You may, however, want something more fine-grained. The auto_update_$type filter (auto_update_core, auto_update_plugin, auto_update_theme, auto_update_translation) is fired for specific updates, as they are ready to be updated. This filter is passed the actual update object that describes what WordPress is about to update. This means you can selectively enable individual plugins or themes to update, for example, or whitelist upcoming core updates.

6. Manipulate whether notification emails are sent

Emails are sent in three situations: a result email after a core auto update, a notification email when WordPress can’t update irself, and a debugging email when running a development version of WordPress (alpha/beta/RC).

The result email comes in three forms:

  • A successful update. Nice!
  • An update that couldn’t occur. As in, WordPress tried to update, but failed early, like an inconsistent permissions error it was able to catch.
  • A critical failure, when the update failed in the middle of copying files.
  • (Note, we’ve yet to see a single critical failure in the wild. Yeah, it’s that reliable.)

    You can stop these emails from being sent by returning false via the auto_core_update_send_email filter:

/* @param bool   $send        Whether to send the email. Default true.
 * @param string $type        The type of email to send.
 *                            Can be one of 'success', 'fail', 'critical'.
 * @param object $core_update The update offer that was attempted.
 * @param mixed  $result      The result for the core update. Can be WP_Error.
 */
apply_filters( 'auto_core_update_send_email', true, $type, $core_update, $result );

Next, the core notification email is controlled by the send_core_update_notification_email filter. By default, administrators are notified when the update offer received from WordPress.org sets a particular flag — of course, only if the install is unable to update. It’ll only email you once per new version, unless the admin email changes. Returning true means the install will always email you when there is a new update, even if the API has not yet instructed the install to do so. Returning false prevents the email from ever being sent.

Finally, the debugging email is a complete log output of all auto updates performed — core, translations, plugins, and themes. It is as if you clicked update yourself and watched the text scroll by; it also includes additional error data if something went wrong. This email controlled by the automatic_updates_send_debug_email filter. Returning false will prevent this email from being sent when running a development install, while returning true will send this email to all versions, including when you’re on a stable release.

***

In the end, please choose wisely. If you’re going to block core updates, use WP_AUTO_CORE_UPDATE.

I’ve gotten this question a few times now: should hosts that offer WordPress update services be disabling self-updates? I have no qualms with hosts who plan to reliably provide their own updates in a timely manner to do updates on their own. Because they have full, unadulterated access to the server, they can do it much more effectively than WordPress can. They can also offer support services. It’s also much more awesome to have a hosting company on board, than to lean back and rest on WordPress’s laurels. Many hosting companies are also setting the bar high and paving the way forward by pushing out major releases after a few weeks of testing, which is fantastic. Of course, this means the host must act immediately and responsibly to push out maintenance and security releases. This is something most of them are already doing, even inside the first 12 hours. (That’s how often installs check WordPress.org to see if there is a pending update.) Honestly, I’m just really excited about how much the community is embracing all of this!

So, if you’re a managed host that handles your own updates, consider using WP_AUTO_CORE_UPDATE versus completely disabling the entire automatic updater. (If you want to send your own emails, block the ones WordPress sends using the send_core_update_notification_email filter.) There’s a lot more to come here, and it would be a shame if users were cut off from its full potential.

If you are a developer who wants to learn more, start with the WP_Automatic_Updater class in wp-admin/includes/class-wp-upgrader.php. Also check out @dd32‘s previous post.

#3-7, #dev-notes

JSON Encoding and SSL for api.wordpress.org Communication in WordPress 3.7

There are two changes to the way WordPress communicates with api.wordpress.org in 3.7: JSON encoding and SSL.

JSON Encoding

In versions prior to WordPress 3.7, data that WordPress sends to (and receives from) api.wordpress.org is serialized using PHP’s native serialization functions. PHP-serialization has two main problems:

  • Security: It has the potential to lead to security exploits via PHP object injection.
  • Portability: It’s hard to unserialize these strings in other languages besides PHP.

In WordPress 3.7, most API calls have now changed so they send and receive JSON encoded data instead. The three major ones are:

  • Core update checks
  • Plugin update checks
  • Theme update checks

The following calls have also changed, but you’re probably not so interested in these:

  • Importers list
  • Credits list
  • Browse Happy (the browser version check)

How might this affect plugin or theme developers?

In general this won’t affect developers at all. If your plugin or theme just consumes the API then you don’t have to make any changes. The API calls that send and received JSON encoded data have all had their version numbers bumped from 1.0 to 1.1 (for example, api.wordpress.org/plugins/update-check/1.1/. If you are consuming the version 1.0 endpoints you’ll continue to get PHP-serialized data. If you want JSON encoded data, you can switch to using the version 1.1 endpoints.

There is one situation that developers may need to account for. If your plugin or theme hooks into the update API requests in order to remove certain plugins or themes from the update checks, your code may need updating.

A common method for removing a plugin or theme from the update checks is to hook into http_request_args, unserialize the data being sent to the API, remove the given theme or plugin from the data, and serialize it again. This will no longer work in WordPress 3.7 and your code will need to be updated so it decodes and encodes the data as JSON instead.

An example of a plugin which has been updated to handle JSON encoding along with fallback support for PHP-serialization (depending on the version number in the API call) can be seen here: https://github.com/cftp/external-update-api/compare/f4d58e2…281a0ef

Note that there are two API calls which have not yet changed to using JSON encoding:

  • Plugin info
  • Theme info

These two calls will most likely be updated to use JSON encoding in WordPress 3.8.

SSL Communication

As part of the hardening process of this release, WordPress 3.7 will only communicate with api.wordpress.org using SSL (HTTPS) when the server supports it. This is an especially important security enhancement, given that automatic background updates are now a part of WordPress. Indeed, automatic background updates are disabled if the server cannot communicate securely with the api.wordpress.org.

How might this affect plugin or theme developers?

Again, this won’t affect developers in general. If your plugin or theme hooks into API calls you may need to update your code to it handles calls to https://api.wordpress.org/ in addition to https://api.wordpress.org/.

JSON encoding and support for SSL means the WordPress.org APIs are in a much better position going forward.

#3-7, #api, #dev-notes

The current plan is that WordPress 3.7 RC2-25890…

The current plan is that WordPress 3.7-RC2-25890 will be the last build of 3.7 before release. Download it here: https://wordpress.org/nightly-builds/wordpress-3.7-latest.zip.

Trunk (and thus the bleeding edge nightly) is 3.8-alpha, but other than the version number, the builds are the exact same.

Per the developer meeting today, we’ll be meeting at Thursday 14:00 UTC in IRC (#wordpress-dev) to see if everything is a go.

#3-7

Have you been receiving update failures with the…

Have you been receiving update failures with the error code mkdir_failed_pclzip? Then why haven’t you reported it? 😉

So, on October 14, I accidentally committed a change that deliberately raised this error when PclZip was used to extract downloaded ZIP files. If ZipArchive is present, it is used instead of PclZip, so this doesn’t affect most installs by default. But, there are a lot of installs that do use PclZip, and they are “stuck” on the October 14 nightly build, and are unable to update past that point.

So, a few options to fix this:

  • You can download the latest nightly build (or WordPress 3.7 RC1 will be released soon) and manually install it. (You probably just need to copy over the file wp-admin/includes/file.php.)
  • Or, you can open up wp-admin/includes/file.php, head to about line 747 and remove the comment on the if statement, as was done in changeset 25799.
  • Or, you can install the Zip PHP extension. 🙂

Of course, it is absolutely critical that an install is always able to update, even in development versions. @dd32 and I take that very seriously. Sorry for the lapse.

#3-7, #blamenacin, #mkdir_failed_pclzip

Agenda for today’s meeting in #wordpress dev at…

Agenda for today’s meeting in #wordpress-dev at 20:00 UTC.

  • Go over the last few days of progress, as well as what we’ve been learning via statistics for background updates. (Spoiler: excellent.)
  • Need a commit or revert on #20316 (garbage collect transients). It has a patch — the queries need testing and review.
  • Need a decision on #24963 (IIS stuff).
  • Finalize the about page. It’s going to be text-heavy as there’s nothing to show off UI-wise. What about a single giant three-column image across the top? I mean, it can be a picture of a sunrise for all I care. Or some screenshot of the WordPress admin. Or some photos from the crowds at WordCamp San Francisco and WordCamp Europe. Anyone have any ideas?
  • WordPress 3.7 Release Candidate 1 (by tonight).
  • Branch WordPress 3.7 (after RC1).

@dd32 and I are working on #10787 this afternoon as well.

#3-7, #agenda

Meeting today: Road to WordPress 3.7 Beta 1

For the meeting today (starts in ~20 minutes), we need to work on getting to 3.7 Beta 1. I think if all goes well, we can release Beta 1 tonight or tomorrow morning. @dd32 has been doing some incredible work on automatic updates. If you haven’t read his post on them, please do!

So, there are a few things we need to discuss and make decisions on:

1. Search results ordered by relevance, #7394. Do we take a chance on this?

2. Should we begin to require that users supply their current password in order to change their password? #20140.

3. Should we consider a slightly more conservative different approach to transient garbage collection? We do not want updates to be blamed for breaking sites. #20316. What would this approach look like?

4. How should individuals be notified via email when it comes to automatic background updates? #10787.

5. How should individuals be notified their own dashboard that their site will be safe if there is a security release?

Proposal for points 4 and 5: A green checkmark on about.php and update-core.php will let them know their install is OK and good to go for automatic background updates for security releases. (We can easily verify this without prompting the user.)

If for some reason we attempt an automatic background update and it fails to complete, then we need to email get_site_option( 'admin_email' ). We need text for this email.

If they don’t have a green checkmark, we should wait five days, after which we email them reminding them a security release is available. A timestamp five days from the point of release will be pushed via the API. Once that time is crossed, an email will be sent. (For a particularly critical situation, we could shorten the timeframe. For a non-security minor release, we might avoid having an email sent all together). We also need text for this email (which will be a fairly nice email).

This does not accomplish all of #10787 (“Email administrators that an update is available for core, plugins, and themes”) but it seems to be a good security balance.

***

My target is Beta 2 early next week, and Release Candidate 1 as early as next Friday. Here is our current progress on tickets:

  • There are no more enhancements or feature requests open on the 3.7 milestone.
  • There are under 20 tasks remaining for 3.7. Many of these are near completion. These need to be cleared by Beta 2.
  • There are 150 bugs open on the 3.7 milestone. We need to reduce this number to about 75 by Friday, and to zero by the end of next week. As in, these need to be cleared by RC1.

We’ll likely branch 3.7 at the WordCamp Europe contributor day on October 7, which means anything that is punted out of 3.7 can still make it into 3.8 starting in just two short weeks from now. A reminder, this is a very short release cycle, and 3.8 is just a few months away and begins in earnest very soon. Here is what our philosophies document says about fast, agile release cycles with crisp deadlines:

The more frequent and regular releases are, the less important it is for any particular feature to be in this release. If it doesn’t make it for this one, it’ll just be a few months before the next one. When releases become unpredictable or few and far between, there’s more pressure to try and squeeze in that one more thing because it’s going to be so long before the next one. Delay begets delay.

If we’re not confident that three weeks is long enough for something to properly soak in trunk, let’s not be afraid to wait until 3.8.

#3-7, #agenda

Automatic Core Updates, an update

Over the last few weeks I’ve been working on bringing Automatic Core Updates to WordPress 3.7 through #22704, Up until now it’s been disabled in trunk while development in progress – today however, that all changes.

As of [25598] WordPress 3.7+ installs will begin updating themselves without the need of user input every time a new security release has been released, or in the case of all us development users, it’ll update daily to the latest nightly if possible.

Automatic Updates are unattended, and by default, will only update WordPress to security releases (for example, from 3.7 to 3.7.1, but not from 3.7.1 to 3.8). Great lengths will be taken to ensure that no site will break as the result of an Automatic update.

Note: Filter and constant names may change pending feedback and discussion.

Edit, October 18: WordPress 3.7 RC1 changed some filter names. This post now reflects the latest names. AUTOMATIC_UPDATER_DISABLED does not work in RC1. This is fixed in 3.7-RC1-25851.

In order for Automatic Updates to be enabled, there are a few simple requirements:

  1. If the install uses FTP for updates (and prompts for credentials), automatic updates are disabled
  2. If the install is running as a SVN or GIT checkout, automatic updates are disabled
  3. If the constants DISALLOW_FILE_MODS or AUTOMATIC_UPDATER_DISABLED are defined, automatic updates are disabled
  4. If the constant WP_AUTO_UPDATE_CORE is defined as false, automatic updates are disabled
  5. Your WordPress install also needs to be able to contact WordPress.org over HTTPS connections, so your PHP install also needs OpenSSL installed and working
  6. Wp-Cron needs to be operational, if for some reason cron fails to work for your install, Automatic Updates will also be unavailable

We’ve also been working on a bunch of related features to make updates even more bulletproof than before, including HTTP, Filesystem, and File verification enhancements, amongst many other things.

How do I test it?

If you’d like to test this out, the simplest way is to simply create a new non-svn checkout of trunk and visit the site regularly to make the cron task run.
If you’d like to test this out, and you’re running SVN/GIT, you can use add_filter( 'auto_upgrade_ignore_checkout_status', '__return_true' ); in a mu-plugin to make it ignore the checkout status, note, that you’ll lose any modifications you’ve made to core files.

After each update, you’ll receive an email with a summary of the actions taken, which will let you know if the upgrade completed, or encountered any problems – the emails are still a work in progress, the email currently in trunk is designed for developers, and may not be the same in the final release.

I don’t like the sound of this, How do I turn it off?

  1. If you’re using a deployment system that uses SVN or GIT, it’s disabled by default
  2. The simplest way to disable it is to add define( 'AUTOMATIC_UPDATER_DISABLED', true ); to your wp-config.php file
  3. You can also make use of the auto_upgrader_disabled automatic_updater_disabled, or, auto_upgrade_core auto_update_core filters

Are there any more hidden features?

This is WordPress we’re talking about, of course there’s a bunch more that it can do!

  1. If you’re using a non-English install, this will also automatically update any plugin/theme Language Packs which are installed, hopefully resulting in a better i18n experience, See #18200 for more information on Language packs, there’ll be a follow up post in the near future explaining how and when Language packs will work
  2. Plugin & Theme Updates! – You can hook into the 'auto_upgrade_plugin' or 'auto_upgrade_theme' 'auto_update_plugin' and 'auto_update_theme' filters to enable auto-updates of one, or many plugins/themes note: Plugin/Theme updates will not be enabled by default in WordPress 3.7
  3. By default, Core Auto-Updates will only apply to WordPress Security & nightly releases, that is, from 3.7.0 to 3.7.1, 3.7.1 will not automatically update to 3.8.0 – This can however be changed,  you can simply add define( 'WP_AUTO_UPDATE_CORE', true ); to your wp-config.php file and it’ll happen automatically!

What can I do to help?

Please test it out and report any bugs you find! Triggering automatic updates can be a little bit difficult since it currently relies upon a twice daily cron job, so the easiest way is to just create a new nightly install and visit the site once a day to cause the cron to be initiated, If you’re game to run it on a production site, please be aware that WordPress will go into Maintenance mode during the upgrade.

You can also head over to Trac and check out report/48, which is a temporary 3.7 report containing anything related to automatic updates and Language packs (more on that in a few days)

#3-7, #updates

Inline Docs Progress for 3.7

After last week’s weekly Inline Docs meeting we decided to begin doing a weekly ticket triage on Wednesdays.

As of last Wednesday, we’re happy to report that 19 patches have been committed to core from 12 different contributors. We’ve also updated the original make/core post to reflect the files that have been completed (please go claim some!). You can see the full report of last week’s triage on Trac. It’s really spectacular, and you should show some gratitude towards Drew for putting the report together.

Finally, beginning this week, our weekly inline-docs office hours are moving to #wordpress-sfd on Freenode. They’ll continue to be held on Wednesdays at 18:00 UTC, and we hope you can join us!

#3-7, #inline-docs