Improved HTTPS detection and migration in WordPress 5.7

WordPress 5.7 will feature a number of enhancements which simplify the migrationMigration Moving the code, database and media files for a website site from one server to another. Most typically done when changing hosting companies. of a site from HTTPHTTP HTTP is an acronym for Hyper Text Transfer Protocol. HTTP is the underlying protocol used by the World Wide Web and this protocol defines how messages are formatted and transmitted, and what actions Web servers and browsers should take in response to various commands. to HTTPSHTTPS HTTPS is an acronym for Hyper Text Transfer Protocol Secure. HTTPS is the secure version of HTTP, the protocol over which data is sent between your browser and the website that you are connected to. The 'S' at the end of HTTPS stands for 'Secure'. It means all communications between your browser and the website are encrypted. This is especially helpful for protecting sensitive data like banking information.. As the foundation for providing the user with more accurate recommendations, WordPress will be able to detect whether the current environment already supports HTTPS. If this is the case, it provides a call-to-action button in the HTTPS status section in Site Health, which switches the site from HTTP to HTTPS with a single click.

Overall guidance in the Site Health section has been improved, now allowing hosting providers to supply a custom URLURL A specific web address of a website or web page on the Internet, such as a website’s URL www.wordpress.org with instructions or for a one-click update.

Walkthrough of the new feature with a quick demo, including highlighting of the problems this addresses

Detecting state of HTTPS and environment support

WordPress 5.7 introduces a new function wp_is_using_https(), which returns true if both the “Site Address” (home_url()) and “WordPress Address” (site_url()) are using HTTPS as their scheme. Essentially, changing both of these URLs to HTTPS formally indicates that the site is using HTTPS. While there are other ways to enable HTTPS partially in WordPress (e.g. with the FORCE_SSL_ADMIN constant), the new detection mechanism focuses on using HTTPS throughout the entire site, i.e. its frontend and backend.

In addition to providing a single function for checking whether HTTPS is being used, a new detection function wp_is_https_supported() can be called to check whether the environment supports HTTPS correctly. This is now used in Site Health to provide more accurate feedback: If the environment already supports HTTPS, the user can make the switch instantly, without involving their hosting company. Under the hood, the detection function is based on a new internal option https_detection_errors, which is controlled by a twice-daily Cron hook that works as follows:

  • It issues a request to the HTTPS version of the site with the sslverify argument enabled.
    • If the request succeeds, there is already a working SSLSSL Secure Sockets Layer. Provides a secure means of sending data over the internet. Used for authenticated and private actions. certificate in place.
      • In this case, the function also checks whether the HTMLHTML HyperText Markup Language. The semantic scripting language primarily used for outputting content in web browsers. body from the response actually belongs to the same WordPress site; this is typically the case, but the extra check is needed to cater for sites and environments that e.g. place custom HTML content under the URL. If the HTML body belongs to the same WordPress site, the environment is ready to be switched to HTTPS. Otherwise, switching to HTTPS then cannot reliably be accomplished by WordPress itself because the content is not entirely controlled by it.
    • If the request fails, it attempts the same request again, except that the sslverify argument is now disabled.
      • If that request succeeds, it means there is an SSL certificate, but it cannot be verified, which for example applies to self-signed certificates.
      • If that request fails as well, it means the site is entirely inaccessible over HTTPS.

The wp_is_https_supported() function simply looks at the https_detection_errors option controlled by the Cron hook, and it returns true if there are no errors stored.

One-click migration to HTTPS

A major pain point in migrating a WordPress site from HTTP to HTTPS has been the need to fix all the hard-coded URLs in existing database content which were still using the HTTP version, to avoid mixed content warnings. These URLs are usually migrated with a database replacement pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party or WP-CLIWP-CLI WP-CLI is the Command Line Interface for WordPress, used to do administrative and development tasks in a programmatic way. The project page is http://wp-cli.org/ https://make.wordpress.org/cli/, but that process is tedious and not intuitive to many WordPress administrators.

WordPress 5.7 introduces a new wp_replace_insecure_home_url() function which is hooked into various pieces of content to replace these insecure URLs on the fly. It relies on another new function wp_should_replace_insecure_home_url() which determines whether the URL replacement logic needs to run or not. All of the following conditions have to be fulfilled for the automatic content rewrites:

  • The site has to be using HTTPS, via wp_is_using_https().
  • A new internal flag option called https_migration_required has to be enabled. The option is automatically enabled when the “Site Address” and “WordPress Address” are switched to their HTTPS counterpart on a site with existing content. (In other words, a fresh_site that is immediately switched to HTTPS does not trigger the content rewrites logic.)
  • The “Site Address” and “WordPress Address” have to be using the same domain.

With the content rewriting of insecure URLs in place, the only change required to switch the site from HTTP to HTTPS is updating the “Site Address” and “WordPress Address” to their HTTPS counterparts. While this only entails updating two text input fields, it can still be simplified; this is why WordPress 5.7 includes another new function wp_update_urls_to_https() which updates both URLs accordingly. It also includes an extra check to verify that this resulted in WordPress correctly recognizing the site as using HTTPS; if not, the change automatically gets reverted to prevent any unexpected issues.

While the one-click migration introduced by WordPress 5.7 does not support advanced site configurations where e.g. “Site Address” and “WordPress Address” differ, it drastically simplifies migration in the common scenario; furthermore, the advanced configurations are often used by more technically savvy users that already know how to migrate to HTTPS anyway.

Administrators that would like to actually replace the URLs in the database can still do so. In that scenario, it is recommended to delete the https_migration_required option, to avoid the URL rewriting logic from running unnecessarily. Alternatively, the URL rewriting function can be unhooked entirely as follows:

remove_filter( 'the_content', 'wp_replace_insecure_home_url' );
remove_filter( 'the_excerpt', 'wp_replace_insecure_home_url' );
remove_filter( 'widget_text_content', 'wp_replace_insecure_home_url' );
remove_filter( 'wp_get_custom_css', 'wp_replace_insecure_home_url' );

Improved Site Health guidance

The HTTPS Status section in Site Health has been improved to guide the user more towards using HTTPS. If the environment already supports HTTPS (via wp_is_https_supported()), the UIUI User interface will now include a button to switch both site URLs with a single click (using wp_update_urls_to_https()). Users will need to have a new update_https metaMeta Meta is a term that refers to the inside workings of a group. For us, this is the team that works on internal WordPress sites like WordCamp Central and Make WordPress. capability in order to perform the switch; by default this capability is granted to every user that can both manage_options and update_core.

HTTPS Status section when HTTPS is not yet supported by the environment
HTTPS Status section when HTTPS is already supported by the environment

Various minor improvements have been made to more accurately describe the site’s configuration. For example, sites that rely on the WP_HOME or WP_SITEURL constant will see this reflected now, since that means WordPress cannot automatically update these URLs.

The severityseverity The seriousness of the ticket in the eyes of the reporter. Generally, severity is a judgment of how bad a bug is, while priority is its relationship to other bugs. of not using HTTPS in the overall HTTPS Status Site Health test is now set to “critical”, whereas before it was “recommended”. This means that sites which do not use HTTPS will now see the “Should be improved” state in the Site Health dashboard widgetWidget A WordPress Widget is a small block that performs a specific function. You can add these widgets in sidebars also known as widget-ready areas on your web page. WordPress widgets were originally created to provide a simple and easy-to-use way of giving design and structure control of the WordPress theme to the user., highlighting this further.

Providing custom support URLs for switching to HTTPS

In order to further help to guide WordPress administrators towards the switch to HTTPS, hosting providers that have their own support content for how to switch to HTTPS can now provide these URLs to WordPress so that users can get to that guidance directly from their administration panel.

  • Hosts which offer their own dedicated HTTPS support content can provide the URL that by setting a WP_UPDATE_HTTPS_URL environment variable or by hooking into a new wp_update_https_url filterFilter Filters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output.. If no such URL is provided, the default URL links to this HTTPS support article.
  • Hosts which offer a utility to automatically switch the site to HTTPS can provide the URL to do so by setting a WP_DIRECT_UPDATE_HTTPS_URL environment variable or by hooking into a new wp_direct_update_https_url filter. If no such URL is provided, the default URL triggers the aforementioned WordPress one-click mechanism.

For reference, see TracTrac An open source project by Edgewall Software that serves as a bug tracker and project management tool for WordPress. tickets #47577 about HTTPS detection and #51437 about HTTPS migration.

Props to @timothyblynjacobs for proofreading

#5-7, #dev-notes, #https, #site-health

Reminder: HTTPS meeting tomorrow

The weekly meetings about HTTPSHTTPS HTTPS is an acronym for Hyper Text Transfer Protocol Secure. HTTPS is the secure version of HTTP, the protocol over which data is sent between your browser and the website that you are connected to. The 'S' at the end of HTTPS stands for 'Secure'. It means all communications between your browser and the website are encrypted. This is especially helpful for protecting sensitive data like banking information. improvements in coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. will resume tomorrow at 16:00 UTC (Friday, September 30, 2016, 16:00 UTC) in the #core-http channel on SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/.. See you there!

#https

HTTPS Working Group

In WordPress 4.4 and 4.5, various pieces of work were done to improve HTTPSHTTPS HTTPS is an acronym for Hyper Text Transfer Protocol Secure. HTTPS is the secure version of HTTP, the protocol over which data is sent between your browser and the website that you are connected to. The 'S' at the end of HTTPS stands for 'Secure'. It means all communications between your browser and the website are encrypted. This is especially helpful for protecting sensitive data like banking information. support in coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress., but not much has been tackled since then. To address this, I’m going to re-start the weekly chats in the #core-http channel in SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/.. Fridays late afternoon UTC/GMT are good for me — does this work for other people who are interested in helping with HTTPS issues?

Although the HTTPS improvements are always ongoing and not tied to a particular release, it would be great to get some improvements into 4.7.

If you run a WordPress site over HTTPS only, support is very good and there are very few issues to contend with. If you’re running a multisitemultisite Used to describe a WordPress installation with a network of multiple blogs, grouped by sites. This installation type has shared users tables, and creates separate database tables for each blog (wp_posts becomes wp_0_posts). See also network, blog, site networknetwork (versus site, blog) on HTTPS there are a few small issues when adding new sites. However, the main HTTPS issues in core come from:

  • Enforcing the HTTPS scheme on assets (such as embedded images in post content, and enqueued JSJS JavaScript, a web scripting language typically executed in the browser. Often used for advanced user interfaces and behaviors. and CSSCSS Cascading Style Sheets.).
  • Enforcing the HTTPS scheme on links, redirects, and canonical URLs.
  • Migrating an existing HTTPHTTP HTTP is an acronym for Hyper Text Transfer Protocol. HTTP is the underlying protocol used by the World Wide Web and this protocol defines how messages are formatted and transmitted, and what actions Web servers and browsers should take in response to various commands. site to HTTPS.
  • Running a site that uses a mixture of HTTP and HTTPS.

The first two points — avoiding mixed content on HTTPS sites — need to be solved via an opt-in system (either via constants or filters) because enforcing these can cause issues with sites that run proxies (for example Cloudflare’s Universal SSLSSL Secure Sockets Layer. Provides a secure means of sending data over the internet. Used for authenticated and private actions.). Overall though, this ought to be a fairly straight forward set of enhancements to implement.

The third point is a potentially complex one which will need a lot of discussion and some ideas putting forward. How can core make life easier for a site owner who wishes to switch their site from HTTP to HTTPS? Should it be a case of being able to change the scheme in the URLURL A specific web address of a website or web page on the Internet, such as a website’s URL www.wordpress.org on the General Settings screen or is there too much risk of breakage? What else can be done post-migrationMigration Moving the code, database and media files for a website site from one server to another. Most typically done when changing hosting companies. to aid the site owner, or will the opt-in enhancements for avoiding mixed content be enough?

The last point is one that, going forward, should be generally discouraged, however it needs to continue to be supported for multisite networks that use domain mapping and can’t serve every domain over HTTPS.

There’s an https keyword on Trac which has been applied to tickets that concern HTTPS issues. We’ll start going through this list in next week’s chat.

Here’s a bunch of further considerations that need to be taken into account while working on HTTPS issues:

  • Differing schemes, domains, and ports in the siteurl and home options.
  • Domain mapping
  • force_ssl_admin() usage
  • Self signed certs
  • No public access to adminadmin (and super admin) URLs
  • Different HTTPS domain on front end (!)
  • HTTP site optionally available over HTTPS

Here’s a list of items that should be considered for enforcing over HTTPS:

  • Enqueued JS and CSS.
  • Post content, images, js, CSS, iframes,srcset, oembeds, forms.
  • How about other fields such as term descriptions, user bios, etc.
  • Force https links. Links to the current site.
  • Force https link in nav menus.
  • Force https redirects and/or canonical.
  • Force HSTS. (Probably not.)
  • Force https rest apiREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/. endpoint.
  • Force https XML RPC.
  • Set https-only on cookies.

Let me know in the comments if you’d like to help out and if Fridays are good for the meeting time!

#https

HTTPS discussion meeting this Wednesday

In recent releases of WordPress there have been various improvements made to support for sites running on HTTPSHTTPS HTTPS is an acronym for Hyper Text Transfer Protocol Secure. HTTPS is the secure version of HTTP, the protocol over which data is sent between your browser and the website that you are connected to. The 'S' at the end of HTTPS stands for 'Secure'. It means all communications between your browser and the website are encrypted. This is especially helpful for protecting sensitive data like banking information.. While support is currently very good, it’s still too easy to end up with mixed content on a site (HTTPHTTP HTTP is an acronym for Hyper Text Transfer Protocol. HTTP is the underlying protocol used by the World Wide Web and this protocol defines how messages are formatted and transmitted, and what actions Web servers and browsers should take in response to various commands. content embedded within an HTTPS page), and especially so when migrating an existing site from HTTP to HTTPS.

There will be a discussion meeting in the #core-http SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. channel on Wednesday, January 27, 2016 at 2000 UTC. This is one hour before the regular weekly meeting in #core. I’d like to discuss three topics:

  1. Implementing an (opt-in) method of forcing a site to use HTTPS.
    • What should this cover? (Embedded content, enqueued scripts/styles, links, redirects)
    • How should it be implemented? (eg. filterFilter Filters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output./constant/automatic)
  2. Defaulting to HTTPS for new installs when it’s available.
    • Only applies when setting up a site over HTTP and it’s available over HTTPS.
    • Need to communicate clearly to the user what this implies, with option to toggle.
  3. Aiding in switching an existing site from HTTP to HTTPS.
    • Migrating existing embedded content.
    • Should this be a feature pluginFeature Plugin A plugin that was created with the intention of eventually being proposed for inclusion in WordPress Core. See Features as Plugins.?

If you’re interested in helping out with any of the above, or with HTTPS improvements in general, join us on Wednesday.

Further reading: the https tag on Core Trac.

#https, #meeting

Request for feedback: Your HTTPS configurations

An ongoing goal of WordPress is to improve the way it works for sites that use HTTPSHTTPS HTTPS is an acronym for Hyper Text Transfer Protocol Secure. HTTPS is the secure version of HTTP, the protocol over which data is sent between your browser and the website that you are connected to. The 'S' at the end of HTTPS stands for 'Secure'. It means all communications between your browser and the website are encrypted. This is especially helpful for protecting sensitive data like banking information., and more specifically sites that run a mixture of schemes (for example, HTTPS in the adminadmin (and super admin) area but HTTPHTTP HTTP is an acronym for Hyper Text Transfer Protocol. HTTP is the underlying protocol used by the World Wide Web and this protocol defines how messages are formatted and transmitted, and what actions Web servers and browsers should take in response to various commands. on the front end).

One of the most visible bugs currently is that media in an HTTPS admin area is served over HTTP unless the ‘WordPress Address’ setting (siteurl) also uses HTTPS, which means that the FORCE_SSL_ADMIN constant isn’t a complete drop-in solution to securing your admin area.

Addressing all the possible configurations of HTTPS is difficult, so I’d like to put out a request for anyone who’s using a particularly interesting HTTPS configuration on your site to let us know what your setup is.

Of particular interest would be a site that’s using different domain names for HTTPS and HTTP, different domain names for the admin area and front end, different ports anywhere, self-signed certs for the admin area, HTTPS admin areas with additional access restrictions, multisites with and without domain mapping that use a mixture of HTTPS and HTTP, etc.

If your site has an interesting HTTPS configuration, and of course if it suffers from scheme related bugs as a result, please let us know in the comments below.

#https

WordPress and HTTP/2

HTTPHTTP HTTP is an acronym for Hyper Text Transfer Protocol. HTTP is the underlying protocol used by the World Wide Web and this protocol defines how messages are formatted and transmitted, and what actions Web servers and browsers should take in response to various commands./2 was finalized by the IETF earlier this year. This update to the HTTP protocol is the first major update in 16 years, since HTTP/1.1 was adopted in 1999. HTTP/2 promises faster websites by reducing latency through various innovations. Additionally, all major browser vendors have announced intentions to support HTTP/2 over TLS/SSLSSL Secure Sockets Layer. Provides a secure means of sending data over the internet. Used for authenticated and private actions. only, meaning that HTTP/2 brings with it an era of HTTPSHTTPS HTTPS is an acronym for Hyper Text Transfer Protocol Secure. HTTPS is the secure version of HTTP, the protocol over which data is sent between your browser and the website that you are connected to. The 'S' at the end of HTTPS stands for 'Secure'. It means all communications between your browser and the website are encrypted. This is especially helpful for protecting sensitive data like banking information. only websites. For a deeper dive into HTTP/2, highly recommend the chapter on HTTP/2 in High Performance Browser Networking.

For the WordPress 4.4 release cycle, Scott Taylor has asked Eric Andrew Lewis and I to investigate what, if anything, WordPress can do to make HTTP/2 integration seamless. We want to get this discussion started with some initial thoughts and an invitation to brainstorm in a dev chat next week.

HTTP/2 has been developed to just work with existing websites, suggesting that there is nothing in particular that we need to do to make WordPress compatible with HTTP/2; however, for all intents and purposes, HTTPS is required for HTTP/2 to work in browsers. One area of focus for making it easier to deployDeploy Launching code from a local development environment to the production web server, so that it's available to visitors. WordPress with HTTP/2 is to focus on tickets related to TLS/SSL issues. John Blackbourn has graciously compiled a list of tickets that could use attention:

In addition to focusing on these tickets, Eric and I have started working on more documentation for setting up TLS/SSL for WordPress, which will naturally lend itself to guides on HTTP/2 deployments for WordPress.

As for HTTP/2 and WordPress, we fortunately do not need to anything to make it work and existing web apps are already compatible. If we chose to, we can try to optimize WordPress for HTTP/2. These optimizations could manifest as optional theme support for HTTP/2 (i.e., add_theme_support() value) or using a constant to unlock HTTP/2 optimizations. We could add support for critical rendering path CSSCSS Cascading Style Sheets. that would allow WordPress to use server push to quickly deliver the assets to the browser. Additionally, we could consider building default themes with non-concatenated JSJS JavaScript, a web scripting language typically executed in the browser. Often used for advanced user interfaces and behaviors. files to optimize delivery for HTTP/2 (we’d still ship the concatenated version for HTTP/1.1 deployments). These are merely early thoughts on what we might be able to do and we are hoping to gather feedback and ideas here and during the dev chat.

For those interested, Eric and I set up a test bed for HTTP/2 and WordPress. We have two sites set up, one with HTTP/1.1 and one with HTTP/2. Currently, they are running the exact same code base (with Twenty Sixteen!) and only differ in the protocol that is used. We intend to experiment with ideas on this setup. The repo is on Github and we plan to make more setup information available soon.

To kick things off, we will have a meeting on September 10 at 20:00 UTC in the #core room on slack.

#agenda, #http2, #https