XML Sitemaps Meeting: March 24th, 2020

In case you were looking for an blog post about the XML Sitemaps feature project last week, worry no more. Work on the plugin is progressing smoothly and steadily, we just didn’t publish an agenda post last week. That means it is time for a double update today!

Meeting Recap: March 10th & 17th

For reference, check my previous blog post from March 10th:

A lot has happened since then. Here’s the summary, not necessarily in the right order:

  • SimpleXML dependency
    We received great feedback from a variety of big hosting providers, all saying that this PHP extension is widely available and we can rely on it safely.
    Current status: no action needed.
  • Rewrite rule conflict with plugins
    As we realized that the new /wp-sitemap.xml URL format clashes with big existing plugins, we decided to look into alternate names for both the rewrite rules as well as the query params. See GitHub issue for details.
    Current status: needs contributors.
  • Rewrite rule issues with custom providers
    It was reported that adding custom sitemap providers might require flushing rewrite rules. Ideally, that shouldn’t be needed.
    Current status: needs decision.
  • Last modified date (lastmod)
    We decided to continue with the proposed PR to remove lastmod from sitemaps (at least for now), but need to make sure there is appropriate documentation. It’s something that can always be added back if needed.
    Current status: has PR, needs documentation.
  • Query Filters
    Valuable feedback emerged from testing, which led to the decision to close the existing PR to make query instances filterable in favor of a simpler approach. In its place, we should make the query arguments filterable, and also add filters to short-circuit queries.
    Current status: needs contributors.

Please let me know in the comments if I got something wrong in this summary!

Agenda: March 24th

The next meeting will be held on Tuesday, March 24 at 16.00 CET.

Today’s agenda is rather straightforward so far:

Want to add anything to the above? Please leave a comment here or reach out on Slack.

This meeting is held in the #core-sitemaps channel , to join the meeting, you’ll need an account on the Making WordPress Slack.

#agenda, #feature-plugins, #feature-projects, #xml-sitemaps

XML Sitemaps Meeting: March 10th, 2020

A lot has happened since last week’s meeting for the XML Sitemaps feature project. Here’s a quick rundown of what we’ve discussed & did, as well as a brief agenda for today’s meeting.

Meeting Recap: March 3rd

For reference, please check out last week’s agenda post:

The tl;dr of our discussion:

  • Disabling sitemaps for private sites
    Mentioned the currently open PR and how it could be used to kill two birds with one stone by making that process filterable; thus making it easier for plugins to disable the sitemaps feature.
    Current status: needs tests
  • Prefixing sitemap URLs
    The main PR for this change has been merged, a new issue has been opened for @kraftbj to handle 404 requests.
  • SimpleXML dependency
    We went over potential alternatives to this extension, but ultimately settled on sticking with the status quo as initial feedback indicated a rather wide availability of SimpleXML. We then discussed how we should gracefully handle the unavailability of said extension and decided on using wp_die to output a nicely formatted error message in XML with HTTP status 501 (“Not implemented”).
    Current status: merged!
  • @joemcgill proposed looking into how to best transition the code base to something more in line with WordPress core. Something that we can discuss in a future meeting, once the plugin is more stable.
  • Added @pbiron, @kraftbj, and @pfefferle as new contributors to the GitHub repository. 🎉

Agenda: March 10th

The next meeting will be held on Tuesday, March 10 at 16.00 CET.

PSA: Unfortunately I won’t be able to lead today’s meeting, but thankfully @tweetythierry stepped up to help out with this.

Today’s agenda is rather straightforward so far:

  • Released version 0.2.0 of the plugin (changelog)
  • Plugin compatibility with new URL structure
    Yoast SEO’s rewrite rules seem to clash with ours
  • SimpleXML dependency: blog post on make/hosting (@pbiron)
  • Currently open issues and pull requests
  • Open floor

Want to add anything to the above? Please leave a comment here or reach out on Slack.

This meeting is held in the #core-sitemaps channel , to join the meeting, you’ll need an account on the Making WordPress Slack.

#agenda, #feature-plugins, #feature-projects, #xml-sitemaps

XML Sitemaps Meeting: March 3rd, 2020

Another week passed by with quite a productive meeting for the XML Sitemaps feature project. Here’s a short summary, as well as the agenda for today’s meeting.

Meeting Recap: February 25th

In case you missed it, I recommend checking out last week’s post with everything that happened so far:

As planned, we went over some of the existing issues, but we also discussed some things that came up on short notice. Here’s the gist:

  • We reiterated on the idea to remove the lastmod field. @swissspidy offered to start a PR that explores this so it can actually be tested in the wild. @joemcgill offered to post some stats about the performance of this last modified date calculation.
  • There was a discussion, also after the meeting, about changing URLs to have a /wp- prefix and whether that prefix should be filterable. The consensus was that a filter is unnecessary. A new PR was added to implement this.
    @kraftbj offered his help to implement automatic redirects from /sitemap.xml to /wp-sitemap.xml for improved discoverability.
  • Next up was the SimpleXML dependency and how the plugin should behave when that PHP extension is missing.
    We tend towards just disabling sitemaps if that’s the case, but perhaps provide some messaging about it.
    @kraftbj offered to try to get some stats about the availability of SimpleXML via Jetpack, as well as to help with a PR.
    @pbiron reached out on the hosting community channel, and is looking for specific questions that we could ask in a make/hosting post.
  • Last but not least, there was an open question about leveraging the REST API for sitemaps. It was not fully clear though how that would be beneficial. As of now, there are no plans to explore this.

Agenda: March 3rd

The next meeting will be held on Tuesday, March 3 at 16.00 CET

This meeting is held in the #core-sitemaps channel , to join the meeting, you’ll need an account on the Making WordPress Slack.

#agenda, #feature-plugins, #feature-projects, #xml-sitemaps

XML Sitemaps Meeting: February 25th, 2020

Last week we held the first of many weekly meetings for the XML Sitemaps feature project on Slack.

Meeting Recap: February 18th

We had quite a few people attending, not all of whom were familiar with the project. Thus, we started off with a small recap of the project’s scope and goals. After that we discussed various different topics:

  • How to modify the sitemaps to include/exclude certain URLS
    A pull request has been opened to add a FAQ section to the readme that aims to answer these kind of questions.
    Also, a new way to filter WP_Query instances used for sitemaps has been proposed.
  • Why are there no changefreq and priority fields?
    Those are optional fields in the sitemaps protocol and not typically consumed by search engines. The feature plugin follows other solutions like Yoast SEO who also don’t include those fields.
    Developers can still add those fields if they really want too.
  • Will there be UI controls to include/exclude content from sitemaps?
    Adding UI controls is currently a non-goal for the project.
  • Calculating the last modified date for URLs
    This is rather difficult and computationally expensive in WordPress. Given that sitemaps are first and foremost a discovery mechanism for content, having this data is not necessarily required. We will explore omitting this functionality (GitHub issue).
  • The default limit of 2000 URLs per sitemap is considered high and might need to be re-evaluated.
  • Potential compatibility issues with other XML Sitemaps plugins have been discussed.
    If a site ends up having two sitemaps by accident that wouldn’t be bad. However, the current /sitemap.xml URL might clash with other plugins. A GitHub issue has been opened to suggesting using /wp-sitemap.xml as the base. This would avoid conflicts in this regard.

Agenda: February 25th

The next meeting will be held on Tuesday, February 25 at 16.00 CET

For tomorrow’s meeting, the agenda is rather brief:

  • Updates since last week (merged changes, new issues)
  • Next steps for proposed lastmod changes
  • Next steps for URL naming change
  • Planning release of version 0.2.0

This meeting is held in the #core-sitemaps channel , to join the meeting, you’ll need an account on the Making WordPress Slack.

#agenda, #feature-plugins, #feature-projects, #seo, #xml-sitemaps

XML Sitemaps Kickoff Meeting Announcement

A few weeks ago an update was posted for the XML Sitemaps feature project to give everyone an idea of where it is heading.

Now, we want to gather more contributors around the feature plugin and get your feedback on the project. For this, we’re kicking off regular meetings in the brand new #core-sitemaps Slack channel.

The first meeting will be held on Tuesday, February 18 at 16.00 CET and will serve as an introduction to the project and an opportunity to discuss the next steps. As such, there is currently no formal agenda for this inaugural meeting.

However, if you have anything specific that you’d like to propose being discussed in this meeting, feel free to leave a comment below.

This meeting is held in the #core-sitemaps channel , to join the meeting, you’ll need an account on the Making WordPress Slack.

#feature-plugins, #feature-projects, #seo, #xml-sitemaps

New Post Type Labels in 5.0

It’s been a while since new post type labels have been introduced in WordPress.

In WordPress 5.0, five additional labels have been made available for custom post types. These get passed in via the labels argument when using register_post_type(). The following labels are new:

  • item_published — The label used in the editor notice after publishing a post. Default “Post published.” / “Page published.”
  • item_published_privately — The label used in the editor notice after publishing a private post. Default “Post published privately.” / “Page published privately.”
  • item_reverted_to_draft — The label used in the editor notice after reverting a post to draft. Default “Post reverted to draft.” / “Page reverted to draft.”
  • item_scheduled — The label used in the editor notice after scheduling a post to be published at a later date. Default “Post scheduled.” / “Page scheduled.”
  • item_updated — The label used in the editor notice after updating a post. Default “Post updated.” / “Page updated.”

Please note the trailing period for all of these strings.

#5-0, #dev-notes, #i18n

JavaScript I18N Meeting Summary: May 8th

After the blog post about JavaScript internationalization, we held a first meeting last Tuesday to discuss how we can improve client-side translation in JavaScript. This is a summary of the JS I18N chat from May 8th, 2018. (Slack log)

Topics

First, we talked about the introductory blog post and the feedback it received so far. While the post gives a great overview of the two main problems — extracting strings from JavaScript and loading the actual translations in a way that makes sense — there are many smaller problems attached to them that need to be tackled.

Thus, we started looking at the string extraction part and the problems which come with it:

  • Plugins bundle their assets in many different ways: minified, non-minified, using simple UglifyJS or multiple individual modules and more complex tools like webpack.
  • If WordPress.org does parse source files, which syntax (e.g. ES2015, ES2018, JSX, etc.) should be supported? WordPress.org needs to set clear boundaries and best practices for plugin and theme developers.
  • Larger plugins like Gutenberg only provide the minified build files on WordPress.org. How can strings be extracted from those?
  • Should such plugins be expected to do the string extraction part themselves and provide a POT file that GlotPress can import? Right now, this is not possible.

Multiple JavaScript I18N tools already exist, e.g. @wordpress/i18n, @wordpress/babel-plugin-makepot, and a WP-CLI i18n command. These can be used to test a new JavaScript string extraction process for individual plugins like Gutenberg and perhaps a dedicated testing plugin.

We then switched over to the other main problem: loading translations for use in JavaScript. Since any solution here as an impact on string extraction and delivering language packs, this ideally should be solved first. The problem areas here are:

  • Plugins can have multiple scripts that can be enqueued each on their own. Loading all translated strings even though you’re only enqueueing one small script handle would have a negative impact on performance.
  • Could performance be improved by caching translations using something like IndexedDB?
  • A script handle could be a minified JavaScript file that originally consisted of multiple smaller JavaScript modules, each containing some translatable string. How can we know which translations should be loaded for the resulting script handle?
  • Can we have a map of script handle => original JavaScript files or perhaps a way of saying “for script handle X, load strings in text domains Y and Z”. How would an implementation for this look like?
  • What if a bundled script contains some third-party components calling wp.i18n.__() with their individual text domain? How could this be handled?

At the end, there were still too many open questions and we figured it would be best to have a look at how other projects handle this in order to not reinvent the wheel.

Next Meeting

We will meet again on Tuesday, May 15th at 15:00 UTC in #core-i18n to further discuss especially the translation loading part.

#i18n

JavaScript Internationalization: The Missing Pieces

Back in 2016, work started on building a proper JavaScript internationalization API and the tooling to support it throughout WordPress core and WordPress.org. Many ideas and patches were being discussed. A summary of that can be found in this blog post. With Gutenberg on the rise, JavaScript I18N is more urgent than ever. WordPress needs a robust solution for that, and some things have already been built. Let’s have a look at where we currently stand.

Status Quo

Right now, Gutenberg is using a custom built JS I18N library that is similar to the one originally proposed in 2016 as part of #20491. It lies on top of a library called Jed which bring Gettext functionality to JavaScript. This means developers can use the same __() function as in PHP and therefore don’t have to learn anything new. WordPress can take it from there.

Unfortunately, WordPress doesn’t yet support JS I18N library. Gutenberg (or any other plugin that uses said library, really) has to jump through quite some hoops to actually localize their JavaScript:

  1. Scan JavaScript files to extract internationalization functions and create a POT file using tools like babel-plugin-makepotPoedit or xgettext-js.
  2. Use that POT file to write the exact same internationalization functions in a “fake” PHP file that can be scanned by the WordPress.org translation platform. This will result in PO and MO files containing all of your plugin’s translations.
  3. Figure out a way to load these translations and make them available to your JavaScript using wp_add_inline_script(). Ideally you’d only load the ones needed by that specific script as you don’t want to print thousands of strings in that inline JS when you only need a few of them.

An example of that process can be found in my demo Gutenberg I18N Block plugin.

At this point you might want to go back to good old wp_localize_script() and simply keep using that for internationalization purposes. I don’t blame you.

However, this complicated process is only needed because the work on JavaScript internationalization is far from done yet. Gutenberg made it quite obvious where things need to be improved.

What’s Missing

Scanning JavaScript files for internationalization functions

First and foremost, the WordPress.org translation platform needs to be able to scan JavaScript files for internationalization functions in addition to just the PHP files. However, that’s not as straightforward as it sounds.

The platform uses a script called makepot.php to scan PHP files all across the WordPress.org ecosystem, i.e. core, meta, and all default themes. In addition to regular Gettext function calls it also scans plugin and theme file headers. Being included in many other libraries, makepot is a widely used tool. Most recently, its functionality was ported to a WP-CLI command to make string extraction easier to use.

On the other side we have babel-plugin-makepot, a tool written in JavaScript to scan JavaScript files. With the ECMAScript standard evolving so quickly, it is natural to write such a tool in the same language. However, it’s not a requirement, as this pull request for said WP-CLI command demonstrates. This opens some questions:

Can we simply use that Babel plugin on WordPress.org? What happens to makepot.php? What are the implications for all the developers out there not hosting their projects on WordPress.org? Not everyone uses the Babel transpiler, and certainly not everyone wants to use two separate tools just to extract some internationalization functions.

Loading only specific set of translations

All translations for a plugin or theme are stored in one single PO / MO file per locale. Loading these translations is a slow process.  We’ve made some improvements in that regard over the years, for example by introducing just-in-time loading of translations in WordPress 4.6.

However, if you only need a handful of translations for a single script in your plugin, it does not make sense to load the entire MO file which can be dozens of kilobytes in size. There’s currently no way to load only a specific set of translations in WordPress. This is something that came up in Gutenberg before, see issue 6015.

Binary MO files don’t make sense in a JavaScript context anyway. Lucky for us, GlotPress—the software that powers translate.wordpress.org—has been able to export translations in a Jed-compatible JSON format since 2016. We just need to use that to export a JSON file for all strings extracted from JavaScript files

So in theory the WordPress.org translation platform could export PO and MO files as usual for strings extracted from PHP files, and a JSON file for all strings coming from JavaScript files. This would be already a huge improvement. But can we take this even further?

Option A

Use a different text domain per JavaScript module. Export a JSON file per text domain. This is appealing, but has to be ruled out quickly: the text domain is not known to GlotPress and is not stored in the database or anything.

Option B

GlotPress doesn’t know about the text domain, but it does know a string’s source file. What if it would export one JSON file per source file it has scanned? This way WordPress has full control over the translations and one could specify which JSON files need to be loaded for a specific module.

The big drawback here: a single module might consist of dozens of source files. Having one JSON file for each of those is not going to scale well.

The built JavaScript file can’t be scanned either, because tools like UglifyJS rename functions and strip out comments.

Option C

Don’t do anything fancy. Sticking with a JSON file already guarantees that a plugin doesn’t unnecessarily load all the translations needed just in PHP. So the file size is definitely smaller. Still, this file alone can be very large for an application like Gutenberg.

Option D

Keep one single JSON file for all translations, but use some PHP code to only ever pass the strings to a module / script handle that it actually needs. However, there’s probably no real benefit in doing so.

Easily load translations

Up until WordPress 4.6, developers needed to use load_plugin_textdomain or load_theme_textdomain() to make sure translations are properly loaded. Now, you only need to use the various translation functions and the rest just works. The only requirement is that your translation files reside in wp-content/languages. This is usually the case when your project is hosted on WordPress.org.

We should aim for a similar experience for JavaScript translations as well. While just-in-time loading of translation files via HTTP isn’t really possible due to the asynchronous nature of JavaScript, WordPress should still make it as easy as possible.

Imagine having a plugin foo-plugin and you’re enqueuing your JavaScript like this:

wp_enqueue_script( 'foo-script', plugins_url( '/foo-script.js' , __FILE__ ) );

Ideally, all you’d need to do to translate it is calling a function like load_js_textdomain( 'foo-plugin' ).  WordPress would then do all the heavy lifting.

However, other options might exist, and this solution would need to be tested in the wild with bigger projects like Gutenberg.

Discussion

Bringing a JavaScript I18N API to WordPress will have a huge impact. We need to make sure we end up with a solid plan that works for as many plugins and themes as possible.

Ideally, we hold a separate JS I18N meeting with all the teams primarily involved: #core-i18n, #core-js, #core-editor, #meta-i18n and #cli. Everyone is welcome to attend though 🎉

I suggest the following date for such a meeting: Tuesday, May 8 15:00 UTC. Of course I’m open for other suggestions. The Slack channel would be #core-i18n.

At this meeting we can discuss the missing pieces outlined in this post and the overall next steps for JavaScript I18N in WordPress.

If you have any questions or concerns about this post or the overall topic, please leave a comment below.

+make.wordpress.org/polyglots

Preferred Languages: The Prototype

A bit over six months ago I set the foundation for the Preferred Languages feature project. After highlighting the problems many WordPress users around the world are facing, some time was spent on researching popular platforms and other systems to see how they handle the issue of setting multiple preferred languages. However, this didn’t really help to move one step forward and the project became dormant — until now.

Personally, I’ve been experiencing the same problems with so-called language fallbacks over and over again. During WordCamp Bilbao I decided to revive the Preferred Languages project. In order to do this I used the previously mentioned GitHub repository to build a super-simple proof-of-concept plugin to have a new basis for discussion. As originally envisioned, this repository can be used as a playground for discussions, prototypes, and eventually a working solution.

This new plugin lets you select multiple preferred languages in your settings. WordPress then tries to load the translations for the first language that’s available, falling back to the next language in your list.

A sneak peek of an early version quickly made the rounds on Twitter:

After some very positive feedback I worked a bit more on it and wanted to share the latest version here with a larger group of people. Here’s what it currently looks like on both the settings screen and when editing your profile:

Without much further ado, please check out the plugin on GitHub, test it on your local WordPress site, and leave some feedback!

#feature-projects, #i18n, #preferred-languages

4.7.4 Release Candidate

After about six weeks of development, a Release Candidate for WordPress 4.7.4 is now available. This maintenance release fixes 46 issues reported against 4.7 and is scheduled for final release on Thursday, April 20, 2017.

Thus far WordPress 4.7 has been downloaded nearly 60 million times since its release on December 6, 2016. Please help us by testing this release candidate to ensure 4.7.4 fixes the reported issues and doesn’t introduce any new ones.

Notable Bug Fixes

There are a few more notable issues being addressed in this release. The first one is about broken video/audio thumbnails when uploading media (#40075). Additionally, an incompatibility between the upcoming Chrome version and the visual editor (#40305) has been solved by updating TinyMCE. Furthermore, the REST API saw some enhancements in relation to date handling (#39854, #40136).

All Changes

Here’s a list of all closed tickets, sorted by component:

Administration

  • #39983 – Consider to don’t use the CSS class button-link for controls that don’t look like links
  • #40056 – Shift-click to select a range of checkboxes isn’t working anymore since 4.7.3 update

Bootstrap/Load

  • #39445 – Add class_exists() check before defining the PasswordHash class

Build/Test Tools

  • #38500 – Automatically cancel pending Travis builds with each commit
  • #39219 – Add assertNotFalse method to WP_UnitTestCase.
  • #39367 – Don’t no-op $user_id in test suite’s wp_set_auth_cookie()
  • #39988 – The theme used during tests should call wp_head() and wp_footer()
  • #40066 – Remove the twentysixteen git clone from the Travis config
  • #40086 – Get Travis tests working again on PHP 7

Bundled Theme

  • #40216 – Twenty Seventeen: Some parts do not escape html attributes
  • #40224 – Twenty Seventeen: navigation.js should be enqueued with jQuery as dependency
  • #40264 – Twenty Seventeen: Incorrect heading hierarchy for front page sections
  • #40461 – Twenty Seventeen: Bump version and update changelog

Customize

  • #31850 – Customizer links should use canonical admin URL
  • #37471 – Widgets: If your theme only has one widget area, we should open it automatically
  • #38953 – Customize Menus: clicking outside of the available menu items panel does not close the panel
  • #39430 – sections and panels that are open and become inactive should be closed
  • #39770 – Client-side notification error is unexpectedly cleared when no corresponding server-side validation
  • #40010 – Template for site icon control fails to check if full image size exists before using
  • #40018 – Selective refresh always falls back to full refreshes when customizing the 404 template
  • #40112 – Can’t preview starter content “Home” menu item in subdirectory installation
  • #40198 – all previewable links are blocked in the customize preview on IE11
  • #40271 – Use get_user_locale() in Customizer
  • #40277 – Adding page created with the dropdown-pages settings to menu creates Custom Link instead of Page
  • #40308 – Video header control fails to use is_header_video_active() for active_callback
  • #40405 – IE9 errors when attempting to generate changeset parameter

Login and Registration

  • #39497 – Can’t log out completely without closing my browser

Media

  • #31071 – media / post_mime_type related queries are very slow on larger sites
  • #40017 – wp_get_image_mime() returns ‘application/octet-stream’ for non-image files.
  • #40075 – Broken video/audio thumbnails because of corrupted blob meta data
  • #40085 – Audio/video uploads are broken in 4.2.13 and 4.3.9
  • #40152 – Crop Image button off-screen on mobile

Networks and Sites

  • #40036 – Re-save Network Settings ruin starter content
  • #40063 – Handle site cache invalidation more specifically for option updates

Posts, Post Types

  • #39986 – Register missing REST API properties on WP_Post_Type

Quick/Bulk Edit

  • #40242 – Bulk edit tag autocomplete layout error

REST API

  • #39854 – Add gmt_offset to base /wp-json response
  • #39881WP_REST_Posts_Controller::check_read_permission() should check if $parent exists before calling itself
  • #40027 – Tags and Categories should have a “slugs” parameter for batch fetching
  • #40136 – Issues with dates and DST
  • #40213 – Users endpoint slug parameter should allow an array of slugs

Taxonomy

  • #39987 – Register missing REST API properties on WP_Taxonomy
  • #40154 – Incorrectly formatted $taxonomies parameter passed to wp_get_object_terms filter
  • #40306 – Term cache isn’t cleared completely when setting and removing object terms

Themes

  • #38292 – Introduce exclusion for WP_Theme::scandir()

TinyMCE

  • #40305 – Image popup toolbar does not support Chrome Beta

Download the Release Candidate now and help us test!

#4-7, #4-7-4, #maintenance, #release