New Post Type Labels in 5.0

It’s been a while since new post type labels have been introduced in WordPress.

In WordPress 5.0, five additional labels have been made available for custom post types. These get passed in via the labels argument when using register_post_type(). The following labels are new:

  • item_published — The label used in the editor notice after publishing a post. Default “Post published.” / “Page published.”
  • item_published_privately — The label used in the editor notice after publishing a private post. Default “Post published privately.” / “Page published privately.”
  • item_reverted_to_draft — The label used in the editor notice after reverting a post to draft. Default “Post reverted to draft.” / “Page reverted to draft.”
  • item_scheduled — The label used in the editor notice after scheduling a post to be published at a later date. Default “Post scheduled.” / “Page scheduled.”
  • item_updated — The label used in the editor notice after updating a post. Default “Post updated.” / “Page updated.”

Please note the trailing period for all of these strings.

#5-0, #dev-notes, #i18n

5.0: JavaScript language packs are here 🎉

News from the JS package inclusion focus: JavaScript language packs are here! Thanks to AWESOME work by @herregroen, @ocean90, @swissspidy, @nerrad, @atimmer and @schlessera we can now translate strings in JavaScript files and distribute them via https://translate.wordpress.org. This functionality will soon be expanded to also work for plugins and themes. This is a major milestone for JavaScript development in WordPress and completes the JavaScript package inclusion focus.

How does it work?

In short, to make it work, you need to

  • use the @wordpress/i18n package for making strings in your JavaScript translatable.
  • let WordPress know that a script has translations by calling wp_set_script_translations( 'my-handle', 'my-domain' ) after you register a script.

Read more about it in the JS i18n devnote.

Can I have a look at the tickets involved?

Sure thing! Here’s an overview of all the work we’ve done in the last few weeks to get to this point:

  1. WP-CLI: Make sure wp i18n CLI command includes JavaScript translations when generating Potfiles. Relevant PRs: https://github.com/wp-cli/i18n-command/pulls?q=is%3Apr+is%3Aclosed+author%3Aherregroen
  2. Meta: generate translations on translate.wordpress.org for core, themes and plugins. Ticket: https://meta.trac.wordpress.org/ticket/3748
  3. Meta: generate and serve JS language packs from translate.wordpress.org for core, themes and plugins. Ticket: https://meta.trac.wordpress.org/ticket/3876
  4. Core: Logic to load JS translations. Ticket: https://core.trac.wordpress.org/ticket/45103
  5. Core: API to register JS translations for a script. https://core.trac.wordpress.org/ticket/45161

What’s next?

With JS language packs we have concluded the package inclusion focus. In the remaining time we will keep focus on two things:

  • Keep the packages up-to-date until 5.0 ships.
  • @herregroen will shift focus towards generating the core JS code documentation on Devhub (https://developer.wordpress.org) and making it searchable. Since this pretty much only involves meta, this is not a release blocker. But with the heaps of JavaScript that 5.0 adds to WordPress, it would be nice to have its documentation searchable and discoverable through Devhub. At the same time it would also make WordPress’ legacy JavaScript (of which a lot was documented in the last years in the JS documentation initiative) more accessible to the broader community. Work on this had already started as part of the JS docs initiative.

#5-0, #dev-notes, #i18n

New! JavaScript i18n support in WordPress 5.0

For years, internationalization (i18n) has been a thing that has been pretty well supported in WordPress when it comes to PHP development. For PHP, WordPress already provides all the tools necessary to make it as easy as possible to localize WordPress core, themes and plugins to any language. Today we are bringing the same capabilities to JavaScript development for WordPress.

How does it work?

When registering your scripts you can add wp-i18n as a dependency to allow you to add translatable strings as you would in PHP:

wp_register_script( 'my-handle', plugins_url( '/js/my-file.js', MY_PLUGIN ), array( 'wp-i18n' ) );

Inside your scripts you will then be able to use wp-18n as follows:

const { __, _x, _n, _nx } = wp.i18n;

__( '__', 'my-domain' );
_x( '_x', '_x_context', 'my-domain' );
_n( '_n_single', '_n_plural', number, 'my-domain' );
_nx( '_nx_single', '_nx_plural', number, '_nx_context', 'my-domain' );

These functions mirror their PHP counterparts and can be used in exactly the same manner. 

The final step is to tell WordPress your script contains translations and of which domain, this is to allow WordPress to selectively load only the necessary translations to ensure everything is as fast as can be:

wp_set_script_translations( 'my-handle', 'my-domain' );

Make sure to also specify the Text Domain for your translations in the header of your plugin file. Otherwise the translations will not be picked up by translate.wordpress.org

Advanced usage

Right now it’s already possible to ship your own translations using the load_textdomain function and passing your own MO file. This is also possible using wp_set_script_translations which accepts an optional third path argument that allows you to tell WordPress to first look elsewhere for translations:

wp_set_script_translations( 'my-handle', 'my-domain', plugin_dir_path( MY_PLUGIN ) . 'languages' );

If passed WordPress will first check if a file in the format of ${domain}-${locale}-${handle}.json exists in the given path and use it as the source of translations if so. Alternatively it will also first check the given path for the md5 filename before defaulting to the WordPress languages directory.

If you want to ship your own translation files these should be in the JED 1.x ( .json ) format. GlotPress is able to create these along with other tools such as po2json. Ideally these files should only contain translations that occur within their respective JS files. With po2json you can generate these files as follows:

po2json translation.po translation.json -f jed

This will generate a JSON in the following format:

{
  "translation-revision-date": "+0000",
  "generator":                 "GlotPress/2.3.0-alpha",
  "domain":                    "messages",
  "locale_data":               {
    "messages": {
      "":                                                             {
        "domain":       "messages",
        "plural-forms": "n != 1",
        "lang":         "en-gb"
      },
      "This file is too big. Files must be less than %d KB in size.": [
        "This file is too big. Files must be less than %d KB in size."
      ],
      "%d Theme Update":                                              [
        "%d Theme Update",
        "%d Theme Updates"
      ],
      "password strength\u0004Medium":                                [
        "Medium"
      ],
      "taxonomy singular name\u0004Category":                         [
        "Category"
      ],
      "post type general name\u0004Pages":                            [
        "Pages"
      ]
    }
  }
}

Behind the screens

When you upload your plugin or theme to wordpress.org all JS files will automatically be parsed the same as is already being done for PHP files. Any detected translations will be added to translate.wordpress.org to allow the community to cooperate to ensure WordPress, plugins and themes are available in as many languages as possible.

In order to parse all JS files the i18n-command for wp-cli is used. This replaces makepot.php to not only allow picking up translations in JS files but also to audit strings, parse strings only of a specific text domain and even pick up a few strings that weren’t detected by makepot.php. This command is freely available and open-source as makepot.php was and it’s recommended that anyone using makepot.php transition over to this much improved replacement.

Based on these parsed translations Language Packs are generated. Traditionally these used to only contain PO and MO files, one pair for each locale. In order to selectively load only the necessary translations regardless of whether it’s used or not a few more files are being added, one JSON file for every JS file that contains translations per locale.

When parsing JS files for translations we don’t know which handle is used to register that file so we’ve had to use an alternate mechanism to find the translations belonging to each file. To do this we’re using the md5 of the relative path of each file. This is appended to the usual name of ${domain}-${locale} in the form of ${domain}-${locale}-${md5}.json.

When you set script translations for a handle WordPress will automatically figure out the relative md5 hash of your source file, check to see if a translations file exists and if so ensure that it’s loaded into wp.i18n before your script runs.

Plugin and theme support

Translation and Language packs support for plugins and themes that are hosted on the repo is expected in the upcoming weeks. The patches are ready and waiting for commit. Plugin and theme authors are encouraged to start using wp-i18n in their JavaScript projects.

Credits

This API wouldn’t have been possible without the long standing efforts of @ocean90, @swissspidy, @nerrad, @atimmer, @schlessera and more recently @herregroen. Thanks a ton for the incredible work you’ve all put into making this a reality! Another thanks to @herregroen for providing the necessary input for this devnote.

#5-0, #dev-notes, #i18n

JavaScript I18N Meeting Summary: May 8th

After the blog post about JavaScript internationalization, we held a first meeting last Tuesday to discuss how we can improve client-side translation in JavaScript. This is a summary of the JS I18N chat from May 8th, 2018. (Slack log)

Topics

First, we talked about the introductory blog post and the feedback it received so far. While the post gives a great overview of the two main problems — extracting strings from JavaScript and loading the actual translations in a way that makes sense — there are many smaller problems attached to them that need to be tackled.

Thus, we started looking at the string extraction part and the problems which come with it:

  • Plugins bundle their assets in many different ways: minified, non-minified, using simple UglifyJS or multiple individual modules and more complex tools like webpack.
  • If WordPress.org does parse source files, which syntax (e.g. ES2015, ES2018, JSX, etc.) should be supported? WordPress.org needs to set clear boundaries and best practices for plugin and theme developers.
  • Larger plugins like Gutenberg only provide the minified build files on WordPress.org. How can strings be extracted from those?
  • Should such plugins be expected to do the string extraction part themselves and provide a POT file that GlotPress can import? Right now, this is not possible.

Multiple JavaScript I18N tools already exist, e.g. @wordpress/i18n, @wordpress/babel-plugin-makepot, and a WP-CLI i18n command. These can be used to test a new JavaScript string extraction process for individual plugins like Gutenberg and perhaps a dedicated testing plugin.

We then switched over to the other main problem: loading translations for use in JavaScript. Since any solution here as an impact on string extraction and delivering language packs, this ideally should be solved first. The problem areas here are:

  • Plugins can have multiple scripts that can be enqueued each on their own. Loading all translated strings even though you’re only enqueueing one small script handle would have a negative impact on performance.
  • Could performance be improved by caching translations using something like IndexedDB?
  • A script handle could be a minified JavaScript file that originally consisted of multiple smaller JavaScript modules, each containing some translatable string. How can we know which translations should be loaded for the resulting script handle?
  • Can we have a map of script handle => original JavaScript files or perhaps a way of saying “for script handle X, load strings in text domains Y and Z”. How would an implementation for this look like?
  • What if a bundled script contains some third-party components calling wp.i18n.__() with their individual text domain? How could this be handled?

At the end, there were still too many open questions and we figured it would be best to have a look at how other projects handle this in order to not reinvent the wheel.

Next Meeting

We will meet again on Tuesday, May 15th at 15:00 UTC in #core-i18n to further discuss especially the translation loading part.

#i18n

Preferred Languages: The Prototype

A bit over six months ago I set the foundation for the Preferred Languages feature project. After highlighting the problems many WordPress users around the world are facing, some time was spent on researching popular platforms and other systems to see how they handle the issue of setting multiple preferred languages. However, this didn’t really help to move one step forward and the project became dormant — until now.

Personally, I’ve been experiencing the same problems with so-called language fallbacks over and over again. During WordCamp Bilbao I decided to revive the Preferred Languages project. In order to do this I used the previously mentioned GitHub repository to build a super-simple proof-of-concept plugin to have a new basis for discussion. As originally envisioned, this repository can be used as a playground for discussions, prototypes, and eventually a working solution.

This new plugin lets you select multiple preferred languages in your settings. WordPress then tries to load the translations for the first language that’s available, falling back to the next language in your list.

A sneak peek of an early version quickly made the rounds on Twitter:

After some very positive feedback I worked a bit more on it and wanted to share the latest version here with a larger group of people. Here’s what it currently looks like on both the settings screen and when editing your profile:

Without much further ado, please check out the plugin on GitHub, test it on your local WordPress site, and leave some feedback!

#feature-projects, #i18n, #preferred-languages

Preferred Languages: Research

Following the first meeting of the Preferred Languages project, some time was spent on researching popular platforms and other systems to see how they handle the issue of setting multiple preferred languages. To recap, this is needed to show things in the most suitable language for users in case their requested language is not installed.

Browsers & Operating Systems

All major browsers and operating systems have some UI where the user can set their preferred languages. Mostly used for the Accept-Language HTTP header, one can set multiple languages and order them by preference. These systems usually know multiple variants like German and German (Switzerland), but not formal / informal variants.

Worth noting that on these systems you can often choose more settings that are influenced by the language, like temperature units and date formats. Related: #18146.

The Accept-Language HTTP Header

RFC 7231 about the HTTP/1.1 standard says the following about this header:

The “Accept-Language” header field can be used by user agents to indicate the set of natural languages that are preferred in the response. Each language-range can be given an associated quality value representing an estimate of the user’s preference for the languages specified by that range.

This header is quite powerful. For example, Accept-Language: da, en-gb;q=0.8, en;q=0.7 would mean: “I prefer Danish, but will accept British English and other types of English”. I can only recommend reading more about it in the RFC, as it helps getting a better understanding of the problems it tries to solve.

This header is usually used by websites to redirect users to the correct version or display a hint like “This content is also available in XY”. While each language in the header has a specific numeric priority, there is usually only a drag & drop interface to determine the order.

Unicode Common Locale Data Repository

The Unicode CLDR provides key building blocks for software to support the world’s languages, with the largest and most extensive standard repository of locale data available. It contains an interesting chart about Language Matching, data that is used to match the user’s desired language/locales against an application’s supported languages/locales.

There’s also a technical standard about Unicode Locale Data Markup Language (LDML), an XML format for the exchange of structured locale data. The section about Locale Inheritance and Language Matching is particularly interesting. For instance, it describes finding the most well suited language using a weighted graph and gives a better picture of dealing with more complex language fallbacks.

It also takes geographic “closeness” into account, arguing that English (Slovakia) should fall back to something within Europe (e.g. British English) in preference to something far away and unrelated like English (Singapore). It’s clearly stated that these fallbacks aren’t as simple as just saying Spanish (Mexico) -> Spanish (Spain) -> English (US).

Note that this technical standard is about finding the best supported locale based on the requested list of languages. The requested list could come from different sources, such as such as the user’s list of preferred languages in the OS Settings, or from a browser’s Accept-Language list.

Wikipedia

Wikipedia is built on the MediaWiki software, which has a hardcoded list of fallback chains for some locales. This is where MediaWiki will fall back on a different language if it cannot find what it needs. For example, French (Cajun) automatically falls back to French (France) when it doesn’t have all messages defined in it. Unfortunately, there’s only little documentation about this.

Apart from that, MediaWiki distinguishes between various kinds of languages: the site content language, the user interface language and the page content language. The latter can differ from the first two and it influences the language the user views the page in, which depends on the user’s preferences, the available languages, and the defined fallbacks.

It’s worth noting that on Wikipedia, you can not only select your preferred language, but also your preferred gender. In addition to that, you can select multiple different locales for both display and input.

Joomla

I tried the official Joomla Demo to test its language management which is a bit overwhelming at first. After installing all the available languages the user is eventually able to select their preferred language for the front end and the back end. There are a few other settings for language switching on multilingual sites, but that’s pretty much it. So basically the same settings as currently in WordPress 4.7.

Joomla User Settings

Joomla User Settings

 

Drupal

Drupal 8 has a rather powerful user interface text language detection mechanism. There is a per session, per user and per browser option in the detection settings. However, users can only choose one language, so they cannot say (in core at least) that they want German primarily and Spanish if German is not available. But the language selected by the user is part of the larger fallback system, so it may fall back further down to other options.

The Language fallback module allows defining one fallback for a language, while the Language Hierarchy module provides a GUI to change the language fallback system. It allows setting up language hierarchies where translations of a site’s content, settings and interface can fall back to parent language translations, without ever falling back to English. This module might be the most interesting one for our research.

Drupal Language Hierarchy

Drupal Language Hierarchy

TYPO3

TYPO3 has had a locale hierarchy since 2011. Quote:

TYPO3 4.6 comes with a clever fallback mechanism when a label is not found in the requested language; instead of returning the default (English) version, it allows you to define your own hierarchy of locales.

By default, French (Canada) will first use French before falling back to English. Similarly, missing labels in Brazilian Portuguese will first try to return Portuguese labels before the English ones. This feature also accommodates completely custom fallbacks.

The same goes for the underlying Flow Framework and the Neos CMS. This is only a front end thing though. On the back end, a user can only configure one language. If there’s no translation in that language, it will fall back to the site’s default and eventually to English.

TYPO3 Back end Language

TYPO3 Back end Language Setting

WordPress.com

On WordPress.com there’s a user interface language selection in the account settings. One cannot select multiple preferred languages though, but only one.

WordPress.com User Interface Language

WordPress.com User Interface Language Setting

Facebook

Facebook only has a really simple language settings. Although a user can select multiple languages they understand for use in the News Feed, they can only choose one specific locale for the overall UI.The locales that Facebook supports are referenced in the Facebook Locales XML file. That file includes multiple variants for various languages, but only one variant for others. For example, there’s only de_DE, but no de_CH. Plus, you can’t choose between formal and informal variants.

Facebook User Language Setting

Facebook User Language Setting

Summary

We’ve now covered a bunch of well-known web platforms and content management systems to see how they are handling this problem. Various techniques exist to assist with finding the right translation. Sometimes they are automated, but most of the time the choice is left up to the individual user.

This research should help with the next steps of the Preferred Languages project as these observations need to be adapted for WordPress so that we can learn from them. Feel free to leave any comments about this research in the comments. After that, I try to schedule a new meeting in December where the next steps can be discussed.

#feature-projects, #i18n, #preferred-languages

User Admin Languages and Locale Switching in 4.7

In WordPress 4.7 users will be able to select their preferred locale (language) when editing their profile. 🎉🎉 This allows for greater personalization of the WordPress admin and therefore a better user experience.

The back end will be displayed in the user’s individual locale while the locale used on the front end equals the one set for the whole site. If the user didn’t specify a locale, the site’s locale will be used as a fallback. The new locale property of the WP_User class can be used to retrieve the user’s locale setting.

The new user language option when editing the profile

The new user language option

To enable sending emails in the language of the recipient and not the current user’s language, an API has been introduced to switch locales and translations at any point during the page load.

Here are some of the new API functions with usage examples.

get_user_locale( $user_id )

This function is used to retrieve the locale of any user. When no user ID is set it uses the current user. The user locale should be used for any user-facing screens in the admin.

The user locale is stored as a user meta locale. Therefore it can be modified with the get_{$meta_type}_metadata filter.
If the meta field is empty the value of get_locale() is returned.

switch_to_locale( $locale )

switch_to_locale() switches the locale and (un)loads text domains according to a given locale. This includes the global $wp_locale object as well. It can be used together with get_locale() or get_user_locale(). Core’s main purpose of this function is to be able to send emails like password/email change notifications in the recipient’s language. Admin emails, emails for  get_option( 'admin_email' ), are using get_locale(), the site language.

When switching a locale several actions are fired which allows one to perform further actions:

  • change_locale: Fires when a locale is switched or restored. Core uses it to re-init post types and taxonomies.
  • switch_locale: Fires when a locale is switched.
  • restore_previous_locale: Fires when a locale is restored to the previous one.

restore_previous_locale() and restore_current_locale()

restore_previous_locale() can be used to restore the previous locale. Example:

$locale_switched = switch_to_locale( get_user_locale() );

// Do something.

if ( $locale_switched ) {
    restore_previous_locale();
}

Use restore_current_locale() if you want to empty the current stack and restore the original locale.

is_locale_switched()

As the name implies, it checks whether switch_to_locale() is in effect.

WP_Locale_Switcher

switch_to_locale(), restore_previous_locale(), restore_current_locale(), and is_locale_switched() are wrapping the same named methods of a new WP_Locale_Switcher class. This class is responsible for holding a stack of locales and filtering the actual locale through the locale filter.

Note about admin-ajax.php

As admin-ajax.php is in the admin, anyone getting translated strings via Ajax will get strings in the user’s locale when they are logged in. You can use switch_to_locale( get_locale() ) to ensure the string is returned in the site’s locale, rather then the user’s locale. Or, ideally, leverage the REST API. 💪🏼

For background information on these changes see #29783, #26511, and #38485.

Related dev note: Changed loading order for current user in 4.7.

#4-7, #dev-notes, #i18n

New Post Type Labels in 4.7

In WordPress 4.7, two additional labels have been made available for custom post types. These get passed in via the labels argument when using register_post_type(). The following labels are new:

  • view_items – The post type archive label used in the toolbar on the edit screen. Default “View Posts” / “View Pages”. See #34113.
  • attributes – The label used for the title of the post attributes meta box (used to select post type templates). Default “Post Attributes”/ “Page Attributes”. See #18375.

 

#4-7, #dev-notes, #i18n

Preferred Languages Chat Summary: October 26

After introducing the Preferred Languages project less than two weeks ago, we held our first meeting on Wednesday. This is a summary of the Preferred Languages chat from October 26. (Slack log)

Attendees: @casiepa, @chantalc, @petya, @swissspidy

Topics

After going through initial feedback on the introduction post, there was a short recap of ideas that have been suggested so far:

  • Have a translated setting of preferred languages, where the translators would define the locales they should “fall back” to in case there’s no translation available. E.g. de_CH states to prefer de_DE and de_DE_formal after that.
    • Pros: simple solution, no UI changes needed
    • Cons: very subjective, unclear how the expected translations would be installed
  • Rely on the Accept-Language header every browser sends, which is set based on the user’s system preferences.
    • Pros: no UI changes needed, it’s a reliable standard
    • Cons: no support for formal language variants, unclear how the expected translations would be installed
  • Extend the UI to allow setting multiple preferred languages
    • Pros: leaves the choice to the user, clear how to install the expected translations, could be used together with Accept-Language
    • Cons: makes the UI option more complex

Besides that, it was suggested to display a note to the user when there’s a missing translation to encourage them to translate WordPress. However, this can be worked on independently in #23348.

Before we get into details and possible solutions, more (non-technical) research needs to be done. We’re mostly interested in how other platforms and content management systems handle this. Think Wikipedia, Facebook, Drupal, Joomla, and so on. WordPress isn’t the only system facing this situation.

Research results can be shared in the comments to this post. Every help in that regard is highly appreciated! So far I’ve collected some information about Drupal and Joomla. After that, I’ll publish the results in a separate blog post.

The next meeting will be on Wednesday, 9 November 2016, 17:00 UTC in the #core-i18n Slack channel.

#feature-projects, #i18n, #preferred-languages, #summary

Preferred Languages

With the introduction of language packs two years ago ([29630]), it’s easier than ever for users to change the main language of their site. However, in some cases a single locale (i.e. the language variant, like Canadian French) is not enough.

Let’s say a site of mine is running German (Switzerland), which there is a language pack for. However, most plugins only have a German (Germany) translation, or perhaps only even a de_DE_formal translation. As a native German speaker, I’d prefer to read a German (Germany) translation instead of English, if a German (Switzerland) version did not exist. Instead of getting translations as I’d wish (as the translations are very similar), WordPress falls back to the original English strings. That’s a poor user experience for many non-English speakers. Now, since the addition of a user-specific language (#29783), this issue is even more important.

There’s been a long discussion about this issue in #28197, where possible solutions have been suggested without any consensus. Instead of directly talking about how this can be technically implemented, we should first explore the actual user experience problems and see what’s possible and how it might look.

For this, I began researching how other software approach this problem. Those of us interested in this problem can learn from existing solutions and proceed from there.

These screenshots should give you a better understanding of what I’m talking about:

As you can see, these software products create a “fallback chain” for the user’s preferred languages. In theory, I could also set my preferred languages to es_ES -> de_DE -> en_GB -> en_US, if that was the order in which I preferred translations.

To keep momentum and continue thinking through this problem, I want to kick off a feature project, about improving the experience for WordPress users who use the product in non-English languages, of which multiple locales may exist.

Get Involved

Your feedback is incredibly important to ensuring we get this right. Leave any thoughts in the comments below. Would you like to help out? Awesome. Let’s have an initial chat on Wednesday, 26 October 2016, 17:00 UTC in the #core-i18n Slack channel and go from there.

I’ve set up a GitHub repository that can be used as a playground for discussions, prototypes, and eventually a working solution. For this, design and accessibility feedback would be very helpful. I’m confident that we can build something that we can propose for inclusion in a future WordPress release!

#feature-projects, #i18n, #preferred-languages