An Update on Preferred Languages

5 years after announcing the Preferred Languages feature project and implementing the first prototype, it’s time for a long overdue update on where things currently stand.

More than half of all WordPress sites in the world use a language other than US English. For these sites and users, the options to change the site and user language are great. But when there’s no translationtranslation The process (or result) of changing text, words, and display formatting to support another language. Also see localization, internationalization. for a given pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party or theme, WordPress falls back to US English. That’s a poor user experience for many non-English speakers.

The Preferred Languages plugin solves this issue by doing the same thing operating systems, browsers, and other types of software have been doing for years. It lets you select multiple preferred languages in your settings. WordPress then tries to load the translations for the first language that’s available, falling back to the next language in your list.

The Preferred Languages UIUI User interface

Recent New Features

After stabilizing the initial prototype, the feature pluginFeature Plugin A plugin that was created with the intention of eventually being proposed for inclusion in WordPress Core. See Features as Plugins. has lived a mostly dormant life, with only irregular updates here and there. Adding support for JavaScript i18n introduced in WordPress 5.0 was the biggest enhancementenhancement Enhancements are simple improvements to WordPress, such as the addition of a hook, a new feature, or an improvement to an existing feature.. With the plugin being stable and used on thousands of sites without issues, there was simply no need for any other change. Yet, the plugin was far from feature complete.

Over the past year, I blew the dust off and made significant improvements to the plugin:

  • Bringing UI and code up-to-date with latest WordPress version
  • Improved Multisitemultisite Used to describe a WordPress installation with a network of multiple blogs, grouped by sites. This installation type has shared users tables, and creates separate database tables for each blog (wp_posts becomes wp_0_posts). See also network, blog, site support, bringing Preferred Languages to Networknetwork (versus site, blog) settings
  • Site Health integration
  • Increased test coverage
  • Improved compatibility with other plugins, especially those accessing the locale user metaMeta Meta is a term that refers to the inside workings of a group. For us, this is the team that works on internal WordPress sites like WordCamp Central and Make WordPress.
  • Added an option to merge incomplete translations to avoid fallbacks to US English

The latter is actually a pretty cool enhancement and can be enabled using a filterFilter Filters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output.. Here’s what it does:

Let’s say your preferred languages are es_CR, es_MX, es_ES, en_US (in this order). With this feature, if some of the translations are incomplete, your site will be displayed in es_CR, with missing strings taken from es_MX, es_ES etc. Without this feature, missing strings would simply be displayed in US English straight away. Merging translations this way makes for a much better user experience.

What’s Still Missing / Open Questions

Textdomain Registry

Since the Preferred Languages feature plugin also needs to work well when switching locales, #39210 has been a missing feature for a long time. While the plugin has its own implementation of a textdomain registry originally created (but then reverted) in that ticketticket Created for both bug reports and feature development on the bug tracker., it is required for this change to finally land in coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress..

My hope is that this can be implemented in WordPress 6.1+.

JavaScriptJavaScript JavaScript or JS is an object-oriented computer programming language commonly used to create interactive effects within web browsers. WordPress makes extensive use of JS for a better user experience. While PHP is executed on the server, JS executes within a user’s browser. https://www.javascript.com/. Code Base

The initial version of the Preferred Languages plugin was built in a pre-GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ era, using jQuery and jQuery UI Sortable. It doesn’t make much sense to potentially merge such a new UI component into core that is built with those technologies.

Rather, some time should be spent to rebuild the client-side code. There are two possible options here:

  1. Rewrite in vanilla JavaScript, replacing jQuery with modern Web APIs. This is most feasible when removing the capability to sort languages using drag & drop, for which jQuery UI Sortable is currently used.

    If we’re okay with dropping drag & drop functionality, then this would be a straightforward change.
  2. Rewrite everything in ReactReact React is a JavaScript library that makes it easy to reason about, construct, and maintain stateless and stateful user interfaces. https://reactjs.org/.. A prototype of this actually exists, so it’s mostly a matter of finishing it up & perhaps submitting some upstream PRs to Gutenberg for any missing features.
    Using React would be more in line with current best practices and expansion of Gutenberg throughout WordPress adminadmin (and super admin). Such a rewrite would also facilitate potential use of the component directly within a Gutenberg context.
    On the other hand, it would significantly increase overall code size and thus maintenance burden, for little immediate benefit.

While I am currently heavily leaning towards the first option, inputs are always welcome!

Of course, if we are okay with keeping jQuery & jQuery UI Sortable, then no change is needed at all.

The Next Steps

The Preferred Languages feature plugin can always use help with development and testing. Right now resolving the two open questions is the main priority before proposing merging this functionality into core.

Active development is taking place on GitHub. If you want to get involved, check out open issues and join the #core-i18n channel on Slack.

#feature-plugins, #feature-projects, #i18n, #polyglots, #preferred-languages

i18n component bug scrub for WordPress 5.6

As part of the 5.6 release, we’ll be hosting a i18ni18n Internationalization, or the act of writing and preparing code to be fully translatable into other languages. Also see localization. Often written with a lowercase i so it is not confused with a lowercase L or the numeral 1. Often an acquired skill. focused bugbug A bug is an error or unexpected result. Performance improvements, code optimization, and are considered enhancements, not defects. After feature freeze, only bugs are dealt with, with regressions (adverse changes from the previous version) being the highest priority. scrub this Wednesday, September 16, 2020, 18:00 UTC in the #core channel on SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/..

They are many tickets for this component that will need feedback and help to move forward, as part of WordPress 5.6 release cycle.

During the scrub, we’ll go through these two TracTrac An open source project by Edgewall Software that serves as a bug tracker and project management tool for WordPress. reports: https://core.trac.wordpress.org/query?component=I18N&milestone=5.6&status=!closed and https://core.trac.wordpress.org/query?component=I18N&id=39295%2C39673%2C39756%2C50040&status=!closed. These two reports contain the i18n tickets with no replies and those slated for 5.6.

#5-6, #bug-scrub, #i18n

More support for JavaScript i18n in WordPress 5.5

In WordPress 5.5, several legacy JavaScriptJavaScript JavaScript or JS is an object-oriented computer programming language commonly used to create interactive effects within web browsers. WordPress makes extensive use of JS for a better user experience. While PHP is executed on the server, JS executes within a user’s browser. https://www.javascript.com/. localization variables have been removed in wp-admin in favor of using the modern form wp.i18n.__().

The i18n package was introduced with WordPress 5.0 to facilitate JavaScript and dynamic UIs translationtranslation The process (or result) of changing text, words, and display formatting to support another language. Also see localization, internationalization.. It was therefore possible to use it instead of other functions used for localization as wp_localize_script().

Several interfaces in wp-adminadmin (and super admin) already use the functions and utilities provided by i18ni18n Internationalization, or the act of writing and preparing code to be fully translatable into other languages. Also see localization. Often written with a lowercase i so it is not confused with a lowercase L or the numeral 1. Often an acquired skill. for translatable strings. With WordPress 5.5, many others are now adopting this new approach.

Here are the related tickets for these changes on TracTrac An open source project by Edgewall Software that serves as a bug tracker and project management tool for WordPress.: #50235, #50525, #50526, #50527, #50535, #50553, #50576, #50577, #50578, #50579, #50596, #50597, #50599, #50601, #50602, #50603, #50604, #50605.

For developers

If you develop WordPress themes or extensions some previously globally accessible localization variables could stop working and cause breaks in your code.

You can see how to use the functions and utilities of i18n package in this p2 post.

Props to @loicblascos for raising this note topic and @francina and @johnbillion for review.

#5-5, #dev-notes, #i18n

The Block Registration API – status update

Build a WordPress.orgWordPress.org The community site where WordPress code is created and shared by the users. This is where you can download the source code for WordPress core, plugins and themes as well as the central location for community conversations and organization. https://wordpress.org/ directory for discovering blocks, and a way to seamlessly install them is one of the 9 priorities in the 2019 roadmap. This post tries to summarize work done so far and identify all the next steps required to land this project in WordPress coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. later this year.

It has been over two months since the MetaMeta Meta is a term that refers to the inside workings of a group. For us, this is the team that works on internal WordPress sites like WordCamp Central and Make WordPress. team published a post intended as a starting point for discussion and new ideas for the Block Directory, and a new type of plugin:

Put briefly, I’d like to propose a new type of WordPress pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party that provides blocks and nothing else: Single BlockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. Plugins. These will be hosted in a separate Block Directory section of the Plugin Directory. They will be JavaScriptJavaScript JavaScript or JS is an object-oriented computer programming language commonly used to create interactive effects within web browsers. WordPress makes extensive use of JS for a better user experience. While PHP is executed on the server, JS executes within a user’s browser. https://www.javascript.com/.-based, and each plugin will register a single Block. And they will be searchable and installable from within the GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ editor itself.

@tellyworth

Currently, new Gutenberg blocks can be provided by plugins, which often register many blocks, and which are managed from outside the editor. The proposal mentioned above outlines a new type of simple block-based plugin that is intended to be seamlessly installed from within Gutenberg itself. It was later followed up with the call for design on installing blocks from within Gutenberg. There was an essential technical aspect highlighted in the post:

The WordPress.org APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. will provide an endpoint for searching for blocks by name and description, and return metadata similar to that of plugins. Gutenberg’s Inserter could use that endpoint to also show relevant block plugins that are available to install, with a button and process for seamless installation.

@tellyworth

This new endpoint is going to be based on the Block Registration API RFC which intends to address the server-side awareness of blocks and simplify the block discovery for the block directory project. In practical terms, it means that we are seeking for a solution where block type registration is declarative and context-agnostic. Any runtime (PHPPHP The web scripting language in which WordPress is primarily architected. WordPress requires PHP 5.6.20 or higher, JSJS JavaScript, a web scripting language typically executed in the browser. Often used for advanced user interfaces and behaviors., Android, iOSiOS The operating system used on iPhones and iPads. or other) should be able to interpret the basics of a block type and should be able to fetch or retrieve the definitions of the context-specific implementation details.

Core Editor team reached the point where we believe that the Request for Comments is close to being finalized. However, there are still some areas where we feel that other WordPress teams could have a significant impact on the proposal.

Internationalization

The way how translations are handled inside JSONJSON JSON, or JavaScript Object Notation, is a minimal, readable format for structuring data. It is used primarily to transmit data between a server and web application, as an alternative to XML. files is something novel for WordPress core. The current proposal for PHP follows the existing get_plugin_data core function which parses the plugin contents to retrieve the plugin’s metadata, and it selectively applies translations dynamically. It would be also similar for JavaScript, with the remark that we plan to implement a custom Babel plugin which would mirror PHP behavior for ESNext code. The transformation would happen during the build step to ensure that files can be statically analyzed before scripts are enqueued. You can find more details in RFC document in the Internationalization section. There is also an issue#15169 opened which describes the technical details of the proposed JavaScript implementation of the Babel plugin.

Core JS team discussed this topic at the end of the weekly meeting on Apr 2nd. We have received great feedback from @swissspidy which helped to shape the proposal. However, we still encourage other Polyglots teamPolyglots Team Polyglots Team is a group of multilingual translators who work on translating plugins, themes, documentation, and front-facing marketing copy. https://make.wordpress.org/polyglots/teams/. members to voice their opinions.

New REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/. endpoints

The long term vision for the block discovery in WordPress includes:

  • Fetching the available block types through REST APIs.
  • Fetching block objects from posts through REST APIs.

The proposed implementation for the server-side awareness of block types should ensure that it stays intact with all recommendations that REST API team might have. That’s why we encourage the REST API team to get involved in the process early on. There is no immediate need to start working on new block type related API endpoints. However, it would be great to have it included on the roadmap. Ideally, they should stay as close as possible to the WordPress.org API and the final shape of the endpoint for searching for blocks.

#block-directory, #core-editor, #gutenberg, #i18n, #meta, #rest-api

New Post Type Labels in 5.0

It’s been a while since new post type labels have been introduced in WordPress.

In WordPress 5.0, five additional labels have been made available for custom post types. These get passed in via the labels argument when using register_post_type(). The following labels are new:

  • item_published — The label used in the editor notice after publishing a post. Default “Post published.” / “Page published.”
  • item_published_privately — The label used in the editor notice after publishing a private post. Default “Post published privately.” / “Page published privately.”
  • item_reverted_to_draft — The label used in the editor notice after reverting a post to draft. Default “Post reverted to draft.” / “Page reverted to draft.”
  • item_scheduled — The label used in the editor notice after scheduling a post to be published at a later date. Default “Post scheduled.” / “Page scheduled.”
  • item_updated — The label used in the editor notice after updating a post. Default “Post updated.” / “Page updated.”

Please note the trailing period for all of these strings.

#5-0, #dev-notes, #i18n

5.0: JavaScript language packs are here 🎉

News from the JSJS JavaScript, a web scripting language typically executed in the browser. Often used for advanced user interfaces and behaviors. package inclusion focus: JavaScriptJavaScript JavaScript or JS is an object-oriented computer programming language commonly used to create interactive effects within web browsers. WordPress makes extensive use of JS for a better user experience. While PHP is executed on the server, JS executes within a user’s browser. https://www.javascript.com/. language packs are here! Thanks to AWESOME work by @herregroen, @ocean90, @swissspidy, @nerrad, @atimmer and @schlessera we can now translate strings in JavaScript files and distribute them via https://translate.wordpress.org. This functionality will soon be expanded to also work for plugins and themes. This is a major milestone for JavaScript development in WordPress and completes the JavaScript package inclusion focus.

How does it work?

In short, to make it work, you need to

  • use the @wordpress/i18n package for making strings in your JavaScript translatable.
  • let WordPress know that a script has translations by calling wp_set_script_translations( 'my-handle', 'my-domain' ) after you register a script.

Read more about it in the JS i18n devnote.

Can I have a look at the tickets involved?

Sure thing! Here’s an overview of all the work we’ve done in the last few weeks to get to this point:

  1. WP-CLIWP-CLI WP-CLI is the Command Line Interface for WordPress, used to do administrative and development tasks in a programmatic way. The project page is http://wp-cli.org/ https://make.wordpress.org/cli/: Make sure wp i18n CLI command includes JavaScript translations when generating Potfiles. Relevant PRs: https://github.com/wp-cli/i18n-command/pulls?q=is%3Apr+is%3Aclosed+author%3Aherregroen
  2. MetaMeta Meta is a term that refers to the inside workings of a group. For us, this is the team that works on internal WordPress sites like WordCamp Central and Make WordPress.: generate translations on translate.wordpress.org for core, themes and plugins. Ticket: https://meta.trac.wordpress.org/ticket/3748
  3. Meta: generate and serve JS language packs from translate.wordpress.org for core, themes and plugins. Ticket: https://meta.trac.wordpress.org/ticket/3876
  4. CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress.: Logic to load JS translations. Ticket: https://core.trac.wordpress.org/ticket/45103
  5. Core: API to register JS translations for a script. https://core.trac.wordpress.org/ticket/45161

What’s next?

With JS language packs we have concluded the package inclusion focus. In the remaining time we will keep focus on two things:

  • Keep the packages up-to-date until 5.0 ships.
  • @herregroen will shift focus towards generating the core JS code documentation on Devhub (https://developer.wordpress.org) and making it searchable. Since this pretty much only involves meta, this is not a release blocker. But with the heaps of JavaScript that 5.0 adds to WordPress, it would be nice to have its documentation searchable and discoverable through Devhub. At the same time it would also make WordPress’ legacy JavaScript (of which a lot was documented in the last years in the JS documentation initiative) more accessible to the broader community. Work on this had already started as part of the JS docs initiative.

#5-0, #dev-notes, #i18n

New! JavaScript i18n support in WordPress 5.0

For years, internationalization (i18ni18n Internationalization, or the act of writing and preparing code to be fully translatable into other languages. Also see localization. Often written with a lowercase i so it is not confused with a lowercase L or the numeral 1. Often an acquired skill.) has been a thing that has been pretty well supported in WordPress when it comes to PHPPHP The web scripting language in which WordPress is primarily architected. WordPress requires PHP 5.6.20 or higher development. For PHP, WordPress already provides all the tools necessary to make it as easy as possible to localize WordPress coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress., themes and plugins to any language. Today we are bringing the same capabilities to JavaScriptJavaScript JavaScript or JS is an object-oriented computer programming language commonly used to create interactive effects within web browsers. WordPress makes extensive use of JS for a better user experience. While PHP is executed on the server, JS executes within a user’s browser. https://www.javascript.com/. development for WordPress.

How does it work?

When registering your scripts you can add wp-i18n as a dependency to allow you to add translatable strings as you would in PHP:

wp_register_script( 'my-handle', plugins_url( '/js/my-file.js', MY_PLUGIN ), array( 'wp-i18n' ) );

Inside your scripts you will then be able to use wp-18n as follows:

const { __, _x, _n, _nx } = wp.i18n;

__( '__', 'my-domain' );
_x( '_x', '_x_context', 'my-domain' );
_n( '_n_single', '_n_plural', number, 'my-domain' );
_nx( '_nx_single', '_nx_plural', number, '_nx_context', 'my-domain' );

These functions mirror their PHP counterparts and can be used in exactly the same manner. 

The final step is to tell WordPress your script contains translations and of which domain, this is to allow WordPress to selectively load only the necessary translations to ensure everything is as fast as can be:

wp_set_script_translations( 'my-handle', 'my-domain' );

Make sure to also specify the Text Domain for your translations in the headerHeader The header of your site is typically the first thing people will experience. The masthead or header art located across the top of your page is part of the look and feel of your website. It can influence a visitor’s opinion about your content and you/ your organization’s brand. It may also look different on different screen sizes. of your pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party file. Otherwise the translations will not be picked up by translate.wordpress.org

Advanced usage

Right now it’s already possible to ship your own translations using the load_textdomain function and passing your own MO file. This is also possible using wp_set_script_translations which accepts an optional third path argument that allows you to tell WordPress to first look elsewhere for translations:

wp_set_script_translations( 'my-handle', 'my-domain', plugin_dir_path( MY_PLUGIN ) . 'languages' );

If passed WordPress will first check if a file in the format of ${domain}-${localeLocale A locale is a combination of language and regional dialect. Usually locales correspond to countries, as is the case with Portuguese (Portugal) and Portuguese (Brazil). Other examples of locales include Canadian English and U.S. English.}-${handle}.jsonJSON JSON, or JavaScript Object Notation, is a minimal, readable format for structuring data. It is used primarily to transmit data between a server and web application, as an alternative to XML. exists in the given path and use it as the source of translations if so. Alternatively it will also first check the given path for the md5 filename before defaulting to the WordPress languages directory.

If you want to ship your own translationtranslation The process (or result) of changing text, words, and display formatting to support another language. Also see localization, internationalization. files these should be in the JED 1.x ( .json ) format. GlotPress is able to create these along with other tools such as po2json. Ideally these files should only contain translations that occur within their respective JSJS JavaScript, a web scripting language typically executed in the browser. Often used for advanced user interfaces and behaviors. files. With po2json you can generate these files as follows:

po2json translation.po translation.json -f jed

This will generate a JSON in the following format:

{
  "translation-revision-date": "+0000",
  "generator":                 "GlotPress/2.3.0-alpha",
  "domain":                    "messages",
  "locale_data":               {
    "messages": {
      "":                                                             {
        "domain":       "messages",
        "plural-forms": "n != 1",
        "lang":         "en-gb"
      },
      "This file is too big. Files must be less than %d KB in size.": [
        "This file is too big. Files must be less than %d KB in size."
      ],
      "%d Theme Update":                                              [
        "%d Theme Update",
        "%d Theme Updates"
      ],
      "password strength\u0004Medium":                                [
        "Medium"
      ],
      "taxonomy singular name\u0004Category":                         [
        "Category"
      ],
      "post type general name\u0004Pages":                            [
        "Pages"
      ]
    }
  }
}

Behind the screens

When you upload your plugin or theme to wordpress.org all JS files will automatically be parsed the same as is already being done for PHP files. Any detected translations will be added to translate.wordpress.org to allow the community to cooperate to ensure WordPress, plugins and themes are available in as many languages as possible.

In order to parse all JS files the i18n-command for wp-cli is used. This replaces makepot.php to not only allow picking up translations in JS files but also to audit strings, parse strings only of a specific text domain and even pick up a few strings that weren’t detected by makepot.php. This command is freely available and open-source as makepot.php was and it’s recommended that anyone using makepot.php transition over to this much improved replacement.

Based on these parsed translations Language Packs are generated. Traditionally these used to only contain PO and MO files, one pair for each locale. In order to selectively load only the necessary translations regardless of whether it’s used or not a few more files are being added, one JSON file for every JS file that contains translations per locale.

When parsing JS files for translations we don’t know which handle is used to register that file so we’ve had to use an alternate mechanism to find the translations belonging to each file. To do this we’re using the md5 of the relative path of each file. This is appended to the usual name of ${domain}-${locale} in the form of ${domain}-${locale}-${md5}.json.

When you set script translations for a handle WordPress will automatically figure out the relative md5 hash of your source file, check to see if a translations file exists and if so ensure that it’s loaded into wp.i18n before your script runs.

Plugin and theme support

Translation and Language packs support for plugins and themes that are hosted on the repo is expected in the upcoming weeks. The patches are ready and waiting for commit. Plugin and theme authors are encouraged to start using wp-i18n in their JavaScript projects.

Credits

This APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. wouldn’t have been possible without the long standing efforts of @ocean90, @swissspidy, @nerrad, @atimmer, @schlessera and more recently @herregroen. Thanks a ton for the incredible work you’ve all put into making this a reality! Another thanks to @herregroen for providing the necessary input for this devnotedev note Each important change in WordPress Core is documented in a developers note, (usually called dev note). Good dev notes generally include: a description of the change; the decision that led to this change a description of how developers are supposed to work with that change. Dev notes are published on Make/Core blog during the beta phase of WordPress release cycle. Publishing dev notes is particularly important when plugin/theme authors and WordPress developers need to be aware of those changes.In general, all dev notes are compiled into a Field Guide at the beginning of the release candidate phase..

#5-0, #dev-notes, #i18n

JavaScript I18N Meeting Summary: May 8th

After the blog post about JavaScript internationalization, we held a first meeting last Tuesday to discuss how we can improve client-side translationtranslation The process (or result) of changing text, words, and display formatting to support another language. Also see localization, internationalization. in JavaScriptJavaScript JavaScript or JS is an object-oriented computer programming language commonly used to create interactive effects within web browsers. WordPress makes extensive use of JS for a better user experience. While PHP is executed on the server, JS executes within a user’s browser. https://www.javascript.com/.. This is a summary of the JSJS JavaScript, a web scripting language typically executed in the browser. Often used for advanced user interfaces and behaviors. I18Ni18n Internationalization, or the act of writing and preparing code to be fully translatable into other languages. Also see localization. Often written with a lowercase i so it is not confused with a lowercase L or the numeral 1. Often an acquired skill. chat from May 8th, 2018. (Slack log)

Topics

First, we talked about the introductory blogblog (versus network, site) post and the feedback it received so far. While the post gives a great overview of the two main problems — extracting strings from JavaScript and loading the actual translations in a way that makes sense — there are many smaller problems attached to them that need to be tackled.

Thus, we started looking at the string extraction part and the problems which come with it:

  • Plugins bundle their assets in many different ways: minified, non-minified, using simple UglifyJS or multiple individual modules and more complex tools like webpack.
  • If WordPress.orgWordPress.org The community site where WordPress code is created and shared by the users. This is where you can download the source code for WordPress core, plugins and themes as well as the central location for community conversations and organization. https://wordpress.org/ does parse source files, which syntax (e.g. ES2015, ES2018, JSX, etc.) should be supported? WordPress.org needs to set clear boundaries and best practices for pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party and theme developers.
  • Larger plugins like GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ only provide the minified build files on WordPress.org. How can strings be extracted from those?
  • Should such plugins be expected to do the string extraction part themselves and provide a POT file that GlotPress can import? Right now, this is not possible.

Multiple JavaScript I18N tools already exist, e.g. @wordpress/i18n, @wordpress/babel-plugin-makepot, and a WP-CLI i18n command. These can be used to test a new JavaScript string extraction process for individual plugins like Gutenberg and perhaps a dedicated testing plugin.

We then switched over to the other main problem: loading translations for use in JavaScript. Since any solution here as an impact on string extraction and delivering language packs, this ideally should be solved first. The problem areas here are:

  • Plugins can have multiple scripts that can be enqueued each on their own. Loading all translated strings even though you’re only enqueueing one small script handle would have a negative impact on performance.
  • Could performance be improved by caching translations using something like IndexedDB?
  • A script handle could be a minified JavaScript file that originally consisted of multiple smaller JavaScript modules, each containing some translatable string. How can we know which translations should be loaded for the resulting script handle?
  • Can we have a map of script handle => original JavaScript files or perhaps a way of saying “for script handle X, load strings in text domains Y and Z”. How would an implementation for this look like?
  • What if a bundled script contains some third-party components calling wp.i18n.__() with their individual text domain? How could this be handled?

At the end, there were still too many open questions and we figured it would be best to have a look at how other projects handle this in order to not reinvent the wheel.

Next Meeting

We will meet again on Tuesday, May 15th at 15:00 UTC in #core-i18n to further discuss especially the translation loading part.

#i18n

Preferred Languages: The Prototype

A bit over six months ago I set the foundation for the Preferred Languages feature project. After highlighting the problems many WordPress users around the world are facing, some time was spent on researching popular platforms and other systems to see how they handle the issue of setting multiple preferred languages. However, this didn’t really help to move one step forward and the project became dormant — until now.

Personally, I’ve been experiencing the same problems with so-called language fallbacks over and over again. During WordCampWordCamp WordCamps are casual, locally-organized conferences covering everything related to WordPress. They're one of the places where the WordPress community comes together to teach one another what they’ve learned throughout the year and share the joy. Learn more. Bilbao I decided to revive the Preferred Languages project. In order to do this I used the previously mentioned GitHub repository to build a super-simple proof-of-concept pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party to have a new basis for discussion. As originally envisioned, this repository can be used as a playground for discussions, prototypes, and eventually a working solution.

This new plugin lets you select multiple preferred languages in your settings. WordPress then tries to load the translations for the first language that’s available, falling back to the next language in your list.

A sneak peek of an early version quickly made the rounds on Twitter:

After some very positive feedback I worked a bit more on it and wanted to share the latest version here with a larger group of people. Here’s what it currently looks like on both the settings screen and when editing your profile:

Without much further ado, please check out the plugin on GitHub, test it on your local WordPress site, and leave some feedback!

#feature-projects, #i18n, #preferred-languages

Preferred Languages: Research

Following the first meeting of the Preferred Languages project, some time was spent on researching popular platforms and other systems to see how they handle the issue of setting multiple preferred languages. To recap, this is needed to show things in the most suitable language for users in case their requested language is not installed.

Browsers & Operating Systems

All major browsers and operating systems have some UIUI User interface where the user can set their preferred languages. Mostly used for the Accept-Language HTTPHTTP HTTP is an acronym for Hyper Text Transfer Protocol. HTTP is the underlying protocol used by the World Wide Web and this protocol defines how messages are formatted and transmitted, and what actions Web servers and browsers should take in response to various commands. headerHeader The header of your site is typically the first thing people will experience. The masthead or header art located across the top of your page is part of the look and feel of your website. It can influence a visitor’s opinion about your content and you/ your organization’s brand. It may also look different on different screen sizes., one can set multiple languages and order them by preference. These systems usually know multiple variants like German and German (Switzerland), but not formal / informal variants.

Worth noting that on these systems you can often choose more settings that are influenced by the language, like temperature units and date formats. Related: #18146.

The Accept-Language HTTP Header

RFC 7231 about the HTTP/1.1 standard says the following about this header:

The “Accept-Language” header field can be used by user agents to indicate the set of natural languages that are preferred in the response. Each language-range can be given an associated quality value representing an estimate of the user’s preference for the languages specified by that range.

This header is quite powerful. For example, Accept-Language: da, en-gb;q=0.8, en;q=0.7 would mean: “I prefer Danish, but will accept British English and other types of English”. I can only recommend reading more about it in the RFC, as it helps getting a better understanding of the problems it tries to solve.

This header is usually used by websites to redirect users to the correct version or display a hint like “This content is also available in XY”. While each language in the header has a specific numeric priority, there is usually only a drag & drop interface to determine the order.

Unicode Common LocaleLocale A locale is a combination of language and regional dialect. Usually locales correspond to countries, as is the case with Portuguese (Portugal) and Portuguese (Brazil). Other examples of locales include Canadian English and U.S. English. Data Repository

The Unicode CLDR provides key building blocks for software to support the world’s languages, with the largest and most extensive standard repository of locale data available. It contains an interesting chart about Language Matching, data that is used to match the user’s desired language/locales against an application’s supported languages/locales.

There’s also a technical standard about Unicode Locale Data Markup Language (LDML), an XML format for the exchange of structured locale data. The section about Locale Inheritance and Language Matching is particularly interesting. For instance, it describes finding the most well suited language using a weighted graph and gives a better picture of dealing with more complex language fallbacks.

It also takes geographic “closeness” into account, arguing that English (Slovakia) should fall back to something within Europe (e.g. British English) in preference to something far away and unrelated like English (Singapore). It’s clearly stated that these fallbacks aren’t as simple as just saying Spanish (Mexico) -> Spanish (Spain) -> English (US).

Note that this technical standard is about finding the best supported locale based on the requested list of languages. The requested list could come from different sources, such as such as the user’s list of preferred languages in the OS Settings, or from a browser’s Accept-Language list.

Wikipedia

Wikipedia is built on the MediaWiki software, which has a hardcoded list of fallback chains for some locales. This is where MediaWiki will fall back on a different language if it cannot find what it needs. For example, French (Cajun) automatically falls back to French (France) when it doesn’t have all messages defined in it. Unfortunately, there’s only little documentation about this.

Apart from that, MediaWiki distinguishes between various kinds of languages: the site content language, the user interface language and the page content language. The latter can differ from the first two and it influences the language the user views the page in, which depends on the user’s preferences, the available languages, and the defined fallbacks.

It’s worth noting that on Wikipedia, you can not only select your preferred language, but also your preferred gender. In addition to that, you can select multiple different locales for both display and input.

Joomla

I tried the official Joomla Demo to test its language management which is a bit overwhelming at first. After installing all the available languages the user is eventually able to select their preferred language for the front end and the back end. There are a few other settings for language switching on multilingual sites, but that’s pretty much it. So basically the same settings as currently in WordPress 4.7.

Joomla User Settings

Joomla User Settings

 

Drupal

Drupal 8 has a rather powerful user interface text language detection mechanism. There is a per session, per user and per browser option in the detection settings. However, users can only choose one language, so they cannot say (in coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. at least) that they want German primarily and Spanish if German is not available. But the language selected by the user is part of the larger fallback system, so it may fall back further down to other options.

The Language fallback module allows defining one fallback for a language, while the Language Hierarchy module provides a GUI to change the language fallback system. It allows setting up language hierarchies where translations of a site’s content, settings and interface can fall back to parent language translations, without ever falling back to English. This module might be the most interesting one for our research.

Drupal Language Hierarchy

Drupal Language Hierarchy

TYPO3

TYPO3 has had a locale hierarchy since 2011. Quote:

TYPO3 4.6 comes with a clever fallback mechanism when a label is not found in the requested language; instead of returning the default (English) version, it allows you to define your own hierarchy of locales.

By default, French (Canada) will first use French before falling back to English. Similarly, missing labels in Brazilian Portuguese will first try to return Portuguese labels before the English ones. This feature also accommodates completely custom fallbacks.

The same goes for the underlying Flow Framework and the Neos CMS. This is only a front end thing though. On the back end, a user can only configure one language. If there’s no translationtranslation The process (or result) of changing text, words, and display formatting to support another language. Also see localization, internationalization. in that language, it will fall back to the site’s default and eventually to English.

TYPO3 Back end Language

TYPO3 Back end Language Setting

WordPress.comWordPress.com An online implementation of WordPress code that lets you immediately access a new WordPress environment to publish your content. WordPress.com is a private company owned by Automattic that hosts the largest multisite in the world. This is arguably the best place to start blogging if you have never touched WordPress before. https://wordpress.com/

On WordPress.com there’s a user interface language selection in the account settings. One cannot select multiple preferred languages though, but only one.

WordPress.com User Interface Language

WordPress.com User Interface Language Setting

Facebook

Facebook only has a really simple language settings. Although a user can select multiple languages they understand for use in the News Feed, they can only choose one specific locale for the overall UI.The locales that Facebook supports are referenced in the Facebook Locales XML file. That file includes multiple variants for various languages, but only one variant for others. For example, there’s only de_DE, but no de_CH. Plus, you can’t choose between formal and informal variants.

Facebook User Language Setting

Facebook User Language Setting

Summary

We’ve now covered a bunch of well-known web platforms and content management systems to see how they are handling this problem. Various techniques exist to assist with finding the right translation. Sometimes they are automated, but most of the time the choice is left up to the individual user.

This research should help with the next steps of the Preferred Languages project as these observations need to be adapted for WordPress so that we can learn from them. Feel free to leave any comments about this research in the comments. After that, I try to schedule a new meeting in December where the next steps can be discussed.

#feature-projects, #i18n, #preferred-languages