Preferred Languages

With the introduction of language packs two years ago ([29630]), it’s easier than ever for users to change the main language of their site. However, in some cases a single locale (i.e. the language variant, like Canadian French) is not enough.

Let’s say a site of mine is running German (Switzerland), which there is a language pack for. However, most plugins only have a German (Germany) translation, or perhaps only even a de_DE_formal translation. As a native German speaker, I’d prefer to read a German (Germany) translation instead of English, if a German (Switzerland) version did not exist. Instead of getting translations as I’d wish (as the translations are very similar), WordPress falls back to the original English strings. That’s a poor user experience for many non-English speakers. Now, since the addition of a user-specific language (#29783), this issue is even more important.

There’s been a long discussion about this issue in #28197, where possible solutions have been suggested without any consensus. Instead of directly talking about how this can be technically implemented, we should first explore the actual user experience problems and see what’s possible and how it might look.

For this, I began researching how other software approach this problem. Those of us interested in this problem can learn from existing solutions and proceed from there.

These screenshots should give you a better understanding of what I’m talking about:

As you can see, these software products create a “fallback chain” for the user’s preferred languages. In theory, I could also set my preferred languages to es_ES -> de_DE -> en_GB -> en_US, if that was the order in which I preferred translations.

To keep momentum and continue thinking through this problem, I want to kick off a feature project, about improving the experience for WordPress users who use the product in non-English languages, of which multiple locales may exist.

Get Involved

Your feedback is incredibly important to ensuring we get this right. Leave any thoughts in the comments below. Would you like to help out? Awesome. Let’s have an initial chat on Wednesday, 26 October 2016, 17:00 UTC in the #core-i18n Slack channel and go from there.

I’ve set up a GitHub repository that can be used as a playground for discussions, prototypes, and eventually a working solution. For this, design and accessibility feedback would be very helpful. I’m confident that we can build something that we can propose for inclusion in a future WordPress release!

#feature-projects, #i18n, #preferred-languages

JavaScript Internationalization

Prequel

Over the last few releases, WordPress has made huge steps in prioritizing internationalization and localization throughout around core. To name a few highlights: language packs for core, language packs for plugins and themes, a language chooser, just-in-time loading for translations, and a user-specific language setting. But core isn’t the only place improvements have been made.

translate.wordpress.org is the platform for making WordPress available in as many languages as possible. Did you know that WordPress 4.6 is currently translated into 72 – s e v e n t y t w o – languages? Recently the number of unique translators passed the 10k mark. 🌎🌍🌏

The platform is powered by GlotPress, a collaborative, web-based software translation tool.

Internationalization is not a feature, it fixes a usability issue for many users whose primary language is not English. And that’s about 47% of our users.

The next big thing

As of today, WordPress doesn’t support a proper internationalization API in JavaScript. There are two, 4-year old tickets, #22229 and #20491, which request such an API. (The former one was closed as a duplicate of #20491 since an internationalization API in JavaScript should provide support for plurals by default.)

And that’s what this post is about: JavaScript Internationalization.

@swissspidy and I (@ocean90) have been thinking about and working on this topic, and now is the time to gather feedback on the direction and get help from everyone interested.

As you may have noticed in the prequel, internationalization spans more than WordPress core. We’ve split the task into four areas: Core, GlotPress, wp-i18n-tools, and language packs.

Jed and wp.i18n (Core)

Since we didn’t want to reinvent any wheels, our proposal is to use Jed as the base library. Jed provides “Gettext Style i18n for Modern JavaScript Apps”.

wp.i18n will be a wrapper for Jed and should be used by WordPress and plugins/themes. One of the biggest benefits of leveraging Jed is that the actual implementation is very similar to the one we already have for PHP code. The latest patch currently introduces these API functions:

  • wp.i18n.getLocale()
  • wp.i18n.loadLocaleData( data )
  • wp.i18n.addTranslations( domain, data )
  • wp.i18n.__( text, domain )
  • wp.i18n._x( text, context, domain )
  • wp.i18n._n( single, plural, number, domain )
  • wp.i18n._nx( single, plural, number, context, domain )
  • wp.i18n.esc_attr__( text, domain )
  • wp.i18n.esc_html__( text, domain )
  • wp.i18n.esc_attr_x( text, context, domain )
  • wp.i18n.esc_html_x( text, context, domain )
  • wp.i18n.numberFormat( number, decimals )
To do
  • Rethink WP_Scripts::load_translation_file() and get_js_i18n_data(), add tests.
  • Get feedback on the proposed API. (That’s this post!)
  • Apply the new API in our JS files.
    • There’s a POC patch that can be worked upon.
    • Do not forget: Underscore templates.
  • Include date of json files in update requests for language packs.
    • See wp_get_installed_translations().
    • See wp_get_json_file_data(), maybe json_decode() might be enough for this case.
    • What’s the format for meta data? See this GitHub issue for discussion.
  • sprintf() in JavaScript – useful for core?

GlotPress (translate.wordpress.org)

Jed supports JSON for translation files. GlotPress currently doesn’t support exporting translations in a JSON format. But there is a pull request which adds the required functionality to GlotPress. The format is going to support “simple” JSON (key -> value) and a Jed 1.x compatible format. The latter will be used by wp.i18n.

To do
  • Get the PR merged. Done. 🎉
  • Find a solution for the missing meta data, see the GitHub issue mentioned before.
  • To enable exports like “get all translations for file common.js”, the current filter system in GlotPress needs to be improved to support search only in file references and search for multiple file references.

wp-i18n-tools (Meta)

A script named MakePOT parses PHP files to extract the original strings and their file references. This script needs to be extended to also parse JavaScript files. A POC has been created.

To do
  • Convert the POC into nice code with tests.
  • Fix #20881 because it’s important for the GlotPress export that all file references remain.

Language Packs (Meta)

Currently a language pack includes a .po and a .mo file. Support needs to be added so that a .json file gets bundled if a project has translations for JavaScript files.

To do
  • Export Jed files when JS translations exist.
  • Include JSON file in language packs.
  • Update API to support version checks for JSON files.

Feedback

Thanks for reading, now it’s your turn. 🙂 Do you have any feedback? Are there things we should do differently? Do you have resources to help us? Please let us know in the comments!

There are currently no meetings scheduled but we’re happy to discuss any topics in #core-i18n. Just say “Hi”!

#i18n, #javascript

I18N Improvements in 4.6

WordPress 4.6 significantly improves the way translations are loaded for plugins and themes which are hosted in the WordPress Plugin and Theme Directory and using language packs via WordPress.org’s translation platform.

Loading Translations in Different Order

Functions like load_plugin_textdomain( $domain ) and load_theme_textdomain( $domain ) are used to load the translations for a specific text domain, where $domain equals the slug of your plugin or theme.

Up until now, these functions first looked inside the plugin/theme folder for available translations before checking the wp-content/languages directory, where language packs reside. With WordPress 4.6, this order has been reversed.
The translation platform for plugins/themes was opened up in December last year. Since then 15,000 plugins and themes were imported and are now benefiting from automated translation updates. That’s why translation files are most likely to be located wp-content/languages directory, hence this change. See [37414] / #34213.

Just-in-time Loading for Translations

By far the biggest change is that you do not have to call load_plugin_textdomain() or load_theme_textdomain() anymore with WordPress 4.6. Of course the same goes for load_muplugin_textdomain().

Again, since translations files are usually inside wp-content/languages, WordPress now scans that directory for available translations and automatically loads them if it encounters a text domain for the first time. That’s handled by the new _load_textdomain_just_in_time() function which gets called in get_translations_for_domain(). See [37415] / #34114.

This so-called just-in-time loading of text domains works really great and removes one more step for developers to worry about. If you do not want this to happen, however, you can still use the override_load_textdomain filter to load your text domains manually.

Also, if you use unload_textdomain() you will have to manually load translations afterwards if you want to use them again because it keeps a record of unloaded text domains.

Localized jQuery UI datepicker

https://make.wordpress.org/core/2016/07/06/jquery-ui-datepicker-localization-in-4-6/

Other Changes

  • Support for comment number declension in get_comments_number_text(). See #13651.
  • Fallback for TextDomain header field in get_plugin_data(). See #36706.
  • Updated list of continents and cities for the timezone selector. See #37554.
  • Support for the German (Switzerland) locale in remove_accents(). See #37076.
  • Improved support for month name declension. See #36790.

 

#4-6, #dev-notes, #i18n

jQuery UI datepicker localization in 4.6

In WordPress 4.6 (#29420) localization defaults were added for the jQuery UI datepicker based on the current locale using the new wp_add_inline_script() function. With 4.6 you can enqueue the datepicker and you are set. It works out of the box.

Instead of using the language files from the jQuery UI project the necessary information, mainly from the WP_Locale class, is directly passed to the datepicker. The list of localized information: closeText, currentText, monthNames, monthNamesShort, nextText, prevText, dayNames, dayNamesShort, dayNamesMin, dateFormat, firstDay, and isRTL.

If you are using the files from jQuery UI in your theme or plugin, please be aware of this change. With 4.6 you can omit calling these files and enqueue the default datepicker which will use the built-in defaults from WordPress.

If you want to change the defaults or add custom settings, you can still use all the arguments from the datepicker. WordPress only sets sane defaults for the chosen locale. But you should check if jQuery UI datepicker is enqueued and you should check if the browser is showing a native datepicker (like Chrome).

@ipstenu has scanned the plugin repo for plugins including the jQuery UI files. The following plugins can be changed to use WordPress’ own translations.

Advanced Booking Calendar, Albo Pretorio On line, Altos Toolbar, AWPCP – Classifieds Plugin, Appointment Calendar, Banner Garden Plugin for WordPress, BBS e-Popup, Beaverlodge Maintenance, Beaverlodge Menu Styler, BigContact Contact Page, Blog Post Calendar Widget, Booking Calendar Contact Form, Events Calendar, Class Booking, Contact Form 7 IE DatePicker and Number Spinner Fix, Custom fields, Datepicker i18n, Easy Calendar, EEXCESS, Event Post Type, Events Made Easy, Events Manager, Social Marketing Scheduler, FormCraft – Form Builder, Date Picker in List Fields for Gravity Forms, Gravity Forms (Spanish), Gravity Forms (nl), inquiry form creator, Jet Event System for BuddyPress, JQuery Accessible Accordion, JQuery Accessible Autocomplete, JQuery Accessible Button, JQuery Accessible Carousel, JQuery Accessible Checkbox, JQuery Accessible Datepicker, JQuery Accessible Dialog, JQuery Accessible Menu, JQuery Accessible Progressbar, JQuery Accessible Slider, JQuery Accessible Tabs, JQuery Accessible Tooltip, JQuery Accessible Tree, Js-appointment, LearnPress – WordPress LMS Plugin, LePress, List Urls, Livemesh SiteOrigin Widgets, Lodgix.com Vacation Rental Listing, Management & Booking Plugin, MailChimp List Subscribe Form, Social Media Share Buttons | MashShare, Member Register, Membership 2, Meta Collections, mind-body, Tribulant Newsletters, Orbis, Order Delivery Date for WooCommerce, OpenWebAnalytics, Pay2Post, PDF Catalog, Pronamic Events, RPB Calendar, Vinum Master Tracking Plugin, Simple Ads Manager, Simple Photos Contest, Sk Multi Tag, Solidres – Hotel booking plugin for WordPress, Cubilis Fastbooker, Surbma – Datepicker localization for Gravity Forms, teachPress, TemplatesNext ToolKit, TheNextTable, Toolset Types, Uiform – Cost Estimation & Payment Form Builder, Uiform – Form Builder, Vacation Rentals, Video Central for WordPress, Visual Form Builder, Webagency Widget – Website builder, WEeventscalendar, WooCommerce Checkout Manager, WooCommerce Quick Donation, WP Agenda, WP-AMD – Global JS and CSS handling, WP Any Form, WP Contact Slider, wp-creator-calculator, WP-CRM – Customer Relations Management for WordPress, WP FEvents Book, wp-greet, WP-Invoice – Web Invoice and Billing, WP Maintenance, WP Online Store, WP-Property – WordPress Powered Real Estate and Property Management, Wp-Recall, WP Scripts Updater, WP Swiper, wpShopGermany Free, and XO Event Calendar.

#4-6, #dev-notes, #external-libraries, #i18n

I18N Kickoff for 4.6 Chat Summary

This is a summary of the I18N Kickoff chat from April 21st.

Attendees: @petya, @SergeyBiryukov, @swissspidy, @ocean90

JavaScript i18n

User Admin Language

  • #32879 was closed as a duplicate of #26511.
  • Having a switch_to_locale() function is a precondition for #29783 and thus this feature is part of the 4.6 milestone.

Language Packs

  • To push language packs to the next level, #34213 (Change priority for loading theme/plugin translations) and #34114 (Remove the requirement to call load_plugin_textdomain() or load_theme_textdomain()) are part of the 4.6 milestone.
  • #34213 should go in early.
  • #34114 needs a refresh and the tests should make use of the WP_LANG_DIR constant.

Database Collation

  • No decision was made for #32405 because @pento wasn’t around.

Other Tasks

  • @swissspidy mentioned #18146 (Add user-level timezone setting) as an UX improvement similarly to #29783
    • @ocean90: “Have to read that ticket in detail but one of the biggest task is probably to define the line between user setting and site setting.”
  • @SergeyBiryukov volunteers to expand https://make.wordpress.org/core/handbook/best-practices/internationalization/
  • @SergeyBiryukov mentioned #13651 and that it should finally get fixed.
    • @ocean90 will review the latest patch.
  • @ocean90 mentioned that the i18n report has a lot of tickets by @ramiy related to “Avoid using HTML tags in translation strings*” which need to be reviewed. One of the reasons why it’s good to have the handbook page ready as soon as possible.
  • @ocean90 mentioned Ginger-MO https://github.com/dd32/ginger-mo, “a minimal .mo reader (with support for PHP & JSON representations), multiple text domains, and multiple loaded locales in the future” by @dd32.
    • @swissspidy asked what this would mean for #17268.
    • (after the chat) @dd32: “As for Ginger-MO, my main intention was so we *didn’t* have to use the native gettext tools. I built it with one aim – To get English (US) and non-English pageloads closer to eachother in terms of performance.”

Bug Scrub

It was decided that the first bug scrub for the i18n component is on Tuesday April 26th, 2016 at 18:00 UTC.

 

Full meeting logs: https://wordpress.slack.com/archives/core-i18n/p1461243825000006

#4-6, #i18n

I18N Kickoff for 4.6

This Thursday, 21 April, 13:00 UTC we’ll have an I18N chat in #core-i18n to discuss the roadmap and ideas for 4.6.

Some of the bigger tasks on which could be worked during the cycle:

JavaScript i18n:

  • #20491: Introduce JavaScript i18n functions
  • #22229: Plurals in JavaScript

User Admin Language:

  • #29783: Main ticket
  • #26511: Separate locale for the admin toolbar
  • #32879: Live switching Language

Language Packs:

  • #34213: Change priority for loading theme/plugin translations
  • #34114: Remove the requirement to call load_plugin_textdomain() or load_theme_textdomain()

Database Collation:

  • #32405: Database collation upgrade routine to support UTF8MB4 collations
  • https://make.wordpress.org/polyglots/2015/11/28/hi-czech-users-have-sometimes-problems-with-ordering/

 

There are currently 77 open tickets in the I18N component which we should try to reduce. It would be nice to get our long planned handbook page about best practices for Internationalization in core (see also Spelling) finalized too.

Please share in the comments below if there are any specific features and tickets that you want to contribute in this next release. Otherwise, please also join us in #core-i18n chat to discuss.

#4-6, #i18n

Language chooser in 4.0

As mentioned in previous posts, WordPress 4.0 includes a language chooser. After selecting a language WordPress will download and install the language pack on the fly. The rest of the install process will then be in that language.

Language Chooser in WordPress 4.0

Language Chooser in WordPress 4.0

A summary

To make this possible we introduced some helper functions or changed existing functions.

translations_api() is based on plugins_api() / themes_api() and retrieves translations from the WordPress Translation API. The first argument $type must be core, theme or plugin. $args is used for additional arguments. For example $args['version'] should be specified for all types. Types theme and plugin must set slug too.

wp_get_available_translations() function is a wrapper for translations_api() and returns core translations for the current installed version. The API result is cached for 3 hours.

As soon as a language is selected and the Continue button is clicked, WordPress will download the language pack in background, wp_download_language_pack() does this with the help of Automatic_Upgrader_Skin and Language_Pack_Upgrader.

Because translations are installed on the fly we had to enhance the existing load_default_textdomain() function. It now supports an optional $locale parameter to allow to switch the default translation. ([29630])

The WPLANG option is now set in single sites too and is populated on upgrade, based on the value of the WPLANG constant, which is now deprecated, see [29630]. get_locale() now includes the global $wp_local_package variable (used in localized packages) and an existing but empty WPLANG option can override the WPLANG constant as an empty WPLANG option is equivalent to en_US.

wp_dropdown_languages() replaces mu_dropdown_languages(), which had many issue like not supporting variants of the same language (like en, en_GB, en_CA and en_AU). The new dropdown is populated by the translation API.

Asynchronous translation updates

In WordPress 3.7 we had introduced Language_Pack_Upgrader::async_upgrade(). Asynchronous translation updates will run after a theme or plugin is installed or updated. What’s the purposes of async updates? One, when you install or update a theme or plugin, you’d expect to get the translations updated for that theme or plugin. But all out of date translations are updated here (even when that plugin or theme was already up to date), in order to capitalize on the fact that we have a filesystem connection (which may be via user-submitted FTP credentials).

In WordPress 4.0 this asynchronous update will no longer run on version-controlled installs. You can also use the async_update_translation filter (which corresponds exactly to the auto_update_translation filter) to disable it, see [29694].

// Disable asynchronous and automatic background translation updates
add_filter( 'async_update_translation', '__return_false' );
add_filter( 'auto_update_translation', '__return_false' );

Other notes

  • Localized packages will skip language chooser, see [29705].
  • For BC it’s allowed to choose a language specified by the WPLANG constant (but not installed), see [29691].
  • General Settings includes an option for the Site Language in single sites now too.
  • WPLANG section from wp-config-sample.php is removed, see make/polyglots post.
  • On install, WordPress will skip the language chooser if it has no access to the filesystem without asking for credentials, see [29673].
  • For a peek at what’s to come in 4.1, check out #29395.

#4-0, #dev-notes, #i18n

Internationalization project updates

Here’s where we are on the five goals for internationalization outlined previously:

1. The first step installing WordPress should be to choose a language. The rest of the install process would then be in that language.

First pass done in #28577. There is a list of things to do in the ticket, which includes:

  • Improved error handling when the API or filesystem isn’t accessible. Working on this.
  • Bring this to setup-config.php. Working on this.
  • Place browser-based language suggestions at the top. Working on this.
  • Use better markup rather than simple select/option HTML, currently being worked on by @jorbin.

2. You should be able to choose/switch a language from the general settings screen, after which the language pack should be downloaded.

This simply requires replacing mu_dropdown_languages() with a new method that handles uninstalled languages gracefully. This is easy to implement and relies on much of the same code as the install process, so it’s simply on hold until that’s done. I’ve also worked out a UX flow with @sonjanyc.

3. You should be able to search from the dashboard for plugins and themes that are available in your language.

This is handled on the WordPress.org side. The updated plugins screen will need to pass a new argument to filter by language, and then remove that argument if the user requests showing plugins in any language. We’ll need to hack in readme/description translation support but that’s a small API change and otherwise WordPress.org work, not core work.

4. All localized packages should be able to be automatically generated and made available immediately as part of the core release process.

A script for this is written. While it needs more work, it was used as a test to build 3.9.1 packages, which are doubling as 4.0-alpha testing packages. This does not require changes in core.

5. Localized packages should only be used for initial downloads from WordPress.org. Instead, language packs should be transparently used for updates.

This is ready. A flag needs to simply be flipped in the API.

Ongoing problems to solve:

  • I have a proposal to type up for how to handle readmes, license files, etc., in both core and plugins. Requires no core changes.
  • No one has picked up the plan to limit the code modifications still done in some locales. This may end up being a July project for me.
  • The relevant APIs we need in core were deployed to WordPress.org. Also, the plugin and theme directories are fully internationalized; we need to get those strings to translators and shoehorn them onto existing international sites.

#4-0, #i18n

Internationalization goals for 4.0

Every few releases we’ve made a major push for improved internationalization. In 3.7, that was laying the groundwork for plugin and theme language packs (more on that soon). In 3.4, it was reducing all of the customizations that many localization teams needed to make. (There’s a page on that here.) In 4.0, I’d like to close the loop on a lot of this. Here’s what I’d like to accomplish, and I’ll need a lot of help.

  1. The first step installing WordPress should be to choose a language. The rest of the install process would then be in that language.
  2. You should be able to choose/switch a language from the general settings screen, after which the language pack should be downloaded.
  3. You should be able to search from the dashboard for plugins and themes that are available in your language.
  4. All localized packages should be able to be automatically generated and made available immediately as part of the core release process.
  5. Localized packages should only be used for initial downloads from WordPress.org. Instead, language packs should be transparently used for updates.

That’s the what. The how poses extensive challenges.

1) Choosing a language when installing WordPress. The rest of the install process would then be in that language.

osx-language

Want.

Split workflows: One issue that came up in #26879 (“friendlier welcome when installing WordPress”) is that setup-config.php is often not the initial entry point. A user installing WordPress through a hosting control panel is likely to be dumped onto the install screen (if not the dashboard directly).

Thus, we need to keep in mind that either screen might be the “intro” screen. This introduces technical challenges: If the user’s first step is setup-config.php, we don’t actually have WordPress fully loaded at this point, which makes actually installing a language pack more difficult. The install step has WordPress loaded in full, just without database tables. We should look into making setup-config.php load “all” of WordPress to make these environments easier to code.

Different approaches: There are two different ways to approach this: download language files immediately upon selection, or bundle barebones language files of the install screens for all supported languages. The latter is not the easiest thing to do (due to error messages and such, strings can come from all over) and also adds a delay to the adoption of new languages. The former would mean that we would want to pull a languages list from WordPress.org. (If we cannot reach WordPress.org, we would have no way of downloading the complete language pack, so this is not a big deal.) It’s still a bit of a challenge due to the split workflows problem.

Recommending a particular language: WordPress.org already recommends a language based on the browser’s settings (Accept-Language header) as well as using a more rough lookup based on IP address blocks. (HTML5 location could be used, but unless we’re also going to use that to set the user’s timezone, it seems excessive for a “choose your language” for the moment, especially since location is not as preferred anyway.)

Both would require the client to hit WordPress.org via JavaScript, versus a server HTTP request. We can do a server-side HTTP request to generate the list, then a client-side request to float recommended ones to the top. It’s possible to do it in two steps: the Accept-Language mapping can be done locally, while the IP-to-location table on WordPress.org has 2.9 million entries and would require a round-trip. Of course, if a user has downloaded a localized package directly from a local WordPress.org site, that would be the top recommendation.

Note that by language or locale we also must consider other translation variants, as there may be a language like Portuguese available in multiple locales (Brazil, Portugal) and further broken down by formal and informal variants. Each of these would be listed as their own “language.” c.f. #28303

2) You should be able to choose/switch a language from the general settings screen, after which the language pack should be downloaded.

WordPress MU had this feature and it still exists in multisite (though it’s pretty broken in terms of how it handles locales, #13069, #15677, #19760). This however only worked for already installed locales.

For single-site, the available languages should be fetched from WordPress.org and cached in a transient. (#13069 suggests using the GlotPress locale list, but as indicated above, we’d want to handle newly added or updated languages. This can then be displayed in a dropdown on the General Settings page. Upon selection and “Save Changes,” the language pack would be downloaded and enabled. We would likely use the oddly named “WPLANG” option since it’s what MU adopted and uses at both the site and network levels.

If we can’t reach WordPress.org, or if a filter is applied, then only installed languages will be listed. (An untrusted multisite network will likely have a default filter in place, to match existing behavior.) We should try to cache failures to .org (generally — not just here) so things aren’t terribly slow when developing a site without internet. We could possibly lazy-fetch the list only when the user expands the dropdown (or clicks a “Change” link or something).

Note this does not address two situations in particular: user-specific languages, or using the dashboard in English. These will be easier to do thanks to improvements in 4.0, but there still exist a number of edge cases where persistent translations can “bleed” into other situations. Examples include a comment moderation email being emailed to an English speaker but sent in the language of the logged-in commenter; or a string stored in the database like image metadata. When using a plugin, these are “edge cases.” When core adopts them, these become “bugs.” We will need to introduce a locale-switching method not unlike switch_to_blog() and also identify and work around any “persistence” issues. Not for 4.0, though we can get started on the groundwork.

3) You should be able to search from the dashboard for plugins and themes that are available in your language.

With language packs (more on that soon), plugins and themes will be able to be translated into any language. The goal would then be to have “localized” plugin and theme directories on the translated WordPress.org sites, listing just the plugins or themes available in their language. Note that even a plugin or theme without strings have a name and description that can be translated.

The plugin and theme install screens in the dashboard should similarly gain this ability. In all cases, there would be an option to expand the search to plugins/themes available in any language, of course — along with the invitation to translate them.

We will need to figure out how to make readme files translatable, which get parsed for the plugin directory (and eventually themes will gain these as well, possibly as part of these initiatives).

4) All localized packages should be able to be automatically generated and made available immediately as part of the core release process.

A localized package should merely consist of core WordPress plus a few translated files (the readme, the license, and the sample config file) and the PO/MO files. No other alterations should occur, which means translation teams will only need to translate, and builds can be automatically generated and made available immediately as part of the core release process.

Local modifications. Before 3.4, every install needed to make modifications to WordPress as part of their localized package, whether by actually replacing core files or using a {$locale}.php file. By my audit, this now applies to only 14 locales.

At the very least, we can allow the current system to work as-is for these locales. Ideally, though, we address the specific concerns of each locale and bring as many as possible into the fold. They include:

  • Some CSS modifications we can incorporate into core.
  • Workarounds for declension issues in a few locales, specifically regarding the names of months. #11226, also #21139, #24899, #22225
  • Adding major locale-specific oEmbed providers. #19980
  • Transliteration such as Romanization for Serbian. We can handle this by having two variants of Serbian, the way we have formal and informal European Portuguese. (But we’d automate these language packs, rather than forcing translations to happen twice.)
  • Significant changes in Uyghur (#19950), Farsi, Chinese, and Japanese (including the multibyte patch plugin).

Additional concerns:

License. Many locales also have an unofficial translation of the license. We should be able to collect any of these in a repository and include it automatically in a package. (Note that link is an explainer; no GPL version 2 translations are available from that page.)

Readme. Some locales directly translate readme.html. Others add a second file and use one of two forms: the translated word “readme” (example: leame.html would be Spanish) or using a suffix (“readme-ja.html” for Japanese). We should standardize this somehow. Technically the readme changes with each version due to the version bump, but we can automate that. Bigger readme changes are fairly rare, but we’ll still need to figure out how to have these translated and tracked.

wp-config-sample.php. This one is especially interesting. If a user has chosen a language on install, we don’t need the readme or license but we do need this file, as setup-config.php will display its contents in a textarea if we’re unable to write the file. We can store this file as a single translated string in a PO/MO file (in some automated fashion) or as part of the API response. We will also need some kind of way to translate and track this file to be included in localized packages that are downloaded.

5) Localized packages should only be used for initial downloads from WordPress.org. Instead, language packs should be transparently used for updates.

The WordPress.org API is set up to return a series of update “offers” — one of each type (“latest” a.k.a. reinstall, “update” and “autoupdate”), in both the site’s language and in English. This results in awkward double-upgrades even when no strings have been changed (updating first to English, then to the locale when it is available for that version) and is a lame experience. While auto-generating localized packages immediately will help, there’s no reason to actually use localized packages for updates. Any locale without local modifications can be set up as a language pack now.

As of 3.7, WordPress core supports receiving instructions for language packs. We’d simply stop issuing localized package offers, start issuing language pack offers, and rip out some code on update-core.php that has been handling the multiple-offer dance. This is the easiest of the five 4.0 tasks but is dependent on the very complex fourth task.

Next steps.

Here’s an overall outline of the things we need to do. I’ll be creating relevant core and meta Trac tickets in the coming days, and I’ve also referenced a few related tickets I was able to find.

Biggest problems to solve:

  • Figure out how to handle readmes, licenses, and wp-config-sample.php. This includes plugin (and theme) readmes as well. Remember we don’t want plugin developers to need to be involved in any way; and also that updates to these files will need to be handled elegantly (such that we have both ‘stable’ and ‘development’ tracks).
  • Eliminate all code modifications across all locales (as much as possible). Related, #20974.

WordPress.org/API work:

  • Create an API for core to use to fetch available locales for download.
  • Create an API for core to use to recommend a locale based on browser settings/location.
  • Auto-generate localized packages and core language packs.
  • Serve core language packs for updates, not localized packages. Related: #23113, #27164, #26914, #27752.
  • Mirror the plugin and theme directories to locale sites, with full translations (including readme data) and with selective searching.

Core work:

  • Add the ability to install a new language pack on demand.
  • Add a “switch to language” method, for languages that are installed. #26511
  • Add a screen that allows a locale to be chosen. May need to change how setup-config.php bootstraps WordPress.
  • Add a language chooser to the General Settings screen. #15677
  • Support searching for plugins/themes in a particular language, as well as leveraging translated data fields that come through from the API response. This mostly just means sending back the site’s language to WordPress.org. It also requires UI to reveal results from any languages. We could possibly filter/hide/show on the client side, if results were not paginated.
  • Take a machete to update-core.php’s localized build handling once everything else is done. Related: #25712, #28199.

#4-0, #i18n

Team Update i18n Our first two week cycle…

Team Update: i18n

Our first two-week cycle comes to a close. Very successful, with more than 20 tickets outright closed, and more in the pipeline or nearing completion. Some highlights since last week’s update:

  • Fixed localization issues when a plugin calls wp_enqueue_script() too early. #19959, #11526
  • Fixed ajax calls for IDN domains (broken in IE and Opera). We partnered with Ryan on this one, to introduce a new ‘relative’ scheme for the URL functions. #18952
  • Hebrew letter nun no longer breaks search results. (Good UTF-8 fix.) #19033
  • We finished off #19852 and, again, #18180, #19924, and #19600
  • Commas (when used to separate tags) were internationalized. A very helpful fix for a number of locales. #7897
  • Finalized RTL styling changes in #19598
  • Made the TinyMCE spellchecker dropdown localizable #19962
  • The document for translators is current (view it here)
  • Awaiting feedback from translation teams in #19980, #8759, and #19601
  • Ryan and I want to revisit #19599

Our next cycle will start in a few days. It will focus on the raw implementation of language packs. Dion will be spearheading the upgrade/install bits while I continue to work on infrastructure as well as the contents of said packs. For now, Sergey is looking at transliteration and character conversion, while I will work on #19597 and #15858.

#3-4, #i18n, #team-update