Dev Chat Summary: May 16th (4.9.6 week 7)

This post summarizes the dev chat meeting from May 16th (agenda, Slack archive).

4.9.6 planning

Updates from focus leads and component maintainers

  • The REST API team has an update on their progress on closing the gap in the Gutenberg merge proposal milestone as well as notes from their office hours last week. Join them Thursdays at 17:00 UTC if you’d like to chat through any questions you have, they especially want input from long-time contributors and leads on the register_meta changes (see #38323).
  • The JavaScript I18N team posted notes from their meeting last week and discussion around extracting strings from JavaScript and loading the actual translations in a way that makes sense. Join them on Tuesday, May 29th at 15:00 UTC to further discuss how to handle translation loading.
  • @omarreiss posted about adding JavaScript build step and folder reorganization as the first part on preparing for WordPress’ JavaScript future. Please give that a review and provide feedback or help as you have time/interest.
  • @rianrietveld posted about Pair programming on the contributors day to join accessibility experts and Gutenberg developers and work to close the 11 remaining issues in the accessibility merge proposal milestone for Gutenberg. If you are a developer that knows your way around the Gutenberg code and are going to WordCamp Europe in Belgrade, then please ping @rianrietveld… thanks!

General announcements

  • @danieltj working on merge proposal for Dark Mode (see #41928), currently working on review for the WP Coding Standards and an a11y review
  • @presskopp looking for UI feedback on #35288, especially if “Search Engine Visibility” be reflected under Settings >> Privacy

Next meeting

The next meeting will take place on May 23, 2018 at 20:00 UTC / May 23, 2018 at 20:00 UTC in the #core Slack channel. Please feel free to drop in with any updates or questions. If you have items to discuss but cannot make the meeting, please leave a comment on this post so that we can take them into account.

#4-9-6, #core, #core-i18n, #core-js, #core-restapi, #dev-chat, #gdpr-compliance, #gutenberg, #summary

JavaScript Internationalization: The Missing Pieces

Back in 2016, work started on building a proper JavaScript internationalization API and the tooling to support it throughout WordPress core and WordPress.org. Many ideas and patches were being discussed. A summary of that can be found in this blog post. With Gutenberg on the rise, JavaScript I18N is more urgent than ever. WordPress needs a robust solution for that, and some things have already been built. Let’s have a look at where we currently stand.

Status Quo

Right now, Gutenberg is using a custom built JS I18N library that is similar to the one originally proposed in 2016 as part of #20491. It lies on top of a library called Jed which bring Gettext functionality to JavaScript. This means developers can use the same __() function as in PHP and therefore don’t have to learn anything new. WordPress can take it from there.

Unfortunately, WordPress doesn’t yet support JS I18N library. Gutenberg (or any other plugin that uses said library, really) has to jump through quite some hoops to actually localize their JavaScript:

  1. Scan JavaScript files to extract internationalization functions and create a POT file using tools like babel-plugin-makepotPoedit or xgettext-js.
  2. Use that POT file to write the exact same internationalization functions in a “fake” PHP file that can be scanned by the WordPress.org translation platform. This will result in PO and MO files containing all of your plugin’s translations.
  3. Figure out a way to load these translations and make them available to your JavaScript using wp_add_inline_script(). Ideally you’d only load the ones needed by that specific script as you don’t want to print thousands of strings in that inline JS when you only need a few of them.

An example of that process can be found in my demo Gutenberg I18N Block plugin.

At this point you might want to go back to good old wp_localize_script() and simply keep using that for internationalization purposes. I don’t blame you.

However, this complicated process is only needed because the work on JavaScript internationalization is far from done yet. Gutenberg made it quite obvious where things need to be improved.

What’s Missing

Scanning JavaScript files for internationalization functions

First and foremost, the WordPress.org translation platform needs to be able to scan JavaScript files for internationalization functions in addition to just the PHP files. However, that’s not as straightforward as it sounds.

The platform uses a script called makepot.php to scan PHP files all across the WordPress.org ecosystem, i.e. core, meta, and all default themes. In addition to regular Gettext function calls it also scans plugin and theme file headers. Being included in many other libraries, makepot is a widely used tool. Most recently, its functionality was ported to a WP-CLI command to make string extraction easier to use.

On the other side we have babel-plugin-makepot, a tool written in JavaScript to scan JavaScript files. With the ECMAScript standard evolving so quickly, it is natural to write such a tool in the same language. However, it’s not a requirement, as this pull request for said WP-CLI command demonstrates. This opens some questions:

Can we simply use that Babel plugin on WordPress.org? What happens to makepot.php? What are the implications for all the developers out there not hosting their projects on WordPress.org? Not everyone uses the Babel transpiler, and certainly not everyone wants to use two separate tools just to extract some internationalization functions.

Loading only specific set of translations

All translations for a plugin or theme are stored in one single PO / MO file per locale. Loading these translations is a slow process.  We’ve made some improvements in that regard over the years, for example by introducing just-in-time loading of translations in WordPress 4.6.

However, if you only need a handful of translations for a single script in your plugin, it does not make sense to load the entire MO file which can be dozens of kilobytes in size. There’s currently no way to load only a specific set of translations in WordPress. This is something that came up in Gutenberg before, see issue 6015.

Binary MO files don’t make sense in a JavaScript context anyway. Lucky for us, GlotPress—the software that powers translate.wordpress.org—has been able to export translations in a Jed-compatible JSON format since 2016. We just need to use that to export a JSON file for all strings extracted from JavaScript files

So in theory the WordPress.org translation platform could export PO and MO files as usual for strings extracted from PHP files, and a JSON file for all strings coming from JavaScript files. This would be already a huge improvement. But can we take this even further?

Option A

Use a different text domain per JavaScript module. Export a JSON file per text domain. This is appealing, but has to be ruled out quickly: the text domain is not known to GlotPress and is not stored in the database or anything.

Option B

GlotPress doesn’t know about the text domain, but it does know a string’s source file. What if it would export one JSON file per source file it has scanned? This way WordPress has full control over the translations and one could specify which JSON files need to be loaded for a specific module.

The big drawback here: a single module might consist of dozens of source files. Having one JSON file for each of those is not going to scale well.

The built JavaScript file can’t be scanned either, because tools like UglifyJS rename functions and strip out comments.

Option C

Don’t do anything fancy. Sticking with a JSON file already guarantees that a plugin doesn’t unnecessarily load all the translations needed just in PHP. So the file size is definitely smaller. Still, this file alone can be very large for an application like Gutenberg.

Option D

Keep one single JSON file for all translations, but use some PHP code to only ever pass the strings to a module / script handle that it actually needs. However, there’s probably no real benefit in doing so.

Easily load translations

Up until WordPress 4.6, developers needed to use load_plugin_textdomain or load_theme_textdomain() to make sure translations are properly loaded. Now, you only need to use the various translation functions and the rest just works. The only requirement is that your translation files reside in wp-content/languages. This is usually the case when your project is hosted on WordPress.org.

We should aim for a similar experience for JavaScript translations as well. While just-in-time loading of translation files via HTTP isn’t really possible due to the asynchronous nature of JavaScript, WordPress should still make it as easy as possible.

Imagine having a plugin foo-plugin and you’re enqueuing your JavaScript like this:

wp_enqueue_script( 'foo-script', plugins_url( '/foo-script.js' , __FILE__ ) );

Ideally, all you’d need to do to translate it is calling a function like load_js_textdomain( 'foo-plugin' ).  WordPress would then do all the heavy lifting.

However, other options might exist, and this solution would need to be tested in the wild with bigger projects like Gutenberg.

Discussion

Bringing a JavaScript I18N API to WordPress will have a huge impact. We need to make sure we end up with a solid plan that works for as many plugins and themes as possible.

Ideally, we hold a separate JS I18N meeting with all the teams primarily involved: #core-i18n, #core-js, #core-editor, #meta-i18n and #cli. Everyone is welcome to attend though 🎉

I suggest the following date for such a meeting: Tuesday, May 8 15:00 UTC. Of course I’m open for other suggestions. The Slack channel would be #core-i18n.

At this meeting we can discuss the missing pieces outlined in this post and the overall next steps for JavaScript I18N in WordPress.

If you have any questions or concerns about this post or the overall topic, please leave a comment below.

+make.wordpress.org/polyglots

Dev Chat Summary: October 12 (4.7 week 8)

This post summarizes the dev chat meeting from October 12th (agenda, Slack archive).

Reminders

Feature Proposals for 4.7

  • REST API: Content API (@kadamwhite)
    • REST API Team Update
    • Content API endpoints have been stable for some time and now have the missing functionality established at the start of this development cycle
    • We pitched merging in the OAuth 1 server alongside those content endpoints, to simplify the process of authenticating from remote applications
    • The auth method in core right now is cookie/nonce-based, so it’s available to any code running within the same domain as the main WordPress install and leverages the existing login cookie
    • plugins and themes that utilize JS to enhance the editing experience within their admin UI have full read/write access, restricted to the capabilities of the authenticated user
    • External applications, e.g. a mobile app, desktop app or external server, would have read-only access
    • readers, aggregators, etc are possible without authentication, and more complex read-write apps can be written by installing one of the available authentication plugins, of which OAuth is but one
    • Reference for options and when/how to use
    • 90 contributors for the rest-api plugin
    • automated test coverage currently at 90.44%
    • 37 more open tickets
    • weekly meeting on Monday at 1400UTC in #core-restapi
  • Discovering and installing themes in the customizer (@celloexpressions)
    • Related Make/Design and Make/Flow posts, but best to post all feedback to the Make/Core feature proposal
    • Working through the accessibility feedback, but it’s a bit unclear what’s specific to this feature versus the customizer in general, and which issues are also present in the admin theme installer
    • Haven’t heard anything from polyglots, security (unlikely to have any issues), or docs
    • There is one known i18n concern where we’ll need a compromise until we have JS i18n in core
    • @celloexpressions to do a final iteration to pick up any remaining feedback on Thursday, then pass to @westonruter for the code review process
  • Custom CSS with live previews (@celloexpressions)
    • seems to have decent support as a feature, with the biggest question in the comments being how to store and output the CSS
    • CSS is output from the db via a style tag, use of `style` tag is key for targeting for `postMessage` live preview
    • Post-based storage allows for revisions built in (ensuring that there are hooks for a plugin that wants to write a file would alleviate many concerns)
    • CSS is theme-specific in the proposal for this, with a post object for each theme that has CSS
    • Scheduling a meeting to go through these concerns and make a decision, coordination to occur in #core-customize
  • Customize Changesets (née Transactions) (@westonruter)
    • Technical Post (audience is developers who have extended the customizer significantly, as there may be impacts to any heavy customizer integrations since changesets touches some very low-level stuff in the customizer)
    • Users will be able to bookmark a customizer session since the changeset UUID will be part of the URL
    • Theme switches will no longer result in changes being lost (no more AYS dialog)
    • a few other open questions and issues that have been noted

Components

#4-7, #core, #dev-chat, #summary

Dev Chat Summary: September 28 (4.7 week 6)

This post summarizes the dev chat meeting from September 21st (agenda, Slack archive).

Reminders

  • Schedule: We are 3 weeks from the final chance to merge in major features. This includes Twenty Seventeen.
  • Tickets: There are currently 196 tickets in the 4.7 milestone. This is 14 more than last week. In just 6 short weeks, this needs to be zero. For any tickets you’ve moved into the milestone, please make sure these are active tickets, with some kind of activity in the last 10 days.
  • Bug Scrubs: We’re looking for people to help run a bug scrub, please reach out to @jorbin if you have interest (details here). Bug scrubs this week plus one on Monday and one on Wednesday next week at yet to be scheduled times.

Components & Features

  • Twenty Seventeen (@davidakennedy, @melchoyce)
    • Latest update
    • Add multi-panel feature to pages through add_theme_support (#37974) & Enable Video Headers in Custom Headers (#38172) need eyes and help the most
    • Additional i18n eyes on Better support for non-latin font fallbacks especially designers who use non-latin alphabets natively to hear suggestions for non-latin font stacks that would look good in the theme
    • Next meeting Friday at 18:00UTC (theme-focused), Tuesday (feature-focused)
  • Media (@mikeschroder, @joemcgill)
    • Latest update
    • Unexpected change to media title behavior in WP 4.6.1 (#37989) – Looks like @sergey added a patch that fixes the remaining issues with some UTF-8 characters. Should be committed soon.
    • Media search doesn’t include file name (#22744) – The current implementation is trampling any preexisting JOINs. Should have a patch a new patch ready to test soon.
    • Also looking at pursuing additional media organization improvements through a feature plugin, details still need discussion, @karmatosed on board to help with design
    • Next meeting Friday at 17:00 UTC
  • Customize (@westonruter, @celloexpressions)
    • Latest update
    • Primary commit for Harden panel/section UI code by removing contents from being logically nested (read: goodbye margin-top hacks) (#34391) is in, and a dev note is scheduled to be published after today’s dev chat
      • Some major changes here, so we need plugin and theme authors to test
    • Received design feedback on A New Experience for Discovering, Installing, and Previewing Themes in the Customizer (#37661) and working on making those revisions by the end of this week and planning to publish a feature proposal on Friday
      • Need to discuss themes again during tomorrow’s #design meeting for final approval before the changes are made
    • Need attention on Provide a better gateway for code-based theme customizations with the Customizer (#35395)
      • Discussion of whether this direction is appropriate lead to tentative consensus that this is likely appropriate for core
      • Next steps will be to loop @folletto in to improve the design and polish up the patch
      • Big other block discussed: sanitizing and validating the CSS & most appropriate corresponding capability
        • Currently rudimentary validation in the patch for balanced braces and comments. Need improvement if relying on it heavily, but it provides instant user feedback
        • Capability solution needs to work for multisite if at all possible, since that’s a primary use case
        • Discussion to continue on the Trac ticket and/or #core-customize
  • i18n (@swissspidy)
    • Feedback/help on Introduce a locale-switching function (#26511) would be appreciated
      • The problem is that labels of custom post types and taxonomies are only evaluated once, so switching locales wouldn’t properly translate those.
      • There’s a proposed fix for built-in types and taxonomies, but we prefer a better solution that works for all of these.
  • Editor (@azaozz, @iseulde)
    • Would like to help with a survey (scratchpad/draft). Need to start gathering user usage stats, should be opt-in, start with a plugin first, and release the aggregated data
    • Weekly data tracking (back-end) meeting Wednesday at 1900 UTC
  • HTTPS (@johnbillion)
  • REST API (@krogsgard, @kadamwhite )
    • Latest update
    • Whether the API should follow core behavior and save a revision every time a post is updated
      • Right now every update to a post creates a revision and can be a bit painful for some clients, so: 1) should that always happen? 2) should we have the ability to turn it off?
      • Decided on: 1) Yes.  2) The ability to use auto-drafts like in core makes sense, but doesn’t need to block merge.
    • How to handle image permissions, specifically for the case where an image is attached (uploaded) to a private post and then featured in a public post
      • Specifically, if I upload an attachment to a private post, its visibility is governed by that post, so it too is private but, in wp-admin I can add it as featured media to another public post. When that public post is queried: what happens!?
      • @joemcgill summary: I happen to think it’s an oversight in WordPress that we allow an image attached to a private post to be set as the featured image of another post (and by an author without permission to view the private parent post). We should probably either close this loophole or detach the attachment from the private post whenever it’s set as a featured image on another post.
      • @kadamwhite to document decision, @rmccue @joemcgill @helen et al will identify core tickets that should be opened.
    • Whether (and how) to expose edit locks through the API
      • Main thing here is whether this is a blocker? Decision: edit locks are great, but doesn’t need to block merge.
    • Next bug scrub is Thursday 1400 UTC; next team meeting is 1400 UTC on Monday, October 3rd

#4-7, #core, #core-editor, #core-http, #core-i18n, #core-media, #core-restapi, #dev-chat, #summary, #twenty-seventeen