Rosetta Forum Upgrades In Progress

The following forums have been upgraded to bbPress 2 and redirected to their new locations:

  • pt.forums.wordpress.org to pt.wordpress.org/support
  • id.forums.wordpress.org to id.wordpress.org/support
  • mya.forums.wordpress.org to mya.wordpress.org/support
  • de.forums.wordpress.org to de.wordpress.org/support
  • zh-cn.forums.wordpress.org to cn.wordpress.org/support
  • bs.forums.wordpress.org to bs.wordpress.org/support
  • th.forums.wordpress.org to th.wordpress.org/support
  • ja.forums.wordpress.org to ja.wordpress.org/support
  • ru.forums.wordpress.org to ru.wordpress.org/support
  • ug.forums.wordpress.org to ug.wordpress.org/support
  • bg.forums.wordpress.org to bg.wordpress.org/support
  • sr.forums.wordpress.org to sr.wordpress.org/support

I’ll continue to update this list as I upgrade forums.

For larger forums, I will set redirects in place about thirty minutes before the end of a upgrade run to avoid having to backfill posted content. This will be noticeable to users when they are forwarded to the in-progress new forum location.

Moderator roles are being transferred from the old forum to the new one. Ported metadata includes user topic and topic tag subscriptions, as well as the topic resolution and WordPress version.

See https://meta.trac.wordpress.org/ticket/1544 for known issues and bug reports.

Plugin Directory v3: Next Steps

The Past

The meta team kicked off the new version of the plugin directory at the end of February. Some of the initial desired improvements included: moving to WordPress, open-sourcing the codebase, improving search, feature parity, and making the plugin review process more scalable.

One aspect that was looked at were taxonomies. An outline of the challenges was posted on make/plugins and followed up by a proposal on how these can be met. Another aspect was the design itself, with prototypes published (#1, #2) and feedback gathered, before they were converted into a theme.

Finally, I spoke at WordCamp Europe about the work done so far, providing more details about some of the decisions made, and asking the community for feedback on the prototype during an open beta phase.

The Present (and Feedback)

We’ve received a lot of feedback about the plugin details page and some improvements that could be made. A number of people mentioned that too many “read more” links can make it hard for users to find relevant information, blending the sections together, and that content may not be indexed properly by search engines. Others were concerned about the effects of partially hiding the plugin description, the loss google site links for sections, and how the accordion behavior may not be ideal on smaller touch devices.

Another part of the new directory that received a lot of feedback is the simplified plugin cards on the front-page and search results. Many reported that they missed some of the specific data, present in the old directory; namely, active install count, last updated date, and author. Without this information they’d have to view the details of each plugin to get more information and make an informed decision.

In terms of search, there are a number of improvements that still need to be made. Various queries includes subjectively irrelevant results and often do not surface a plugin despite searching an exact match of its name. One suggestion to mitigate that we’ll explore, is to emphasize matches in tags and matches in a plugin’s description.

A future enhancement we’d like to add that has been discussed quite a bit is showing categories on the front page instead of (or in addition to) curated sections to show the breath and depth of the directory. An additional section could also include “trending” plugins, with an algorithm backing it.

Plugin authors really like the dashboard, but there’s no reason that it can’t be entirely public, instead of housed within wp-admin. If a plugin author is logged in, this view would show options for them. For other visitors, an overview of every plugin from a plugin author could be helpful.

One of the decisions we made early on was to switch from freeform tags to categories. However, this switch limits the ability for plugin authors to change tags at-will and doesn’t allow the same flexibility as freeform tags, requiring editorial decisions on category names. Conversely, unlimited freeform tags have been abused and are not helpful to users. One idea is to bring back freeform tags, but limit the number show (and indexed) to five.

In bringing parts of the readme.txt file into the UI, we attempted to make it easier for developers to manage their plugins without making a commit. However, there are other ways we could solve this. For example, compatibility could be crowdsourced automatically from installed versions of plugins and WordPress. The developer-provided value would become less interesting in this case. Knowing if the user thinks it works is more interesting than knowing if the developer thinks it works. In other cases, we could also allow editing of the readme.txt file within a web UI, automatically committing these changes to SVN for the plugin author.

The Future

Going forward we’d like to discuss community feedback and review some of the design decisions, as well as plan how the interfaces for plugin reviews and plugin authors should look like and behave. An important first step will be to reinstate weekly meetings, starting next week, and get back into a rhythm of setting milestones, attaching tickets to them, and complete them in due time. The next Plugin Directory meeting will be August 4, 2016 at 00:00 UTC.

#plugin-directory

Meta Environment Now Supports Contributing with Git

The Meta Environment has always managed its core functionality in Git, but up until now, it used Subversion to check out the actual files that make up each Meta website. New contributors are much more likely to be familiar with Git than SVN, though. Requiring them to learn SVN in addition to all of  our other tools and processes introduced an unnecessary burden.

The goal of the Environment is to help make contributing as easy as possible, so it makes sense to switch to Git instead. As of today, that switch is complete, and new installations will use Git automatically.

The two exceptions are: 1) Any plugin or theme that doesn’t have an official Git mirror; 2) the entire translate.wordpress.org site, since it’s in the process of migrating to GlotPress 2. In those cases, SVN is still used.

If you have an existing installation, it will continue to use Subversion until you git pull the latest copy of the Environment, and run vagrant provision. During that provision, each site’s public_html folder will be backed up to public_html-old-svn-backup, and a new Git-based public_html folder will be provisioned. If you’re working on any unfinished patches, just copy the modified files from the old directory to the new one.

If you’d like to learn more about contributing with Git, check out this guide from the Meta Handbook:

Contributing With Git

No DevHub chat this week

We won’t be having a DevHub chat this week. I will be unavailable to run the chat and it sounds like @coffee2code won’t be available either. Chats are normally held on Tuesdays at 18:00 UTC in the #meta-devhub channel.

We’ll pick up scrubbing the remaining tickets in the DevHub component report next week at the usual time and place.

Forum Upgrade Update

Hello again! It is time for a progress update on the forum upgrade project.

Plugins
Meta has had volunteers working on converting various forum-related plugins for the past year or so, and we’re now at the point where those plugins are being committed. When doing conversions, I’ve tried to concentrate on the spirit of the code, rather than having an exact match. We can’t adhere strictly to the filters and actions that were previously available in bbPress 1, as these do not always map well to a bbPress 2 code path. Additionally, in some cases bbPress 2 has a better hook available, so we can avoid needing to code conditionals into the new plugins by choosing a more precise hook.

I’m currently working through and committing the backlog of converted plugins. These are being updated to use common meta.trac.wordpress.org code structures and the ‘wporg’ text domain for translations. Generally speaking, simple filtering plugins are being committed as classes, while more complex plugins that require an admin or user interface use namespaces. All of these plugins are open source (GPL2+), and can be used in other bbPress installations. All are also easily testable with a local installation of bbPress 2 or the meta environment.

Some plugins which are integral to all of the forums are:

  • Topic Resolution — this allows topics to be set to ‘unresolved’, ‘resolved’, and ‘not a support question’ (formerly known as Support Forums)
  • Version Dropdown — this allows the user to set the WordPress version relevant to a given topic
  • Subscribe to Tags — this allows the user to subscribe to a given tag and will play an important part in the support forum for plugins and themes
  • User Moderation — this sets a user’s posts to be automatically moderated


Because all of the above plugins actually create and store data, rather than just filtering it, it is important that they be ported prior to forum upgrade. I’d rather spend the time to get all the moving pieces in place rather than convert a forum and then discover later on that I’ve left something important behind, like tag subscriptions. It is far easier to run the complete conversion once rather than trying to backfill unported data from the old tables, especially when it involves user metadata.

Props go to everyone who has contributed code to the plugin overhauls, including @nullbyte, @justingreerbbl, @Clorith, @coffee2code, @Kenshino, @SergeyBiryukov, and @netweb.

URL redirects
I’ve written a bbPress 1 plugin to handle requests to the original forum location; this will allow me to incrementally convert *.forums.wordpress.org to *.wordpress.org/support without needing to constantly update the nginx rewrite rules. Once all the forums have been converted, then the nginx rewrite rules can be updated once and plugin removed.

Conversion script
In my first pass, I took care of most of the topic meta and user options that were plugin related, but missed the tag subscriptions and user favorites. I’ve had to go back and code some parts of the conversion script to handle this data, as well as dealing with iterating over the usermeta table.

I’ll be committing the required plugins all week so that I can start running the conversion script. Plugin testing is welcome; please ping in Slack on #meta if you have any questions or suggestions regarding the ported plugins.

#forums

Plugin Directory v3 Open Beta

We’ve been working on the new, WordPress-based version of the Plugin Directory and it’s at a point where the meta team would like to open it up to the community to foster participation and receive feedback. You can find the latest iteration of it under the temporary URL:

https://wordpress.org/plugins-wp/

If you have commit access to a plugin in the existing directory and are logged in to your WordPress.org account, you also have access to the provisional developer interface.

Please keep in mind that it is by far not a finished product. There is still a lot of work to be done, mainly around front-end technology, search, and developer facing interfaces. Every decision that has been made so far is up for discussion, which is the purpose of this post. You can find a history of all previous discussions at #plugin-directory.

We’ve already received a good amount of feedback after my WCEU session on meta trac, the summary post of our last chat, and in blog posts throughout the community.

But we’re looking for more! 🙂 Please feel free to leave general feedback as comments on this post, and specific bug reports as new tickets on meta Trac (but make sure it doesn’t exist yet). The meta team will go through all feedback, derive a prioritized list of suggestions from it, and work it into the iteration schedule.

Thank you for your participation!

Forum Design Research

As some may have noticed there are efforts to improve the support forums on WordPress.org. In addition to the backend changes, this seems like a good time to make some design upgrades as well. To kick things off, I’ve done a bit of research into various forum designs in use online (similar to the web store explorations for the plugin directory).

If you’re interested in the forums, please take some time to review the research here and leave your feedback. What stands out from these examples as something you’d like to see implemented on WordPress.org? Which design feature is your favorite and why?

Feature comparison

Feature comparison

Below are some screenshots from each of the forums reviewed.

Non-bbPress Support Forums

bbPress Support Forums

Thank You

Thank you for your feedback! If there are other forums designs you’ve seen on the internet that should be reviewed, leave a comment here, along with any other feedback you have.

#design, #forums

DevHub Weekly Chats are restarting

Beginning Tuesday, July 12, 2016 18:00 UTC, we’ll be restarting weekly DevHub chats in the #meta-devhub channel on Slack.

For the uninitiated, the Developer Hub, or “DevHub”, largely serves as the new official source for developer documentation on WordPress.org. The DevHub team’s responsibilities cover the phpdoc-parser project, as well as infrastructure behind the Code Reference, Theme and Plugin and Developer handbooks, resources pages, and more.

Getting weekly chats restarted should allow us to gather more regularly as a team and get group feedback, as well as set direction for new features or improvements. See you on Tuesday!

Plugin Directory v3 Chat Summary (6/1)

This is a summary of the Plugin Directory chat from June 1. (Slack log)

Attendees: @dd32, @samuelsidler@mapk, @pento, @tellyworth, @obenland

Topics:

  • Review M4
    M4 had eight tickets assigned to it, four were fixed by the time of the meeting. #1575 had a patch for the non-open-sourced part that was committed a day later and fixed the ticket. There was some follow-up work that re-instated using the old directory as the data source but largely satisfied the requirements. #1579 was waiting for a commit. #1722 was moved into M5, waiting for a final commit later that week. #1692 was also moved into M5, waiting for more updates.
  • Plan M5
    After our prioritization pow wow the week before, M5 almost planned itself. #1720, #1573, #1578, and #1691 sat atop of the list and need most attention. #1742 was also added to the milestone, as a left-over task from #1575.

The next meeting is on Thursday June 9, 00:00 UTC.

#plugin-directory

WordPress.org Forums: User Meta

My self-set deadline for the successful conversion of three forums was last Friday. I did successfully import the pt.forums.wordpress.org forum by the deadline, but restricted the import set to only include content and tags. I did not update subscriptions and favorites because it was a test upgrade, but I will include those in a final scheduled upgrade. No one will lose their subscriptions or favorites during this move.

The import class that I am using needs to scale so that it works with the https://wordpress.org/support forum, which has over ten million posts. There are already some cracks showing when dealing with user roles across the forums because of the shared user tables which allow one user to have a role on multiple forums and blogs on the WordPress.org sites. In the import class, I use the existing bbPress queries as a basis for custom MySQL INSERT VALUES statements, which combine many small INSERTs into far fewer database writes. This has given me some insight into queries that will not perform well against the larger support forum dataset.

Favorites are stored solely in user metadata and keyed by forum in both bbPress 1 and bbPress 2. This is a fairly simple query against the usermeta table as it is fetched on a per-user basis.

bbPress 1 stores user subscriptions as taxonomy terms. As the importer class adds topics and their associated terms to the new site tables, it stores subscriptions as a temporary array of user ids in the postmeta for that topic. Once a forum is upgraded, I can page through the topic postmeta and add the appropriate subscriptions on a user-by-user basis.

In bbPress 2, subscriptions for forums are stored in the usermeta table and are fetched for a given topic using a FIND_IN_SET MySQL flag. We end up having to traverse all of the user subscriptions for a given forum in order to find the subscriptions for a single topic. This query is performed in bbPress 2 whenever new topic or reply notifications are sent. Given the size of the support forum, I cannot see this query scaling well for forum and topic subscriptions.

After a conversation with @jjj about optimizing this in bbPress, he opened a new issue to handle improving user subscription and favorite queries. I will continue to upgrade forums using the old style of favorite and subscription storage, but will also be working on that Trac ticket in order to allow subscriptions to work on large forum datasets.

How does this affect the conversion milestones?

I will still be converting a number of WordPress.org forums to their bbPress 2 equivalent, including their users’ current subscriptions and favorites. It takes the import class less than fifteen minutes to run on a forum with approximately ten thousand posts and the associated terms, with a lot of sleep()s thrown in “just in case”. User meta will add to that, but not much.

The upgrade process should run as follows:

  • Post to forum with a scheduled time and date for the upgrade. I will try to schedule upgrades for local off-hours.
  • Close forum to new posts when scheduled. It will still be searchable and available until the redirects are in place.
  • Run the upgrade script.
  • Confirm that redirects are in place for *.forums.wordpress.org to *.wordpress.org/support. At this point, the old forum will redirect users to the newly upgraded forum.

Larger forum imports will require backfilling the majority of the forum while it is still live, then doing a partial import of those topics and replies that were not backfilled. After that, the usermeta import can be staged for forum roles, favorites, and subscriptions.

July 1st is still the deadline for the plugin conversion; most of the overhead is now in working on plugins, as the import script runs fairly quickly and can do forum conversions “live”.

#forums