Dive Into Distributed Unit Tests

Recently, I have spent some time looking into using pre-built tools for distributed unit tests. What I found was that all of them were built to fetch the test results. What we need, for this project, is to have the results sent to us. At this point it seems like a custom WordPress solution will be the best route.

There was some discussion in #hosting-community about how the tests would be triggered. Some hosts have scripts that run nightly, so slipping tests into that process would make implementation easy. However, using a nightly solution would make it more difficult to determine exactly which commit caused an issue.

If we move forward with running tests on a per-commit basis, do hosts watch for commits and then report the results? Or do we ping hosts? It seems like pinging hosts would be ideal, but it might not be an option for all hosts. It would be really nice to get some feedback here, so your thoughts would be appreciated.

The first step is to start on the method used to report results. This will be a WordPress based application, with a REST API. Hosts will submit the results with the commit SHA to be aggregated. I’m going to dig into this over the next several weeks and will share a GitHub link when I have something up and running.

Call for Best Practices Documentation

One of the topics that has been largely discussed during #hosting-community meetings is coming up with a list of best practices for hosting companies to follow. Currently, we know that virtually every hosting company has a different server OS, default php.ini values, NGINX/Apache configs, etc. We’re looking to get some of those things aggregated to provide suggested defaults and best practices.

Here is some of the data we are looking for:

  • Any plugins or mu-plugins installed by default
  • Any modifications made to wp-config.php by default
  • PHP versions offered
  • Modules available in each PHP version
  • Default php.ini files
  • Default caching configs (Varnish, Memcache, Redis, etc)
  • Default web server configs (NGINX, Apache, etc)

Of course, we’re aware that some things (i.e. security configurations) may be sensitive. We don’t want those things.

This information will be compared, discussed, and picked apart to determine best practices to follow. Our goal is not to debate which hosting company is better. This should be an objective analysis by involved individuals from many hosting companies and potentially individuals not affiliated with any hosting company.

If you would like to contribute to the formation of WordPress hosting best practices and have information to offer, please contact @voldemortensen on Slack.

#documentation

Hosting Meeting Notes: February 8, 2017

Here’s the summary of the fourth meeting in #hosting-community on Wednesday, February 8, 2017 at 1600 UTC (Slack archive).

Promoting WordCamps

Distributed Unit Testing

  • @octalmage’s post for Make Hosting collecting thoughts around last week’s conversation needs review.

“Best Practices” Documentation

  • @voldemortensen is in the process of working up a draft for the Make Hosting blog.

Community Summit

Group Description

  • Discussion around the group description for the main make.wordpress.org page.
    • “We work together on hosting best practices and tools for the community.” was the frontrunner.

Feedback

Miss this meeting? Want to pitch in on one of the initiatives above? Please feel free to drop into the comments and share your thoughts.

Next Meeting

Our next meeting will be in #hosting-community on Wednesday, February 15, 2017 at 1600 UTC. If you can fit that into your schedule, we’d love it if you could attend.

#notes, #weekly-hosting-chat

Building a Machine-Readable List of WordPress Events

In #hosting-community, it was brought up that several hosts are interested in collecting data of local WordCamps and WordPress Meetups to promote attendance to customers. With many events happening around the world, the first task is to build a reliable source of data. The most important WordPress-related events are collected in two places: WordCamp Central and Meetup. Both sites have APIs to retrieve these lists.

WordCamp Central Events

The WordCamp Central API does not require an API key, so no setup is required. To pull the last 100 modified WordCamp events use the following request:

[user@localhost]$ curl -g 'https://central.wordcamp.org/wp-json/posts?type=wordcamp&filter[posts_per_page]=100'

The response is formatted as JSON. If filter[posts_per_page] isn’t added, you’ll get only the default amount of results (8, at the time of writing this). To get a more readable output, you can use jq and less on Linux/MacOS (you may need to install jq using your package manager):

[user@localhost]$ curl -g 'https://central.wordcamp.org/wp-json/posts?type=wordcamp&filter[posts_per_page]=100' | jq '.' | less

The top level element is a list. Each element in the list is an object with key/value pairs. The post_meta element is a list that contains objects for metadata including location, Twitter hashtag, start and end date (in epoch format), and other useful details of approved WordCamps. The location does not follow a strict format and is usually “city, country” or “city, state” if in the United States, but some WordCamps follow a slightly different format, and the Community Team has suggested an effort to standardize format and normalize the previous values. A good next step is to connect with the Community team to see if the Hosting team can help out here, since that will make it so that the country retrieved will always be accurate.

Meetup Events

The other source of events is on Meetup Their API requires a developer key, which can be created for free. To get a JSON formatted object of all scheduled WordPress Meetups, use:

[user@localhost]$ curl -g "https://api.meetup.com/2/events?member_id=72560962&key=$KEY&sign=true"

Be sure to change $KEY in the URL to your API key or it will not work. Again, you can use jq and less to parse the JSON:

[user@localhost]$ curl -g "https://api.meetup.com/2/events?member_id=72560962&key=$KEY&sign=true" | jq '.' | less

The member_id value 72560962 is the id for the WordPress Foundation Meetup account, and by specifying this, the API returns all WordPress official events, rather than the events for a single Meetup group.

The JSON that the Meetup API returns has a different schema than the WordCamp Central API JSON responses. For Meetup, the top level entity is an object, and the main key to look at is results. This contains a list of objects, and each object represents a WordPress Meetup. Each Meetup has a lot of metadata associated with it. Some important ones to know about are time, which is in epoch format * 1000, venue, which is an object with information about the venue, name, which is the name of the WordPress Meetup, and event_url, which is the website for the event. The venue‘s localized_country_name and city keys can be used together to replicate the WordCamp API’s location object.

Bringing it All Together

To get the full list of WordCamps and Meetups, a script is needed to pull from both sources and combine them into one list. An example script can be found on GitHub to see how a host might retrieve the data and combine the two sources. There is also an Official WordPress Events plugin, which creates this page, that can be used.

#promoting-wordcamps

Hosting Meeting Notes: February 1, 2017

Here is the summary of Make Hosting’s third meeting in #hosting-community on Wednesday, February 1, 2017 at 1600 UTC (Slack archive).

Updates

“Best Practices” Documentation

  • Some documentation has been sent in from a few sources.
    • Keep it coming!
  • @voldemortensen will try to get a call for docs up ASAP on the Make Hosting site.

Promoting WordCamps

  • @calebb and @smaffulli are working on documentation outlining the available APIs for the Make Hosting site.
    • Should be posted in the next few days.

Distributed Unit Tests

  • This project is still in need of a lead.
    • You’d chat with folks interested in working on the various parts to organize. Everyone will pitch in, but this needs a champion if we’re going to move forward.
    • C’mon, people… Doesn’t anybody want to do this?
  • Update from @octalmage re: defining a method for reporting back to .org.
    • Looked into using a pre-built tool (Jenkins, gocd) but both required custom plugins.
    • Those tools are built to grab results vs. having the reports submitted and have good integration with tools like Trac and GitHub.
    • Building something on our own would be a huge task.
  • The OpenStack project has a system with similar scope (RefStack) but it might be too hard to generalize.
    • The approach is that a client runs the tests, gets an XML output and feeds that into the RefStack server. (Example test results)
  • The tests need to be run on the exact server setup that users are going to be on.
    • This way, things like the ImageMagick issue that recently affected a few hosts will be caught early.
  • There was a long discussion around potential methods that could be used.
    • @octalmage agreed to collect the thoughts around this conversation for a future Make Hosting post.

Security

Feedback

Miss this meeting? Want to pitch in on one of the initiatives above? Please feel free to drop into the comments and share your thoughts.

Next Meeting

Our next meeting will be in #hosting-community on Wednesday, February 8, 2017 at 1600 UTC. If you can make that work with your schedule, we’d love to have you. Hope to see even more of you next week! 😃

#notes, #weekly-hosting-chat

Hosting Meeting Agenda: February 1, 2017

Another week, another weekly hosting meeting. Funny how that works, isn’t it? 😉

Here’s the agenda for the meeting on Wednesday, February 1, 2017 at 1600 UTC in #hosting-community on Slack:

  • “Best Practices” Documentation updates.
    • Progress?
    • Next steps.
  • Promoting WordCamps updates.
    • WordCamp & Meetup API documentation progress.
    • Next steps.
  • Distributed Unit Tests updates.
    • Finding a project lead.
    • .org reporting progress.
    • Next steps.
  • Open discussion.

Should you have any items to add to the agenda or specific comments related to the outline above — especially if you are unable to attend the meeting at its scheduled time — please leave a comment. Looking forward to seeing y’all virtually!

#agenda, #weekly-hosting-chat

Hosting Meeting Notes: January 25, 2017

Here’s the summary of the second meeting in #hosting-community on Wednesday, January 25, 2017 at 1600 UTC (Slack archive).

Updates

“Best Practices” Documentation

The current idea is to build a public handbook of the best practices that are generally used in hosting. This could be a variety of topics (and doesn’t need to be a single team).

  • Will probably include things like:
    • Server configuration requirements.
    • Recommendations.
    • Common scaling configurations.
    • Deployment.
  • Need reps from each host that wants to participate to send a sanitized copy of the following information to @voldemortensen (contact on Slack for E-Mail address if needed):
    • Default php.ini.
    • Default nginx, Varnish, httpd configs.
    • What modules you include with PHP.
    • PHP versions offered
    • Any changes you make to default wp-config.php.
    • Any mu-plugins you include and their purpose.
  • Data will be aggregated so it can be grokked and discussion can happen around the differences.
    • There’ll be a Make Hosting blog post on the aggregated data.
  • It was pointed out that some hosts might not want to share all of the above. Whatever they would like to share is welcome.
  • Request for additional volunteers.

Promoting WordCamps

  • WordCamps have an official list available, with a simple API call.
    • Returns JSON with all approved events.
  • Meetup.com has an API.
    • All WordPress meetups are in one single, easy to identify spot.
  • PoC code pulling from WordPress.org and Meetup.com.
    • https://github.com/squidboylan/wordpress-meetup-scraper
    • Builds a combined JSON containing events from both.
      • Easy to filter out recent events only, or by proximity.
  • To start, we need a post/page that details what information the current APIs offer, and how to reach them.
    • Where are these URLs?
    • What is returned in the JSON?
    • Link existing PoC tools.
    • @smaffulli is down to do this.
  • Scope is narrow enough that no additional people appear to be needed at this time.

Security Discussion

WordPress Security Czar @aaroncampbell wants to make sure the WordPress Security Team and hosts work together as much as possible.

  • WordPress has a security notifications list.
    • It’s only for established hosting companies.
    • It’s meant to give you a day or two to prepare for any security issues before a security release.
  • Also trying out a channel meant for hosts to be able to discuss security issues, potential fixes, etc.
  • Private message @aaroncampbell for more information on list and/or channel.
    • He’ll give a primer on what is expected/needed from you, what can’t be shared, etc.

Distributed Unit Tests

The idea is to end up with a common framework for hosts running WordPress tests on their platforms. Also, we’re looking to send reports on tests back to .org. This will allow for more complete testing on actual deployments.

  • @jorbin has some posts about it, with some scoping here:
    • https://daily.jorb.in/2016/05/31/wordpress-automated-testing-next-gen/
  • @danielbachhuber shared some tools he worked on for Pantheon in last week’s meeting notes:
    • https://make.wordpress.org/hosting/2017/01/18/hosting-meeting-notes-january-18-2017/#comment-10
  • The current biggest need is to have someone lead the cause.
    • The project can’t get started without someone to steer the ship.
      • It doesn’t need to be a host that coordinates it.
        • It’s helpful if someone who at least hosts their own installs.
        • At the very least they should have a testbed for the necessary parts.
    • @mikehansenme, @mikeschroder, @joshkoenig, and @octalmage are willing to champion this at their respective hosts.
    • @octalmage kindly offered to look into defining the method for reporting back to .org.

Feedback

Miss this meeting? Want to pitch in on one of the initiatives above? Please feel free to drop into the comments and share your thoughts.

Next Meeting

Our next meeting will be in #hosting-community on Wednesday, February 1, 2017 at 1600 UTC. If you can fit that into your schedule, great! We’d love it if you could attend. Hope to see even more of you next week! 😃

#notes, #weekly-hosting-chat

Hosting Meeting Agenda: January 25, 2017

Here’s the agenda for the weekly hosting meeting on Wednesday, January 25, 2017 at 1600 UTC in #hosting-community on Slack:

  • “Best Practices” Documentation updates.
    • Plan?
    • Progress?
    • Request for additional volunteers.
  • Promoting WordCamps updates.
    • Plan?
    • Existing scripts & tools to pull data.
    • Request for additional volunteers.
  • Distributed Unit Tests update.
    • Plan?
    • Existing tool discussion.
    • What additional tools are needed?
    • Request for additional volunteers.
  • Security discussion.
    • WordPress Security Czar @aaroncampbell would like to have a quick chat about hosts and security.
  • Open discussion.

If you have any items to add to the agenda or specific comments related to the outline above — especially if you are unable to attend the meeting at its scheduled time — please leave a comment. Hope to see you there!

#agenda

#weekly-hosting-chat

Hosting Meeting Notes: January 18, 2017

Here’s the summary of our very first meeting in #hosting-community on Wednesday, January 18, 2017 at 1600 UTC (Slack archive).

Purpose

  • Kicking things off by restating the purpose of the group.

Ideas

  • Hosts should band together and work through things that the core group can never do.
    • Come up with solutions for things like:
      • Upgrading PHP for non-managed users.
      • How we can all generally make WordPress feel and work better for everyone.
      • If core needs PHP module X, can we get enough of us on the same page for core to rely on that?
  • Best practices documentation (see: WordPress Hosting Handbook).
    • Server setups are going to vary, but there are definitely best practices that we can document.
      • Who puts default settings in wp-config.php?
      • System plugins.
      • Standard “default” php.ini values.
        • Hosts should offer feedback on things like upping default memory limits in core. (#29341)
      • Scaling and performance optimizations.
        • The REST API and plugins like WooCommerce are changing how people use WordPress.
  • Promoting and encouraging WordCamp attendance.
    • Hosts are uniquely positioned to promote regional WordCamps to customers.
      • WordPress has great brand awareness amongst customers.
      • WordCamp? Not so much.
      • Knowledge sharing “in your own backyard” is a big draw for some people.
    • Can leverage geographical customer data to promote upcoming local camps.
      • “As a WordPress user, here’s an event near you that you might find educational and informative that you didn’t even know was a thing!”
    • May require support from the #community-events team to help automate it.
      • An API would make it easy to tie into.
  • Making sure new features are scalable on implementations of all sizes.

Assignments

Feedback

Miss the meeting? Interested in helping on one of the initiatives above? Please feel free to drop into the comments and weigh in.

Next Meeting

Our next meeting will be in #hosting-community on Wednesday, January 25, 2017 at 1600 UTC. If that works with your schedule, we’d love it if you could attend. Hope to see more of you next week!

#notes, #weekly-hosting-chat

First Meeting Schedule

After collecting availability in #hosting-community, the first weekly meeting is scheduled for Wednesday, January 18, 2017 at 1600 UTC.

Due to interest from many time zones (a good problem to have!), there wasn’t a time that would work for everyone, but it can change in the future as is helpful.

For the first meeting, I’d like to go over the purpose of the group and find out what hosting related community projects folks are interested in helping out with, so that we can start building project groups and get things going.

Want to get a head start? Please comment with any topics or projects that you’d like to see discussed or are interested in helping out with.

Look forward to seeing you there!

#schedule