RFC: Automated Hosting Test Runner Service

Summary

The HostingHosting A web hosting service is a type of Internet hosting service that allows individuals and organizations to make their website accessible via the World Wide Web. Team has received feedback that the current process for running hosting tests is cumbersome and requires significant setup beyond simply providing hosting space. This RFC proposes the development of a service that simplifies and automates the process of preparing and executing the hosting tests.

Problem

Currently, contributors and hosting providers who want to run the hosting tests must:

Manually clone and configure the test suite
Set up the proper environment
Understand all dependencies and edge cases
Execute and interpret the results themselves

This level of complexity creates a barrier for participation, especially for smaller hosts or non-technical contributors.

Proposed Solution

Create an automated service that prepares and runs the hosting test suite on behalf of the user. The workflow would be as follows:

User Input:
- SFTPSFTP The SSH File Transfer Protocol (SFTP) is a network protocol that provides file access, file transfer, and file management over any reliable data stream. Like FTP, but under a secure protocol./SSHSSH SSH or Secure Shell is a cryptographic network protocol for operating network services securely over an unsecured network. SSH provides a secure channel over an unsecured network by using a client–server architecture, connecting an SSH client application with an SSH server. credentials
- DatabaseDatabase A database is an organized collection of data. Access to this data is usually provided by a "database management system" (DBMS) consisting of an integrated set of computer software that allows users to interact with one or more databases and provides access to all of the data contained in the database. Because of the close relationship between them, the term "database" is often used casually to refer to both a database and the DBMS used to manipulate it. credentials (host, name, user, password)
- WordPress.orgWordPress.org The community site where WordPress code is created and shared by the users. This is where you can download the source code for WordPress core, plugins and themes as well as the central location for community conversations and organization. https://wordpress.org/ username and app password/token
Response:
- The service returns a public SSH key to be added to the user’s hosting environment
Execution:
- Once the public key is in place, the service connects to the hosting environment
- Prepares the test suite
- Moves all the required files to the host
- Executes the tests
- Sends the result to make.wordpress.org

Security Considerations

All credentials must be securely transmitted and stored (if at all)
The public key approach ensures that credentials are not reused or stored long-term
A unique key-pair per session can further isolate the testing instance
Connections should be limited to specific IP addresses and short-lived

Benefits

Lower barrier to entry: Users do not need to understand the full setup process
Consistency: Every test runs in a standard, validated environment
Scalability: Easier onboarding of new contributors and hosts
Insights: Results can be aggregated to better understand trends and common issues

Next Steps

Discuss and approve the general concept
Create a prototype of the service
Review and iterate based on initial testing
Plan deployment and documentation for broader use

Request for Feedback

The Hosting Team would appreciate feedback on the following, especially from those hosts who raised concerns about the current setup:

Is this the right approach to lowering the barrier?
Are there any privacy or security concerns that must be addressed?
Would hosts be willing to test this service in an early access phase?

Thanks to @zunaid321 @amykamala @wesrapyd for the review

David E. Smith 02:18 on 2025-06-26

I work for a host that specializes in WordPress specifically, and part of our offering involves our managing the users’ WordPress installs. (They’re symlinked into the customers’ web space, owned by a different user, so that the customer cannot modify WordPress.) This is usually good for security, and for making it easy for customers to update WP. This is very bad for something like the test runner. 😀

Does the test runner simply need to run code in the environment, or does it need to be able to access itself over the web? i.e. could I leave our managed WP install in the root, then put this in a subdirectory or similar, so that the tests can run in effectively an isolated environment?
David E. Smith 02:34 on 2025-06-26

Who will be hostingHosting A web hosting service is a type of Internet hosting service that allows individuals and organizations to make their website accessible via the World Wide Web. this project, i.e. to whom am I giving SSHSSH SSH or Secure Shell is a cryptographic network protocol for operating network services securely over an unsecured network. SSH provides a secure channel over an unsecured network by using a client–server architecture, connecting an SSH client application with an SSH server. access to one of my servers, and who is maintaining/securing the whole thing? Will we be giving access to one of our competitors? (I can certainly see many hosts being hesitant to do this.)

I think my preference, at least, would be to have this run and maintained by a community entity, but I’m honestly not sure who that would be. (As an example, “the folks behind wordpress.orgWordPress.org The community site where WordPress code is created and shared by the users. This is where you can download the source code for WordPress core, plugins and themes as well as the central location for community conversations and organization. https://wordpress.org/” might be a tough sell to some, because they’re often perceived as being part of Automattic, and we’re back to that “competitors” thing.)
- Tiffany Bridge 14:11 on 2025-06-26
  
  I’m not sure that I’m worried about the “competitor SSHSSH SSH or Secure Shell is a cryptographic network protocol for operating network services securely over an unsecured network. SSH provides a secure channel over an unsecured network by using a client–server architecture, connecting an SSH client application with an SSH server. access to my serverServer A server is a piece of computer hardware or software that provides functionality for other programs or devices. Typical servers are database servers, file servers, mail servers, print servers, web servers, game servers, and application servers.” thing. It looks like the idea is that the credentials needed are just for a customer-level user, right? So the same SSH creds I would give to any customer. Literally nothing prevents any of my competitors from just buying a plan from me and getting that access.
  
  My question is whether the test suite requires stuff I wouldn’t let a customer install. If the idea is to test WordPress in a real environment like the ones my customers would use, it’s one thing for me to get a VPSVPS A virtual private server (VPS) is a virtual machine sold as a service by an Internet hosting service. A VPS runs its own copy of an operating system (OS), and customers may have superuser-level access to that operating system instance, so they can install almost any software that runs on that OS. (where a customer would have root access and the right to install nearly anything) and let the test suite setup go nuts, but it’s quite another for me to point it at our Managed WordPress environment, which is much more opinionated and locked-down by design, and then have it try to install a bunch of stuff that a customer user doesn’t have privileges for.