Matt Mullenweg 6:48 pm on July 19, 2024

The Crowdstrike disaster is good…

The Crowdstrike disaster is a good opportunity for us to brainstorm and review our “defense in depth” around updates. If I committed a die(); to coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress., what happens? What if I modify the file on the server directly? What automated fail-stops can we build in? How can WordPress clients also protect themselves? (We do some decent stuff here already for failed updates.)

It’s our ethical duty as engineers to make sure these systems fail gracefully when something goes wrong, because it’s guaranteed that it will at some point.

Matt Cromwell 8:26 pm on July 19, 2024

*Nods in acknowledgement*

Just don’t do those things, Matt!

Kidding aside. This is a healthy question for a platform as significant as WP, and a rollback option doesn’t feel quite serious enough.
Javier Casares 8:08 am on July 20, 2024

Maybe, when something fails (and shouldn’t) a checksum verification should be in order. Now we have the screen and the link to “what may happen”, but having the possibility to write a report in the file system (like a log or something) and having a link to that in the email or in some place to access quickly should be a good start.

When WordPress fails, tech people may know what to do or where to look, but users / editors should have a start point where a little summary at the beginning with a little report may be useful. Like “file check: OK, plugins: OK, theme: FAILED…” so, at least, they may know where to start looking.

I’m sure countless sysadmins, with the CrowdStrike “thing” had in their minds “WTF is happening, and why all the systems are failing” not knowing where to start.
Bike Gremlin 8:44 am on July 20, 2024

Thinking out loud (not claiming to be right 🙂 ):

Say no to auto updates, for a start (and definitely don’t force-push them as CrowdStrike did). 🙂

WordPress seems very eager to “nudge” the users to update ASAP (every page in my backend shows the “Please update now” button).

Ideally, making it easy (“idiot-friendly”) to deployDeploy Launching code from a local development environment to the production web server, so that it's available to visitors. a staging environmentStaging Environment A staging environment is a non-production copy of your site. This is a private place to build the site -- design, copy, and code -- until your client approves it for production or live. Sometimes used in addition to, or as a Development Environment. would be awesome. In some hosting environments, Softaculous software offers that option (it’s a bit of a hit-and-miss in terms of working though LOL) – you can one-click make a staging copy, then push only updates to live, without overwriting the WooCommerce article list, for example.
Paul 4:18 pm on July 20, 2024

Just my take on the issue as i cloudn’t stay mute on that one 😉

Auto updates as well as stressing users to update is the root of many troubles. Running around like chickens and shouting update just to follow up with another update, quick fixing vulnerabilities, indroducing new issues, then scream oh we messed up but here is another fix… you get the idea, sound like more like nervous breakdown than sound development.

Especially for non techy users, wouldn’t it be way more convinient for them to install an update when they have someone handy who is able to help them if something breaks? Instead of updsting cause they are stressed to or worse autoupdating and then running wild in the support forums and on phones of devs?

Also for devs like us: I cannot count how many fatals i fixed in woocommerce or wordpress themes that wouldn’t even happen if the user updates 3 days later cause devs already took care of it. And yes if no one uses the new version issues are not discoverd that fast but let’s not kid ourselfes… most users want to have a new version but also want to be able to take their time, save an hour when it suits them, maybe run a proper staging install and then roll out to production.

Take woocommerce for instance, when was the last update where there wasn’t a serious(more or less) issue discovered in the first 24h? i cannot remember.

The wp update i guess it was 6.4 breaking curl calls… still in my head. Took a sleepless night to fix it… 2 days later all good.

Yes there are critical vulnerabilities, but if they have been in there for weeks, rolling a new version instantly that potentially breaks other things and most likely introducing new vulnerabilities solves exactly nothing, except wasting lifetime, worries and dev resources.

Just read through the wordpress support forums after every update where people are complaining about sites breaking, style issues en mass missing translations unreachable adminadmin (and super admin) screens and so on.

I think if we would all take a breath, think about when and how we want to roll an update… issues would be less, the community healthy and the communication in the forums friendly( if i read through some threads, i lose trust in humanity how stresst folks can be)

Just my take on the issue 🙂
Not saying it’s the only optinion but after 18+years in software development i have seen my share on the update front…

Cheers and keep up the good work!
Svetoslav Marinov 6:11 pm on July 20, 2024

There has to be different testing areas and approaches and lots of people should be tasked to test a future release especially if it’s a mission critical software. If it had to be done at the end of the week there must be a really good reason to do so.
Miriam Schwab 8:08 pm on July 20, 2024

Thanks for raising this. If I understand correctly, I think it ties into some ideas I planned on raising for discussion related to the integrity of WP infra and systems, and the limited number of people who can work on resolving issues if and when they arise.

Gradual roll-outs

I was going to raise this for plugins, but it could be relevant for CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. too. It might be good for us to have a gradual roll-out system for Core and for themes/plugins (at least for those with large user bases). That way, if there are issues, it will be discovered before everyone gets it, and a fix can be released to everyone. I believe this is common practice in software development in general, particularly when dealing with large user bases.

I’m aware this is pretty complicated, but I’m putting it out there as a way that might potentially mitigate the impact of serious issues in software updates. We at Elementor would love to contribute to this type of approach if it’s relevant.

Taking another look at metaMeta Meta is a term that refers to the inside workings of a group. For us, this is the team that works on internal WordPress sites like WordCamp Central and Make WordPress. infra and the limited number of people who have access to systems

Admittedly I don’t know how WP’s systems work behind the scenes, but something happened recently that brought up some concerns about the integrity of said systems, and how issues with them can affect millions of users.

About two months ago, something happened with the Theme repository’s APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. that prevented users from updating Elementor’s Hello Theme from the repository.

Here’s the ticket about this explaining the issue. At the time I also reached out to whoever I could in Make WordPress SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. to get help.

I learned that there are only a handful of people who have access to key systems and could have looked into this, but they were AFK. After a few days of helplessness, someone suggested we release an empty version update and see if that works. It did, and the ability to updated the theme from the WP Adminadmin (and super admin) was restored. But what’s also concerning is that to date, no one knows what happened or why. Also, how do we, or any other pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party or theme, handle this type of system fail if and when it happens again?

We would very much like to discuss and contribute to efforts to bolster system integrity, and the expansion of support should meta issues arise.

Apologies if this was not the right place to raise this, or I misunderstood the direction of this post.
peq42 9:28 pm on July 20, 2024

Maybe the best way is to have it all be as transparent as possible, git style.

Add to wordpress website’s main menu a page that lists any commit changes, and if a file is modified outside of a commit(through say, FTPFTP FTP is an acronym for File Transfer Protocol which is a way of moving computer files from one computer to another via the Internet. You can use software, known as a FTP client, to upload files to a server for a WordPress website. https://codex.wordpress.org/FTP_Clients.), add it + the IP(store other information too but privately) to the list as well.
Rene Hermenau 12:20 pm on July 21, 2024

> If I committed a die(); to coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress., what happens?
What happens when I push a die(); to core?

There should be mechanisms where even a (super)adminadmin (and super admin) can’t just push something to trunktrunk A directory in Subversion containing the latest development code in preparation for the next major release cycle. If you are running "trunk", then you are on the latest revision. without first creating a branchbranch A directory in Subversion. WordPress uses branches to store the latest development code for each major release (3.9, 4.0, etc.). Branches are then updated with code for any minor releases of that branch. Sometimes, a major version of WordPress and its minor versions are collectively referred to as a "branch", such as "the 4.0 branch". and assigning a bunch of developers to review it first.

GithubGitHub GitHub is a website that offers online implementation of git repositories that can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/, for example, has so-called branch protection rules that prohibit any direct changes in `main` until a branch has been created and approved.

And GitGit Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. Git is easy to learn and has a tiny footprint with lightning fast performance. Most modern plugin and theme development is being done with this version control system. https://git-scm.com/. also has something like this, called “pre-push”.

SVNSVN Subversion, the popular version control system (VCS) by the Apache project, used by WordPress to manage changes to its codebase.? I don’t know of such a function there.

To name a well-calculated risk factor, the question is how many people have direct access to the wp core distribution package or are able to merge something into the core before creating the core package and thus possibly bypassing the review process.

Even if everyone is fully trusted, what happens if one of these accounts is compromised? Is there some other factor that prevents that compromised account from doing damage to the dist package?

I can only give some food for thought here, not offer any concrete solutions, as I am not involved in the core release process, but I trust in the competence of those responsible at the wp core to do everything possible to prevent such a crowdstrike-like disaster.
Scott Kingsley Clark 6:25 pm on July 21, 2024

Once the release is already out, that’s all you can do from the WP side of things at that point. If it broke all of the WP installs, you’ll have to rely on the generous support for WordPress by many hosting platforms that can move quickly on solutions. I think that’s the answer here. Work out strategies with hosting providers for scenarios like this so there’s a “fire alarm” plan and what tooling is needed by hosts to prepare for scenarios like this or others. All of the hosts should communicate and share their plans so they can provide as uniform of a response as possible.
- Courtney Robertson 3:28 pm on July 22, 2024
  
  +1
- Pixelbart 7:35 am on July 29, 2024
  
  +1
thomasprice61 7:32 pm on July 21, 2024
Look what others do… hint, it’s about management.. the tech and processes will follow.
Surely partners can assist .. look for a long-term win-win
Review then start adopting the most applicable standard(s) like:
Courtney Robertson 3:27 pm on July 22, 2024
How OpenSSF Scorecards Can Help WordPress

The CrowdStrike disaster serves as a crucial reminder for enhancing our defenses, especially around updates. Here’s a practical breakdown of how OpenSSF Scorecards can address the challenges mentioned and improve WordPress security:

Automated Checks for Early Detection
- Problem: If a die(); statement or other harmful code is committed to the coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress., it could cause widespread failures.
- Scorecards Solution: Scorecards continuously monitor the repository for such problematic commits. By setting up automated scans, Scorecards can detect and flag potentially harmful code changes before they are merged into the core. This helps prevent disruptive updates from being deployedDeploy Launching code from a local development environment to the production web server, so that it's available to visitors..
Handling Direct Modifications on Servers
- Problem: Direct file modifications on the server can bypass regular security checks, leading to unforeseen failures.
- Scorecards Solution: Scorecards include checks for consistent deployment practices. They ensure that all changes go through proper channels (e.g., code reviews, automated tests) and flag any direct modifications that could bypass these controls.
Implementing Automated Fail-Stops
- Problem: Ensuring systems fail gracefully is critical to maintaining service continuity.
- Scorecards Solution: Scorecards evaluate the presence of automated testing, CI/CD integration, and other fail-safes. They continuously assess the effectiveness of these measures, helping to ensure that fail-stops like rolling back to a previous stable version or isolating faulty components are in place and functioning correctly.
Enhancing Client-Side Protections
- Problem: Users need ways to protect their installations from failed updates or other issues.
- Scorecards Solution: Scorecards check for best practices in dependency management and update mechanisms. By encouraging these practices in pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party and theme development, Scorecards help ensure third-party code is secure and reliable. Additionally, providing guidelines for users on setting up automated backups and update tests can further protect client installations.
Practical Steps for Implementation
- Set Up Scorecards in the Repository:
  Integrate OpenSSF Scorecards with the WordPress GitHubGitHub GitHub is a website that offers online implementation of git repositories that can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/ repository. This involves setting up configurations to allow Scorecards to run automated checks on the codebase.
- Regular Security Audits:
  Schedule regular automated security audits using Scorecards. These audits will check for common vulnerabilities, improper coding practices, and other security issues.
- Enforce Code Review Policies:
  Utilize Scorecards to enforce strict code review policies. Ensure that every code change is reviewed and tested before being merged into the core. Scorecards can help flag any deviations from this process.
- Enhance CI/CD Integration:
  Integrate Scorecards with the CI/CD pipeline to ensure continuous security assessment during the development lifecycle. This ensures that every build is checked for security compliance.
- Educate and Empower the Community:
  Share the importance of these security measures with the WordPress community. Provide documentation and training on using Scorecards, setting up automated backups, and performing regular security checks on their installations.
Our Ethical Duty

It’s our ethical duty to ensure systems fail gracefully when something goes wrong because it inevitably will at some point. While I believe integrating OpenSSF Scorecards could help build robust fail-safes and proactive security measures, I recommend that WordPress Core research and evaluate if this tool is a good fit for our needs. Regularly updating and maintaining these security practices is essential to safeguarding the platform and its users.

For more details on implementing OpenSSF Scorecards, visit the OpenSSF Scorecards page.
Stefano Picco 11:33 am on July 29, 2024

Hi all 🙂

Just a spontaneous thought. How about differentiating between versions, like LibreOffice does, for example?

They offer a current/newest version, but also a stable version that is not the newest, so to speak, but has shown in use that everything is fine with it.

Dell also does something similar with the Windows updates – I don’t know whether these were really not affected by the CrowdStrike problem.

Comments are closed.

Welcome!

Communication

The Crowdstrike disaster is good…

Welcome!

Communication

Share this: