Ian Dunn 7:01 pm on March 5, 2018
Tags: prio1 ( 55 )

WordCamp Let’s Encrypt Script Broken

We received warnings from Let’s Encrypt that many of the WordCamp.org certs have not been renewed, and will invalidate on March 25th.

I’m guessing there are some necessary things that didn’t get transfered from LAX to ORD.

There may be some details in the letsencrypt-update.log, but I don’t have access to that on production anymore.

I tried setting up a test environment on my sandbox, but couldn’t because pip isn’t available. That may be the problem on production as well, but I can’t test any further until it’s on our sandboxes.

@barry, can you please take a look as soon as you have time? Let me know if there’s anything I can do on my end.

#prio1

Ian Dunn 10:52 pm on March 5, 2018

Thanks for the details @762e5e74. It looks like there’s something wrong with the script; for the past 2 weeks it’s been renewing every domain every time the script runs, rather than skipping the ones that have been renewed within the past 60 days.

I’m wondering if that’s somehow related to the `ORD` migrationMigration Moving the code, database and media files for a website site from one server to another. Most typically done when changing hosting companies., like if the `data.jsonJSON JSON, or JavaScript Object Notation, is a minimal, readable format for structuring data. It is used primarily to transmit data between a server and web application, as an alternative to XML.` file isn’t writable or something.

In order me for to troubleshoot more, could you please do these 2 things:

1) Temporarily disable the cron job, so that we stop hitting the rate limit until the script is fixed.
2) Install `pip` (and anything else necessary for running `issue.py`) on @coreymckrill and my sandboxes, so that we can manually test the script and find the bug.
Ian Dunn 8:53 pm on March 12, 2018

@762e5e74, I was able to successfully run the entire script on my sandbox, now that the rate limits have expired.

I think these are the steps necessary to restore everything back to normal.

1) On my sandbox: `cp -R {path/to/my/sandbox}/output {path/to/role}/output`
2) Track new files, this should work: `svn add $(svn status | grep [^?] | awk ‘{print $2}’)`
3) Commit & deployDeploy Launching code from a local development environment to the production web server, so that it's available to visitors. those changes.
4) On production: `nginxNGINX NGINX is open source software for web serving, reverse proxying, caching, load balancing, media streaming, and more. It started out as a web server designed for maximum performance and stability. In addition to its HTTP server capabilities, NGINX can also function as a proxy server for email (IMAP, POP3, and SMTP) and a reverse proxy and load balancer for HTTP, TCP, and UDP servers. https://www.nginx.com/. -t && service nginx reload`.
5) Disable the proxying of `.well-known` to my sandbox.
6) Manually run `.env/bin/python issue.py` on production. If there are any problems, let me know.
7) If not, then go ahead and re-enable the cron job.

Thanks!
Ian Dunn 5:29 pm on May 22, 2018

@762e5e74, it looks like the cron job to update the cert checkout from SVNSVN Apache Subversion (often abbreviated SVN, after its command name svn) is a software versioning and revision control system. Software developers use Subversion to maintain current and historical versions of files such as source code, web pages, and documentation. Its goal is to be a mostly compatible successor to the widely used Concurrent Versions System (CVS). WordPress core and the wordpress.org released code are all centrally managed through SVN. https://subversion.apache.org/. is still disabled on my sandbox, so I get errors when trying to browse to to newer sites.

I don’t have permission to modify `etc/cron.d/letsencrypt-update` or manually `svn up`. Would you mind uncommenting the jobs?
- abbe 7:52 pm on May 22, 2018
  
  Sorry about that. It is enabled now.
  - Ian Dunn 8:58 pm on May 22, 2018
    
    Thanks!