WordCamp Deployments broken

Deploys were initially working after they were setup, but the past few days it it looks like they’ve stopped working, or are only partially working. The script still reports that it finished successfully, but the effects of commits are not showing up anymore.

It used to take about a minute to finish, but now it finished almost immediately, so I wonder if part of it is dying early or something. If I svn up on my sandbox, the commits are there; it’s just production that doesn’t have them.

Here’s two examples:

  • https://meta.trac.wordpress.org/changeset/6625

Browse to https://central.wordcamp.org/wp-admin/post.php?post=3044783&action=edit&classic-editor, look at the Contact Information metabox. If the commit is applied, you’ll see * at the end of most of the input fields.

  • https://github.com/coreymckrill/wordcamp-reports/commit/31c6f66566fde5e3986b44a845798df622977cf7

This is setup as an svn:external to plugins/wordcamp-reports. We’ve never had problems using GitHub as an external before.

Browse to https://central.wordcamp.org/wp-admin/index.php?page=wordcamp-reports&report=ticket-revenue, you should see a field instead of the field


PDF generation doesn’t work on new WordCamp server

Hi, it doesn’t look like wkhtmltopdf has all of its dependencies on the new server:

wkhtmltopdf -d 300 -T ’10’ -R ’10’ -B ’10’ -L ’10’ ‘/tmp/wcdocs_5a831fa5c3b96/sponsorship-agreement.pdf.html’ ‘/tmp/wcdocs_5a831fa5c3b96/sponsorship-agreement.pdf’

wkhtmltopdf: error while loading shared libraries: libXrender.so.1: cannot open shared object file: No such file or directory


Expired SSL certificate for jobs.wordpress.net

https://jobs.wordpress.net/ needs a new certificate.


Automating ip2location updates

I’m working on automating updates for the ip2location database tables. Barry created the beginnings of a script in [dotorg13140], but it uses mysqlimport, which doesn’t appear to exist on our sandboxes.

Would it be possible to get mysql or mysqlimport installed, or is there another approach that Systems would prefer?

If it’s possible, I’d like to get the script done before WP 4.8 is released on the 8th, since the data in those tables is important for a new feature being launched.


Codex returns 500 error when WordPress cookies are set

Since March 14th Codex is sometimes returning a 500 error. It seems like this only happens if the wporg_logged_in cookie is set.

➜  ~ curl -v --cookie "wporg_logged_in=fooo%7C123456789%7C25ce9fd05b1aewf00509d6g5a602ae23" "https://codex.wordpress.org/index.php?title=Special:UserLogin&returnto=Help%3AContents"
*   Trying
* Connected to codex.wordpress.org ( port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.wordpress.org
* Server certificate: Go Daddy Secure Certificate Authority - G2
* Server certificate: Go Daddy Root Certificate Authority - G2
> GET /index.php?title=Special:UserLogin&returnto=Help%3AContents HTTP/1.1
> Host: codex.wordpress.org
> User-Agent: curl/7.51.0
> Accept: */*
> Cookie: wporg_logged_in=fooo%7C123456789%7C25ce9fd05b1aewf00509d6g5a602ae23
< HTTP/1.1 500 MediaWiki exception
< Server: nginx
< Date: Fri, 17 Mar 2017 10:40:10 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Content-Type-Options: nosniff
< Strict-Transport-Security: max-age=31536000
<div style="border:1px solid #ffd0d0;padding:1em;">This is a cached copy of the requested page, and may not be up to date.</div><hr /><h1>Sorry! This site is experiencing technical difficulties.</h1><p>Try waiting a few minutes and reloading.</p><p><small>(Cannot contact the database server)</small></p><hr /><div style="margin: 1.5em">You can try searching via Google in the meantime.<br />
<small>Note that their indexes of our content may be out of date.</small>
<form method="get" action="//www.google.com/search" id="googlesearch">
	<input type="hidden" name="domains" value="http://codex.wordpress.org" />
	<input type="hidden" name="num" value="50" />
	<input type="hidden" name="ie" value="UTF-8" />
	<input type="hidden" name="oe" value="UTF-8" />

	<input type="text" name="q" size="31" maxlength="255" value="" />
	<input type="submit" name="btnG" value="Search" />
		<label><input type="radio" name="sitesearch" value="http://codex.wordpress.org" checked="checked" />Codex</label>
		<label><input type="radio" name="sitesearch" value="" />WWW</label>
* Curl_http_done: called premature == 0
* Connection #0 to host codex.wordpress.org left intact



Deploy Key and Mirroring for GitHub Repo

Ref: https://meta.trac.wordpress.org/ticket/633

git.develop is being mirrored by GitHub, but their mirror process only checks every few hours, which is way too slow to be useful. Instead, we want to replace it with a post-receive push.

To do that, we need a deploy key created, per GitHub’s docs.

The private key needs to live on the git server, either in ssh agent, or with a reference in the ssh config file:

Host github.com
    IdentityFile ~/.ssh/id_rsa_github

The push command then needs to be added to the post-receive script.

git push -f --mirror git@github.com:WordPress/wordpress-develop.git

I’d like to get this done quickly, so we can get GitHub integrations up and running. Thanks!


New table column for translate.wordpress.org

Hi, for an upcoming update of GlotPress I need a new column user_id_last_modified for the translations table.
Background: https://github.com/GlotPress/GlotPress-WP/issues/293

The query:
ALTER TABLE translate_translations ADD COLUMN user_id_last_modified bigint(20) DEFAULT NULL;
Running this query on my local dump took 9 min 32.67 sec (35207550 rows affected).

@barry: Can you run the query on each server like you did for the index change? Thank you!


502 Bad Gateway errors

The number of reports for 502 Bad Gateway errors is increasing recently.

My first report for 502 errors was for https://*.wordpress.org/?fetch-custom-header=/plugins/ requests on Jan 7th. This was solved by @barry, “some memory corruption probably caused by a bug in pecl-memcache that we are working on fixing“.

Around April 3rd I got a few reports for translate.wordpress.org but couldn’t confirm them.

On April 13th I reported that the https://*.wordpress.org/?fetch-custom-header=/plugins/ issue happens again. This issue is still there, see #dotorg-warnings.

Since this Monday we’re getting reports for 502 Bad Gateway errors on our make sites. For example  https://make.wordpress.org/polyglots/feed/p2.ajax returns a 502 which prevents adding new comments.


But there are also new reports for translate.wordpress.org (when submitting translations) and for localized sites:

Image 2016-06-07 at 9.56.15 am

The 502 errors do not happen on our sandboxes, only in production. Can someone please look into this? Thank you!