Codex returns 500 error when WordPress cookies are set

Since March 14th Codex is sometimes returning a 500 error. It seems like this only happens if the wporg_logged_in cookie is set.

➜  ~ curl -v --cookie "wporg_logged_in=fooo%7C123456789%7C25ce9fd05b1aewf00509d6g5a602ae23" "https://codex.wordpress.org/index.php?title=Special:UserLogin&returnto=Help%3AContents"
*   Trying 66.155.40.241...
* TCP_NODELAY set
* Connected to codex.wordpress.org (66.155.40.241) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.wordpress.org
* Server certificate: Go Daddy Secure Certificate Authority - G2
* Server certificate: Go Daddy Root Certificate Authority - G2
> GET /index.php?title=Special:UserLogin&returnto=Help%3AContents HTTP/1.1
> Host: codex.wordpress.org
> User-Agent: curl/7.51.0
> Accept: */*
> Cookie: wporg_logged_in=fooo%7C123456789%7C25ce9fd05b1aewf00509d6g5a602ae23
>
< HTTP/1.1 500 MediaWiki exception
< Server: nginx
< Date: Fri, 17 Mar 2017 10:40:10 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Content-Type-Options: nosniff
< Strict-Transport-Security: max-age=31536000
<
<div style="border:1px solid #ffd0d0;padding:1em;">This is a cached copy of the requested page, and may not be up to date.</div><hr /><h1>Sorry! This site is experiencing technical difficulties.</h1><p>Try waiting a few minutes and reloading.</p><p><small>(Cannot contact the database server)</small></p><hr /><div style="margin: 1.5em">You can try searching via Google in the meantime.<br />
<small>Note that their indexes of our content may be out of date.</small>
</div>
<form method="get" action="//www.google.com/search" id="googlesearch">
	<input type="hidden" name="domains" value="http://codex.wordpress.org" />
	<input type="hidden" name="num" value="50" />
	<input type="hidden" name="ie" value="UTF-8" />
	<input type="hidden" name="oe" value="UTF-8" />

	<input type="text" name="q" size="31" maxlength="255" value="" />
	<input type="submit" name="btnG" value="Search" />
	<p>
		<label><input type="radio" name="sitesearch" value="http://codex.wordpress.org" checked="checked" />Codex</label>
		<label><input type="radio" name="sitesearch" value="" />WWW</label>
	</p>
* Curl_http_done: called premature == 0
* Connection #0 to host codex.wordpress.org left intact
</form></body></html>

#prio1

#ffd0d0

Theme previewer missing a theme.

web1 appears to be missing the “tempo” theme for the theme previewer. I tried the normal commands to get it to update as well as reapproving the theme manually, to no effect. Need somebody to take a look at it, see if something is wrong with that directory.

It’ll be in /extend/theme-preview/wp-content/themes/tempo. Should have version 0.0.33 in there like the other webs do.

Marx cPanel License Expired

When I log in to cPanel, I get a License not activated error instead of the normal dashboard.

#prio2

Marx blacklisted by Barracuda

I noticed a legitimate message in the WordCamp Help Scout account was flagged as spam because 66.155.40.24 is blacklisted by Barracuda.

Lookup | Removal

I can request removal, but it seems like it’ll just get added again if we don’t figure out what outbound messages caused the blacklisting and prevent that from happening again.

#prio2

nginx rewrite from `/forums` to `/support`

For two international forums I need a nginx rewrite from /forums to /support.

  • https://ary.wordpress.org/forums -> https://ary.wordpress.org/support
  • https://it.wordpress.org/forums -> https://it.wordpress.org/support

That seems to be something for the wporg-rosetta config:

location = /forums {
    return 301 /support/;
}

location ~ ^/forums/(.*) {
    return 301 /support/$1;
}

Not sure if we need to restrict that to the both hosts. We have a similar rule for wordpress.org/forums.

Once the configs are deployed the site URL needs to be updated as well:
* https://global.wordpress.org/wp-admin/network/site-info.php?id=332
* https://global.wordpress.org/wp-admin/network/site-info.php?id=352

Could someone please add the rules if they are looking good? Thank you!

#prio2

Whitelisted WordCamp Production Data for Dev Environments

Right now WordCamp devs use a small subset of the production database that was manually created, because it wouldn’t be safe to keep copies of the production database in local environments.

That works good enough for most things, but we keep running into situations where reproducing bugs and testing fixes is much harder, and takes much longer, than it would if we had real-world data to work with.

So, I’d like to create a way to safely use a whitelisted copy of production data in local environments. Here’s how I envision it working:

  1. Create a script that runs on the production web server once a day
  2. It would create a copy of the primary database on the production database server
  3. Then run lots of SQL commands against that copy in order to redact anything that hasn’t been whitelisted
  4. Have another script in dev environments that uses sftp to download a copy of the whitelisted database once a day

The whitelist would contain a list of tables, columns, and keys that have been determined to not have any sensitive data. For example:

  • wp_users – The table itself would be whitelisted, but only the ID, user_login, user_nicename, user_registered, user_status, display_name, spam, and deleted fields would be whitelisted. Because user_pass, user_email, and user_activation_key would not be in the whitelist, the script would replace the contents of those columns with [redacted] (or in the case of user_email, redacted@example.org).
  • wp_usermeta – The table itself would be whitelisted, along with the umeta_id, user_id, meta_key, and meta_value columns, but only certain meta_key rows would be whitelisted. For instance, first_name, last_name, description, and wp_capabilities would be whitelisted, but session_tokens and wordcamp-qbo-oauth would not be.

Additionally, the script would have some logic to redact potentially sensitive values within whitelisted columns. For example, any e-mail addresses inside a meta_value value would be replaced with redacted@example.org.

What does Systems think about that? I’d do all the work to build the script, but I want to make sure you don’t have any security/privacy concerns.

#prio2

Can we get irclogs.wordpress.org updated…

Can we get irclogs.wordpress.org updated from the latest svn?

A recent commit was made to restore the listing of archives (re: #meta-2448).

#irclogs #prio3

smtp-fwd reverse DNS

As reported on meta.trac it appears that smtp-fwd is lacking the rDNS which might explain why we’re seeing a higher level of slack invite failures of recent.

Could we add a reverse DNS for that mail forwarder please?

FYI; SPF and DKIM appear to be succeeding according to gmail.
#prio2

One of the servers seems…

One of the servers seems to not be updating for planet.wordpress.org. Every once in a while you can refresh the site and will get a page from December. Like one of the webservers isn’t updating properly or otherwise not running the process.

I have no visibility into the production servers, so can somebody take a look and see if one of them is “stuck” somehow?

Kathryn Presner has notified us…

Kathryn Presner has notified us that she is not receiving some emails from Themes Trac. I’ve been investigating this issue and found that the emails do get sent out, but they are sometimes broken – they are sometimes missing the ‘To’ header.
I’ve tried to find the root cause of this, but with no luck. Our trac install has plenty of custom modules, and a few local patches. One of them does modify the notification.py file, but i couldn’t locate any code that could cause this issue.
I’ve looked through the themes mailbox for automattic user, and found around ~1100 emails since 2014 that did not contain the To header (and were from themes trac), which is quite a big number as there’s not that many emails coming from themes trac.

Can somebody who has worked with customising / writing patches for wporg Trac have a look at this issue? I know @nacin did some, not sure who else.

An example comment for which the To header was missing is this one. I can find more if needed.

Example raw content of a broken email:

Return-Path: <noreply@wordpress.org>
X-Original-To: themes@automattic.com
Delivered-To: themes@automattic.com
Received: from localhost (localhost.localdomain [127.0.0.1])
        by smtp1.lax.automattic.com (Postfix) with ESMTP id 4D01C16C605A;
        Tue,  3 Jan 2017 20:25:43 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at wordpress.com
X-Spam-Flag: NO
X-Spam-Score: 0.072
X-Spam-Level:
X-Spam-Status: No, score=0.072 tagged_above=-999 required=5
        tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
        DKIM_VALID_AU=-0.1, MISSING_HEADERS=1.021,
        REPLYTO_WITHOUT_TO_CC=1.552, RP_MATCHES_RCVD=-0.5, SPF_PASS=-0.001]
        autolearn=no
Authentication-Results: smtp1.lax.automattic.com (amavisd-new);
        dkim=pass (1024-bit key) header.d=wordpress.org;
        domainkeys=pass (1024-bit key) header.from=noreply@wordpress.org
        header.d=wordpress.org
Received: from smtp1.lax.automattic.com ([127.0.0.1])
        by localhost (smtp1.lax.automattic.com [127.0.0.1]) (amavisd-new, port 10024)
        with ESMTP id mji5gHMhfkVq; Tue,  3 Jan 2017 20:25:42 +0000 (UTC)
Received: from mail.wordpress.org (lb2.lax.wordpress.org [66.155.40.19])
        (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
        (No client certificate requested)
        by smtp1.lax.automattic.com (Postfix) with ESMTPS id D818016C6094
        for <themes@automattic.com>; Tue,  3 Jan 2017 20:25:42 +0000 (UTC)
Received: from mail.wordpress.org (localhost.localdomain [127.0.0.1])
        by mail.wordpress.org (Postfix) with ESMTP id BA70D214159;
        Tue,  3 Jan 2017 20:25:42 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=wordpress.org; h=
        mime-version:content-type:content-transfer-encoding:from:date
        :reply-to:subject:message-id:references:in-reply-to; s=wp1; bh=4
        RMdfkSspyf94FiOiv44TS+QOpY=; b=fsqVz4XlXzJl4vkCdss1hxBmnUmhe1PDr
        nUq22LBvQ06Wf8P7hg410wP3lj9lAseMa408lEIapTpr/RV2MeQu9FC5JPtrXQNV
        qtR0j6ziwKPBAKEO4T3ELRL0p5N6DuOv0Fi2XaJjhOh/1ZP8BuNyQURQ317wu1bB
        G+vuLirP18=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=wordpress.org; h=
        mime-version:content-type:content-transfer-encoding:from:date
        :reply-to:subject:message-id:references:in-reply-to; q=dns; s=
        wp1; b=aAJpRBd8dOO1D4a/Gj1/kPKkmz+VU71wx8qkx2xwxg6f10x4MjKgVStOF
        vUJvK4AYXWnOWJypHIFOsSCdLsOhllcocSZR0kvCvjlHod8mgppfIWM8qexOh2xV
        EW5qlb/Y66t51MCNSzPol8AjY1XCoYsSkczkk1zd6ScTR/DT9U=
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
From: "WordPress Themes" <noreply@wordpress.org>
X-Trac-Version: 1.0.1
Precedence: bulk
Auto-Submitted: auto-generated
X-Mailer: Trac 1.0.1, by Edgewall Software
X-Trac-Project: WordPress Themes
Date: Tue, 03 Jan 2017 20:25:13 -0000
Reply-To: theme-reviewers@lists.wordpress.org
X-URL: https://themes.trac.wordpress.org/
Subject: =?utf-8?b?UmU6IFtXb3JkUHJlc3MgVGhlbWVzXSAjMzQ0NzY6IFRIRU1FOiBS?=
 =?utf-8?b?ZWJhbGFuY2Ug4oCTIDEuMS4y?=
X-Trac-Ticket-URL: https://themes.trac.wordpress.org/ticket/34476#comment:7
Message-ID: <071.a6e2191aa2ba7a4f4c3caee661ef5a26@wordpress.org>
References: <056.00d47f9a37a313e887ad08a208d6a56a@wordpress.org>
X-Trac-Ticket-ID: 34476
In-Reply-To: <056.00d47f9a37a313e887ad08a208d6a56a@wordpress.org>

IzM0NDc2OiBUSEVNRTogUmViYWxhbmNlIOKAkyAxLjEuMgotLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQpSZXBvcnRlcjogIGF1dG9tYXR0aWMgICAg
ICAgfCAgICAgICBPd25lcjogIGFjb3NtaW4KICAgIFR5cGU6ICB0aGVtZSAgICAgICAgICAgIHwg
ICAgICBTdGF0dXM6ICByZXZpZXdpbmcKUHJpb3JpdHk6ICBuZXcgdGhlbWUgICAgICAgIHwgIFJl
c29sdXRpb246CktleXdvcmRzOiAgdGhlbWUtcmViYWxhbmNlICB8Ci0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCgpDb21tZW50IChieSBncmFwcGxl
cnVscmljaCk6CgogQGthcm1hdG9zZWQgQ2FuIHlvdSBjaGVjayB0aGF0IGFuIHVwZGF0ZSBpcyB1
cGxvYWRlZCBub3cgdGhhdCBob2xpZGF5cyBhcmUKIG92ZXI/IFRoZXJlIGFyZSB0d28gb3RoZXIg
dGhlbWVzIHRoYXQgbmVlZCB1cGRhdGVzIHRvby4KCi0tClRpY2tldCBVUkw6IDxodHRwczovL3Ro
ZW1lcy50cmFjLndvcmRwcmVzcy5vcmcvdGlja2V0LzM0NDc2I2NvbW1lbnQ6Nz4KV29yZFByZXNz
IFRoZW1lcyA8aHR0cHM6Ly90aGVtZXMudHJhYy53b3JkcHJlc3Mub3JnLz4KV29yZFByZXNzLm9y
ZyBUaGVtZSBEaXJlY3RvcnkgUmV2aWV3cwo=