I’ve been looking into #21098 since yesterday, trying to figure out how to minimize out of memory errors on media uploads through XML-RPC. Results were a bit unexpected, but interesting nonetheless. Any ideas on how to reduce memory usage are more than welcome.

I’m still not sure where all the memory goes, for instance a 1MB file upload takes 50M of memory, while a 10M file takes 153M. The base64 encoding represents just a 33% overhead, so that’s not it. Also, it seems that even if you disable $HTTP_RAW_POST_DATA, php keeps the whole POST body in memory, but still, that’s a lot of memory to account for.

Fix #1: read from php://input directly

After a lot of playing with xdebug, IXR_Message->parse() didn’t look too efficient, so I tried https://gist.github.com/koke/5720860.

The current implementations relies a lot on substr, which works fine for smaller requests, but not so much on media upload. After some early testings, my solution seemed to require a bit more memory on small requests, but then performed much better as request size increased.

How did I test

I added this pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party (xmlrpc-test-upload.php), which adds a new wp.testUpload method: it replicates metaWeblog.newMediaObject, but it also returns peak memory usage. I had some problems with integer overflows so I ended up using the values output to the error log.

Then did a quick client on test.php and created a bunch of blank videos with file sizes from 1 to 200 mb. For instance, to get a ~1MB video:

[code]

ffmpeg -t 32 -s 1280×720 -f rawvideo -pix_fmt rgb24 -r 25 -i /dev/zero -y test-1M.mpg

[/code]

Memory results

The memory reductions weren’t as good as I expected, but still worth a try

Time results

After some trial/error process while running the tests, I did notice the new version felt faster, so I tested for that as well. Results were more pleasing this time:

What’s next

Even after this, I’m not sure yet where most of the memory goes. Running a xdebug trace on the new code, with a 200MB file upload:

• When xmlrpc.php is called, xdebug already reports a memory usage of 786MB.
• Once WordPress is initialized and IXR_Server->serve() is called, memory usage is 1083MB.
• Just before IXR finishes parsing, it reaches peak memory usage at 1607MB.
• At the end of execution memory usage is 1345MB.

I’ve uploaded the xdebug trace (trace-200.txt) in case someone else can spot another improvements.