I ran four users through the new themes experience in the customizer (#37661) today. I sat with them for each test to take notes and hand them the test steps as they went (written on slips of paper). All of the testers are somewhat technical but not particularly experienced with WordPress; they all used Chrome on Mac with two on laptops and two on desktops (using their own devices). We started out with a clean install, then I added an import of posts from another site before the third test (the featured images didn’t carry over).
I’ve been living with this feature for the past two months, and also witnessed all of the tests directly, so independent analyses of the videos would be much appreciated in the comments.
There were no major issues that everyone had trouble with. However, each test revealed a few things that we should improve. Most of these items are unrelated to the feature at question – the new theme browser/installer – but they could all use thoughts and in some cases trac tickets. The biggest potential change to the feature at this point would be to add an additional notice about unsaved changes before opening the themes panel – I’m looking for opinions on whether that’s needed here. Also looking for thoughts on whether we need the install without previewing functionality, or if maybe it should only be available in the details modal?
It was interesting to see how different each test was, despite all of the testers being extremely demographically similar. A lot of this also stemmed from the variety of themes that they looked at. Several of the themes violate various theme review team guidelines, and I made a list that I’ll send to someone in the TRT to address, as unfortunately I still haven’t had time to go through and find ticket for all of them (cc @greenshady).
Each of the videos and the key takeaways from each individual test are available after the jump.