User Testing the New Customizer Themes Experience

I ran four users through the new themes experience in the customizer (#37661) today. I sat with them for each test to take notes and hand them the test steps as they went (written on slips of paper). All of the testers are somewhat technical but not particularly experienced with WordPress; they all used Chrome on Mac with two on laptops and two on desktops (using their own devices). We started out with a clean install, then I added an import of posts from another site before the third test (the featured images didn’t carry over).

I’ve been living with this feature for the past two months, and also witnessed all of the tests directly, so independent analyses of the videos would be much appreciated in the comments.

Summary

There were no major issues that everyone had trouble with. However, each test revealed a few things that we should improve. Most of these items are unrelated to the feature at question – the new theme browser/installer – but they could all use thoughts and in some cases trac tickets. The biggest potential change to the feature at this point would be to add an additional notice about unsaved changes before opening the themes panel – I’m looking for opinions on whether that’s needed here. Also looking for thoughts on whether we need the install without previewing functionality, or if maybe it should only be available in the details modal?

It was interesting to see how different each test was, despite all of the testers being extremely demographically similar. A lot of this also stemmed from the variety of themes that they looked at. Several of the themes violate various theme review team guidelines, and I made a list that I’ll send to someone in the TRT to address, as unfortunately I still haven’t had time to go through and find ticket for all of them (cc @greenshady).

Each of the videos and the key takeaways from each individual test are available after the jump.

Test 1: Arim

Arim is a Communication student at USC Annenberg School for Communication and Journalism. She has a blog on WordPress.com, but hasn’t customized it much. Here are the key takeaways from her test:

  • Completely misses the big blue customize button in the dashboard the first time.
  • Repeatedly closes the customizer to go back to changing the theme; then gets that you can switch within the customizer after the first few tries.
  • Clicked in and out of a lot of settings in the customizer without changing them.
  • The customizer randomly closed at one point; this is probably a bug with customize-loader.js, which we’ve been thinking about eliminating anyway.
  • She installed and previewed all of the themes when asked to install multiple – skipped over the install button.
  • After the test, she said that she realized that the live previews looked different from the theme images because there weren’t any images, etc. in the site yet (even though she didn’t say anything about that during the test).

Test 2: Kai

Kai is an artist and a recent graduate from USC’s Roski School of Art and Design. She has minimal experience with WordPress but has used it before. She took a coding class and had an assignment to make and customize a WordPress site. She currently has a portfolio site that she coded from scratch because she thought it would take too long to learn how to do it in a WordPress theme (I’ve offered to help her transition it). Notes:

  • I forgot to switch it back to a default theme, so she struggled through some unfortunately unuseful options in the theme that was previously used.
  • Wanted to start looking for themes to add before looking at the installed themes (this is a good thing).
  • Tried using the browser’s back button to go back. Note: the customizer browser history project enables this functionality.
  • Got a bit confused with menus, although there isn’t a lot of content yet.
  • Also skips over the install (without previewing) functionality despite my direction.
  • Takes a second to figure out how to go back to the theme browser, but gets it pretty quickly.
  • There aren’t very many text-oriented themes in the repo.
  • Really likes the Site Icon feature!
  • The device preview buttons were cut off/misaligned on her screen (he didn’t notice).
  • Wishes that there was a better correlation between theme screenshots and how it looks on your site/with dummy content. Or if there were a curated demo for them before looking at them with your site. “It would be cool to see another site using it, to its fullest.” Listen to the end of the video for the details. Related: #38114.

Test 3: Erika

Note: I had Erika play with this less formally last week so she’s seen it before. She offered to run through the test steps and record it though. This is probably a good example of how users might approach the interface the next time they switch themes.

Erika is a Design Specialist at the USC Annenberg Digital Lounge. She recently switched her personal site to WordPress and also oversees content creation on the digital lounge website. Notes:

  • She had some issues with menu items – thought that the add new page functionality was a search form for that type of post. She was also confused because I had imported all of the posts from the site she oversees.
  • The “Automatically add new top-level pages to this menu” option is confusing in the customize content. Also, how does this work relative to pages created in the customizer?
  • She was kind of confused with the social menus implementation. This is the recommended approach for social menus right now. Social menus is a good example of when it would be nice to be able to control what types of available menu items are shown for menus in given locations. Related: #37974.
  • Clean installs only have one page, making the front page/posts page options even more confusing. We should really be able to create pages from here.
  • It’s easy to miss the unsaved changes notice at the top of the page. We should do everything we can to prevent them from getting to the browser’s are-you-sure message. One of her previous suggestions was to add a notice to the change theme button at the top level of the customizer.
  • Really likes device preview! Thinks it may be a little too hidden.

Test 4: Triet

Please excuse the background noise during this test. There were a lot of distractions, so I tried to crop those out as much as possible.

Triet is a Media Arts and Practices major at USC’s School of Cinematic Arts. He’s never used WordPress before. Does use Squarespace because it’s easy to make sites look nice with their themes. Takeaways:

  • Thinking about using a theme: “Can I prev… Oh, Preview!”
  • A lot of the themes he tried had too many options, so he got a bit lost.
  • He was a bit confused with the testing setup/purpose of the site, so there were some questions around that.
  • Likes the options in the feature filter, but the subject tags need to be radio buttons (he selected several, which didn’t match themes close to what he was looking for).
  • Also tried using the browser back button to go back.
  • Suggested that results should show up as soon as you click feature filter options. We could change this interaction, but it currently matches the themes admin page.
  • Was initially confused by install vs. install & preview – I asked him to clarify and he correctly figured it out.
  • Got a little confused about what content was on the site, and the options that are specific to the different themes.
  • Would generally prefer being able to edit the site inline, maybe even with drag & drop for the layout. See Customize Posts.
  • Thinks that it would be better if everything in the screenshot were available in the preview once it’s on your site.
  • Had a lot of trouble getting to the front end at the end of the test.
  • “The menu (customizer navigation options) was hard to navigate”. This is because themes are improperly using panels and adding too many options, primarily.
  • Anyone who doesn’t agree with decisions, not options should watch this video. He does a great job of expressing how it’s nice to be able to change things, but when there are too many options it becomes overwhelming and actually makes it harder to change anything. Throughout the test, he ultimately has much more success with themes like Twenty Fifteen that generally align with core philosophies than option-heavy themes, even though his initial reaction was in favor of lots of options. He was also able to verbalize and identify the problem in the end, as an end user trying WordPress for the first time.

Testing Steps

Here are the steps I gave them during testing:

  • Throughout the test, talk through everything you’re thinking. Why you’re clicking on something, making a choice, whether you have trouble finding something, etc. Also read the instructions out loud, step by step.
  • Visit http://annenbergdl.org/test/wp-admin to log in, with username “tester” and password “ascj200”.
  • Customize your site. Play with a few of the options and save your changes.
  • Change your theme to one of the installed themes. Explain why you chose the one you picked. Live preview the theme, change some of the options, and save and publish your changes.
  • Change your theme again. This time, install a few different themes to try. Look in the different sections, and also try searching until you find a few you like. Preview one of the themes you installed, customize and save it.
  • Change your theme again, and preview each of the other themes you installed. Use the buttons at the bottom of the screen to preview the theme on different sized devices. Navigate to different pages in the preview to compare them. Pick your favorite theme from all of the ones you’ve tried, customize it and publish your changes.
  • Visit your site. Discuss any areas that you had trouble with during the test and any areas you think could be improved.

Special thanks to the USC Annenberg Digital Lounge staff for testing this new feature!

#customize, #theme, #theme-install, #theme-switcher