Usability Testing for WordPress Customizer Theme Switcher Improvements

I did five usability tests for theme switcher and sliding panels in the customizer.

Summary

  • Users found the customizer very user-friendly overall.
  • Expected clicking on a theme thumbnail to open a preview, not the details modal.
  • Difficulty navigating back using the current 4.2-alpha design (pre-31289.diff).
  • Expected clicking the “Save & Activate” button to not kick them to the front end.
  • “Save & Publish” is clicked a lot more often for sliding panels.
  • I asked two users (note small sample size) which navigation style they preferred, both picked accordion.

Full Videos

  1. 1406491a-usertesting.mp4 (17m5s)
  2. 1408362a-usertesting.mp4 (8m53s)
  3. 1408760a-usertesting.mp4 (7m58s)
  4. 1408936a-usertesting.mp4 (11m15s)
  5. 1414874a-usertesting.mp4 (15m35s)

Site Setup Details

Note how a couple different configurations were tested.

  1. Sliding panels + theme switcher changes: WP trunk r31615 + 31289.diff + 31336.3.diff
  2. Theme switcher changes only: WP trunk r31656 + 31289.diff
  3. Sliding panels only: WP trunk r31662 + 31336.3.diff
  4. Sliding panels + theme switcher changes: WP trunk r31757 + 31289.3.diff + 31336.3.diff + modification to swap the preview & details buttons
  5. Same setup as test 4

Successes

  • One user really liked how you can use arrows to navigate in the theme details modal.
  • People love widgets. 🙂 cc @westonruter (because you do awesome work with widgets)
  • Not sure if it’s just UserTesting.com tester-speak, but the phrase “user-friendly” came up a lot.

(1m47s)

Points of Confusion

  • Expected clicking the theme thumbnail to show a preview (38s)

  • Difficulty navigating back from the themes panel pre-31289.diff (1m42s)

  • Surprised that clicking “Save & Activate” takes you out to the front end (1m21s)

  • Clicked “Save & Publish” after every small change (1m13s)

Oddities

  • Back button takes you back to the dashboard instead of one step back inside the customizer (9s)
  • Tries to save by pressing the Enter key, note: that’s intentionally disabled, probably for a good reason but I don’t know the history (15s)

Bugs

Note: it’s quite possible these bugs are due to my beginner patching skills—I combined patches from tickets 31289 and 31336 for some of the tests.

  • Theme count should’t fall to a new line, Customize button is offset (#)
  • Widgets get stuck half way when first selected (#, #)

Survey Responses

1. Is there anything missing from the customizer that you would like to see added?

  • I think a little more explanation about certain features of the customizer would be great. For example, I think that explanation about what different kinds of menus are and what they entail would be great.
  • There is nothing that I would like to add. It has very good options.
  • The theme page seems a bit confusing. I think it would be helpful to have a screen shot of what the “themes” would look like on their page. The large graphic image doesn’t really help me picture what it’s going to look like on my blog. [Testing note: she was confused at first thinking that featured images displayed in the preview on the right were themes but it was just how the active theme layout happened to look.]
  • Yes, I would like to see more options and maybe an option which allows you to integrate sound when you open up a specific web page.
  • Customizing fonts. For example changing the size of fonts or making fonts BOLD and maybe preset customization themes that would change the entire page by just selecting a theme.

2. After customizing the appearance for a new blog, what is the very next thing you would want to do?

  • I would want to publish a post, or customize the “About” page.
  • I would want to come up with some very good content to add to the blog.
  • I would want to play around with the background and images a bit to make it feel the most like me. I’d also like to be able to easily link to other social media to be able to tell people that I have a new blog. 🙂
  • I would want to view the blog in full screen and take a look at every little detail and make sure it looks the way I want it to.
  • I would like to view the page in its entirety.

3. Have you used WordPress before? If yes, have you used the customizer before?

  • I have not used WordPress before.
  • I personally haven’t used WordPress, but with all the options I would definitely consider it.
  • I’ve used wordpress very little a long time ago, but I use another blogging platform as of now. It definitely looks very different than how I remember it.
  • I have used WordPress before and customizer is one of the reasons why I use WordPress, to customize.
  • Yes I have used WordPress before and have used the customizer.

4. How likely are you to recommend WordPress to a friend or colleague (0=Not at all likely, and 10=Very Likely)?

  • 10
  • 10
  • 7
  • 9
  • 10

Browser & Computer Info

  1. Safari 8.0.3 on Mac OS X 10.10.2
  2. Chrome 40.0.2214 on Windows 8
  3. Chrome 40.0.2214 on Windows 8
  4. Firefox 36.0 on Mac OS X 10.10
  5. Chrome 41.0.2272 on Windows 7 6.1

Tasks Used in Tests 1 to 3

  • You have just setup a new blog, and you would like to customize the appearance including picking a theme, changing colors, adding a header and background image, and setting up at least one widget.
  1. Please sign in.
  2. Go to Appearance > Customize in the dashboard.
  3. Try making a change in each one of the panels. This is the last task, so please take your time looking through all the settings.

Tasks Used in Tests 4 & 5

  • The purpose of this test is to compare two different styles of navigation for blog settings. One uses accordion/dropdown panels and the other uses sliding panels. The goal is to figure out which navigation style works better for you and why.
  1. Go to http://whyistheseasalty.com/wp-login.php and log in.
  2. Go to Appearance > Customize in the dashboard.
  3. Make a change in each one of the panels. Talk about how easy or hard it is to navigate between areas. Save changes at the end.
  4. Overall, this task was: [Rating Scale: Very Difficult to Very Easy]
  5. Exit the customizer and the dashboard and view the site as a normal visitor would view it.
  6. Go to http://whatisinfinity.com/wp-login.php and log in.
  7. Go to Appearance > Customize in the dashboard.
  8. Make a change in each one of the panels. Talk about how easy or hard it is to navigate between areas. Save changes at the end.
  9. Overall, this task was: [Rating Scale: Very Difficult to Very Easy]
  10. Exit the customizer and the dashboard and view the site as a normal visitor would view it.
  11. Comparing the sliding panels in Site A to the accordion/dropdown panels in Site B, which navigation style worked better for you and why? [Verbal Response]
  12. Were the sliding panels in Site A better or worse compared to the drop-down panels in Site B? [Rating Scale: Much Worse to Much Better]

Testing Notes

  • In test 3, the user mistakenly thought the featured images in the “Codium Now” theme were themes at first. That’s possibly a side effect of being thrown into an already-created site, but I can also try to avoid that problem by starting with a theme that looks more like a typical blog, such as Twenty Fifteen.
  • Holy slow loading, Batman. My testing site was fast sometimes but really, really slow other times! (5m28s) I ran the tests with SCRIPT_DEBUG turned on because I found that was the easiest way to patch since patches were based on develop.svn and my test sites were made using core.svn—I am sure there’s a better way to apply these patches for testing to boost my test site performance and I just have to figure out what they are.

[UPDATE: On Testing Notes]

  • I received some feedback about this testing, which is: asking users what they would prefer is a bit flawed. I agree with this feedback. When doing user testing, it’s really important to what what user do and not simply accept what they say they want.
  • It’s also worth noting that these are tests of the patches currently available for tickets 31289 and 31336 and not the complete design, so this testing is just a start.
  • I am always open to learning more about testing best practices!! Critical feedback is welcomed, and I find it especially helpful.

My Observations

  • When asked directly, users said they preferred accordion style navigation. This is interesting because I expected sliding panels to perform better. It could indicate more work is needed on the sliding panels implementation before it’s good enough to replace accordion—these tests can help inform that effort.
  • Fear of losing changes happened more often with sliding panels, evidenced by users clicking the save button a lot more often.
  • I swapped the “Theme Details” and “Live Preview” buttons for tests 4 & 5, and that seemed to reduce confusion but also made the details modal much less noticeable. I think this is a good tradeoff.

My Suggestions

The following suggestions are based on these tests only.

  • Swap “Theme Details” and “Live Preview” button actions in the theme switcher panel.
  • Stay in the customizer after clicking “Save & Activate,” matching the “Save & Publish” button action.
  • Watch out for inconsistencies in themes vs widgets UX—I see them diverging but think they should stay consistent.
  • Keep working on sliding panels vs accordion.

/ht @folletto for the design work, reviewing the tests, and posting some great suggestions in the Slack #core-customize channel.

/cc @celloexpressions @folletto @westonruter (for the widgets notes) @ocean90 (since you commented on ticket 31289)

#accordion-vs-sliding-panels, #customizer, #theme-switcher, #usability-testing-2