4.7 Beta 1 User Testing in the Customizer and with Twenty Seventeen

I ran four users through some tasks in the customizer with 4.7 beta 1 and Twenty Seventeen. The format was pretty free, but due to time considerations I gave them some level of direction at times. Arim, Kai, and Triet participated in the previous testing for customizer themes, Teresa was using WordPress for the first time.

Arim

Notes:

  • Bug: add menu items not closed when clicking in preview
  • Can’t see whether an external link in a menu actually works
  • No easy path to cropping a currently selected header image, but she found a way.
  • Doesn’t understand a lot of the theme feature filter options (she lists specifics!)
  • Header video (from YouTube) didn’t work for her at first, this was actually a ticket with a patch somewhere regarding broken selective refresh I think (it showed up after a full refresh).
  • Clicks the customizer close button a lot, before realizing that that’s not how you go back.
  • “How would I put something in the second column if I use the two column layout”
  • Bug: she hit an infinite loop in the customizer preview 🙁 Not sure why.
  • Previewing video on device preview is broken in Twenty Seventeen. Also, can we hide the video when entering device preview modes to reflect that it’s not displayed? That could probably be done with CSS.
  • Thought that the logo being “bigger” (relative to the size of the screen/video) on smaller screens was weird.
  • Expected more to change with custom colors in Twenty Seventeen. But thinks it’s more aesthetically pleasing the way it works.
  • After recording, she said she likes the idea of having a video header, and also being able to use a YouTube video was really helpful. She likes that there’s an option to use an .mp4 directly, but feels like she’d have access to a YouTube video more often.
  • Thinks the placeholder is actual text in the custom css textarea.
  • I knew that she knows what CSS is, so I had her try changing something. She wasn’t successful, but felt like she could do it if she cared enough and appreciated that there was a way to do it.

Kai

Notes:

  • Calling a menu “top” implies that it’s shown at the top of the screen and was confusing, since it’s at the bottom of the screen. This led to menu locations confusion.
  • Edit shortcuts: “that’s cool”. Wants them to just be there without clicking.
  • Weird that you can’t edit the titles of dummy content pages. This conflicts with the usefulness of being able to create new pages within the customizer – you have to create a new one, but then also have to figure out how to delete the old one.
  • Also tries static front page well before theme options to get to the front page section options.
  • Bug: something went wrong with save/publishing – but the content came back when she re-opened the customizer. It looks like the front page options got cleared.
  • Add new page under each option is confusing, thinks it’ll add another front page content section.
  • Menu locations are confusing, but “display location” helps a little bit.
  • Prefers edit shortcuts over finding options in the section/panel navigation.
  • Even on this very large screen, the feature filter has far too many options.
  • “They’re all like shopping themes”. She was browsing in the “portfolio” subject.
  • Didn’t see any major issues with the theme installer.
  • Has concerns about not being able to really see a demo of themes with content.

Triet

Notes:

  • Conceptually, icons and shift-click make sense, but the details get confusing. For widgets with a list of categories, it’s unclear that the individual categories are coming from the widget instead of being able to be edited directly. The difference between widgets and menus led to the confusion here, I think.
  • Can’t find static front page section, even when prompted with specific terminology. Granted, this was testing on a random test site with a bunch of imported content that may have added to the confusion.
  • Doesn’t know where to click to edit different content. Preview vs. controls areas are very unclear to him.
  • Was generally confused by the fact that not everything could be edited with the icons, leading to confusion about how to edit other things on the site.
  • The front page sections in Twenty Seventeen made sense once he was directed to them. Again, the content on the test site led to some confusion.
  • Found selecting text as the way to show/hide icons, as opposed to clicking. Wasn’t able to figure it out even with some hints.
  • After he stopped the recording he suggested having outlines on the site for what areas can be edited, with edit icons there and being able to edit in the preview instead of having to switch between the site and the controls “bar”.
  • Bug: Front page section placeholders are shown before changing the sections in Twenty Seventeen.

Teresa

(video is hopefully coming soon, there were some issues with her computer saving the recording)

Notes:

  • I started her from a clean install to test starter content.
  • Her computer/internet was really slow, so keep that in mind. (it was her computer, not WP)
  • She also hit an infinite reload.
  • Can’t find a way to remove content from a front page section in Twenty Seventeen.
  • Looks in static front page for front page sections, eventually find it in theme options but says it was confusing. We should reconsider the placement and naming of these.
  • No issues with browsing/installing themes.

Action Items

Some initial changes we need to make that I noted during the tests:

  • We need a better link for custom CSS help.
  • For CSS, we can add validation if someone pastes in a `<font>` or some sort of markup with a `style=` attribute, and provide a more useful user feedback message that whatever they pasted isn’t actually CSS.
  • The “randomize” options for custom headers shouldn’t show up unless there’s more than one image.
  • Menu item screen options are all enabled by default. They should all be disabled by default. This is a regression, possibly dating to 4.3.
  • Dummy content doesn’t show up in available nav menu items. That fees like a bug.
  • Based on these results, it would make more sense to put the front page section controls in the static front page section than in theme options in Twenty Seventeen.

#4-7, #custom, #customize, #theme, #theme-install, #theme-switcher, #twenty-seventeen

User Testing the New Customizer Themes Experience

I ran four users through the new themes experience in the customizer (#37661) today. I sat with them for each test to take notes and hand them the test steps as they went (written on slips of paper). All of the testers are somewhat technical but not particularly experienced with WordPress; they all used Chrome on Mac with two on laptops and two on desktops (using their own devices). We started out with a clean install, then I added an import of posts from another site before the third test (the featured images didn’t carry over).

I’ve been living with this feature for the past two months, and also witnessed all of the tests directly, so independent analyses of the videos would be much appreciated in the comments.

Summary

There were no major issues that everyone had trouble with. However, each test revealed a few things that we should improve. Most of these items are unrelated to the feature at question – the new theme browser/installer – but they could all use thoughts and in some cases trac tickets. The biggest potential change to the feature at this point would be to add an additional notice about unsaved changes before opening the themes panel – I’m looking for opinions on whether that’s needed here. Also looking for thoughts on whether we need the install without previewing functionality, or if maybe it should only be available in the details modal?

It was interesting to see how different each test was, despite all of the testers being extremely demographically similar. A lot of this also stemmed from the variety of themes that they looked at. Several of the themes violate various theme review team guidelines, and I made a list that I’ll send to someone in the TRT to address, as unfortunately I still haven’t had time to go through and find ticket for all of them (cc @greenshady).

Each of the videos and the key takeaways from each individual test are available after the jump.

Continue reading

#customize, #theme, #theme-install, #theme-switcher

Usability Testing for WordPress Customizer Theme Switcher Improvements

I did five usability tests for theme switcher and sliding panels in the customizer.

Summary

  • Users found the customizer very user-friendly overall.
  • Expected clicking on a theme thumbnail to open a preview, not the details modal.
  • Difficulty navigating back using the current 4.2-alpha design (pre-31289.diff).
  • Expected clicking the “Save & Activate” button to not kick them to the front end.
  • “Save & Publish” is clicked a lot more often for sliding panels.
  • I asked two users (note small sample size) which navigation style they preferred, both picked accordion.

Full Videos

  1. 1406491a-usertesting.mp4 (17m5s)
  2. 1408362a-usertesting.mp4 (8m53s)
  3. 1408760a-usertesting.mp4 (7m58s)
  4. 1408936a-usertesting.mp4 (11m15s)
  5. 1414874a-usertesting.mp4 (15m35s)

Continue reading

#accordion-vs-sliding-panels, #customizer, #theme-switcher, #usability-testing-2