User Testing the Gutenberg Editor

Development on the new editor for WordPress is steaming ahead towards a big reveal at WordCamp Europe in June this year. Last week, I tested the Gutenberg prototype using the community-created Gutenberg testing script (a big shout out to @martinlugton, @j-falk, @lucijanblagonic, and @karmatosed for the hard work on this!).

Here are the insights from the analysis of the 3.5 hours of user test footage. Being a total UX nerd, I also took the opportunity to test the test.

Summary of findings

As the user test script contained 18 tasks, the detailed observations and video footage make this for a very long post. If you are after a quick summary, the main takeaways are that participants love the overall minimalist and modern direction of Gutenberg (kudos to all the WordPressers!) and they find the general concept of blocks and contextual menus really intuitive to use, confirming earlier user test results.

The main insights from this user test session pertain to perfecting the finer, detailed-level interactions:

  • Keyboard shortcuts, tooltips and search are a must
  • Inserting a new block is really, really hard
  • Inserting a new block is also a little confusing as new blocks lack definition and are not in-focus, making them easily confused for a whitespace
  • Moving blocks with the up and down arrows causes confusion due to displacement
  • Block drag-and-drop was expected
  • No conceptual separation between the block-level and content-level controls, which leads to some confusion
  • Fine-level interactions, such as when context menus pop-up,  are really important to get perfectly right

Although some of the things involve some re-learning, so to speak, once people see how to do something in the new editor, the uptake is almost immediate. I don’t think that being different, and involving a little re-learning, should be seen as a deterrent. It’s an interesting tradeoff. People are really excited about the new and modern changes, and I think they will absorb some re-learning.

User Test Setup

Participants were presented with the following setup and asked to complete the task as set out in the instructions were placed on the left. The changes were to be made in the Gutenberg editor window on the right. On average, participants took around 21 minutes to complete the 18 tasks in the instruction set.

A total of 7 (+2) participants took part in the test. All participants were WordPress users or developers with a wide range of experience. Two of the participants were in groups of two, the first group was a WordPress developer and his novice user client (both aged 30-50); the second group were two experienced WordPress developers (both aged 20-30). The remaining participants were a mix of WordPress developers (the first aged 20-30, the second also aged 20-30), experienced WordPress administrators (the first aged 30-50, the second 20-30), and a WordPress novice (aged 50+).

Detailed Observations

Task 1: You want to edit the heading to “1.0 Is The Starting Number”

Participants really liked the clean, modern feel of the Gutenberg block style editor. There were no observed barriers to using the editor in a general sense: participants were able to get in there and change text to complete task one without any major issues. They commented favourably about the contextual (floating) menus and were able to “get” these quite intuitively.

Yes, I like this… the required tools up the top… so the tools are coming to me instead of me having to track them down.

Contextual menus make things easy.

Task 2: You want to change the heading to a subheading

In the second task, there was some confusion about the meaning of “sub-heading”.  We thought that it perhaps detracted from the task at hand. Some participants focused on creating a heading, and adding a sub-heading rather than changing the heading level of the existing heading, as was intended by the task.

Some participants noted (and were confused) by the H-zero label on the top level heading, while others took a while to understand that H0, H1, H2 corresponded to different levels of headings. One participant made the suggestion that these should be called Heading, and Sub-Heading as these terms have meaning to people who write blog posts (as opposed to designers and developers).

I’m not used to seeing H0… to me, you have H1, and your entire universe lives below that.

Does H1, H2 make sense to people? If you are in publishing or in web, it’s very familiar but for “my mum”, would she get it? NO. Sub-heading, heading are more useful terms than H1 H2.

The interactions in this task began to highlight the importance of getting the fine-level behaviors perfected. As an example, throughout the user test there were cases where clicking in the block, or highlighting text in a block, did not activate the contextual menus. This observation led a lot of participants to look for these menus in other places. We lost a lot of people down the “right click menu” rabbit hole – in my tests, the right click menu was the default Safari browser menu, which turned out to be the source of a lot of confusion.

A second example that can be seen if you look at the footage closely is that participants were not making the distinction between block-level menus (the dropdown that switches between block types) and the content-level menus (the buttons that apply actions to blocks). This is evident in the footage from this task when you look at how many people click on the drop-down list in an attempt to create a sub-heading.

Task 3: You want to move the subheading down below the image

This task proved to be surprisingly difficult. Almost all participants expressed that they were expecting the blocks to be drag and drop enabled. Many intuitively looked to cut and paste as a way to move blocks around (the expectation was that keyboard shortcuts and/or right-click menus would work).

I believe that overall, much of the difficulty in this task is attributed to the cognitive displacement that happened when the block-move arrows were clicked. Participants were quite confused as to what happened when they clicked the up and down arrow. It’s possible that the right kind of animation on block move will help to reduce the displacement and improve the move experience. Adding drag and drop, and keyboard shortcuts for cut and paste are likely also must have features.

Task 4: You want to add a new page heading at the top titled “Numbers”

As a general observation, block insertion was really hard for participants. In this task, participants were unable to insert a paragraph at the top of the page. When prompted, they alluded to wanting some kind of element that would let them insert a new block – either at the top of the page, or more generally, between blocks. A number of participants can be seen exploring the move block arrows to see if there is a way to add a block above/below the current block from this element.

I am looking for something up here [top of the page] to add it [new heading block] in… so is that the only way to add things in? To add them to the bottom and then keep pressing up, up, up?… that’s less than awesome.

For the participants who were able to add in a new block, a number can be seen to not notice that a new block has been inserted – to many it just looked like an extra whitespace, not a new block. Once again, the detailed level interactions of inserting a block, and possibly giving it a stronger default presence and/or animation on insertion would help to improve the issue.

Oh… the new block is not focused when added, so it just looks like an empty whitespace.

Task 5: Make the name Walt Mossberg in the first paragraph bold

This task was completed with no observed issues. There was an expectation that the keyboard shortcuts would work, which they did!

Task 6: Separate the paragraph into two after the ellipsis (…)

This task was completed with no observed issues. Only a few participants noticed that two returns created a new block. One participant tried backspacing to see if the blocks would join back up again – which they did, as expected.

Task 7: You want to link the word Apple to ‘www.apple.com’

This task was completed with no observed issues. Participants expected keyboard shortcuts to work. A few wanted the ability to test the link to see that it was correct.

Task 8: You want to center align the paragraph

This task was completed with no observed issues, however, all participants selected the text, then clicked align. No one noticed that “align” was a block-level action.

Task 9: You want to add the caption for the first image with the content “Snow mountains”

This task highlighted the power of the block editing concept. The interaction with the image block, complete with the placeholder text for the caption was totally intuitive to use.

There was a glitch that was noted by one of the participants in the caption field – when they tried to backspace, the cursor jumped to the end of the word.

Task 10:  You want to align the first image to the right

This task was completed with no observed issues, however, a few participants raised questions about how this would work with themes. A few expressed surprise that the image changed size, but I suspect this was related to the terminology in the task instructions related to “alignment” versus “wrapping.”

I expected it [the image] to keep it’s size. It was simple to do [right align] but had an unexpected result for me

A number of participants noted that the image looked really strange when floated right next to a heading block, as this created a bunch of white space. One suggested that some blocks should be made “full-width”, i.e. not allow images to be floated next to them.

Although this was not directly mentioned by the participants, I suspect that they would have found image resize handles quite natural in this block.

Task 11: You want to make the second image full width

This task should have been quite trivial but raised a couple of interesting observations. Firstly, tooltips are needed on all buttons. Almost all the participants can be seen pausing before clicking on the full-width button. Some directly expressed that a tooltip would have been welcome.

…I wonder how this would work outside of this… how this would work with templating, most templates would have a wrap around the entire contents section… I don’t know how this would work with most themes, or anything with a side-bar as well. So I think that’s a pretty cool feature but I wonder how that would work in real-life templates

Some participants struggled with understanding what full width meant, in the context of the text being presumed as being full width already. I suspect that a slight terminology change could resolve this issue, perhaps full-bleed or hero image (we’d have to test these :-).

I guess that’s full width… it’s funny… I thought that this part down here kind of counted as full width, not the full page… so that was a bit unexpected

Task 12: You want to remove the caption for the second image

This task was completed with no observed issues. One participant pressed backspace twice and deleted the whole image, which raised the issue of keyboard shortcuts for undo.

Task 13: You want to change the quote to your favourite Steve Jobs quote, “It’s more fun to be a pirate than to join the Navy.”

This task proved to be a little hard as participants had trouble finding the quote that they were supposed to edit. For those that found the quote block, changing the text was not difficult.

This task highlights again the lack of distinction between block level and content level contextual menus – you can see participants exploring the block type switcher dropdown and the two quote type buttons and being unsure about what these are.

A number of participants also noticed a glitch in the link part of the quote block. Clicking into the link part of the quote did not always show the link button in its active state, and sometimes, did not display the URL in the add link dialog.

Image of Link and Dialog glitch

Task 14: The date for the Steve Jobs quote is now wrong. It should be 1982

This task was conceptually easy to complete, but without exception, proved to be really fiddly as editing the last digit did not include the edit in the existing link. This caused all manner of fiddling around by participants. Kudos to @spocke for implementing the new link boundary feature coming soon to TinyMCE which will (hopefully) eliminate this user angst forever!

Task 15: You want to update the quote attribution link to ‘en.wikiquote.org/wiki/Steve_Jobs’

This task was conceptually easy, but re-surfaced some of the glitches around link insertion. From observations of the user tests, it would be really handy if a first click in the attribution link section selected the entire link. This should be the case with the updated link boundary in the next release of TinyMCE.

Task 16: You want to change how the quote is presented. Change it to the alternate quote style

This task was completed with no major issues – most participants had already noticed the quote style buttons and changed the quote style easily. There were a few participants that did not get the quote style buttons, and proceeded to jump right into the right click menu rabbit hole. You can see some using the font menu to apply italic font, and marking their task as completed with success.

Task 17: You don’t just want to share quotes by Steve Jobs. Create a new quote block with the quote: “Your most unhappy customers are your greatest source of learning” from Bill Gates in 1999

This task should have been completed with no issues, however, it re-surfaced the difficulty that participants had in creating (inserting) a new block.

“Now this is where it will get more difficult as I still can’t see an easy way to get text [a new block] in…”

Participants liked the pre-defined structure of the quote, although found it tricky to click into the attribution section. Some managed by tabbing into the field.

Task 18: You decide that you don’t want this block to be a quote any more. Change this block to a normal paragraph of text

This task was completed with no observed issues. However, only two participants noticed that the conversion from quote to paragraph made them lose the text that was in the attribution field. One of the participants remarked that it may be a good idea to concatenate the text from these fields, rather than lose it.

As a side observation, this task raises the question of whether it makes sense to permit block transformations that can result in data loss? For example, converting a paragraph to a quote makes sense, but does the opposite (quote to paragraph) make sense, or will it lead to unintended angst over data loss?

 

If you’d like to conduct your own user testing session, follow these setup instructions from the Make WordPress Design team. To learn more about how to conduct user tests, and what pitfalls to avoid when testing, check out my post here.