This week’s design meeting agenda for April 27th

This week’s chat is happening today 20:00 UTC.

This weeks agenda is:

  • Half an hour ticket triage in UI/UX queue: bring any tickets you want looked at.
  • Open floor

Feel free to leave a comment and we can discuss anything people want.

See you in the #design channel in Slack

Testing the Gutenberg User Test

On 13th April 2017, I ran a user testing session at the Brisbane WordPress meetup to test out the Gutenberg editor. In the process, I also took the opportunity to reflect on the user test itself. Here are my notes on testing the test.

About the User Test

The user test, created during WordCamp London contributor day (a big shout out to @martinlugton, @j-falk, @lucijanblagonic, and @karmatosed for the hard work) consisted of 18 tasks. In my session, the set of tasks took around 21 minutes to complete on average.

The test was set up as an observational user test. The script gave the participants a task on each page, and asked them to rate the experience on a scale of 1 (bad) to 5 (good) scale. There was also an area for comments to be typed in. A sample question is shown below.

Image of the Gutenberg Test Script

In my session, I ended up with 3.5 hours of video footage, which took about 20 hours to process.


Wording of tasks

The tasks were very clear and unambiguous, perhaps with the exception of Task 2: You want to change the heading to a subheading. Some participants got hung-up on trying to make a heading, and inserting a sub-heading below it, but overall, this was a really minor issue.

Another minor suggestion is to remove the leading phrase “You want to…” from each task description. This is purely to reduce the number of words in the instructions, so that it takes a little less time for participants to process what to do.

Ordering of tasks

After running through a few participants, it became clear that the first few tasks were the hardest in the test. I’d suggest re-ordering the tasks to give a few easy wins for the participants, before throwing them into the more challenging waters.

Suggested task order: Task 1, Tasks 5-9, remaining tasks in order.

Processing video footage

The test was quite long, however, I don’t think this was an issue in the administration of the user tests… only in the video post-processing phase.

To make the video processing a little easier, it would help to make each question a different colour. This would make it a easier to find the right video segment when looking at each participant’s reel:

Image of participant reel

Survey questions

While it is always really tempting to include survey style questions to give us immediate feedback and nice, easy to graph results, I am not a big fan of including this type of question in observational user tests.

The reason for this is that there is often a big disconnect between how a user performs a task, and the rating that they assign to the task.

Looking at this particular testing session, here are some interesting discrepancies:

  • Task 2. One participant failed to complete the task, and gave it a rating of 2. Another completed the task successfully, but rated it as 2 as they got a little confused at the start.
  • Task 3. One participant rates a 5, but was evidently confused about the arrow interactions and movement of blocks. Another participant rates a 3, while providing all kinds of excuses why they failed to complete the task.
  • Task 4. Participant rates a 4 while articulating why they did not like the way the feature worked.
  • Task 11. Participant rates a 2 even though they completed the task successfully.
  • Task 14. Participant struggles, does not complete the task and rates it a 4.
  • Task 15. Participant rates a 5 even though they failed to complete the task.
  • Task 16. Two separate participants rate a 4 even though they fail to complete the task.
  • Task 15. One participant rates a 5 even they did not perform the task correctly. Another participant rates a 2, even though they did complete the task but found it fiddly.

Participants really hate getting things wrong, or disappointing the tester, so will tend to rate things higher. Although it is more time-consuming, I personally find that you get a lot more insight from observing, and really listening

“I would like to see the [heading] options go to H6 to be totally honest… in my experience, I generally probably only use H1 to H2… H3… very rarely will you go down to H6.”

It’s not unusual for participants to claim that they must have a feature, or can’t live without something, only to tell you a moment later that actually, they never use it!


Congratulations to the WordPress folks who put this test together. While I have some nerdy, fine tuning suggestions and changes, it is a really well thought out test and made for a great and enjoyable testing session.

Happy Testing, everyone! If you are new to user testing, or just getting started, check out my tips here.

User Testing the Gutenberg Editor

Development on the new editor for WordPress is steaming ahead towards a big reveal at WordCamp Europe in June this year. Last week, I tested the Gutenberg prototype using the community-created Gutenberg testing script (a big shout out to @martinlugton, @j-falk, @lucijanblagonic, and @karmatosed for the hard work on this!).

Here are the insights from the analysis of the 3.5 hours of user test footage. Being a total UX nerd, I also took the opportunity to test the test.

Summary of findings

As the user test script contained 18 tasks, the detailed observations and video footage make this for a very long post. If you are after a quick summary, the main takeaways are that participants love the overall minimalist and modern direction of Gutenberg (kudos to all the WordPressers!) and they find the general concept of blocks and contextual menus really intuitive to use, confirming earlier user test results.

The main insights from this user test session pertain to perfecting the finer, detailed-level interactions:

  • Keyboard shortcuts, tooltips and search are a must
  • Inserting a new block is really, really hard
  • Inserting a new block is also a little confusing as new blocks lack definition and are not in-focus, making them easily confused for a whitespace
  • Moving blocks with the up and down arrows causes confusion due to displacement
  • Block drag-and-drop was expected
  • No conceptual separation between the block-level and content-level controls, which leads to some confusion
  • Fine-level interactions, such as when context menus pop-up,  are really important to get perfectly right

Although some of the things involve some re-learning, so to speak, once people see how to do something in the new editor, the uptake is almost immediate. I don’t think that being different, and involving a little re-learning, should be seen as a deterrent. It’s an interesting tradeoff. People are really excited about the new and modern changes, and I think they will absorb some re-learning.

User Test Setup

Participants were presented with the following setup and asked to complete the task as set out in the instructions were placed on the left. The changes were to be made in the Gutenberg editor window on the right. On average, participants took around 21 minutes to complete the 18 tasks in the instruction set.

A total of 7 (+2) participants took part in the test. All participants were WordPress users or developers with a wide range of experience. Two of the participants were in groups of two, the first group was a WordPress developer and his novice user client (both aged 30-50); the second group were two experienced WordPress developers (both aged 20-30). The remaining participants were a mix of WordPress developers (the first aged 20-30, the second also aged 20-30), experienced WordPress administrators (the first aged 30-50, the second 20-30), and a WordPress novice (aged 50+).

Detailed Observations

Task 1: You want to edit the heading to “1.0 Is The Starting Number”

Participants really liked the clean, modern feel of the Gutenberg block style editor. There were no observed barriers to using the editor in a general sense: participants were able to get in there and change text to complete task one without any major issues. They commented favourably about the contextual (floating) menus and were able to “get” these quite intuitively.

Yes, I like this… the required tools up the top… so the tools are coming to me instead of me having to track them down.

Contextual menus make things easy.

Task 2: You want to change the heading to a subheading

In the second task, there was some confusion about the meaning of “sub-heading”.  We thought that it perhaps detracted from the task at hand. Some participants focused on creating a heading, and adding a sub-heading rather than changing the heading level of the existing heading, as was intended by the task.

Some participants noted (and were confused) by the H-zero label on the top level heading, while others took a while to understand that H0, H1, H2 corresponded to different levels of headings. One participant made the suggestion that these should be called Heading, and Sub-Heading as these terms have meaning to people who write blog posts (as opposed to designers and developers).

I’m not used to seeing H0… to me, you have H1, and your entire universe lives below that.

Does H1, H2 make sense to people? If you are in publishing or in web, it’s very familiar but for “my mum”, would she get it? NO. Sub-heading, heading are more useful terms than H1 H2.

The interactions in this task began to highlight the importance of getting the fine-level behaviors perfected. As an example, throughout the user test there were cases where clicking in the block, or highlighting text in a block, did not activate the contextual menus. This observation led a lot of participants to look for these menus in other places. We lost a lot of people down the “right click menu” rabbit hole – in my tests, the right click menu was the default Safari browser menu, which turned out to be the source of a lot of confusion.

A second example that can be seen if you look at the footage closely is that participants were not making the distinction between block-level menus (the dropdown that switches between block types) and the content-level menus (the buttons that apply actions to blocks). This is evident in the footage from this task when you look at how many people click on the drop-down list in an attempt to create a sub-heading.

Task 3: You want to move the subheading down below the image

This task proved to be surprisingly difficult. Almost all participants expressed that they were expecting the blocks to be drag and drop enabled. Many intuitively looked to cut and paste as a way to move blocks around (the expectation was that keyboard shortcuts and/or right-click menus would work).

I believe that overall, much of the difficulty in this task is attributed to the cognitive displacement that happened when the block-move arrows were clicked. Participants were quite confused as to what happened when they clicked the up and down arrow. It’s possible that the right kind of animation on block move will help to reduce the displacement and improve the move experience. Adding drag and drop, and keyboard shortcuts for cut and paste are likely also must have features.

Task 4: You want to add a new page heading at the top titled “Numbers”

As a general observation, block insertion was really hard for participants. In this task, participants were unable to insert a paragraph at the top of the page. When prompted, they alluded to wanting some kind of element that would let them insert a new block – either at the top of the page, or more generally, between blocks. A number of participants can be seen exploring the move block arrows to see if there is a way to add a block above/below the current block from this element.

I am looking for something up here [top of the page] to add it [new heading block] in… so is that the only way to add things in? To add them to the bottom and then keep pressing up, up, up?… that’s less than awesome.

For the participants who were able to add in a new block, a number can be seen to not notice that a new block has been inserted – to many it just looked like an extra whitespace, not a new block. Once again, the detailed level interactions of inserting a block, and possibly giving it a stronger default presence and/or animation on insertion would help to improve the issue.

Oh… the new block is not focused when added, so it just looks like an empty whitespace.

Task 5: Make the name Walt Mossberg in the first paragraph bold

This task was completed with no observed issues. There was an expectation that the keyboard shortcuts would work, which they did!

Task 6: Separate the paragraph into two after the ellipsis (…)

This task was completed with no observed issues. Only a few participants noticed that two returns created a new block. One participant tried backspacing to see if the blocks would join back up again – which they did, as expected.

Task 7: You want to link the word Apple to ‘’

This task was completed with no observed issues. Participants expected keyboard shortcuts to work. A few wanted the ability to test the link to see that it was correct.

Task 8: You want to center align the paragraph

This task was completed with no observed issues, however, all participants selected the text, then clicked align. No one noticed that “align” was a block-level action.

Task 9: You want to add the caption for the first image with the content “Snow mountains”

This task highlighted the power of the block editing concept. The interaction with the image block, complete with the placeholder text for the caption was totally intuitive to use.

There was a glitch that was noted by one of the participants in the caption field – when they tried to backspace, the cursor jumped to the end of the word.

Task 10:  You want to align the first image to the right

This task was completed with no observed issues, however, a few participants raised questions about how this would work with themes. A few expressed surprise that the image changed size, but I suspect this was related to the terminology in the task instructions related to “alignment” versus “wrapping.”

I expected it [the image] to keep it’s size. It was simple to do [right align] but had an unexpected result for me

A number of participants noted that the image looked really strange when floated right next to a heading block, as this created a bunch of white space. One suggested that some blocks should be made “full-width”, i.e. not allow images to be floated next to them.

Although this was not directly mentioned by the participants, I suspect that they would have found image resize handles quite natural in this block.

Task 11: You want to make the second image full width

This task should have been quite trivial but raised a couple of interesting observations. Firstly, tooltips are needed on all buttons. Almost all the participants can be seen pausing before clicking on the full-width button. Some directly expressed that a tooltip would have been welcome.

…I wonder how this would work outside of this… how this would work with templating, most templates would have a wrap around the entire contents section… I don’t know how this would work with most themes, or anything with a side-bar as well. So I think that’s a pretty cool feature but I wonder how that would work in real-life templates

Some participants struggled with understanding what full width meant, in the context of the text being presumed as being full width already. I suspect that a slight terminology change could resolve this issue, perhaps full-bleed or hero image (we’d have to test these :-).

I guess that’s full width… it’s funny… I thought that this part down here kind of counted as full width, not the full page… so that was a bit unexpected

Task 12: You want to remove the caption for the second image

This task was completed with no observed issues. One participant pressed backspace twice and deleted the whole image, which raised the issue of keyboard shortcuts for undo.

Task 13: You want to change the quote to your favourite Steve Jobs quote, “It’s more fun to be a pirate than to join the Navy.”

This task proved to be a little hard as participants had trouble finding the quote that they were supposed to edit. For those that found the quote block, changing the text was not difficult.

This task highlights again the lack of distinction between block level and content level contextual menus – you can see participants exploring the block type switcher dropdown and the two quote type buttons and being unsure about what these are.

A number of participants also noticed a glitch in the link part of the quote block. Clicking into the link part of the quote did not always show the link button in its active state, and sometimes, did not display the URL in the add link dialog.

Image of Link and Dialog glitch

Task 14: The date for the Steve Jobs quote is now wrong. It should be 1982

This task was conceptually easy to complete, but without exception, proved to be really fiddly as editing the last digit did not include the edit in the existing link. This caused all manner of fiddling around by participants. Kudos to @spocke for implementing the new link boundary feature coming soon to TinyMCE which will (hopefully) eliminate this user angst forever!

Task 15: You want to update the quote attribution link to ‘’

This task was conceptually easy, but re-surfaced some of the glitches around link insertion. From observations of the user tests, it would be really handy if a first click in the attribution link section selected the entire link. This should be the case with the updated link boundary in the next release of TinyMCE.

Task 16: You want to change how the quote is presented. Change it to the alternate quote style

This task was completed with no major issues – most participants had already noticed the quote style buttons and changed the quote style easily. There were a few participants that did not get the quote style buttons, and proceeded to jump right into the right click menu rabbit hole. You can see some using the font menu to apply italic font, and marking their task as completed with success.

Task 17: You don’t just want to share quotes by Steve Jobs. Create a new quote block with the quote: “Your most unhappy customers are your greatest source of learning” from Bill Gates in 1999

This task should have been completed with no issues, however, it re-surfaced the difficulty that participants had in creating (inserting) a new block.

“Now this is where it will get more difficult as I still can’t see an easy way to get text [a new block] in…”

Participants liked the pre-defined structure of the quote, although found it tricky to click into the attribution section. Some managed by tabbing into the field.

Task 18: You decide that you don’t want this block to be a quote any more. Change this block to a normal paragraph of text

This task was completed with no observed issues. However, only two participants noticed that the conversion from quote to paragraph made them lose the text that was in the attribution field. One of the participants remarked that it may be a good idea to concatenate the text from these fields, rather than lose it.

As a side observation, this task raises the question of whether it makes sense to permit block transformations that can result in data loss? For example, converting a paragraph to a quote makes sense, but does the opposite (quote to paragraph) make sense, or will it lead to unintended angst over data loss?


If you’d like to conduct your own user testing session, follow these setup instructions from the Make WordPress Design team. To learn more about how to conduct user tests, and what pitfalls to avoid when testing, check out my post here.

This week’s design meeting agenda for April 20th

This week’s chat is happening Today 20:00 UTC.

This weeks agenda is:

  • Open floor

Feel free to leave a comment and we can discuss anything people want.

See you in the #design channel in Slack

X-post: Nearby WordPress Events

X-post from Nearby WordPress Events

Editor weekly office hours for April 5th

It’s spring, it’s busy, and everyone is deep in work on the plugin. So we’re making a small change to the format. Instead of weekly meetings, we’re doing weekly office hours. That is, you’ll be guaranteed to find someone in chat, ready to discuss any topic you’d like to bring up.

The office hours start at 17:00 UTC, 19:00 CEST. Here’s a handy link: Wednesday April 5th 17:00 UTC. Yes, we’ve double-checked that.

Since last time:

  • Work is proceeding on the plugin. Follow it in Github, for example this link.
  • Feel free to leave a comment and we can discuss anything people want.

Remember, even if there are now office hours, the #core-editor channel in Slack is open 24/7! 💫

Some results from user testing the prototype Gutenberg Core Editor

During WordCamp London contributor day, Martin Lugton (@martinlugton), Johan Falk (@j-falk) and Lucijan Blagonic (@lucijanblagonic) created a user testing script to test the features of the Gutenberg editor. This is a guest post from them of their findings.

Issues observed:

  • When an image is aligned left or right just above the heading, the heading block encompasses the image block. Could probably be corrected with adding CSS to clear the floated element?
  • Adding links when no text is selected creates a link with no text. Maybe force the user to select something?
  • Dragging and dropping of blocks. In prototype 3, the user can drag and drop blocks. But this is only allowed if a user clicks on the left- or right-hand border of the block. Could we allow users to click and drag a block from its top or bottom border as well? Could we allow users to click and drag a block, just by clicking somewhere in its body?
  • Adding new block at the top of the page. Do we want to allow and facilitate this?
  • When adding new blocks, the user clicked on the HTML link instead of Paragraph.
    Could we rearrange the links in a more meaningful way? (Headings, Paragraphs, Images, Embed, Quote and HTML (last)).

Gutenberg Core Editor User Testing

This is a guest post by Martin Lugton (@martinlugton), Johan Falk (@j-falk) and Lucijan Blagonic (@lucijanblagonic).

During WordCamp London contributor day, we created a user testing script to test the features of the Gutenberg editor. The aim was to build a set of instructions so that people can easily carry out user testing, and quickly share the results.

We prepared 18 questions that cover the functionality of the prototype, and created a form for recording the results. This form populates a spreadsheet that gives an overview of each task and each test. This will allow us to identify which tasks are easy for users to complete, and which ones need some further design work.

We did one test run (with only a handful of questions) and one full run with a test subject (Thanks, @Jip). The whole interview is 24 minutes long and can be viewed here:

Part 1, Part 2 and Part 3.

Tammie Lister (@karmatosed) suggested we add another text field where the user can write down comments. We also added conditional formatting to the results spreadsheet so that tasks which were rated with 3 or lower (out of 5) are marked red. If you want to use this form you can or create your own based on it.

How to do the tests:

  • Open the interview form and resize to 1/3 of the width of the screen
  • Open Editor Prototype and resize to 2/3 the width of the screen
  • Use a screen recording app or the Chrome plugin Screencastify to record the whole screen. The free version of Screencastify version caps at 10 minutes, but you can record multiple videos.

Video showing how to test up the test environment:

How you can get involved in user testing the new editor:

  • Test the prototype yourself:
    • Follow the above instructions (“How to do the tests”)
    • Post a comment on this post with your test videos, and highlighting any key problem areas.
  • Find some users to test the prototype
    • Send them the above instructions, so that they can fill out the form themselves
    • Watch the video of their session. Make a note of anything interesting.

Post a comment on this post linking to your test videos, noting any key insights.

This week’s design meeting agenda for March 30th

This week’s chat is happening Thursday March 30 20:00 UTC.

This weeks agenda is:

  • Open floor

Feel free to leave a comment and we can discuss anything people want.

See you in the #design channel in Slack

This week’s editor meeting agenda for March 29th

This week’s chat is happening Wednesday March 29 19:00 UTC. Edit: Due to an unfortunate typo by yours truly, this weeks meeting was kept very short. More details here.

The agenda is:

  • Work is proceeding fast on the editor plugin. Anyone have something to share or discuss?
  • Anyone have test results to share?
  • With DST here, is there a better time for the weekly meeting?
  • Anything else?

As a reminder, day to day work happens in the GitHub repository. Feel free to leave a comment and we can discuss anything people want.

See you in the #core-editor channel in Slack! ✨