During WCUS, we had a ton of volunteers staff the Gutenberg testing booth (affectionately called the “Gutenbooth.”) A huge thank you to everyone who volunteered their time and ran tests throughout the weekend!

At the end of the weekend, we asked volunteers for some feedback about common trends they saw, along with recommendations for improving the testing process. Here’s the feedback we received:

Did you see any issues come up repeatedly while you were watching people test?

• Aligning caption citation, converting paragraphs to list, changing block type when clicking into “Write your story”.
• Typing the quote into the paragraph block, then trying to format it to match. The ability to change the block type was surprising info, as was the quote block having different styles available.
• People couldn’t find the second quote style.
• Right aligning quote blocks instead of changing the quote style.
• People didn’t realize they should use the quote block type.
• People didn’t know there were two quote styles.
• May be more issues of the test itself, but most often people didn’t think to make the text style a quote block. They would be looking for font controls. If they did discover blocks sometimes they wouldn’t see the quote block.
• No one really noticed the second quote style. They were more likely to find the Block settings, so maybe we move the quote style options to the quote > Block > Settings for easier discoverability.
• People often missed the existence of the Quote block and used two Paragraph blocks instead, and when they did find the Quote block, they often didn’t know they could select different quote styles.
• It feels like there are too many places for block settings. The icon for the second quote style was unclear. Some people didn’t know it was a quote and made paragraphs and styled those. Developers added inline CSS to make it match the design.
• Insert block and then edit was not default mindset (at this point). Discovery of the ability to transform wasn’t strong.
• There were plenty of issues with people not finding block creation or navigation intuitive. block controls cover up the previous block bottom line of text. If it’s a short line of text you might not see it at all. People sometimes got confused thinking their text was gone.
• Undo/Redo is not intuitively discovered.
• It wasn’t obvious what was behind the three vertical dots.
• /slash commands could be interesting to search and use. We may consider a walk-thru wizard for new users to get them acquainted with now the new blocks can work.
• Some tech glitches.
• Mostly related to the test setup (e.g. not knowing to switch tabs to Gutenberg / switch tabs back to finish survey).

tl;dr: The two separate quote styles were the biggest pain point, followed by trouble learning the editor and block interface, particularly switching blocks, the ••• menu, and block controls.

Is there anything you think we should change about the test?

• I suspect part of the issue with caption alignment was due to the task of the test to be imitation, not creation, so I think it leads people to think in terms of alignment, not necessarily style.
• Make the screenshot not achievable using Gutenberg / Provide people with content and let them do much more free-form style.
• Maybe written instructions instead of asking people to “mimic” output, because it this is not the way people write content in general, they do not “copy” something.
• I’d have the sample printed out and set next to the laptop to keep the user from having to swap between windows.
• Automate screen recording start/stop, one-button reset for survey etc.
• Yes. I think expecting people to know they should be recreating a block quote without telling them that is what it is, skews the test results a bit. We need to try to replicate a more natural publishing process somehow.
• No, this was a good example to make people search for options and solutions.

tl;dr: Imitating an existing design make people focus too much on the details and not as much on the editing experience, we need to print out whatever instructions we provide, and better automation.

If you attended WCUS and ran through the Gutenberg usability test, we’d also love your feedback with how you think the test can be improved!

#gutenberg, #wcus