WCUS 2019 Contributor Day

The Test Team gathered at the table for Contributor Day at WCUS 2019. We talked about the various ways people can test and where to go to test things out. After this, we split into groups and wrote usability testing scripts. Once scripts were written, we took some time to install a local WordPress setup and began testing Gutenberg issues labeled with needs-testing.

The team included

Test table at WCUS 2019 Contributor Day

@dll416, @suziwilson, @dickmoomaw, Allie Brock, @brezocordero, @chrisp, @c3zh, @christopher-pollock, @royho, @mapk, @jg-visual

Issues/PRs that were tested

Scripts created for usability testing

  1. Create a new post.
  2. Title the post: “Gutenberg Recipe”
  3. Add a text block describing the recipe:
    “This is a test recipe of a delicious pie. We love pies as they are tart and sweet at the same time.”
  4. Add a two-column block for the ingredients.
  5. In the first column, type “Crust”. Then add a bulleted list:
    • 1 ½ cup cookie crumbs
    • ¼ cup sugar
  6. In the second column, type “Curd”. Then add a bulleted list:
    • 1 (12 ounce) package of cranberries
    • 1/2 cup Sugar
    • 1 orange, zested
  7. Add a single-column block titled “Directions”
  8. In the block add a numbered list:
    1. Preheat oven to 350 degrees F. Spray a 9-inch two-piece tart pan with cooking spray.
    2. In a medium bowl, stir together cookie crumbs, sugar, stevia, salt, and butter. Press mixture evenly in bottom and up sides of tart pan. Bake until set, about 15 minutes.
    3. In a medium saucepan, combine cranberries, sugar, stevia, and orange zest and juice. Bring to a simmer over medium-high heat and cook until cranberries burst, about 8 minutes.

  1. Go to Posts->Add New
  2. Add a title to the post
  3. Click + to add a 3 columns block
  4. On the first column add a header block and an image block
  5. On the second column, add an image
  6. On the third column add text
  7. Resize the image on the second column to make it larger (any size)
  8. Add a fourth column by clicking the Block Navigation icon on the upper left of the editing page.
  9. Add a slideshow in the fourth column.
  10. Click Preview to see how your post appears
  11. Publish the post (private if you want)

These scripts will be used in future usability testing scenarios that help inform the design and development direction in Gutenberg.

Thank you, everyone, for your wonderful contributions today! Keep testing all the things!

#contributors, #wcus

WCUS Gutenberg Testing: Volunteer Feedback

During WCUS, we had a ton of volunteers staff the Gutenberg testing booth (affectionately called the “Gutenbooth.”) A huge thank you to everyone who volunteered their time and ran tests throughout the weekend!

At the end of the weekend, we asked volunteers for some feedback about common trends they saw, along with recommendations for improving the testing process. Here’s the feedback we received:

Did you see any issues come up repeatedly while you were watching people test?

  • Aligning caption citation, converting paragraphs to list, changing block type when clicking into “Write your story”.
  • Typing the quote into the paragraph block, then trying to format it to match. The ability to change the block type was surprising info, as was the quote block having different styles available.
  • People couldn’t find the second quote style.
  • Right aligning quote blocks instead of changing the quote style.
  • People didn’t realize they should use the quote block type.
  • People didn’t know there were two quote styles.
  • May be more issues of the test itself, but most often people didn’t think to make the text style a quote block. They would be looking for font controls. If they did discover blocks sometimes they wouldn’t see the quote block.
  • No one really noticed the second quote style. They were more likely to find the Block settings, so maybe we move the quote style options to the quote > Block > Settings for easier discoverability.
  • People often missed the existence of the Quote block and used two Paragraph blocks instead, and when they did find the Quote block, they often didn’t know they could select different quote styles.
  • It feels like there are too many places for block settings. The icon for the second quote style was unclear. Some people didn’t know it was a quote and made paragraphs and styled those. Developers added inline CSS to make it match the design.
  • Insert block and then edit was not default mindset (at this point). Discovery of the ability to transform wasn’t strong.
  • There were plenty of issues with people not finding block creation or navigation intuitive. block controls cover up the previous block bottom line of text. If it’s a short line of text you might not see it at all. People sometimes got confused thinking their text was gone.
  • Undo/Redo is not intuitively discovered.
  • It wasn’t obvious what was behind the three vertical dots.
  • /slash commands could be interesting to search and use. We may consider a walk-thru wizard for new users to get them acquainted with now the new blocks can work.
  • Some tech glitches.
  • Mostly related to the test setup (e.g. not knowing to switch tabs to Gutenberg / switch tabs back to finish survey).

tl;dr: The two separate quote styles were the biggest pain point, followed by trouble learning the editor and block interface, particularly switching blocks, the ••• menu, and block controls.

Is there anything you think we should change about the test?

  • I suspect part of the issue with caption alignment was due to the task of the test to be imitation, not creation, so I think it leads people to think in terms of alignment, not necessarily style.
  • Make the screenshot not achievable using Gutenberg / Provide people with content and let them do much more free-form style.
  • Maybe written instructions instead of asking people to “mimic” output, because it this is not the way people write content in general, they do not “copy” something.
  • I’d have the sample printed out and set next to the laptop to keep the user from having to swap between windows.
  • Automate screen recording start/stop, one-button reset for survey etc.
  • Yes. I think expecting people to know they should be recreating a block quote without telling them that is what it is, skews the test results a bit. We need to try to replicate a more natural publishing process somehow.
  • No, this was a good example to make people search for options and solutions.

tl;dr: Imitating an existing design make people focus too much on the details and not as much on the editing experience, we need to print out whatever instructions we provide, and better automation.

If you attended WCUS and ran through the Gutenberg usability test, we’d also love your feedback with how you think the test can be improved!

#gutenberg, #wcus

Gutenberg Usability Testing at WordCamp US

Hello folks! We’re going to be running a Gutenberg usability testing station at WordCamp US, and are looking for volunteers to help staff the testing booth throughout the conference.

Volunteers will:

  • Welcome people interested in testing Gutenberg.
  • Set up testers with the Gutenberg test survey, the Gutenberg testing site, and start the provided screen recording app.
  • After the test is complete, save the test recording and reset everything.
  • Chat with testers about their experiences and their thoughts on Gutenberg, taking notes where possible.

Shifts are a half-hour each and you can sign up for as many as you think you can commit to. If you’re interested and available, you can sign up here:

https://docs.google.com/spreadsheets/d/1GxtoZT3ztCf3NzpXbcGEP8zfWczwvceW-tpEDSVZ0Go/edit?usp=sharing

We’ll be using pre-set tests. Folks will have limited time and attention, so we want to get them in and out of the booth in a timely manner. For more information, check out these previous posts from @annaharrison:

Hope to see you there!

+make.wordpress.org/core
+make.wordpress.org/design
+make.wordpress.org/community

#gutenberg, #wcus