The Test Team helps manage testing and triage across the WordPress ecosystem. They focus on user testing of the editing experience and WordPress dashboard, replicating and documenting bug reports, and supporting a culture of review and triage across the project.
The Test Team gathered at the table for Contributor DayContributor DayContributor Days are standalone days, frequently held before or after WordCamps but they can also happen at any time. They are events where people get together to work on various areas of https://make.wordpress.org/ There are many teams that people can participate in, each with a different focus. https://2017.us.wordcamp.org/contributor-day/https://make.wordpress.org/support/handbook/getting-started/getting-started-at-a-contributor-day/. at WCUS 2019. We talked about the various ways people can test and where to go to test things out. After this, we split into groups and wrote usability testing scripts. Once scripts were written, we took some time to install a local WordPress setup and began testing GutenbergGutenbergThe Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ issues labeled with needs-testing.
Add a text blockBlockBlock is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. describing the recipe: “This is a test recipe of a delicious pie. We love pies as they are tart and sweet at the same time.”
Add a two-column block for the ingredients.
In the first column, type “Crust”. Then add a bulleted list: • 1 ½ cup cookie crumbs • ¼ cup sugar
In the second column, type “Curd”. Then add a bulleted list: • 1 (12 ounce) package of cranberries • 1/2 cup Sugar • 1 orange, zested
Add a single-column block titled “Directions”
In the block add a numbered list:
Preheat oven to 350 degrees F. Spray a 9-inch two-piece tart pan with cooking spray.
In a medium bowl, stir together cookie crumbs, sugar, stevia, salt, and butter. Press mixture evenly in bottom and up sides of tart pan. Bake until set, about 15 minutes.
In a medium saucepan, combine cranberries, sugar, stevia, and orange zest and juice. Bring to a simmer over medium-high heat and cook until cranberries burst, about 8 minutes.
Go to Posts->Add New
Add a title to the post
Click + to add a 3 columns block
On the first column add a headerHeaderThe header of your site is typically the first thing people will experience. The masthead or header art located across the top of your page is part of the look and feel of your website. It can influence a visitor’s opinion about your content and you/ your organization’s brand. It may also look different on different screen sizes. block and an image block
On the second column, add an image
On the third column add text
Resize the image on the second column to make it larger (any size)
Add a fourth column by clicking the Block Navigation icon on the upper left of the editing page.
Add a slideshow in the fourth column.
Click Preview to see how your post appears
Publish the post (private if you want)
These scripts will be used in future usability testing scenarios that help inform the design and development direction in Gutenberg.
Thank you, everyone, for your wonderful contributions today! Keep testing all the things!
During WCUS, we had a ton of volunteers staff the GutenbergGutenbergThe Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ testing booth (affectionately called the “Gutenbooth.”) A huge thank you to everyone who volunteered their time and ran tests throughout the weekend!
At the end of the weekend, we asked volunteers for some feedback about common trends they saw, along with recommendations for improving the testing process. Here’s the feedback we received:
Did you see any issues come up repeatedly while you were watching people test?
Aligning caption citation, converting paragraphs to list, changing blockBlockBlock is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. type when clicking into “Write your story”.
Typing the quote into the paragraph block, then trying to format it to match. The ability to change the block type was surprising info, as was the quote block having different styles available.
People couldn’t find the second quote style.
Right aligning quote blocks instead of changing the quote style.
People didn’t realize they should use the quote block type.
People didn’t know there were two quote styles.
May be more issues of the test itself, but most often people didn’t think to make the text style a quote block. They would be looking for font controls. If they did discover blocks sometimes they wouldn’t see the quote block.
No one really noticed the second quote style. They were more likely to find the Block settings, so maybe we move the quote style options to the quote > Block > Settings for easier discoverability.
People often missed the existence of the Quote block and used two Paragraph blocks instead, and when they did find the Quote block, they often didn’t know they could select different quote styles.
It feels like there are too many places for block settings. The icon for the second quote style was unclear. Some people didn’t know it was a quote and made paragraphs and styled those. Developers added inline CSSCSSCSS is an acronym for cascading style sheets. This is what controls the design or look and feel of a site. to make it match the design.
Insert block and then edit was not default mindset (at this point). Discovery of the ability to transform wasn’t strong.
There were plenty of issues with people not finding block creation or navigation intuitive. block controls cover up the previous block bottom line of text. If it’s a short line of text you might not see it at all. People sometimes got confused thinking their text was gone.
Undo/Redo is not intuitively discovered.
It wasn’t obvious what was behind the three vertical dots.
/slash commands could be interesting to search and use. We may consider a walk-thru wizard for new users to get them acquainted with now the new blocks can work.
Some tech glitches.
Mostly related to the test setup (e.g. not knowing to switch tabs to Gutenberg / switch tabs back to finish survey).
tl;dr: The two separate quote styles were the biggest pain pointPain pointPain points are “places where you know from research or analytics that users are currently getting hung up and have to ask questions, or are likely to abandon the site or app.” — Design for Real Life, followed by trouble learning the editor and block interface, particularly switching blocks, the ••• menu, and block controls.
Is there anything you think we should change about the test?
I suspect part of the issue with caption alignment was due to the task of the test to be imitation, not creation, so I think it leads people to think in terms of alignment, not necessarily style.
Make the screenshot not achievable using Gutenberg / Provide people with content and let them do much more free-form style.
Maybe written instructions instead of asking people to “mimic” output, because it this is not the way people write content in general, they do not “copy” something.
I’d have the sample printed out and set next to the laptop to keep the user from having to swap between windows.
Automate screen recording start/stop, one-button reset for survey etc.
Yes. I think expecting people to know they should be recreating a block quote without telling them that is what it is, skews the test results a bit. We need to try to replicate a more natural publishing process somehow.
No, this was a good example to make people search for options and solutions.
tl;dr: Imitating an existing design make people focus too much on the details and not as much on the editing experience, we need to print out whatever instructions we provide, and better automation.
If you attended WCUS and ran through the Gutenberg usability test, we’d also love your feedback with how you think the test can be improved!
Hello folks! We’re going to be running a GutenbergGutenbergThe Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ usability testing station at WordCampWordCampWordCamps are casual, locally-organized conferences covering everything related to WordPress. They're one of the places where the WordPress community comes together to teach one another what they’ve learned throughout the year and share the joy. Learn more. US, and are looking for volunteers to help staff the testing booth throughout the conference.
Welcome people interested in testing Gutenberg.
Set up testers with the Gutenberg test survey, the Gutenberg testing site, and start the provided screen recording app.
After the test is complete, save the test recording and reset everything.
Chat with testers about their experiences and their thoughts on Gutenberg, taking notes where possible.
Shifts are a half-hour each and you can sign up for as many as you think you can commit to. If you’re interested and available, you can sign up here:
We’ll be using pre-set tests. Folks will have limited time and attention, so we want to get them in and out of the booth in a timely manner. For more information, check out these previous posts from @annaharrison: