The Test team patrols, curates, and tests the WordPress experience. We use a QA mindset to do visual records, using, observing, user research, manual testing, and user testing. We also document and triage top flows.
Help us Analyse the WCUS Gutenberg Usability Videos!
Huge thanks to everyone who participated in usability testing at WCUS. Our next step is to analyze the video footage – and we need your help!
The rest of this post tells you how you can help us with the analysis, even if you are new to usability testing and have never done this before.
If you have any questions comment below or better yet ping us in #core-flow on WordPress Slack.
How to analyze video footage:
- Go to the Volunteer Reviewers tab to fill out your name, Slack username and your WordPress.org username.
- Go to the Survey Results tab to assign yourself at least 5 videos to review. Enter your name in column B (or column D if column B is NOT empty) of the videos you are reviewing.
- Go to the session tab using the ID link listed in column A of the Survey Results tab.
- Review the tester’s survey results before watching the video.
- Follow the video link to watch the session and enter the following information in the highlighted cells:
- Start time – when the tester started using Gutenberg.
- End time – when the tester completed the task in Gutenberg.
- Summarize the participant’s WordPress experience, if they mentioned it. (Some participants were asked or may have mentioned it during their session.)
- Take notes about the user experience in the note section below. Include the timestamp from the video for the note, select note type (Bug, Pain point, or Insight) and your name with each note.
- Return to the Volunteer Reviewers tab to summarize the sessions in column E. Include any notable or/and recurring themes. If you feel a video should be reviewed by the Gutenberg team, specify this in column F.
For your reference, these are the two sample posts users were asked to replicate using Gutenberg: Test 1, Test 2.
Tips on Reviewing:
- When taking notes on the session, please be sure to keep your feedback separate from the participant feedback. Notes on the individual session tabs should be from the participant. If you have suggestions or conclusions of your own, keep those on the reviewer’s tab, and be sure to flag any videos you think worthwhile for additional review.
- Watch for and note any instances where user confuses functionality, i.e. looks for caption when should be attempting to add a block quote, etc. or moments of delight, was there anything they mentioned that they liked?
- Note whether the participant succeed in completing the task or if they gave up. Try to note where or why they stopped.
- In attempting to complete the task, does the participant pause for any length of time? Do they hover the mouse or appear to randomly click around? Does the participant get “lost”? Try to note what they were attempting to find or trying to do during this time.
- Watch for visual cues from the participant like a furrowed brow of concentration, or audible cues like “hmmm…”? If these occur, note what the participant was trying to do at the time. If they solved it, are you able to note how?
- If available, watch through the participant completing the Part 3 end survey. Do their responses provide any additional insight on pain points they experienced during the task? Sometimes participants will provide additional explanation of why they seemed to struggle. Is there anything they say but don’t type into the form?
Special thanks to @betsela, @fuyuko, @annaharrison, and @lynneux for developing and refining these plans.
WCUS Gutenberg Testing: Volunteer Feedback
During WCUS, we had a ton of volunteers staff the Gutenberg testing booth (affectionately called the “Gutenbooth.”) A huge thank you to everyone who volunteered their time and ran tests throughout the weekend!
At the end of the weekend, we asked volunteers for some feedback about common trends they saw, along with recommendations for improving the testing process. Here’s the feedback we received:
Did you see any issues come up repeatedly while you were watching people test?
- Aligning caption citation, converting paragraphs to list, changing block type when clicking into “Write your story”.
- Typing the quote into the paragraph block, then trying to format it to match. The ability to change the block type was surprising info, as was the quote block having different styles available.
- People couldn’t find the second quote style.
- Right aligning quote blocks instead of changing the quote style.
- People didn’t realize they should use the quote block type.
- People didn’t know there were two quote styles.
- May be more issues of the test itself, but most often people didn’t think to make the text style a quote block. They would be looking for font controls. If they did discover blocks sometimes they wouldn’t see the quote block.
- No one really noticed the second quote style. They were more likely to find the Block settings, so maybe we move the quote style options to the quote > Block > Settings for easier discoverability.
- People often missed the existence of the Quote block and used two Paragraph blocks instead, and when they did find the Quote block, they often didn’t know they could select different quote styles.
- It feels like there are too many places for block settings. The icon for the second quote style was unclear. Some people didn’t know it was a quote and made paragraphs and styled those. Developers added inline CSS to make it match the design.
- Insert block and then edit was not default mindset (at this point). Discovery of the ability to transform wasn’t strong.
- There were plenty of issues with people not finding block creation or navigation intuitive. block controls cover up the previous block bottom line of text. If it’s a short line of text you might not see it at all. People sometimes got confused thinking their text was gone.
- Undo/Redo is not intuitively discovered.
- It wasn’t obvious what was behind the three vertical dots.
- /slash commands could be interesting to search and use. We may consider a walk-thru wizard for new users to get them acquainted with now the new blocks can work.
- Some tech glitches.
- Mostly related to the test setup (e.g. not knowing to switch tabs to Gutenberg / switch tabs back to finish survey).
tl;dr: The two separate quote styles were the biggest pain point, followed by trouble learning the editor and block interface, particularly switching blocks, the ••• menu, and block controls.
Is there anything you think we should change about the test?
- I suspect part of the issue with caption alignment was due to the task of the test to be imitation, not creation, so I think it leads people to think in terms of alignment, not necessarily style.
- Make the screenshot not achievable using Gutenberg / Provide people with content and let them do much more free-form style.
- Maybe written instructions instead of asking people to “mimic” output, because it this is not the way people write content in general, they do not “copy” something.
- I’d have the sample printed out and set next to the laptop to keep the user from having to swap between windows.
- Automate screen recording start/stop, one-button reset for survey etc.
- Yes. I think expecting people to know they should be recreating a block quote without telling them that is what it is, skews the test results a bit. We need to try to replicate a more natural publishing process somehow.
- No, this was a good example to make people search for options and solutions.
tl;dr: Imitating an existing design make people focus too much on the details and not as much on the editing experience, we need to print out whatever instructions we provide, and better automation.
If you attended WCUS and ran through the Gutenberg usability test, we’d also love your feedback with how you think the test can be improved!
WordCamp Milano testing the Gutenberg tests
At WordCamp Milano contribution day the tests recently written about here. Thank you to everyone that helped. @emanuelblagonic gets a shout out for helping co-lead on the day.
So, what happened? Well we started by running the tests. They were self run by people who hadn’t done the tests before. We had about 3-4 people doing this. From there others split into testing on another person and also into even testing on mobile.
It’s worth noting this is a brief overview, as the tests are run through bugs will be reported and enhancements made. However, it’s good to report early as this was a test of the tests.
The stats:
- Task C: 2 tests
- Task B: 4 tests
- Task A: 3 tests
Videos collected: we have 9 videos collected from the testing. These are still being processed. One test was on mobile, the rest desktop.
The test changes
Based on the feedback we got a few things were changed:
- Wording was made more explicit.
- The image to make up itself was changed and copy provided as this was big piece of feedback as an issue.
- Task B was easier than task A so we modified this.
- Video uploading was an issue and we now have a Google drive to use.
- The age range was better at 50-60 and then 60+
Gutenberg changes from test
One major thing was adjusted, the block toolbar was put back by the block, over being fixed. This was something previous tests had suggested worked, but in testing it really didn’t. We still have the switch and more A/B testing needs to happen with a wider audience.
Interesting feedback
One ‘food for thought’ feedback was this that when saying ‘make this image’ one user looked for a block that looked like that. It is important to note and see if this is common. We could direct more, but we maybe can hold this to review.
Thanks to everyone that helped run the tests, your work counts and by testing we make Gutenberg a better product.
Don’t forget there is going to be testing happening at WordCamp US. If you are attending, please sign up to help with the tests or come along and discover Gutenberg for yourself. We’ll be sure to post here the findings.
Gutenberg Usability Testing at WordCamp US
Hello folks! We’re going to be running a Gutenberg usability testing station at WordCamp US, and are looking for volunteers to help staff the testing booth throughout the conference.
Volunteers will:
- Welcome people interested in testing Gutenberg.
- Set up testers with the Gutenberg test survey, the Gutenberg testing site, and start the provided screen recording app.
- After the test is complete, save the test recording and reset everything.
- Chat with testers about their experiences and their thoughts on Gutenberg, taking notes where possible.
Shifts are a half-hour each and you can sign up for as many as you think you can commit to. If you’re interested and available, you can sign up here:
https://docs.google.com/spreadsheets/d/1GxtoZT3ztCf3NzpXbcGEP8zfWczwvceW-tpEDSVZ0Go/edit?usp=sharing
We’ll be using pre-set tests. Folks will have limited time and attention, so we want to get them in and out of the booth in a timely manner. For more information, check out these previous posts from @annaharrison:
- Testing Flow in Gutenberg: Instructions for how to contribute to usability testing
- Gutenberg Usability Testing Plan
Hope to see you there!
+make.wordpress.org/core
+make.wordpress.org/design
+make.wordpress.org/community
Testing Flow in Gutenberg: Instructions for how to contribute to usability testing
The new editing experience is now ready to be tested! In this round of usability testing, we invite anyone from the WordPress community to either complete a test, or moderate an observational usability testing session. We have worked hard to make the tests simple to administer, so you can contribute even if you have zero usability testing experience.
How you can help!
- Run through one of the three usability tests yourself (A, B C)
- Help others run through one of the three usability tests
- Recruit your [mother | sister | partner | grandfather | child | puppy | entire social media network] to run though one of the three user tests
What feedback are we looking for?
We welcome all kinds of feedback (our doors are always open, join the conversation in #core-editor on the core WordPress Slack). To make sure we get a diverse range of perspectives, our key focus is to deploy the user tests to a wide, global audience. In particular, we are really keen to reach participants outside of the WordPress community.
About the test
There are three usability tests that you can choose from. The tests are almost identical, except for the complexity of the task that you will be asked to complete. Each test scales in time to take and difficulty. You are free to complete any of these tests, irrespective of whether you have prior WordPress experience or not.
Each usability test consist of three sections:
- Part 1 asks you some general questions about your experience with using WordPress
- Part 2 asks you to complete a task using the Gutenberg editor
- Part 3 asks some questions about your editing experience. You also have the option to upload your screen recording and answer some questions about the video footage in this section
Running the usability test
If you are new to usability testing, please have a quick look at our Guide to observational usability testing. There are some handy tips that will help you run a fun usability testing session. We also have a quick Guide to screen recordings for those who have not done that before.
In order to complete the usability test, you will need to set up a few things:
- Get your hands on a device (laptop, tablet, desktop or mobile device)
- Ensure that you are familiar with the process for how to run a usability test
- Ensure that you know how to do a screen recording on your device
- Open two browser windows, one with the test instructions (either A, B or C) and one with the Gutenberg editor loaded with the Twenty Seventeen theme. Example setup here.
- Follow the instructions in the test
- Optionally, upload your screen recording
- Optionally, write up a blog post about your observations*
How to report your results
There are three ways in which you can report back your user test results:
- You can simply answer all the questions in the test instructions
- You can optionally analyse your screen recording footage by answering the video coding questions in Part 3 of the test instructions
- In addition, you are welcome to write up your test results in a blog post like this one. Share your link with us in the comments below and in #core-editor on the core WordPress Slack
If you are new to usability testing, you may find it confusing to figure out how to provide feedback. A really easy rule to follow is to articulate what the user did when they hit an obstacle. A common mistake for new user testers is to jump straight into solution or resolution mode. While we all love to solve problems (us too!!), doing this during the testing phase is likely to cloud our ability to uncover the underlying causes of the friction. For this reason, we prefer to keep identifying the friction separate from fixing the friction.
Get Involved
Run through one of the three tests yourself, set up a user testing session or join one at a WordCamp near you.
Gutenberg Usability Testing Plan – Feedback Needed
We are getting ready to roll out a new round of usability tests for Gutenberg mid-November. In this next round, we will focus on testing writing flow. We are also very keen to widen the net for participant feedback, including testing with participants who are not current WordPress users.
In order to make this happen we have drafted a usability testing plan – and we need your feedback and suggestions (in the comments below) before the next meeting in #core-flow on Thursday 9 Nov at 18:00 UTC. The next steps will be to test the user test on a small sample set (week of Nov 10th), refine at the #core-meeting on Tuesday 14 Nov 17:00 UTC, and roll out to a wider audience starting from 15th Nov.
We will also have a usability testing section at WCUS, so if you are attending please drop by the and join in!
Proposed Usability Tests
To test the flow of writing, we propose to construct a three part usability test:
- General demographics: including prior WordPress experience, age and device used. This information will help us to segment findings
- The main task: participants will be asked to re-create the post shown in an image. There will be three images to select from, mapping roughly to a beginner, intermediate and advanced level
- Follow up questions: a few questions about the experience of re-creating the post
Participants will be optionally invited to upload their screen recording, and answer a few questions about their video footage.
Testing Script
Question 1: Do you currently use WordPress?
- Yes
- No
Question 2: Would you describe yourself mostly as a…
- Developer
- Designer
- Blogger
- Business Owner
- Other: ________
Question 3 (optional): How old are you?
- Under 18
- 18 – 30
- 31 – 40
- 41 – 50
- 50 – 60
- Over 60
Question 4: What device are you using?
- Mobile phone
- Tablet
- Laptop
- Desktop
Question 5: Let’s get set up!
Check that you have the following items ready as you will need them to complete the task
- Open Gutenberg editor in a new browser window
- Ensure that you have the Twenty Seventeen theme selected
- Open the task image in a new window [ beginner | intermediate | advanced ]
- Start a screen recording
- Remember to talk out loud as you complete the task
Your task is to re-create the page that you see in the image using the Gutenberg editor. Remember to start your screen recorder, and talk out loud as you complete the task! When you are finished, continue on to answer a few questions about your editing experience.
Question 6: Did the task take long or shorter than you expected?
- It took longer than I expected
- It took about the amount of time that I expected
- It took less time than I expected
Question 7: Can you tell us why?
Question 8: Was the task easier or harder than you expected?
- It was harder than I expected
- It was about what I expected
- It was easier than I expected
Question 9: Can you tell us why?
Question 10: Are you more or less likely to use the Gutenberg editor in the future?
- I am not likely to use Gutenberg in the future
- I am unsure
- I am likely to use Gutenberg in the future
Optional section: screen recording analysis
It would help us out a lot if you could upload your screen recording and answer a few questions about your recording
Question 11: Save your screen recording, and upload your file here…
Question 12: How long did it take to complete the task?
Question 13: Was the title added correctly?
- Yes
- No
Question 14: Was the quote added correctly?
- Yes
- No
Question 15: Was the image added correctly?
- Yes
- No
Question 16: Where were the main sources of friction?
Thank you so much for your help!
If you would like to be kept in the loop with the progress of Gutenberg testing, please leave us your email below and we will add you to the Make.WordPress user testing mailing list
Test Setup
The test can be completed by a participant, or used as a run sheet for an observational research session.
In order to complete the test, participants will need to:
- Get their hands on a device (laptop, tablet, desktop or mobile device)
- Ensure that they know how to do a screen recording on the device
- Load up the user test
- Follow the instructions in the test
- Upload the screen recording to the cloud
- Optionally, code the results from the screen recording
- Optionally, write up a blog post and tell us what you found
Reporting usability test results
There are three ways in which you can report back your user test results:
- You can simply answer all the questions in the test instructions. Remember to upload your screen recording at the end
- You can optionally analyse your screen recording footage by answering the optional questions in the final section of the test instructions
- In addition, you are welcome to write up your test results in a blog post
Get Involved
Have an idea on how to improve the usability testing plan? Have your say in the comments below before the next meeting in #core-flow on Thursday 9 Nov at 18:00 UTC. Once we have collected all feedback we will post a link to the test script and open the call for user testing!
Gutenberg usability testing meeting three
Last week we had a usability meeting. The meeting time wasn’t good for everyone, so let’s change the time this week.
When? Tuesday 31st October 19:00 UTC.
Where? wordpress.org Slack #core-flow: the testing channel.
Who should come? Anyone interested in helping test Gutenberg, all skill levels welcome.
Last week we talked about plans for testing. We will continue that conversation this week.
Gutenberg usability testing meeting two
Some work has already been doing after our first meeting, which you can find out about here. It’s time to build on that work, let’s have another meeting.
When? Tuesday 24th October 18:00 UTC
Where? wordpress.org Slack #core-flow: the testing channel
Who should come? Anyone interested in helping test Gutenberg, all skill levels welcome.
Writing my first Gutenberg Block
Inspired by a comment I read a week or so ago, I wrote my first Gutenblock. Here are my impressions of that process.
I initially planned on copying an existing block, but quickly found the example in the blocks directory README.
This is the ultimate result, you should open the example and the result to follow along how things changed.
Getting from one to the other was not too tricky, but did have a handful of surprises and confusing bits.
I started with changing the block name, which caused the first problem – I got an error that it needed to be namespaced. That was easily rectified.
Changing the title was easy enough, but I didn’t know what I should change the category to. I got an error when I left it blank, so I set it to common.
Next up, changing the attribute definition. Changing the type to ‘int’ was a lucky guess, there was nothing to suggest what the valid types are. source confused me for some time, as I didn’t really know what it needed to be. I figured I needed to store the value as a data attribute on the HTML that Star returned, so I wrote a basic version of the Star element, and tried to save it there. It… didn’t work. I tried several variations of this before noticing in the docs that source is optional. So, I tried deleting it, and it worked! That bit of magic made me smile, but I was a bit annoyed I spent so much time on a non-problem.
Writing the setStars() method was easy enough, even without really knowing what the props.setAttributes() call did.
Trying to add the Stars render to the children array gave me a weird error, complaining that the element needed an identifier. I tried adding an id attribute to the HTML that Stars rendered, but that didn’t help. Googling the error message showed me that I needed to set the key attribute, which was kind of weird, and I didn’t understand why it was necessary now, when the example didn’t need it. But it worked, so I moved on.
Adding the input element to children was also easy. I got the same error as before about needing an identifier, so I added a key attribute to that, too. I added all the extra attributes for fun, but realised when defining the min / max attributes that there was no nice way to validate the data being put into the stars attribute.
It was around this time that I discovered the Stars element wasn’t actually rendering, the problem was that I didn’t know how to put the text inside the div element. Did I need to append it? Was there a createTextNode() method I needed to call? As it turns out, it just magically accepted the string, but I only found that out by guessing. (Excuse the hacky math to add the “½”, I got a little carried away.)
One annoyance throughout was that every time I changed the attribute definition, or the output of Stars, Gutenberg threw a warning about the format changing. It seems like this warning would show whenever a plugin upgrade included a block change, though I didn’t test that.
And that’s about the sum of it – I found it to be a generally pleasant experience, it was nice that I could do this all without needing to compile from JSX, but I knew that option was there for if I wanted it. It’s still a bit of a raw experience, but it shows a lot of promise for creating a delightful development experience.
