Tag Archives: usability-testing

Mark Uraine 12:58 pm on June 4, 2019
Tags: , usability-testing   

Gutenberg Usability Testing for May 2019

As noted in last month’s post, I’m conducting a usability test every week on userbrain.net for Gutenberg. These tests are a way to keep track of Gutenberg’s progress and improvements. It also helps reveal any additional UX issues that may not have been evident.

This month focused on image and video blocks. It also involved editing content in relation to those blocks like moving them up and down in the post, and adding captions.

Testing Script

Imagine you are on assignment to write an article for a new travel blog. Luckily, they are sending you to your favorite vacation spot! Your instructions are to choose a topic you think will get people as excited as you are about this place. It’s up to you to pick a focus (food, activities, lodging, etc.).

  1. Log in.
  2. Create a new post.
  3. Give your post a title and add one sentence.
  4. Add two images side-by-side.
  5. Add a YouTube video (https://www.youtube.com/watch?v=-PcrLWkxZSk).
  6. Add a caption to the video you added.
  7. Swap one of your previously added images with another image.
  8. Add an image on the left and some text on the right and a paragraph below it.
  9. Move the video to the bottom of the content.
  10. Once you’re done, publish the post.

Videos

Video 1 (May 6, 2019)

Video 2 (May 13, 2019)

Video 3 (May 20, 2019)

Video 4 (May 28, 2019)

Feedback

Please watch the videos and comment on anything you noticed particularly interesting about how these users used Gutenberg. It would be great to call out points of struggle which can be connected to opportunities, or even points of success that can be positive indicators of improvements.

As a reminder, if you’re interested in helping with usability testing, there’s a number of things you can try:

  • Join the #research channel in slack.
  • You can write a test script that can be usability tested for Gutenberg.
  • Or simply watch the videos and leave some feedback below.

Thanks for watching and contributing anywhere you can.

#gutenberg, #usability-testing

Mark Uraine 12:22 am on May 8, 2019
Tags: , usability-testing   

Gutenberg Usability Testing for April 2019

To keep a steady pulse on the user experience in Gutenberg, the Research Team is conducting weekly usability tests on userbrain.net. These tests are meant to be non-moderated tasks for new and experienced WordPress users. One test per week is given and reviewed to help highlight how people might go about completing tasks within Gutenberg.

The tasks will remain the same for each user over the course of two months. This will help compare a few usability tests within a similar range of time throughout the year.

I’ll gather the results at the end of each month to share.

Each user was new to WordPress, but wasn’t screened for demographic data.

Testing script

Imagine your friend has requested that you build a webpage on their site using excerpts from the novel, Moby Dick. They sent you a mockup of what they’d like you to build. And to help you further, they’ve already uploaded all the media and content you’ll need for this. They’ve also created a draft page for you to design called, “Moby Dick Excerpt.”

  1. In a new tab in your browser, review the mockup at: https://cldup.com/SAtxaMGvA0.png
  2. Now go back to: https://wpuxtesting.com/wp-admin
  3. Log in with
    username: XXXX
    password: XXXX
  4. Find and open the “Moby Dick Excerpt” draft page.
  5. Begin editing the content so that it looks like the mockup. Try to get as close as you can to the mockup’s design before completing. All the assets are already added to the site in the Media Library.
  6. Once you’re done, publish the page.

Please make sure to talk through everything out loud as you complete the tasks.

Videos

Video 1 (April 15, 2019)

Video 2 (April 22, 2019)

Video 3 (April 30, 2019)

Feedback

  1. While watching the above videos, did you have any findings or key takeaways?
  2. Do you have any thoughts on how this script might be improved?
  3. Would you like to create a usability test script? If so, share in the comments below.

Caveat: Because these are non-moderated usability tests, sometimes the people going through the tasks treat them as such – simple tasks they need to finish. This can lead to a lack of creativity and exploration on the part of the user and may not reflect real-life situations when building a site.

#gutenberg, #usability-testing

designsimply 10:20 pm on December 19, 2018
Tags: , usability-testing   

Comparing the Classic Editor and Gutenberg

Some videos of people using the Classic Editor were shown in the State of the Word 2018. These videos illustrate some of the pain points found in past usability testing, studies, surveys, and feedback. We added a few tests of Gutenberg with same tasks and are sharing them here.

Demographics

  • Classic Editor: 10 participants (6 female, 4 male), 3 countries, age range: 20-39, web expertise: average or advanced, WP experience: novice or average, NPS: 6.4.
  • Gutenberg: 3 participants (2 female, 1 male), 2 countries, age range: 19-36, web expertise: average or advanced, WP experience: novice or average, NPS: 8.3.

Tasks

  • You are on assignment to write an article for a new travel blog. Luckily, they are sending you to your favorite vacation spot! Your instructions are to choose a topic you think will get people as excited as you are about this place. It’s up to you to pick a focus (food, activities, lodging, etc.). Start at https://plastic-fox.jurassic.ninja/wp-admin/post-new.php
  1. Log in with username demo and password pass037538
  2. Give your post a title and add one sentence.
  3. Add two images side-by-side. 
  4. Add a YouTube video.
  5. Click “Preview” to check your work so far.
  6. Go back to editing and add a caption to the video you added before.
  7. Add an image on the left and some text on the right and a paragraph below it.
  8. Move the video or images from one part of the content to another (i.e. from the top to the bottom or vice versa).
  9. You love to close each article with the same snippet. Open the following link, copy the HTML, and add it to the end of your post.
1 <img draggable=”false” class=”emoji” alt=”<img draggable=”false” class=”emoji” alt=”” src=”https://s0.wp.com/wp-content/mu-plugins/wpcom-smileys/twemoji/2/svg/1f44b.svg”>” src=”https://s.w.org/images/core/emoji/2.3/svg/1f44b.svg”>
  1. Publish the post!

Classic Editor Highlight Reels

Gutenberg Highlight Reels

Although the questions were initially created for the Classic Editor, the same tasks in Gutenberg were compared.

Net Promoter Score (NPS)

We asked, “based on your experience today, how likely are you to recommend this editor to a friend or colleague? (0=Not at all likely, and 10=Very Likely)?” after each test. Users rated the Classic Editor experience at an average of 6.4 and the Gutenberg experience 8.3.

Links to Full Videos

  1. Session 1 — VideoPress
  2. Session 2 — VideoPress
  3. Session 3 — VideoPress
  4. Session 4 — VideoPress
  5. Session 5 — VideoPress
  6. Session 6 — VideoPress
  7. Session 7 — VideoPress
  8. Session 8 — VideoPress
  9. Session 9 — VideoPress
  10. Session 10 —VideoPress
  11. Session 1 — VideoPress
  12. Session 2 — VideoPress
  13. Session 3 — VideoPress
    (One task was modified here accidentally: “Add two images side-by-side in a gallery” and this little bit helped the user discover how to complete the task.)

There’s More!

A sitebuilding study is under way, and you can volunteer to get involved or follow along on the Make WordPress Design blog.

#gutenberg, #usability-testing

Anna Harrison 9:40 pm on November 6, 2017
Tags: , core-meeting, , usability-testing   

Gutenberg Usability Testing Plan – Feedback Needed

We are getting ready to roll out a new round of usability tests for Gutenberg mid-November. In this next round, we will focus on testing writing flow. We are also very keen to widen the net for participant feedback, including testing with participants who are not current WordPress users.

In order to make this happen we have drafted a usability testing plan – and we need your feedback and suggestions (in the comments below) before the next meeting in #core-flow on Thursday 9 Nov at 18:00 UTC. The next steps will be to test the user test on a small sample set (week of Nov 10th), refine at the #core-meeting on  Tuesday 14 Nov 17:00 UTC, and roll out to a wider audience starting from 15th Nov.

We will also have a usability testing section at WCUS, so if you are attending please drop by the and join in!

Proposed Usability Tests

To test the flow of writing, we propose to construct a three part usability test:

  1. General demographics:  including prior WordPress experience, age and device used. This information will help us to segment findings
  2. The main task: participants will be asked to re-create the post shown in an image. There will be three images to select from, mapping roughly to a beginner, intermediate and advanced level
  3. Follow up questions: a few questions about the experience of re-creating the post

Participants will be optionally invited to upload their screen recording, and answer a few questions about their video footage.

Testing Script

Question 1: Do you currently use WordPress?

  • Yes
  • No

Question 2: Would you describe yourself mostly as a…

  • Developer
  • Designer
  • Blogger
  • Business Owner
  • Other: ________

Question 3 (optional): How old are you?

  • Under 18
  • 18 – 30
  • 31 – 40
  • 41 – 50
  • 50 – 60
  • Over 60

Question 4: What device are you using?

  • Mobile phone
  • Tablet
  • Laptop
  • Desktop

Question 5: Let’s get set up!

Check that you have the following items ready as you will need them to complete the task

  • Open Gutenberg editor in a new browser window
  • Ensure that you have the Twenty Seventeen theme selected
  • Open the task image in a new window [ beginner | intermediate | advanced ]
  • Start a screen recording
  • Remember to talk out loud as you complete the task

Your task is to re-create the page that you see in the image using the Gutenberg editor. Remember to start your screen recorder, and talk out loud as you complete the task! When you are finished, continue on to answer a few questions about your editing experience.

Question 6:  Did the task take long or shorter than you expected?

  • It took longer than I expected
  • It took about the amount of time that I expected
  • It took less time than I expected

Question 7: Can you tell us why?

Question 8: Was the task easier or harder than you expected?

  • It was harder than I expected
  • It was about what I expected
  • It was easier than I expected

Question 9: Can you tell us why?

Question 10: Are you more or less likely to use the Gutenberg editor in the future?

  • I am not likely to use Gutenberg in the future
  • I am unsure
  • I am likely to use Gutenberg in the future

Optional section: screen recording analysis

It would help us out a lot if you could upload your screen recording and answer a few questions about your recording

Question 11:  Save your screen recording, and upload your file here…

Question 12: How long did it take to complete the task?

Question 13: Was the title added correctly?

  • Yes
  • No

Question 14: Was the quote added correctly?

  • Yes
  • No

Question 15: Was the image added correctly?

  • Yes
  • No

Question 16: Where were the main sources of friction?

Thank you so much for your help!

If you would like to be kept in the loop with the progress of Gutenberg testing, please leave us your email below and we will add you to the Make.WordPress user testing mailing list

Test Setup

The test can be completed by a participant, or used as a run sheet for an observational research session.

In order to complete the test, participants will need to:

  1. Get their hands on a device (laptop, tablet, desktop or mobile device)
  2. Ensure that they know how to do a screen recording on the device
  3. Load up the user test
  4. Follow the instructions in the test
  5. Upload the screen recording to the cloud
  6. Optionally, code the results from the screen recording
  7. Optionally, write up a blog post and tell us what you found

Reporting usability test results

There are three ways in which you can report back your user test results:

  1. You can simply answer all the questions in the test instructions. Remember to upload your screen recording at the end
  2. You can optionally analyse your screen recording footage by answering the optional questions in the final section of the test instructions
  3. In addition, you are welcome to write up your test results in a blog post

Get Involved

Have an idea on how to improve the usability testing plan? Have your say in the comments below before the next meeting in #core-flow on Thursday 9 Nov at 18:00 UTC. Once we have collected all feedback we will post a link to the test script and open the call for user testing!

 

#gutenberg, #usability-testing