Gutenberg Usability Testing for July 2019

July usability testing has completed for GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/. I kept the same testing script as last month to be able to make a comparison over the course of these last two months. The users were mostly new to WordPress so expect a novice level of interaction.

Testing script

Imagine you’re writing a blog post about your favorite bakery. Your blog should have a title, some images of desserts, and a couple content sections. Please make sure to talk through everything out loud as you complete the tasks. Let’s get started.

  1. Log in.
  2. Create a new post.
  3. Give your post a title.
  4. Add a Cover blockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. to your post with a message overlaying the image that reads, “The sweetest place on earth”.
  5. Make the Cover block full-width.
  6. Add a paragraph below that reads, “This bakery is committed to using the finest ingredients from all over the world ensuring that quality remains the cornerstone of their business. In fact, quality is the number one ingredient in everything they do.”
  7. Now add a heading for the cupcakes section.
  8. In this section, add an image of cupcakes with some text along the right side of it.
  9. Add a sentence or two below to finish off the cupcake section.
  10. Add a new heading about cookies.
  11. In this section, add a Media+Text block with an image of cookies on the right, and some text on the left side. Make this area full width.
  12. Below that, add a gallery of 5 more images.
  13. Now add a quote you heard someone say about this bakery. Maybe something like, “This bakery is almost as good as Grandma’s house!”
  14. At the bottom, add a horizontal divider.
  15. Finally, include a link to the bakery site. Type the bakery name, and then link it to a website (any website is fine).
  16. Publish your post.

Videos

Video 1 (July 4)

[User adds a Cover block] Okay, now I’m just going to begin typing in it. [completely skips the Cover block’s instructions in the placeholder]

The hardest part was having to figure out what part to click on to add the text.

I’m not really sure how I can add [a block at the bottom of the page]

Video 2 (July 10)

Oh, the text is here. I couldn’t even tell. I wish there was an indication. [referring to where to add text in the page]

So it’s kinda like a puzzle. I can move these elements around.

Video 3 (July 17)

I’m afraid you’re using terminology that doesn’t mean that much to me.

I’m going to give up on this. [referring to the Media Library and adding items to gallery]

This isn’t obvious or intuitive at all.

Video 4 (July 24)

Feedback

As with all the prior videos being shared, please leave some feedback! What are some patterns that pop out to you? What successes do these users have in navigating and creating a post? What challenges become evident in their flowFlow Flow is the path of screens and interactions taken to accomplish a task. It’s an experience vector. Flow is also a feeling. It’s being unselfconscious and in the zone. Flow is what happens when difficulties are removed and you are freed to pursue an activity without forming intentions. You just do it.

Flow is the actual user experience, in many ways. If you like, you can think of flow as a really comprehensive set of user stories. When you think about user flow, you’re thinking about exactly how a user will perform the tasks allowed by your product.Flow and Context
?

As a reminder, if you’re interested in helping with usability testing, there’s a number of things you can try:

  • Join the #research channel in slackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/..
  • You can write a test script that can be usability tested for Gutenberg.
  • Or simply watch the videos and leave some feedback below.

Thanks for watching and contributing anywhere you can.

#gutenberg, #usability-testing

Gutenberg Usability Testing for June 2019

Continuing the usability tests on GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/, here’s three from the month of June. The insights found in these videos help inform decisions going forward with Gutenberg. Keep in mind that due to the platform being used for these videos, many of the users might be completely new to WordPress.

This month focused on a test similar to the WCEU usability tests that were run. You can read up on those from WCEU here: part one and part two. It focused on building a post using common blocks including media + text, headings, etc.

Testing script

Imagine you’re writing a blog post about your favorite bakery. Your blog should have a title, some images of desserts, and a couple content sections. Please make sure to talk through everything out loud as you complete the tasks. Let’s get started.

  1. Log in.
  2. Create a new post.
  3. Give your post a title.
  4. Add a Cover blockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. to your post with a message overlaying the image that reads, “The sweetest place on earth”.
  5. Make the Cover block full-width.
  6. Add a paragraph below that reads, “This bakery is committed to using the finest ingredients from all over the world ensuring that quality remains the cornerstone of their business. In fact, quality is the number one ingredient in everything they do.”
  7. Now add a heading for the cupcakes section.
  8. In this section, add an image of cupcakes with some text along the right side of it.
  9. Add a sentence or two below to finish off the cupcake section.
  10. Add a new heading about cookies.
  11. In this section, add a Media+Text block with an image of cookies on the right, and some text on the left side. Make this area full width.
  12. Below that, add a gallery of 5 more images.
  13. Now add a quote you heard someone say about this bakery. Maybe something like, “This bakery is almost as good as Grandma’s house!”
  14. At the bottom, add a horizontal divider.
  15. Finally, include a link to the bakery site. Type the bakery name, and then link it to a website (any website is fine).
  16. Publish your post.

Videos

Video 1 (June 5, 2019)

I’m not sure how I’m supposed to make it go on the right.

I want to add a paragraph so I’ll just write some lines.

Video 2 (June 13, 2019)

I’m not sure how to do that but I’m going to give it a try.

I cannot figure out for the life of me how to get the writing exactly right next to it.

Video 3 (June 25, 2019)

This looks good!

How do I add an image?

Feedback

As with all the prior videos being shared, please leave some feedback! What are some patterns that pop out to you? What successes do these users have in navigating and creating a post? What challenges become evident in their flowFlow Flow is the path of screens and interactions taken to accomplish a task. It’s an experience vector. Flow is also a feeling. It’s being unselfconscious and in the zone. Flow is what happens when difficulties are removed and you are freed to pursue an activity without forming intentions. You just do it.

Flow is the actual user experience, in many ways. If you like, you can think of flow as a really comprehensive set of user stories. When you think about user flow, you’re thinking about exactly how a user will perform the tasks allowed by your product.Flow and Context
?

As a reminder, if you’re interested in helping with usability testing, there’s a number of things you can try:

  • Join the #research channel in slackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/..
  • You can write a test script that can be usability tested for Gutenberg.
  • Or simply watch the videos and leave some feedback below.

Thanks for watching and contributing anywhere you can.

#gutenberg, #usability-testing

WCEU usability test results: part one

During WCEU there were a number of usability tests run. For each test, there was a form to follow through and questions. In this first part, five videos will be shared with the answers, then a second post with another five will be made. Thank you every person that both helped run tests and also took part in them. It’s really important to have opportunities to test like this.

As you watch through have a think about a few things:

  • What bugs or enhancements could be made from this?
  • What insights do you gain in the way people use GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/?
  • Any ideas that spring up from this of ways to make the editor even better for people?

Share anything you have as a comment and part two will be along in a few days.

Continue reading

#gutenberg, #usability-testing, #wceu-usability-testing

Gutenberg Usability Testing for May 2019

As noted in last month’s post, I’m conducting a usability test every week on userbrain.net for GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/. These tests are a way to keep track of Gutenberg’s progress and improvements. It also helps reveal any additional UXUX UX is an acronym for User Experience - the way the user uses the UI. Think ‘what they are doing’ and less about how they do it. issues that may not have been evident.

This month focused on image and video blocks. It also involved editing content in relation to those blocks like moving them up and down in the post, and adding captions.

Testing Script

Imagine you are on assignment to write an article for a new travel blog. Luckily, they are sending you to your favorite vacation spot! Your instructions are to choose a topic you think will get people as excited as you are about this place. It’s up to you to pick a focus (food, activities, lodging, etc.).

  1. Log in.
  2. Create a new post.
  3. Give your post a title and add one sentence.
  4. Add two images side-by-side.
  5. Add a YouTube video (https://www.youtube.com/watch?v=-PcrLWkxZSk).
  6. Add a caption to the video you added.
  7. Swap one of your previously added images with another image.
  8. Add an image on the left and some text on the right and a paragraph below it.
  9. Move the video to the bottom of the content.
  10. Once you’re done, publish the post.

Videos

Video 1 (May 6, 2019)

Video 2 (May 13, 2019)

Video 3 (May 20, 2019)

Video 4 (May 28, 2019)

Feedback

Please watch the videos and comment on anything you noticed particularly interesting about how these users used Gutenberg. It would be great to call out points of struggle which can be connected to opportunities, or even points of success that can be positive indicators of improvements.

As a reminder, if you’re interested in helping with usability testing, there’s a number of things you can try:

  • Join the #research channel in slackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/..
  • You can write a test script that can be usability tested for Gutenberg.
  • Or simply watch the videos and leave some feedback below.

Thanks for watching and contributing anywhere you can.

#gutenberg, #usability-testing

Gutenberg Usability Testing for April 2019

To keep a steady pulse on the user experience in GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/, the Research Team is conducting weekly usability tests on userbrain.net. These tests are meant to be non-moderated tasks for new and experienced WordPress users. One test per week is given and reviewed to help highlight how people might go about completing tasks within Gutenberg.

The tasks will remain the same for each user over the course of two months. This will help compare a few usability tests within a similar range of time throughout the year.

I’ll gather the results at the end of each month to share.

Each user was new to WordPress, but wasn’t screened for demographic data.

Testing script

Imagine your friend has requested that you build a webpage on their site using excerpts from the novel, Moby Dick. They sent you a mockup of what they’d like you to build. And to help you further, they’ve already uploaded all the media and content you’ll need for this. They’ve also created a draft page for you to design called, “Moby Dick ExcerptExcerpt An excerpt is the description of the blog post or page that will by default show on the blog archive page, in search results (SERPs), and on social media. With an SEO plugin, the excerpt may also be in that plugin’s metabox..”

  1. In a new tab in your browser, review the mockup at: https://cldup.com/SAtxaMGvA0.png
  2. Now go back to: https://wpuxtesting.com/wp-admin
  3. Log in with
    username: XXXX
    password: XXXX
  4. Find and open the “Moby Dick Excerpt” draft page.
  5. Begin editing the content so that it looks like the mockup. Try to get as close as you can to the mockup’s design before completing. All the assets are already added to the site in the Media Library.
  6. Once you’re done, publish the page.

Please make sure to talk through everything out loud as you complete the tasks.

Videos

Video 1 (April 15, 2019)

Video 2 (April 22, 2019)

Video 3 (April 30, 2019)

Feedback

  1. While watching the above videos, did you have any findings or key takeaways?
  2. Do you have any thoughts on how this script might be improved?
  3. Would you like to create a usability test script? If so, share in the comments below.

Caveat: Because these are non-moderated usability tests, sometimes the people going through the tasks treat them as such – simple tasks they need to finish. This can lead to a lack of creativity and exploration on the part of the user and may not reflect real-life situations when building a site.

#gutenberg, #usability-testing

Comparing the Classic Editor and Gutenberg

Some videos of people using the Classic Editor were shown in the State of the WordState of the Word This is the annual report given by Matt Mullenweg, founder of WordPress at WordCamp US. It looks at what we’ve done, what we’re doing, and the future of WordPress. https://wordpress.tv/tag/state-of-the-word/. 2018. These videos illustrate some of the pain points found in past usability testing, studies, surveys, and feedback. We added a few tests of GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ with same tasks and are sharing them here.

Demographics

  • Classic Editor: 10 participants (6 female, 4 male), 3 countries, age range: 20-39, web expertise: average or advanced, WP experience: novice or average, NPS: 6.4.
  • Gutenberg: 3 participants (2 female, 1 male), 2 countries, age range: 19-36, web expertise: average or advanced, WP experience: novice or average, NPS: 8.3.

Tasks

  • You are on assignment to write an article for a new travel blog. Luckily, they are sending you to your favorite vacation spot! Your instructions are to choose a topic you think will get people as excited as you are about this place. It’s up to you to pick a focus (food, activities, lodging, etc.). Start at https://plastic-fox.jurassic.ninja/wp-admin/post-new.php
  1. Log in with username demo and password pass037538
  2. Give your post a title and add one sentence.
  3. Add two images side-by-side. 
  4. Add a YouTube video.
  5. Click “Preview” to check your work so far.
  6. Go back to editing and add a caption to the video you added before.
  7. Add an image on the left and some text on the right and a paragraph below it.
  8. Move the video or images from one part of the content to another (i.e. from the top to the bottom or vice versa).
  9. You love to close each article with the same snippet. Open the following link, copy the HTMLHTML HTML is an acronym for Hyper Text Markup Language. It is a markup language that is used in the development of web pages and websites., and add it to the end of your post.
1<img draggable=”false” class=”emoji” alt=”<img draggable=”false” class=”emoji” alt=”” src=”https://s0.wp.com/wp-content/mu-plugins/wpcom-smileys/twemoji/2/svg/1f44b.svg”>” src=”https://s.w.org/images/core/emoji/2.3/svg/1f44b.svg”>
  1. Publish the post!

Classic Editor Highlight Reels

Gutenberg Highlight Reels

Although the questions were initially created for the Classic Editor, the same tasks in Gutenberg were compared.

Net Promoter Score (NPS)

We asked, “based on your experience today, how likely are you to recommend this editor to a friend or colleague? (0=Not at all likely, and 10=Very Likely)?” after each test. Users rated the Classic Editor experience at an average of 6.4 and the Gutenberg experience 8.3.

Links to Full Videos

  1. Session 1 — VideoPress
  2. Session 2 — VideoPress
  3. Session 3 — VideoPress
  4. Session 4 — VideoPress
  5. Session 5 — VideoPress
  6. Session 6 — VideoPress
  7. Session 7 — VideoPress
  8. Session 8 — VideoPress
  9. Session 9 — VideoPress
  10. Session 10 —VideoPress
  11. Session 1 — VideoPress
  12. Session 2 — VideoPress
  13. Session 3 — VideoPress
    (One task was modified here accidentally: “Add two images side-by-side in a gallery” and this little bit helped the user discover how to complete the task.)

There’s More!

A sitebuilding study is under way, and you can volunteer to get involved or follow along on the Make WordPress Design blog.

#gutenberg, #usability-testing

Gutenberg Usability Testing Plan – Feedback Needed

We are getting ready to roll out a new round of usability tests for GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ mid-November. In this next round, we will focus on testing writing flowFlow Flow is the path of screens and interactions taken to accomplish a task. It’s an experience vector. Flow is also a feeling. It’s being unselfconscious and in the zone. Flow is what happens when difficulties are removed and you are freed to pursue an activity without forming intentions. You just do it.

Flow is the actual user experience, in many ways. If you like, you can think of flow as a really comprehensive set of user stories. When you think about user flow, you’re thinking about exactly how a user will perform the tasks allowed by your product.Flow and Context
. We are also very keen to widen the net for participant feedback, including testing with participants who are not current WordPress users.

In order to make this happen we have drafted a usability testing plan – and we need your feedback and suggestions (in the comments below) before the next meeting in #core-flow on Thursday 9 Nov at 18:00 UTC. The next steps will be to test the user test on a small sample set (week of Nov 10th), refine at the #core-meeting on  Tuesday 14 Nov 17:00 UTC, and roll out to a wider audience starting from 15th Nov.

We will also have a usability testing section at WCUS, so if you are attending please drop by the and join in!

Proposed Usability Tests

To test the flow of writing, we propose to construct a three part usability test:

  1. General demographics:  including prior WordPress experience, age and device used. This information will help us to segment findings
  2. The main task: participants will be asked to re-create the post shown in an image. There will be three images to select from, mapping roughly to a beginner, intermediate and advanced level
  3. Follow up questions: a few questions about the experience of re-creating the post

Participants will be optionally invited to upload their screen recording, and answer a few questions about their video footage.

Testing Script

Question 1: Do you currently use WordPress?

  • Yes
  • No

Question 2: Would you describe yourself mostly as a…

  • Developer
  • Designer
  • Blogger
  • Business Owner
  • Other: ________

Question 3 (optional): How old are you?

  • Under 18
  • 18 – 30
  • 31 – 40
  • 41 – 50
  • 50 – 60
  • Over 60

Question 4: What device are you using?

  • Mobile phone
  • Tablet
  • Laptop
  • Desktop

Question 5: Let’s get set up!

Check that you have the following items ready as you will need them to complete the task

  • Open Gutenberg editor in a new browser window
  • Ensure that you have the Twenty Seventeen theme selected
  • Open the task image in a new window [ beginner | intermediate | advanced ]
  • Start a screen recording
  • Remember to talk out loud as you complete the task

Your task is to re-create the page that you see in the image using the Gutenberg editor. Remember to start your screen recorder, and talk out loud as you complete the task! When you are finished, continue on to answer a few questions about your editing experience.

Question 6:  Did the task take long or shorter than you expected?

  • It took longer than I expected
  • It took about the amount of time that I expected
  • It took less time than I expected

Question 7: Can you tell us why?

Question 8: Was the task easier or harder than you expected?

  • It was harder than I expected
  • It was about what I expected
  • It was easier than I expected

Question 9: Can you tell us why?

Question 10: Are you more or less likely to use the Gutenberg editor in the future?

  • I am not likely to use Gutenberg in the future
  • I am unsure
  • I am likely to use Gutenberg in the future

Optional section: screen recording analysis

It would help us out a lot if you could upload your screen recording and answer a few questions about your recording

Question 11:  Save your screen recording, and upload your file here…

Question 12: How long did it take to complete the task?

Question 13: Was the title added correctly?

  • Yes
  • No

Question 14: Was the quote added correctly?

  • Yes
  • No

Question 15: Was the image added correctly?

  • Yes
  • No

Question 16: Where were the main sources of friction?

Thank you so much for your help!

If you would like to be kept in the loopLoop The Loop is PHP code used by WordPress to display posts. Using The Loop, WordPress processes each post to be displayed on the current page, and formats it according to how it matches specified criteria within The Loop tags. Any HTML or PHP code in the Loop will be processed on each post. https://codex.wordpress.org/The_Loop. with the progress of Gutenberg testing, please leave us your email below and we will add you to the Make.WordPress user testing mailing list

Test Setup

The test can be completed by a participant, or used as a run sheet for an observational research session.

In order to complete the test, participants will need to:

  1. Get their hands on a device (laptop, tablet, desktop or mobile device)
  2. Ensure that they know how to do a screen recording on the device
  3. Load up the user test
  4. Follow the instructions in the test
  5. Upload the screen recording to the cloud
  6. Optionally, code the results from the screen recording
  7. Optionally, write up a blog post and tell us what you found

Reporting usability test results

There are three ways in which you can report back your user test results:

  1. You can simply answer all the questions in the test instructions. Remember to upload your screen recording at the end
  2. You can optionally analyse your screen recording footage by answering the optional questions in the final section of the test instructions
  3. In addition, you are welcome to write up your test results in a blog post

Get Involved

Have an idea on how to improve the usability testing plan? Have your say in the comments below before the next meeting in #core-flow on Thursday 9 Nov at 18:00 UTC. Once we have collected all feedback we will post a link to the test script and open the call for user testing!

 

#gutenberg, #usability-testing