Gutenberg Usability Testing for July 2019

July usability testing has completed for Gutenberg. I kept the same testing script as last month to be able to make a comparison over the course of these last two months. The users were mostly new to WordPress so expect a novice level of interaction.

Testing script

Imagine you’re writing a blog post about your favorite bakery. Your blog should have a title, some images of desserts, and a couple content sections. Please make sure to talk through everything out loud as you complete the tasks. Let’s get started.

  1. Log in.
  2. Create a new post.
  3. Give your post a title.
  4. Add a Cover block to your post with a message overlaying the image that reads, “The sweetest place on earth”.
  5. Make the Cover block full-width.
  6. Add a paragraph below that reads, “This bakery is committed to using the finest ingredients from all over the world ensuring that quality remains the cornerstone of their business. In fact, quality is the number one ingredient in everything they do.”
  7. Now add a heading for the cupcakes section.
  8. In this section, add an image of cupcakes with some text along the right side of it.
  9. Add a sentence or two below to finish off the cupcake section.
  10. Add a new heading about cookies.
  11. In this section, add a Media+Text block with an image of cookies on the right, and some text on the left side. Make this area full width.
  12. Below that, add a gallery of 5 more images.
  13. Now add a quote you heard someone say about this bakery. Maybe something like, “This bakery is almost as good as Grandma’s house!”
  14. At the bottom, add a horizontal divider.
  15. Finally, include a link to the bakery site. Type the bakery name, and then link it to a website (any website is fine).
  16. Publish your post.

Videos

Video 1 (July 4)

[User adds a Cover block] Okay, now I’m just going to begin typing in it. [completely skips the Cover block’s instructions in the placeholder]

The hardest part was having to figure out what part to click on to add the text.

I’m not really sure how I can add [a block at the bottom of the page]

Video 2 (July 10)

Oh, the text is here. I couldn’t even tell. I wish there was an indication. [referring to where to add text in the page]

So it’s kinda like a puzzle. I can move these elements around.

Video 3 (July 17)

I’m afraid you’re using terminology that doesn’t mean that much to me.

I’m going to give up on this. [referring to the Media Library and adding items to gallery]

This isn’t obvious or intuitive at all.

Video 4 (July 24)

Feedback

As with all the prior videos being shared, please leave some feedback! What are some patterns that pop out to you? What successes do these users have in navigating and creating a post? What challenges become evident in their flow?

As a reminder, if you’re interested in helping with usability testing, there’s a number of things you can try:

  • Join the #research channel in slack.
  • You can write a test script that can be usability tested for Gutenberg.
  • Or simply watch the videos and leave some feedback below.

Thanks for watching and contributing anywhere you can.

#gutenberg, #usability-testing

Gutenberg Usability Testing for June 2019

Continuing the usability tests on Gutenberg, here’s three from the month of June. The insights found in these videos help inform decisions going forward with Gutenberg. Keep in mind that due to the platform being used for these videos, many of the users might be completely new to WordPress.

This month focused on a test similar to the WCEU usability tests that were run. You can read up on those from WCEU here: part one and part two. It focused on building a post using common blocks including media + text, headings, etc.

Testing script

Imagine you’re writing a blog post about your favorite bakery. Your blog should have a title, some images of desserts, and a couple content sections. Please make sure to talk through everything out loud as you complete the tasks. Let’s get started.

  1. Log in.
  2. Create a new post.
  3. Give your post a title.
  4. Add a Cover block to your post with a message overlaying the image that reads, “The sweetest place on earth”.
  5. Make the Cover block full-width.
  6. Add a paragraph below that reads, “This bakery is committed to using the finest ingredients from all over the world ensuring that quality remains the cornerstone of their business. In fact, quality is the number one ingredient in everything they do.”
  7. Now add a heading for the cupcakes section.
  8. In this section, add an image of cupcakes with some text along the right side of it.
  9. Add a sentence or two below to finish off the cupcake section.
  10. Add a new heading about cookies.
  11. In this section, add a Media+Text block with an image of cookies on the right, and some text on the left side. Make this area full width.
  12. Below that, add a gallery of 5 more images.
  13. Now add a quote you heard someone say about this bakery. Maybe something like, “This bakery is almost as good as Grandma’s house!”
  14. At the bottom, add a horizontal divider.
  15. Finally, include a link to the bakery site. Type the bakery name, and then link it to a website (any website is fine).
  16. Publish your post.

Videos

Video 1 (June 5, 2019)

I’m not sure how I’m supposed to make it go on the right.

I want to add a paragraph so I’ll just write some lines.

Video 2 (June 13, 2019)

I’m not sure how to do that but I’m going to give it a try.

I cannot figure out for the life of me how to get the writing exactly right next to it.

Video 3 (June 25, 2019)

This looks good!

How do I add an image?

Feedback

As with all the prior videos being shared, please leave some feedback! What are some patterns that pop out to you? What successes do these users have in navigating and creating a post? What challenges become evident in their flow?

As a reminder, if you’re interested in helping with usability testing, there’s a number of things you can try:

  • Join the #research channel in slack.
  • You can write a test script that can be usability tested for Gutenberg.
  • Or simply watch the videos and leave some feedback below.

Thanks for watching and contributing anywhere you can.

#gutenberg, #usability-testing

WCEU usability test results: part one

During WCEU there were a number of usability tests run. For each test, there was a form to follow through and questions. In this first part, five videos will be shared with the answers, then a second post with another five will be made. Thank you every person that both helped run tests and also took part in them. It’s really important to have opportunities to test like this.

As you watch through have a think about a few things:

  • What bugs or enhancements could be made from this?
  • What insights do you gain in the way people use Gutenberg?
  • Any ideas that spring up from this of ways to make the editor even better for people?

Share anything you have as a comment and part two will be along in a few days.

Continue reading

#gutenberg, #usability-testing, #wceu-usability-testing

Gutenberg Usability Testing for May 2019

As noted in last month’s post, I’m conducting a usability test every week on userbrain.net for Gutenberg. These tests are a way to keep track of Gutenberg’s progress and improvements. It also helps reveal any additional UX issues that may not have been evident.

This month focused on image and video blocks. It also involved editing content in relation to those blocks like moving them up and down in the post, and adding captions.

Testing Script

Imagine you are on assignment to write an article for a new travel blog. Luckily, they are sending you to your favorite vacation spot! Your instructions are to choose a topic you think will get people as excited as you are about this place. It’s up to you to pick a focus (food, activities, lodging, etc.).

  1. Log in.
  2. Create a new post.
  3. Give your post a title and add one sentence.
  4. Add two images side-by-side.
  5. Add a YouTube video (https://www.youtube.com/watch?v=-PcrLWkxZSk).
  6. Add a caption to the video you added.
  7. Swap one of your previously added images with another image.
  8. Add an image on the left and some text on the right and a paragraph below it.
  9. Move the video to the bottom of the content.
  10. Once you’re done, publish the post.

Videos

Video 1 (May 6, 2019)

Video 2 (May 13, 2019)

Video 3 (May 20, 2019)

Video 4 (May 28, 2019)

Feedback

Please watch the videos and comment on anything you noticed particularly interesting about how these users used Gutenberg. It would be great to call out points of struggle which can be connected to opportunities, or even points of success that can be positive indicators of improvements.

As a reminder, if you’re interested in helping with usability testing, there’s a number of things you can try:

  • Join the #research channel in slack.
  • You can write a test script that can be usability tested for Gutenberg.
  • Or simply watch the videos and leave some feedback below.

Thanks for watching and contributing anywhere you can.

#gutenberg, #usability-testing

Gutenberg Usability Testing for April 2019

To keep a steady pulse on the user experience in Gutenberg, the Research Team is conducting weekly usability tests on userbrain.net. These tests are meant to be non-moderated tasks for new and experienced WordPress users. One test per week is given and reviewed to help highlight how people might go about completing tasks within Gutenberg.

The tasks will remain the same for each user over the course of two months. This will help compare a few usability tests within a similar range of time throughout the year.

I’ll gather the results at the end of each month to share.

Each user was new to WordPress, but wasn’t screened for demographic data.

Testing script

Imagine your friend has requested that you build a webpage on their site using excerpts from the novel, Moby Dick. They sent you a mockup of what they’d like you to build. And to help you further, they’ve already uploaded all the media and content you’ll need for this. They’ve also created a draft page for you to design called, “Moby Dick Excerpt.”

  1. In a new tab in your browser, review the mockup at: https://cldup.com/SAtxaMGvA0.png
  2. Now go back to: https://wpuxtesting.com/wp-admin
  3. Log in with
    username: XXXX
    password: XXXX
  4. Find and open the “Moby Dick Excerpt” draft page.
  5. Begin editing the content so that it looks like the mockup. Try to get as close as you can to the mockup’s design before completing. All the assets are already added to the site in the Media Library.
  6. Once you’re done, publish the page.

Please make sure to talk through everything out loud as you complete the tasks.

Videos

Video 1 (April 15, 2019)

Video 2 (April 22, 2019)

Video 3 (April 30, 2019)

Feedback

  1. While watching the above videos, did you have any findings or key takeaways?
  2. Do you have any thoughts on how this script might be improved?
  3. Would you like to create a usability test script? If so, share in the comments below.

Caveat: Because these are non-moderated usability tests, sometimes the people going through the tasks treat them as such – simple tasks they need to finish. This can lead to a lack of creativity and exploration on the part of the user and may not reflect real-life situations when building a site.

#gutenberg, #usability-testing

Call for Testing: Gutenberg 5.0

This release includes two new blocks and a way to change the focal point in a cover image block. It also implements a performance improvement when adding/removing blocks, several enhancements, bug fixes, documentation improvements, tooling updates, and removed PHP code now maintained in Core.

The goal of Gutenberg is to simplify the creation of rich pages and posts in WordPress by replacing old custom HTML, CSS, and shortcodes with native Blocks. The Gutenberg plugin is currently the main focus for the testing group and Gutenberg is the new editing experience in WordPress. To help test, please download gutenberg.zip from the releases page or use the button above, install/activate the plugin, and try testing any of the items listed below. You can also see other options for getting setup for testing in the handbook. When testing, use the latest stable release of WordPress (5.0.3) and the Gutenberg 5.0 plugin (download using the links above). All testing is welcome even if it’s just one or two items!
  1. Add an RSS feed using the new RSS block. (7966)
  2. Find the Amazon instant preview link for your favorite book and embed it into a post. (13510)
  3. Add a Cover block and change the focal point for the image. (10925)
  4. Add several blocks in quick succession (it should feel faster than in the previous release). (13067)
  5. Try to create a reusable block as a contributor (you should not be able to). (12378)
  6. Log in as an admin and create a reusable block. Log in as an author or contributor and try to delete it (you should not be able to). (12378)
  7. Log in as an author and try to edit the reusable block created in the previous step (you should not be able to). (12378)
  8. Add a SmugMug photo or slideshow embed. Check to make sure the embed works on the front-end but you do not see a preview in the editor (12961).
  9. Add a gallery block with 4 images and set the columns to 4 then click the pencil icon in the gallery block toolbar and remove an image. Check that the columns option changes to 3. (13423)
  10. Create a new category with no published posts in it. Start a new post and try to assign the new category to it (you should not be able to). (13549)
  11. Add a columns block with an odd number of columns and check that the margins on the right and left work as expected. Try from different themes. (12199)
  12. Add a list block and use the indent and outdent buttons on various list items in the list. (12667)
  13. Accessibility: using a screen reader, go to the More menu and toggle between the views for Top Toolbar, Spotlight Mode, and Fullscreen Mode and check that the change is announced properly. (13385)
  14. Accessibility: using a screen reader, use the block inserter to search for any block and check that the label “Search for a block” is announced correctly followed by the number of results. (13388)
  15. For more, see pull requests merged in the 5.0 milestone.
If you find a new bug, please file it in gutenberg on GitHub. Thank you! Please join us in #core-test on WordPress Slack any time if you have questions about testing!

#5-0, #call-for-testing, #gutenberg

Call for Testing: Gutenberg 4.9

This release includes a lot of tweaks, bug fixes and performance improvements. It includes a new Async Rendering Mode for blocks so it’s important to test it with custom blocks.

The goal of Gutenberg is to simplify the creation of rich pages and posts in WordPress by replacing old custom HTML, CSS, and shortcodes with native Blocks. The Gutenberg plugin is currently the main focus for the testing group and Gutenberg is the new editing experience in WordPress. To help test, please download gutenberg.zip from the releases page or use the button above, install/activate the plugin, and try testing any of the items listed below. You can also see other options for getting setup for testing in the handbook. When testing, use the latest stable release of WordPress (5.0.3) and the Gutenberg 4.9 plugin (download using the links above). All testing is welcome even if it’s just one or two items!
  1. Open a large document and quickly type a short word. Typing should not feel slow, if it does please note the document size you tested. (13056)
  2. Check custom blocks from many available plugins. (13056)
  3. Mark a post as pending review and check that the publish date still says “immediately”. (13178)
  4. Upload an image with a title in its metadata and check that the title in the media library is the same. (13193)
  5. Try applying background colors for the table block stripes. (10611)
  6. Try using the fullscreen mode in different browsers including Edge and notice that you only have a single scrollbar on long posts. (13327)
  7. Try using the alignment toolbar in the Cover and Categories blocks .
  8. For more, see pull requests merged in the 4.8 milestone.
If you find a new bug, please file it in gutenberg on GitHub. Thank you! Please join us in #core-test on WordPress Slack any time if you have questions about testing! #4-9, #call-for-testing, #gutenberg

Call for Testing: Gutenberg 4.8

This release includes 5.0.3 fixes and other small improvements, including another round of performance improvements. It will line up with includes WordPress 5.0.3 fixes but it doesn’t align with it completely and can be first tested as plugin version 4.8.

The goal of Gutenberg is to simplify the creation of rich pages and posts in WordPress by replacing old custom HTML, CSS, and shortcodes with native Blocks. The Gutenberg plugin is currently the main focus for the testing group and Gutenberg is the new editing experience in WordPress. To help test, please download gutenberg.zip from the releases page or use the button above, install/activate the plugin, and try testing any of the items listed below. You can also see other options for getting setup for testing in the handbook. When testing, use the latest stable release of WordPress (5.0.2) and the Gutenberg 4.8 plugin (download using the links above). All testing is welcome even if it’s just one or two items!
  1. Open a large document and quickly type a short word. Typing should not feel slow, if it does please note the document size you tested. (12312)
  2. Check that drag and drop works as expected when adding media before/after blocks and on the media placeholder. (12852)
  3. Open the editor and try to drag an image that is larger than the max file upload limit. You should see a relevant error message. (10224)
  4. Add a File block, upload a large file, click the Copy URL button. Make sure the URL copied does not start with “blob”. Throttle your connection speed if needed. (12499)
  5. Add a gallery of images, set an alignment, convert the gallery into images and verify the alignment is preserved. (12242)
  6. Create a post with multiple paragraphs, ensure you have a large number of plugins active that make the left hand menu long (e.g. 10 custom post types), publish or refresh, attempt to scroll to the end. (12644)
  7. Add a Latest Posts block, add an extra class in Additional CSS Class box, make sure the date position does not change. (12725)
  8. Add a Latest Posts block, click on Align Center, and check that the content centers as expected. (12306)
  9. Add a short sentence to a paragraph block, enable dropcaps from block options, hover over the block and note the hover area is shown around the dropcap and content. (12177)
  10. Create a classic post, insert an image with a caption AND link to the media file, publish the classic post, edit the post in the block editor, convert the classic block to blocks. There should not be a duplicate image. (12315)
  11. Open any post, run this command in the console dispatch action command 'wp.data.dispatch('core/editor').lockPostSaving('lockName');' then click on “Update”. Update button should not do any action if it is locked. (11809)
  12. If you use custom post types with taxonomies, ensure all taxonomies have show_in_rest, and check that you can see more than 10 taxonomies. (12971)
  13. If you use custom posts types with 'publicly_queryable' => false, check that you can save the post without any problems. (12800)
  14. For more, see issues in the 4.8 milestone.
If you find a new bug, please file it in gutenberg on GitHub. Thank you! Block developers: make sure to be aware of deprecations. Please join us in #core-test on WordPress Slack any time if you have questions about testing! #4-8, #call-for-testing, #gutenberg

Comparing the Classic Editor and Gutenberg

Some videos of people using the Classic Editor were shown in the State of the Word 2018. These videos illustrate some of the pain points found in past usability testing, studies, surveys, and feedback. We added a few tests of Gutenberg with same tasks and are sharing them here.

Demographics

  • Classic Editor: 10 participants (6 female, 4 male), 3 countries, age range: 20-39, web expertise: average or advanced, WP experience: novice or average, NPS: 6.4.
  • Gutenberg: 3 participants (2 female, 1 male), 2 countries, age range: 19-36, web expertise: average or advanced, WP experience: novice or average, NPS: 8.3.

Tasks

  • You are on assignment to write an article for a new travel blog. Luckily, they are sending you to your favorite vacation spot! Your instructions are to choose a topic you think will get people as excited as you are about this place. It’s up to you to pick a focus (food, activities, lodging, etc.). Start at https://plastic-fox.jurassic.ninja/wp-admin/post-new.php
  1. Log in with username demo and password pass037538
  2. Give your post a title and add one sentence.
  3. Add two images side-by-side. 
  4. Add a YouTube video.
  5. Click “Preview” to check your work so far.
  6. Go back to editing and add a caption to the video you added before.
  7. Add an image on the left and some text on the right and a paragraph below it.
  8. Move the video or images from one part of the content to another (i.e. from the top to the bottom or vice versa).
  9. You love to close each article with the same snippet. Open the following link, copy the HTML, and add it to the end of your post.
1 <img draggable=”false” class=”emoji” alt=”<img draggable=”false” class=”emoji” alt=”” src=”https://s0.wp.com/wp-content/mu-plugins/wpcom-smileys/twemoji/2/svg/1f44b.svg”>” src=”https://s.w.org/images/core/emoji/2.3/svg/1f44b.svg”>
  1. Publish the post!

Classic Editor Highlight Reels

Gutenberg Highlight Reels

Although the questions were initially created for the Classic Editor, the same tasks in Gutenberg were compared.

Net Promoter Score (NPS)

We asked, “based on your experience today, how likely are you to recommend this editor to a friend or colleague? (0=Not at all likely, and 10=Very Likely)?” after each test. Users rated the Classic Editor experience at an average of 6.4 and the Gutenberg experience 8.3.

Links to Full Videos

  1. Session 1 — VideoPress
  2. Session 2 — VideoPress
  3. Session 3 — VideoPress
  4. Session 4 — VideoPress
  5. Session 5 — VideoPress
  6. Session 6 — VideoPress
  7. Session 7 — VideoPress
  8. Session 8 — VideoPress
  9. Session 9 — VideoPress
  10. Session 10 —VideoPress
  11. Session 1 — VideoPress
  12. Session 2 — VideoPress
  13. Session 3 — VideoPress
    (One task was modified here accidentally: “Add two images side-by-side in a gallery” and this little bit helped the user discover how to complete the task.)

There’s More!

A sitebuilding study is under way, and you can volunteer to get involved or follow along on the Make WordPress Design blog.

#gutenberg, #usability-testing

Call for Testing: Gutenberg 4.7

This Gutenberg release includes several bug fixes and focuses on improving the performance of the block editor, particularly when editing large posts. It will line up with WordPress 5.0.1 and can be first tested as a plugin version 4.7.

The goal of Gutenberg is to simplify the creation of rich pages and posts in WordPress by replacing old custom HTML, CSS, and shortcodes with native Blocks. The Gutenberg plugin is currently the main focus for the testing group and Gutenberg is the new editing experience in WordPress. To help test, please download gutenberg.zip from the releases page or use the button above, install/activate the plugin, and try testing any of the items listed below. You can also see other options for getting setup for testing in the handbook. Currently, you should be testing with WordPress 5.0 stable alongside the Gutenberg 4.7 plugin. This is because 5.1-alpha trunk is currently undergoing some cleanup (normally, testing with trunk is fine). All testing is welcome even if it’s just one or two items!
  1. Create a very long post with thousands of words. (12480)
  2. Open a very long post and begin typing to add new content. (12386, 12460, 12521, 12547)
  3. Create a large post with many different types of blocks. (12384, 12510)
  4. Add an image to the Media & Text block and try to resize the image. (12619)
  5. Add an image by selecting one from the media library, click undo, and make sure the placeholder shows properly at that point. (12567)
  6. Copy and paste content from one block to another. Switch to Code Editor mode. (12543)
  7. Insert several images into a classic block and make sure they are inserted into the correct location where the cursor was positioned. (10509)
  8. Try tabbing through blocks and check to see if it works as expected even after pressing Ctrl + backtick twice to jump to the top toolbar and back to the content area. (12390)
  9. Create, delete, add, then trash a reusable block. After it is trashed checked to see that a “Deleted or unavailable” message appears in its place in the editor. (12345)
  10. Copy content from a word processing app and confirm no spaces are collapsed unexpectedly. (12166)
  11. Edit a paragraph as HTML. Remove a space and insert a line break (use ENTER). Edit visually. You should see a space where you inserted the line break. Edit the content. Edit as HTML. The line break should be replaced by a space. (12166)
  12. Disable a core block, try pasting content that will trigger that block, expect it to. (11057)
  13. Install ACF and make sure hidden meta boxes stay hidden. (12628)
  14. Install a plugin which uses TinyMCE in a textarea in a meta box area, such as Memberlite Elements or Meta Box (and can you think of any others we can test?), then create a post using that content and make sure it saves. (12568)
  15. For more, see issues in the 4.7 milestone.
If you find a new bug, please file it in gutenberg on GitHub. Thank you! Block developers: make sure to be aware of deprecations. Please join us in #core-test on WordPress Slack any time if you have questions about testing!

#4-7, #call-for-testing, #gutenberg