Gutenberg Needs Testing Areas

The goal of Gutenberg is to simplify the creation of rich pages and posts in WordPress by replacing old custom HTML, CSS, and shortcodes with native Blocks. Gutenberg development is moving fast right now and there are a few general areas which could use extra help with testing.

If you are here to help, thank you so much! To triage issues that have been reported by others, use the steps for testing bugs in the make/test handbook as a good starting place and guideline. To test in general, see the lists below for ideas about what to test and then either file an issue, post a comment here, or mention your findings in #core-test on WordPress Slack. All testing is welcome!

Some general areas that need testing are:

  1. Everything in the Needs Testing label. Use the steps for testing bugs. 😊
  2. RTL languages.
  3. Roles/Capabilities.
  4. Undo/Redo.

Areas that need testing from a developer perspective:

  1. Everything in the Needs Technical Feedback label. 😊
  2. Backwards Compatibility label.

Testing ideas to help you get started:

  1. Switch to an RTL language and try adding as many different types of blocks as possible.
  2. Switch to an RTL language and test all of the sidebar settings such as categories, tags, changing the publish date, etc.
  3. Create and schedule a new post. Try from a different role.
  4. Create and publish a post with several different blocks using the Editor role.
  5. Create and publish a post with several different blocks using the Author role.
  6. Create and submit a post with several different blocks using the Contributor role.
  7. Add a pending post as one user and try to approve it as another user.
  8. Edit the demo post, make changes, undo and redo the changes as you go.
  9. Edit a page and try making changes and then undoing the changes.
  10. Publish a post with a block type you haven’t used before yet.
  11. Developers: reach out to those with questions at Needs Technical Feedback and chime in if you can confirm a bug or give an alternate code-related solution to a help request. ❤️
  12. If you have an idea about something else to test, add it in the comments!

If you find a new bug, please file it in gutenberg on GitHub. Thank you!

If you are new to testing Gutenberg, the first thing you’ll want to do is get setup for testing with one of the following options:

The stable release is the easiest to test and the local dev env is a bit more technical. Testing any of these will be a great option at this stage. Choose whichever works well for you! If you’re not sure which one to pick, ask and someone will help. 🤓

Any questions? Please ask here or in #core-test on WordPress Slack any time! We would love to see you there.

#call-for-testing, #gutenberg

Call for Testing: Gutenberg 4.0 Pre-release

Gutenberg is currently the main focus for the testing group and it is the new editing experience in WordPress. The goal of Gutenberg is to simplify the creation of rich pages and posts in WordPress by replacing old custom HTML, CSS and shortcodes with native Blocks.

To help test, please download gutenberg.zip from the releases page, or use the button above, and try testing any of the following things:

  1. Change the overlay color on the Cover Image block. (10291)
  2. Try changing the style variation and colors for pullquotes. (9763)
  3. Edit a post already being edited by another user and check that post looking works as expected. (4217)
  4. Align different blocks. Try on mobile. (10085)
  5. Turn on the option to skip the publish confirmation and check that posts publish immediately. (9760)
  6. Exit the code editor using the link at top right. Try from different browser versions. (10061)
  7. Nest some bulleted list items and check that the indented ones are circles. (10358)
  8. Search for terms when adding categories. (10138)
  9. Add a text color to a paragraph with links and see that the links change color too. (10171)
  10. Insert images using the classic block. (10306)
  11. Embed a few videos and check that they look correct at different screen sizes. (10161, 10213)
  12. Paste content from Word or LibreOffice and check that unwanted style tags aren’t showing. (10019)
  13. Use access + z (ctrl + option + z on a Mac / shift + alt + z elsewhere) to remove blocks. (10008)
  14. Bonus: test with Yoast SEO.
  15. Bonus: write and publish a few posts while logged as a user with a non-admin role such as editor, author, or contributor.
  16. What else would you like to see tested?!

If you find a new bug, please file it in gutenberg on GitHub. Thank you!

There are a number of updates in this release and you can check the list on the releases page if you’d like to test more or different stuff. If you are a developer creating custom blocks, check out that list for updates that will affect coding.

Block developers: please keep an eye on deprecations in 4.0. Important note:  wp.editor.RichTextProvider is deprecated and is being from Gutenberg in 4.0.0.

Please join us in #core-test on WordPress Slack any time!

#4-0, #call-for-testing, #editor, #gutenberg

Call for Testing: Gutenberg 3.9.0 RC 2

Some updates have been made based on previous testing and issues filed since RC 1 was released.

To help test, please install RC 2 by downloading gutenberg.zip from the releases page or use the button below.

Help is needed testing anything from the list in the Call for Testing: Gutenberg 3.9 Pre-release post or from the following testing list:

  1. Add a classic block, add a latest posts block, save, refresh. (9968)
  2. Add a Vimeo video embed and check for vertical scrollbars. (9920)
  3. Try exporting reusable blocks in IE 11 or Firefox. (10000)
  4. Enable spotlight mode and verify unselected blocks have a lower opacity. (9951)

Please join us in #core-test on WordPress Slack if you have questions or want to bounce ideas around!

#3-9, #call-for-testing, #editor, #gutenberg

Call for Testing: Gutenberg 3.9 Pre-release

Gutenberg is the new editing experience in WordPress, and it is currently the main focus for the testing group. The goal of Gutenberg is to simplify the creation of rich pages and posts in WordPress by replacing old custom HTML, CSS and shortcodes with native Blocks.

To help test, please download gutenberg.zip from the releases page, or use the button above, and try testing any of the following things:

  1. Select multiple blocks and save them as a reusable block. (9732)
  2. Create, export, import, and modify some reusable blocks. (9788)
  3. Trigger an invalid block warning that says “This block has been modified externally” by modifying the HTML for a paragraph or image block (can you suggest another example for how to trigger an invalid block?) and try out the “Compare Conversion” option in the more menu. (7995)
  4. Find and test the new drag handle in various browsers. (9569)
  5. From a small screen width, test that collapsible toolbar components (such as alignment on paragraph blocks) work as expected. (9687)
  6. Convert a Cover Image block to an Image block and back. (9743)
  7. Open a new post and check whether title field is focused. (9608)
  8. Embed videos with different aspect ratios (16:9, 4:3), check that they look correct. (9500)
  9. Add a gallery as a non-admin role and make sure it loads and is editable. (9875)
  10. Add galleries using IE 11 and Edge. (9622)
  11. Add content using the classic block. (9680)
  12. Find a theme that styles captions for video blocks & make sure styling looks correct. (9609)
  13. Edit a block’s HTML, delete all the content, deselect the block and check that it reverts back to its default state. (9546)
  14. Check that you can insert a Columns block but not a Column (singular) block. (9523)
  15. Use the keyboard shortcut to unlink. (9903)
  16. Bonus: How-to and FAQ blocks with Yoast SEO 8.2  (released Sep 11).
  17. Bonus: write and publish a few posts while logged as a user with a non-admin role such as editor, author, or contributor.
  18. What else would you like to see tested?!

If you find a new bug, please file it in gutenberg on GitHub. Thank you!

Plugin developers: please keep an eye on deprecations in 3.9, especially if you work with colors. Also note that wp.editor.RichTextProvider is deprecated and will be removed from Gutenberg in 4.0.0. Please use wp.data.select( 'core/editor' ) methods instead.

Please join us in #core-test on WordPress Slack any time!

#3-9, #call-for-testing, #editor, #gutenberg

Call for Testing: Gutenberg 3.8 Pre-release

There is a new editing experience in WordPress called Gutenberg—it has been a huge team effort and it is currently the main focus for the testing group.  The goal of Gutenberg is to simplify the creation of rich pages and posts in WordPress by replacing old custom HTML, CSS and shortcodes with native Blocks.

Gutenberg 3.8 included several improvements and bug fixes and had a short release cycle. There are also a couple test cases from 3.7 thrown in. The testing list tries to cover the major updates as a good starting point for testing.

 

To help with pre-release testing, please download the gutenberg.zip file from the releases page (or use the button above) and try testing the following things:

UPDATE: the following items were added since they made it in to the RC after this post was originally published.

  1. In the top editor toolbar, go to More > Fullscreen Mode and test in that mode. (9567)
  2. Check that the block settings menu works as expected after being moved from the top right of the block to the block toolbar. Note the browser version you tested. (9572)
  3. Add several reusable blocks then bulk delete them. (9588)
  4. Bonus: enable editor styles for a theme you are developing and try writing CSS without adding extra specificity to target the edit area. (9008)

Here is the original testing list:

  1. Drag and drop a block and test that it works as expected: the block being dragged is shown as a grey box, the dragging event shows an image that is a clone of the block being dragged. (9311)
  2. Hover over blocks  and see that the block label appears after a short delay. (9197)
  3. Try to upload a filetype not allowed: `.exe`, empty text file, file w/ no extension. (9578)
  4. Center an image and check that it works as expected on the front end. (9509)
  5. Check that block library icons are aligned properly in various browsers/screen sizes. (9497)
  6. Create a table with 2 columns, remove a column, remove another column. (8767)
  7. Create a table, toggle fixed width table cells, check that alignment works. (8767)
  8. Check to make sure categories still work when the Yoast SEO 8.1.1 plugin is active.
  9. Bonus: test with Yoast SEO 8.1.1 which was released Sep 3.
  10. Bonus: test with WooCommerce 3.4.5 which was released Aug 29.
  11. Bonus: test with Jetpack 6.5 which was released Sep 4.
  12. What else? Are there any major plugins you think should be added to this list?

If you find a new bug, please file it in gutenberg on GitHub. Thank you!

Note for testers/developers: some scripts such as wp.api and underscore are no longer used by Gutenberg. Please keep in mind that plugins should always add their dependencies explicitly when registering scripts, even if it’s already there somehow registered by another script. (9163, 9374)

Please feel free to discuss testing practices or issues here in the comments or in #core-test on WordPress Slack any time.

#call-for-testing, #editor, #gutenberg

Call for Testing: Gutenberg 3.7 Pre-release

Gutenberg is the project name of the new editing experience in WordPress. The goal of Gutenberg is to simplify the creation of rich pages and posts in WordPress by replacing old custom HTML, CSS and shortcodes with native Blocks. Creating Gutenberg has been a huge team effort – and right now, the team needs your help with testing!

Today, we have a small window for regression testing Gutenberg 3.7 because the release is planned for tomorrow.

If you would like to help with pre-release testing, please install locally so you are using master or download 3.7 as a zip file and install it to your Plugins folder (make sure to delete older versions of the plugin first!), and try testing some of these updates from 3.7:

  1. Write/edit/publish a few posts using the Unified Toolbar (previously called “Fixed Toolbar”).
  2. Change alignment for various images.
  3. Make sure icons and dropdown arrow look correct on hover for block conversion.
  4. Create an Audio block by drag-and-dropping an audio file.
  5. Search for blocks and check that it works as expected (i.e. start with `/` via the inserter).
  6. Convert a multiline-paragraph into a list.
  7. Add links to text.
  8. Add a link by selecting a URL and clicking the link icon.
  9. Upload an image with a space in the filename and check that the title does not have a hyphen added.
  10. Add a post image in a video block.
  11. Add videos and check that all works as expected.
  12. There were several other improvements and bug fixes in the 3.7 release and this list tries to cover the more major updates so we can have a good starting point for testing. Alternately, just run a general test writing or editing posts and pages as you would normally!

If you find a new bug, file it in gutenberg on GitHub.

Please feel free to discuss testing practices or issues here in the comments or in #core-test on WordPress Slack any time.

#3-7, #call-for-testing, #editor, #gutenberg

Would you like to help test Gutenberg?

We would love to have you! Gutenberg releases are happening every few weeks right now.

To help with general testing after opting in to the “Try Gutenberg” callout in the WordPress 4.9.8 dashboard, please comment here or say hello in #core-test on WordPress Slack.

To help with pre-release testing, or regression testing, first get a testing environment setup.

If you find a new bug, file it in gutenberg on GitHub.

Anyone can jump in to help with issues labeled Needs Testing and if you are a developer then issues labeled Needs Technical Feedback would be a great place to lend a hand. See the bug testing guidelines in the handbook for some tips.

If you want to help and aren’t sure where to start, just ask!

#editor

X-post: Two Factor Authentication on WP.org

X-post from +make.wordpress.org/meta: Two Factor Authentication on WP.org

Help us Analyse the WCUS Gutenberg Usability Videos!

Huge thanks to everyone who participated in usability testing at WCUS. Our next step is to analyze the video footage – and we need your help!

The rest of this post tells you how you can help us with the analysis, even if you are new to usability testing and have never done this before.

If you have any questions comment below or better yet ping us in #core-flow on WordPress Slack.

How to analyze video footage:

  1. Go to the Volunteer Reviewers tab to fill out your name, Slack username and your WordPress.org username.
  2. Go to the Survey Results tab to assign yourself at least 5 videos to review. Enter your name in column B (or column D if column B is NOT empty) of the videos you are reviewing.
  3. Go to the session tab using the ID link listed in column A of the Survey Results tab.
  4. Review the tester’s survey results before watching the video.
  5. Follow the video link to watch the session and enter the following information in the highlighted cells:
  • Start time – when the tester started using Gutenberg.
  • End time – when the tester completed the task in Gutenberg.
  • Summarize the participant’s WordPress experience, if they mentioned it. (Some participants were asked or may have mentioned it during their session.)
  • Take notes about the user experience in the note section below. Include the timestamp from the video for the note, select note type (Bug, Pain point, or Insight) and your name with each note.
  1. Return to the Volunteer Reviewers tab to summarize the sessions in column E. Include any notable or/and recurring themes. If you feel a video should be reviewed by the Gutenberg team, specify this in column F.

For your reference, these are the two sample posts users were asked to replicate using Gutenberg: Test 1, Test 2.

Tips on Reviewing:

  • When taking notes on the session, please be sure to keep your feedback separate from the participant feedback. Notes on the individual session tabs should be from the participant. If you have suggestions or conclusions of your own, keep those on the reviewer’s tab, and be sure to flag any videos you think worthwhile for additional review.
  • Watch for and note any instances where user confuses functionality, i.e. looks for caption when should be attempting to add a block quote, etc. or moments of delight, was there anything they mentioned that they liked?
  • Note whether the participant succeed in completing the task or if they gave up. Try to note where or why they stopped.
  • In attempting to complete the task, does the participant pause for any length of time? Do they hover the mouse or appear to randomly click around? Does the participant get “lost”? Try to note what they were attempting to find or trying to do during this time.
  • Watch for visual cues from the participant like a furrowed brow of concentration, or audible cues like “hmmm…”? If these occur, note what the participant was trying to do at the time. If they solved it, are you able to note how?
  • If available, watch through the participant completing the Part 3 end survey. Do their responses provide any additional insight on pain points they experienced during the task? Sometimes participants will provide additional explanation of why they seemed to struggle. Is there anything they say but don’t type into the form?

Special thanks to @betsela, @fuyuko, @annaharrison, and @lynneux for developing and refining these plans.

WCUS Gutenberg Testing: Volunteer Feedback

During WCUS, we had a ton of volunteers staff the Gutenberg testing booth (affectionately called the “Gutenbooth.”) A huge thank you to everyone who volunteered their time and ran tests throughout the weekend!

At the end of the weekend, we asked volunteers for some feedback about common trends they saw, along with recommendations for improving the testing process. Here’s the feedback we received:

Did you see any issues come up repeatedly while you were watching people test?

  • Aligning caption citation, converting paragraphs to list, changing block type when clicking into “Write your story”.
  • Typing the quote into the paragraph block, then trying to format it to match. The ability to change the block type was surprising info, as was the quote block having different styles available.
  • People couldn’t find the second quote style.
  • Right aligning quote blocks instead of changing the quote style.
  • People didn’t realize they should use the quote block type.
  • People didn’t know there were two quote styles.
  • May be more issues of the test itself, but most often people didn’t think to make the text style a quote block. They would be looking for font controls. If they did discover blocks sometimes they wouldn’t see the quote block.
  • No one really noticed the second quote style. They were more likely to find the Block settings, so maybe we move the quote style options to the quote > Block > Settings for easier discoverability.
  • People often missed the existence of the Quote block and used two Paragraph blocks instead, and when they did find the Quote block, they often didn’t know they could select different quote styles.
  • It feels like there are too many places for block settings. The icon for the second quote style was unclear. Some people didn’t know it was a quote and made paragraphs and styled those. Developers added inline CSS to make it match the design.
  • Insert block and then edit was not default mindset (at this point). Discovery of the ability to transform wasn’t strong.
  • There were plenty of issues with people not finding block creation or navigation intuitive. block controls cover up the previous block bottom line of text. If it’s a short line of text you might not see it at all. People sometimes got confused thinking their text was gone.
  • Undo/Redo is not intuitively discovered.
  • It wasn’t obvious what was behind the three vertical dots.
  • /slash commands could be interesting to search and use. We may consider a walk-thru wizard for new users to get them acquainted with now the new blocks can work.
  • Some tech glitches.
  • Mostly related to the test setup (e.g. not knowing to switch tabs to Gutenberg / switch tabs back to finish survey).

tl;dr: The two separate quote styles were the biggest pain point, followed by trouble learning the editor and block interface, particularly switching blocks, the ••• menu, and block controls.

Is there anything you think we should change about the test?

  • I suspect part of the issue with caption alignment was due to the task of the test to be imitation, not creation, so I think it leads people to think in terms of alignment, not necessarily style.
  • Make the screenshot not achievable using Gutenberg / Provide people with content and let them do much more free-form style.
  • Maybe written instructions instead of asking people to “mimic” output, because it this is not the way people write content in general, they do not “copy” something.
  • I’d have the sample printed out and set next to the laptop to keep the user from having to swap between windows.
  • Automate screen recording start/stop, one-button reset for survey etc.
  • Yes. I think expecting people to know they should be recreating a block quote without telling them that is what it is, skews the test results a bit. We need to try to replicate a more natural publishing process somehow.
  • No, this was a good example to make people search for options and solutions.

tl;dr: Imitating an existing design make people focus too much on the details and not as much on the editing experience, we need to print out whatever instructions we provide, and better automation.

If you attended WCUS and ran through the Gutenberg usability test, we’d also love your feedback with how you think the test can be improved!

#gutenberg, #wcus