Testing Flow in Gutenberg: Instructions for how to contribute to usability testing

The new editing experience is now ready to be tested! In this round of usability testing, we invite anyone from the WordPress community to either complete a test, or moderate an observational usability testing session. We have worked hard to make the tests simple to administer, so you can contribute even if you have zero usability testing experience.

How you can help!

  1. Run through one of the three usability tests yourself (A, B C)
  2. Help others run through one of the three usability tests
  3. Recruit your [mother | sister | partner | grandfather | child | puppy | entire social media network] to run though one of the three user tests

What feedback are we looking for?

We welcome all kinds of feedback (our doors are always open, join the conversation in #core-editor on the core WordPress Slack). To make sure we get a diverse range of perspectives, our key focus is to deploy the user tests to a wide, global audience. In particular, we are really keen to reach participants outside of the WordPress community.

About the test

There are three usability tests that you can choose from. The tests are almost identical, except for the complexity of the task that you will be asked to complete. Each test scales in time to take and difficulty. You are free to complete any of these tests, irrespective of whether you have prior WordPress experience or not.

Each usability test consist of three sections:

  1. Part 1 asks you some general questions about your experience with using WordPress
  2. Part 2 asks you to complete a task using the Gutenberg editor
  3. Part 3 asks some questions about your editing experience. You also have the option to upload your screen recording and answer some questions about the video footage in this section

Running the usability test

If you are new to usability testing, please have a quick look at our Guide to observational usability testing. There are some handy tips that will help you run a fun usability testing session. We also have a quick Guide to screen recordings for those who have not done that before.

In order to complete the usability test, you will need to set up a few things:

  1. Get your hands on a device (laptop, tablet, desktop or mobile device)
  2. Ensure that you are familiar with the process for how to run a usability test
  3. Ensure that you know how to do a screen recording on your device
  4. Open two browser windows, one with the test instructions (either A, B or C) and one with the Gutenberg editor loaded with the Twenty Seventeen theme. Example setup here.
  5. Follow the instructions in the test
  6. Optionally, upload your screen recording
  7. Optionally, write up a blog post about your observations*

How to report your results

There are three ways in which you can report back your user test results:

  1. You can simply answer all the questions in the test instructions
  2. You can optionally analyse your screen recording footage by answering the video coding questions in Part 3 of the test instructions
  3. In addition, you are welcome to write up your test results in a blog post like this one. Share your link with us in the comments below and in #core-editor on the core WordPress Slack

If you are new to usability testing, you may find it confusing to figure out how to provide feedback. A really easy rule to follow is to articulate what the user did when they hit an obstacle. A common mistake for new user testers is to jump straight into solution or resolution mode. While we all love to solve problems (us too!!), doing this during the testing phase is likely to cloud our ability to uncover the underlying causes of the friction. For this reason, we prefer to keep identifying the friction separate from fixing the friction.

Get Involved

Run through one of the three tests yourself, set up a user testing session or join one at a WordCamp near you.

Gutenberg Usability Testing Plan – Feedback Needed

We are getting ready to roll out a new round of usability tests for Gutenberg mid-November. In this next round, we will focus on testing writing flow. We are also very keen to widen the net for participant feedback, including testing with participants who are not current WordPress users.

In order to make this happen we have drafted a usability testing plan – and we need your feedback and suggestions (in the comments below) before the next meeting in #core-flow on Thursday 9 Nov at 18:00 UTC. The next steps will be to test the user test on a small sample set (week of Nov 10th), refine at the #core-meeting on  Tuesday 14 Nov 17:00 UTC, and roll out to a wider audience starting from 15th Nov.

We will also have a usability testing section at WCUS, so if you are attending please drop by the and join in!

Proposed Usability Tests

To test the flow of writing, we propose to construct a three part usability test:

  1. General demographics:  including prior WordPress experience, age and device used. This information will help us to segment findings
  2. The main task: participants will be asked to re-create the post shown in an image. There will be three images to select from, mapping roughly to a beginner, intermediate and advanced level
  3. Follow up questions: a few questions about the experience of re-creating the post

Participants will be optionally invited to upload their screen recording, and answer a few questions about their video footage.

Testing Script

Question 1: Do you currently use WordPress?

  • Yes
  • No

Question 2: Would you describe yourself mostly as a…

  • Developer
  • Designer
  • Blogger
  • Business Owner
  • Other: ________

Question 3 (optional): How old are you?

  • Under 18
  • 18 – 30
  • 31 – 40
  • 41 – 50
  • 50 – 60
  • Over 60

Question 4: What device are you using?

  • Mobile phone
  • Tablet
  • Laptop
  • Desktop

Question 5: Let’s get set up!

Check that you have the following items ready as you will need them to complete the task

  • Open Gutenberg editor in a new browser window
  • Ensure that you have the Twenty Seventeen theme selected
  • Open the task image in a new window [ beginner | intermediate | advanced ]
  • Start a screen recording
  • Remember to talk out loud as you complete the task

Your task is to re-create the page that you see in the image using the Gutenberg editor. Remember to start your screen recorder, and talk out loud as you complete the task! When you are finished, continue on to answer a few questions about your editing experience.

Question 6:  Did the task take long or shorter than you expected?

  • It took longer than I expected
  • It took about the amount of time that I expected
  • It took less time than I expected

Question 7: Can you tell us why?

Question 8: Was the task easier or harder than you expected?

  • It was harder than I expected
  • It was about what I expected
  • It was easier than I expected

Question 9: Can you tell us why?

Question 10: Are you more or less likely to use the Gutenberg editor in the future?

  • I am not likely to use Gutenberg in the future
  • I am unsure
  • I am likely to use Gutenberg in the future

Optional section: screen recording analysis

It would help us out a lot if you could upload your screen recording and answer a few questions about your recording

Question 11:  Save your screen recording, and upload your file here…

Question 12: How long did it take to complete the task?

Question 13: Was the title added correctly?

  • Yes
  • No

Question 14: Was the quote added correctly?

  • Yes
  • No

Question 15: Was the image added correctly?

  • Yes
  • No

Question 16: Where were the main sources of friction?

Thank you so much for your help!

If you would like to be kept in the loop with the progress of Gutenberg testing, please leave us your email below and we will add you to the Make.WordPress user testing mailing list

Test Setup

The test can be completed by a participant, or used as a run sheet for an observational research session.

In order to complete the test, participants will need to:

  1. Get their hands on a device (laptop, tablet, desktop or mobile device)
  2. Ensure that they know how to do a screen recording on the device
  3. Load up the user test
  4. Follow the instructions in the test
  5. Upload the screen recording to the cloud
  6. Optionally, code the results from the screen recording
  7. Optionally, write up a blog post and tell us what you found

Reporting usability test results

There are three ways in which you can report back your user test results:

  1. You can simply answer all the questions in the test instructions. Remember to upload your screen recording at the end
  2. You can optionally analyse your screen recording footage by answering the optional questions in the final section of the test instructions
  3. In addition, you are welcome to write up your test results in a blog post

Get Involved

Have an idea on how to improve the usability testing plan? Have your say in the comments below before the next meeting in #core-flow on Thursday 9 Nov at 18:00 UTC. Once we have collected all feedback we will post a link to the test script and open the call for user testing!

 

#gutenberg, #usability-testing

Gutenberg usability testing meeting three

Last week we had a usability meeting. The meeting time wasn’t good for everyone, so let’s change the time this week.

When? Tuesday 31st October 19:00 UTC.
Where? wordpress.org Slack #core-flow: the testing channel.
Who should come? Anyone interested in helping test Gutenberg, all skill levels welcome.

Last week we talked about plans for testing. We will continue that conversation this week.

#gutenberg

Gutenberg usability testing meeting two

Some work has already been doing after our first meeting, which you can find out about here. It’s time to build on that work, let’s have another meeting.

When? Tuesday 24th October 18:00 UTC
Where? wordpress.org Slack #core-flow: the testing channel
Who should come? Anyone interested in helping test Gutenberg, all skill levels welcome.

#gutenberg

Writing my first Gutenberg Block

Inspired by a comment I read a week or so ago, I wrote my first Gutenblock. Here are my impressions of that process.

I initially planned on copying an existing block, but quickly found the example in the blocks directory README.

This is the ultimate result, you should open the example and the result to follow along how things changed.

Getting from one to the other was not too tricky, but did have a handful of surprises and confusing bits.

I started with changing the block name, which caused the first problem – I got an error that it needed to be namespaced. That was easily rectified.

Changing the title was easy enough, but I didn’t know what I should change the category to. I got an error when I left it blank, so I set it to common.

Next up, changing the attribute definition. Changing the type to ‘int’ was a lucky guess, there was nothing to suggest what the valid types are. source confused me for some time, as I didn’t really know what it needed to be. I figured I needed to store the value as a data attribute on the HTML that Star returned, so I wrote a basic version of the Star element, and tried to save it there. It… didn’t work. I tried several variations of this before noticing in the docs that source is optional. So, I tried deleting it, and it worked! That bit of magic made me smile, but I was a bit annoyed I spent so much time on a non-problem.

Writing the setStars() method was easy enough, even without really knowing what the props.setAttributes() call did.

Trying to add the Stars render to the children array gave me a weird error, complaining that the element needed an identifier. I tried adding an id attribute to the HTML that Stars rendered, but that didn’t help. Googling the error message showed me that I needed to set the key attribute, which was kind of weird, and I didn’t understand why it was necessary now, when the example didn’t need it. But it worked, so I moved on.

Adding the input element to children was also easy. I got the same error as before about needing an identifier, so I added a key attribute to that, too. I added all the extra attributes for fun, but realised when defining the min / max attributes that there was no nice way to validate the data being put into the stars attribute.

It was around this time that I discovered the Stars element wasn’t actually rendering, the problem was that I didn’t know how to put the text inside the div element. Did I need to append it? Was there a createTextNode() method I needed to call? As it turns out, it just magically accepted the string, but I only found that out by guessing. (Excuse the hacky math to add the “½”, I got a little carried away.)

One annoyance throughout was that every time I changed the attribute definition, or the output of Stars, Gutenberg threw a warning about the format changing. It seems like this warning would show whenever a plugin upgrade included a block change, though I didn’t test that.

And that’s about the sum of it – I found it to be a generally pleasant experience, it was nice that I could do this all without needing to compile from JSX, but I knew that option was there for if I wanted it. It’s still a bit of a raw experience, but it shows a lot of promise for creating a delightful development experience.

Widgets, Menus and Customize Changesets

Earlier today “converted” a Squarespace user, sat there and watched them struggle with hierarchical categories, pages (and why they don’t show on the site), then creating a menu and adding to it. Then widgets… At the end had to explain how all these work, they were pretty happy to set and use them.

From this user (well, hoping to find more “uninitiated” soon), it looks like a good thing would be to tie creating pages with adding them to the menu. Also adding to the menu on the separate Categories and Tags screens.

Apparently the separate widgets screen was easier to grasp for a first-timer, possibly because it’s all about widgets/not that overloaded with all kinds of (new) things.

Menus were easier as soon as a menu was added to the proper location. However the flow of making a page and then needing to add to the menu so the page shows on the site sucked. It was even more confusing when I tried to explain that a tag can be created and then added to the menu and it will show an “archive” of all the posts that were tagged with it. The separate Tags screen makes this somewhat clearer, and having an “Add to the menu” button/link for each tag would probably be a winner.

Thinking that we need “Annoyingly Aggressive Help” for new users 🙂 Maybe something like a modal overlay when they go to a page that is full of new concepts and “things” (of course that modal will also need “Don’t show again!” button).

I’ve been looking at the discussions around Customize Changesets. They seem like a good idea at first look, but IMHO they are an advanced feature that only few users need, and only a fraction of these users will understand and use. Helping out this person earlier made me think that what we have is already hard enough 🙂

Next steps for Gutenberg testing

There was an open Gutenberg testing chat held on Wednesday, you can find out more about this and catch the video and notes here. There were a few outcomes from this and some next steps.

Firstly, what has happened so far?

Organised testing for Gutenberg has been in the form of 2 unmoderated usability tests. You can find out more about those here. These were a freeform and a task based test.

The test so far worked whilst Gutenberg was in early stages. However, it was limited in user type and response. The goal now is to scale up testing and make sure there are usability tests with each user type defined and prioritized, as well as non-WordPress users.

What are we going to do now?

In the group chat it was decided, the next step was to define user types, goals for testing, and the tasks for testing with each of the various user types defined.

To begin this, let’s use a collaborative spreadsheet to gather ideas. The scope of what we’re going to add is:

  • User types: In this tab let’s braindump a list of user types. For a starting point, the ones outlined in Matt’s post here have been added. What more should we add? Then let’s define:
    • Goals: What are the main goals of this user type? What do they want to achieve with an online publishing process?
    • Notes: Do you have anything else you think needs noting about this user type?
  • Tasks: In this tab the focus is tasks that people will do with Gutenberg. What can be broken down into tasks and test?

Add your thoughts on the above to the existing spreadsheet over the course of this week. If it’s easier you can leave a comment on this post with your thoughts and it will get added to the spreadsheet. Do you have some past research you can link to help add to this?

Are you a member of an agency, University, or are you in contact with a user type that can help the project reach? If so, let us know in the comments below, saying what you are able to help with for testing outreach.

Once the week is up, we can have a meeting again in #core-flow in wordpress.org Slack (time to be announced). We will go over the types and look at how we can begin testing.

If there is anything else you think should be considering in this testing focus, please let add a comment.

Get involved!

If you’d like to be involved in helping test Gutenberg, please leave a comment below and you can be added to the list of amazing people helping test this project.

This post was composed with the help of @msdesign21.

Gutenberg testing planning group chat

As the Gutenberg project develops, it’s now time to think about a testing plan. A way to get this tested on as many users as possible, on as many environments as possible.

As a first step, come and join the first Gutenberg testing planning group chat at 19:00 UTC on Wednesday. Join #core-flow to get the link to the group chat just before it starts – it will also be added as a comment here.

Why a chat to plan? Coming up with a scalable plan on how to progress testing is important. A few points for discussion:

  • What are our initial testing goals?
  • What format should the testing be in?
  • How do we reach outside WordPress to test with as many people as possible?
  • How do we scale testing?
  • How can we improve gathering feedback

… If there are any other points you think we should add please put them as a comment.

This chat will be recorded and a video put up after to ensure that everyone gets a chance to keep up. Notes will also be taken.

From this chat a plan will be formed that will be posted for iteration from feedback. This will then form the structure for testing Gutenberg.

+make.wordpress.org/design +make.wordpress.org/core

Updating plugins with .zip uploads

As detailed in this Trac ticket, when you try to upload a new version of a plugin, you get an error. Meaning, if I have a plugin on my WordPress site that needs to be updated to the latest version, or even to a development version, I can’t just upload a new .zip file to do this. Instead I have to either update from the plugins page or upload a new folder via SFTP.

Following are the steps to replicate the existing issue where you get blocked when trying to upload a new version of a plugin:

X-post: Community Conduct Project – Kick off meeting scheduled for 17:00 UTC on the 5th September 2017

X-comment from +make.wordpress.org/updates: Comment on Community Conduct Project – Kick off meeting scheduled for 17:00 UTC on the 5th September 2017