Gutenberg usability testing meeting three

Last week we had a usability meeting. The meeting time wasn’t good for everyone, so let’s change the time this week.

When? Tuesday 31st October 19:00 UTC.
Where? wordpress.org Slack #core-flow: the testing channel.
Who should come? Anyone interested in helping test Gutenberg, all skill levels welcome.

Last week we talked about plans for testing. We will continue that conversation this week.

#gutenberg

Gutenberg usability testing meeting two

Some work has already been doing after our first meeting, which you can find out about here. It’s time to build on that work, let’s have another meeting.

When? Tuesday 24th October 18:00 UTC
Where? wordpress.org Slack #core-flow: the testing channel
Who should come? Anyone interested in helping test Gutenberg, all skill levels welcome.

#gutenberg

Writing my first Gutenberg Block

Inspired by a comment I read a week or so ago, I wrote my first Gutenblock. Here are my impressions of that process.

I initially planned on copying an existing block, but quickly found the example in the blocks directory README.

This is the ultimate result, you should open the example and the result to follow along how things changed.

Getting from one to the other was not too tricky, but did have a handful of surprises and confusing bits.

I started with changing the block name, which caused the first problem – I got an error that it needed to be namespaced. That was easily rectified.

Changing the title was easy enough, but I didn’t know what I should change the category to. I got an error when I left it blank, so I set it to common.

Next up, changing the attribute definition. Changing the type to ‘int’ was a lucky guess, there was nothing to suggest what the valid types are. source confused me for some time, as I didn’t really know what it needed to be. I figured I needed to store the value as a data attribute on the HTML that Star returned, so I wrote a basic version of the Star element, and tried to save it there. It… didn’t work. I tried several variations of this before noticing in the docs that source is optional. So, I tried deleting it, and it worked! That bit of magic made me smile, but I was a bit annoyed I spent so much time on a non-problem.

Writing the setStars() method was easy enough, even without really knowing what the props.setAttributes() call did.

Trying to add the Stars render to the children array gave me a weird error, complaining that the element needed an identifier. I tried adding an id attribute to the HTML that Stars rendered, but that didn’t help. Googling the error message showed me that I needed to set the key attribute, which was kind of weird, and I didn’t understand why it was necessary now, when the example didn’t need it. But it worked, so I moved on.

Adding the input element to children was also easy. I got the same error as before about needing an identifier, so I added a key attribute to that, too. I added all the extra attributes for fun, but realised when defining the min / max attributes that there was no nice way to validate the data being put into the stars attribute.

It was around this time that I discovered the Stars element wasn’t actually rendering, the problem was that I didn’t know how to put the text inside the div element. Did I need to append it? Was there a createTextNode() method I needed to call? As it turns out, it just magically accepted the string, but I only found that out by guessing. (Excuse the hacky math to add the “½”, I got a little carried away.)

One annoyance throughout was that every time I changed the attribute definition, or the output of Stars, Gutenberg threw a warning about the format changing. It seems like this warning would show whenever a plugin upgrade included a block change, though I didn’t test that.

And that’s about the sum of it – I found it to be a generally pleasant experience, it was nice that I could do this all without needing to compile from JSX, but I knew that option was there for if I wanted it. It’s still a bit of a raw experience, but it shows a lot of promise for creating a delightful development experience.

Next steps for Gutenberg testing

There was an open Gutenberg testing chat held on Wednesday, you can find out more about this and catch the video and notes here. There were a few outcomes from this and some next steps.

Firstly, what has happened so far?

Organised testing for Gutenberg has been in the form of 2 unmoderated usability tests. You can find out more about those here. These were a freeform and a task based test.

The test so far worked whilst Gutenberg was in early stages. However, it was limited in user type and response. The goal now is to scale up testing and make sure there are usability tests with each user type defined and prioritized, as well as non-WordPress users.

What are we going to do now?

In the group chat it was decided, the next step was to define user types, goals for testing, and the tasks for testing with each of the various user types defined.

To begin this, let’s use a collaborative spreadsheet to gather ideas. The scope of what we’re going to add is:

  • User types: In this tab let’s braindump a list of user types. For a starting point, the ones outlined in Matt’s post here have been added. What more should we add? Then let’s define:
    • Goals: What are the main goals of this user type? What do they want to achieve with an online publishing process?
    • Notes: Do you have anything else you think needs noting about this user type?
  • Tasks: In this tab the focus is tasks that people will do with Gutenberg. What can be broken down into tasks and test?

Add your thoughts on the above to the existing spreadsheet over the course of this week. If it’s easier you can leave a comment on this post with your thoughts and it will get added to the spreadsheet. Do you have some past research you can link to help add to this?

Are you a member of an agency, University, or are you in contact with a user type that can help the project reach? If so, let us know in the comments below, saying what you are able to help with for testing outreach.

Once the week is up, we can have a meeting again in #core-flow in wordpress.org Slack (time to be announced). We will go over the types and look at how we can begin testing.

If there is anything else you think should be considering in this testing focus, please let add a comment.

Get involved!

If you’d like to be involved in helping test Gutenberg, please leave a comment below and you can be added to the list of amazing people helping test this project.

This post was composed with the help of @msdesign21.

Gutenberg testing planning group chat

As the Gutenberg project develops, it’s now time to think about a testing plan. A way to get this tested on as many users as possible, on as many environments as possible.

As a first step, come and join the first Gutenberg testing planning group chat at 19:00 UTC on Wednesday. Join #core-flow to get the link to the group chat just before it starts – it will also be added as a comment here.

Why a chat to plan? Coming up with a scalable plan on how to progress testing is important. A few points for discussion:

  • What are our initial testing goals?
  • What format should the testing be in?
  • How do we reach outside WordPress to test with as many people as possible?
  • How do we scale testing?
  • How can we improve gathering feedback

… If there are any other points you think we should add please put them as a comment.

This chat will be recorded and a video put up after to ensure that everyone gets a chance to keep up. Notes will also be taken.

From this chat a plan will be formed that will be posted for iteration from feedback. This will then form the structure for testing Gutenberg.

+make.wordpress.org/design +make.wordpress.org/core

Updating plugins with .zip uploads

As detailed in this Trac ticket, when you try to upload a new version of a plugin, you get an error. Meaning, if I have a plugin on my WordPress site that needs to be updated to the latest version, or even to a development version, I can’t just upload a new .zip file to do this. Instead I have to either update from the plugins page or upload a new folder via SFTP.

Following are the steps to replicate the existing issue where you get blocked when trying to upload a new version of a plugin:

Call for testing: Gutenberg

Gutenberg is now in beta, with that comes a new opportunity for testing.

There is now a testing page dedicated to Gutenberg, you can find it right here. There are currently 2 types of testing being looked for, each has a central feedback form.

Looking for other ways to give feedback? You can also write a blog post (let the editor team know about that in Slack #core-editor) or add an issue to the GitHub repository.

Your feedback and testing is really important at this stage of Gutenberg, it really does matter and help make this as good as it can be.

+make.wordpress.org/core +make.wordpress.org/design

#needs-testing

A new call for testing…

A new call for testing is out for the prototype editor. It looks neat! If you’d like to help test, remember that it’s in very early prototype stages so functionality is limited and they’ve asked for help with the following (other functions haven’t been built yet):

  1. Add a new paragraph.
  2. Add an image.

Written response:

  1. Is it clear what each section does?
  2. How do the flows for adding paragraphs and images feel overall?

Post testing notes on the thread at https://make.wordpress.org/design/2017/02/09/initial-gutenberg-prototype-editor-testing/

Report bugs (search first!) at https://github.com/WordPress/gutenberg/issues

#needs-testing

WordCamp Contributor Day: Workshop Findings

December 4, 2016

WordCamp Contributor Day: Workshop Findings

BACKGROUND

Goals

This test allowed Flow Patrol to 1) observe sign-up and log-in flow, as well as 2) gauge user experience of creating an entirely new site solely through the WordPress Mobile App version 6.2 on Android 5.1.

Tester

The tester has experience using WordPress, but has procrastinated making a blog for a long time. He has used WordPress for other things, commercial sites for example, but has never made a blog because he finds it difficult to get into writing. The tester is a recent Android user, who is new to using smartphones in general, and often expressed that he dislikes prolonged typing on mobile devices.

 

OBSERVATIONS

Sign-up

Since the tester has trouble typing on mobile devices and was self-conscious about making mistakes, he began looking for a “show password” button as he navigated the sign-up page. He commented that it needs to be more visible and thinks perhaps the image of a crossed-out eye is too abstract for some users.

The tester found that an account already existed under the name he wanted and figured it was his old one. However, he was required to back out of the sign-up page before navigating to the log-in page to try entering it. He commented that it would be nice to simply be taken to the log-in page for the account once WP discovered it, or at least to be prompted to try to log into that site.

The tester clicked on “forgot password” and was redirected to the browser version of WPs’ sign in page, which caused some confusion. He said that he expected to leave the app to retrieve an email confirmation from his email inbox, but he didn’t understand why he was taken outside of the app to reset the password. He said that changing platforms like this takes a person out of their work-flow. He admitted that this left him with a bad impression, explaining that if WordPress can claim to power over a fourth of the Internet, he expects that it “should have this figured out by now.”

 

Log-in/Log-out

Done with sign-up, the tester tried to log in with what was believed to be his “username”, but could not get in because it turned out that he was using his “public display name.” In the site navigation screen, the “public display name” is emphasized – it’s a darker, larger font and above “username.” Log-in using email address was successful.

The tester attempted to log out to figure out which name was to be used to log in. A “log-out” button was difficult to find, because it was expressed as “disconnect.” The tester expressed fear at the confirmation prompt, which seemed like a warning about permanent deletion of accounts, sites etc. He suggested that it simply be called, “log-out.” Only after two successful log-ins did tester realize the difference between and significance of “username” and “public display name.”

 

Site Creation

Once logged in again, the user saw past sites displayed on the opening screen. The tester explored other parts of the site looking for a “create new site” button. He openly questioned where he should go, even exploring “account settings.” The tester expressed that he felt stupid and surprised that it was difficult for him to find. After prompting from the interviewer that he search back at the “show sites” screen where he started, he found a plus sign which turned out to be what he was looking for all along.

When asked where he expected it to be, the tester explained that maybe front and center was not appropriate because he already had sites and doesn’t create them all the time. Still, he found “show sites” to be misleading because it implies that it would take him to view existing sites only. He only found it when looking for it intently and with extra help. The tester suggested that the plus sign makes sense, but could be emphasized by making it a brighter color.

 

First Post

The tester was intent on creating a quick post just to see how it would look. He successfully created a simple post with a title and a sentence in the body. When he then viewed the post, it opened up in the app itself, which he liked. He then checked the post on the mobile browser and was satisfied with the way everything looked. The end result matched his expectation.

 

VISUAL RECORD

A visual record of this test does not exist. Subsequent testing will hopefully provide such data.

 

CONCLUSIONS

At the end of the test, the tester offered this feedback:

  • Publishing was a great experience and everything went as expected.
  • “Switch sites” proved to be misleading and he might not have looked there without guidance.
  • Logging in was frustrating and “could have been done better.” The tester was not a strong typist, and was made to do it several times. He admitted that if it wasn’t for the sake of test completion, he would have put the app down and “done it later,” perhaps not revisiting it for some time.
  • After the test, the tester admitted that he would likely never pick up the app again anyway. Writing is not something the tester would do on the go; he prefers instead, a laptop or desktop to work from.
  • The tester would instead want the app for notifications that would help him keep up to date with happenings on his site. He said that an ideal app would allow him to make quick corrections, approve work submitted by other writers on a shared blog, track business/e-commerce information, and generally keep track of his site when away from keyboard.

Better Dogfooding Tips

This was originally written by @karmatosed. I’ve adapted it for WordPress.org.


Dogfooding is a great way we can use WordPress and test our features. By testing WordPress ourselves, we get to unearth all manner of bugs, issues and fun things. Try setting up a new site from scratch, especially thinking about specific use-cases like a new photoblog, or a company website.

I thought it would be great to come up with some tips for so new folks doing dogfooding can have a resource to start with. So, without further ado here’s just that!

Format

How you present the post really helps for people to read and also how we can use later on. A rough format for reporting your observations via video that works is:

  • Title: Make sure your title has the word “dogfooding” in it somewhere, so it’s easy to scan. Also include what feature you’re testing in title. ex: Dogfooding: Theme Setup
  • Use the more tag to shorten posts. It helps keep the p2 streamlined so it’s easier to scan.
  • Talk about your perspective as a user, and what you’re trying to accomplish.
  • Embed the video in your post.
  • Under the video, pull out any key points at minute marks. ex: 01:00: here’s where I spent a whole minute trying to search for directory themes from my installed screen. Keep this list short; don’t worry about pulling everything out, as if this is long, people will be less likely to read. Just make sure to list the biggest finds.
  • After that, it’s often useful to chop up the video (if it’s very long) and pull out significant clips. Making clips like this is useful for bug reporting and summaries. Clips can go after or next to minute marks, and also next to bugs for easier visual reference. Just consider that not everyone will want to watch your entire video. 🙂
  • Bugs: List out any bugs you found while testing. You can even do a fun thing by putting the bug emoji at end to make them stand out: 🐛
  • Summary: Summarize the main issues and conclusions you found from testing. ex: I stumbled a lot with the theme installation process because… If we did x or y it might improve this feature…
  • Tags: Tag it #dogfooding and anything else relevant.

Here’s what the above would look like laid out:

  1. Dogfooding: Feature You’re Testing
  2. As a user, I’m trying to accomplish x
  3. Video embed
  4. 1:00: something amazing, 2:00: something else amazing
  5. Significant clips
  6. Bugs 🐛
  7. Summary
  8. #dogfooding

Reporting

If you see a bug, report a bug. It’s important we follow through on the bugs and don’t just leave them on this p2.

The same goes for things you think are issues, or even enhancements. Sometimes you may think of something awesome nobody else has thought of before, and it could become something that changes the life of our users. Pretty cool right? That’s why it’s very important to report on trac.

Enjoy and please dogfood 🐶


 

This has also been x-posted to make/design.