It’s summary time.
Recap
We’ve run 3 users through the same set of scenarios (via usertesting.com). You can review the results here, here, and here.
Disclaimer
Now… What exactly are we trying to accomplish with all of this?
It’s important to note that we have to think about these discoveries in the context of the typical WP user. But to be honest, I’m not entirely sure I know what the “typical WP user” looks like (but I hope to dig into that over the next couple of months
). Sure, I have assumptions, as I’m sure you do, but my assumptions aren’t backed by any sort of data.
The important thing to keep in mind here is that these 3 people were paid to perform a list of tasks. While we can benefit from watching their interactions (and looking for patterns of complexity/confusion), we need to recognize that the flow they follow is prescribed, and likely different in some ways from an actual user.
Overview of discoveries
Here’s an unfiltered list of observations across all of the tests:
Background
- Had trouble figuring out how the color selector worked (x3)
Dashboard
- Unsure what “QuickPress” was
- Unsure where “1 post”, “1 page” came from under “Right Now”
- Had lots of trouble finding a link to view their site
Media Upload
- User clicked “Select files” when they actually had a URL to paste (and should have used the “From URL” tab). (x3)
Post Add New
- Lost all changes when she clicked a link taking her away from the new post page (after already making some changes)
- Didn’t notice “Insert into post” button when adding an image in the media modal
- Clicked “Preview” button, but didn’t realize a new tab had opened (causing confusion)
- Doesn’t appear to be a clear path for users that just wish to post a photo
Settings
- Had trouble knowing where to go to change her site title (x3)
Toolbar
- User didn’t notice links in toolbar (x2)
- User never noticed “+ new” dropdown (x2)
- The blog dropdown menu is completely different when in the admin vs. on site (should it be?)
Twenty Eleven
- Confusion over why the header image was changing (x2)
Next steps
This is where I need your insight/input.
1) First, is there anything that you noticed that’s not on this list?
2) Next, which of these do you think we should attempt to address (are there any quick wins that would benefit all WP users)?
3) If you had to prioritize your list, what would you tackle first?
I’ll hold off for a day before I post my thoughts. Don’t be shy!
I’d love to hear what you think.
Drew Jaynes 12:53 am on June 28, 2012 Permalink
1) Something I think that is starkly missing is mention if the seeming importance of the Welcome Screen and the way in which these 3 users relied so heavily on it. The users weren’t dismissing the screen and using the Dashboard so much as returning to it over and over for guidance.
2) I think one quick win would be to revisit the “Add Media” button. I know I’ve seen some discussion on trace of a labeled button or a larger icon. Another option might be something like an idea suggested by @kovshenin for dragging media directly into the editor. On the topic of the ‘site title’ issues, it seems like we’re moving more and more toward consolidating settings and previews into the Customizer. Perhaps some clearer verbiage on the Welcome Screen about what the Customizer is for (and some pointers once you get there) would be more helpful.
3) Not to rehash too much, but I’d say 1) Add Media, 2) Clearer connection between Welcome Screen & Customizer.
Matt Mullenweg 11:49 am on June 28, 2012 Permalink
Good points, especially on the permanence of the “temporary” welcome screen.
Helen Hou-Sandi 10:15 pm on June 28, 2012 Permalink
1. Maybe I missed it, but I didn’t see any mention of the featured image/post thumbnail functionality – there are quite a few tickets already about how weird and difficult that workflow is.
2. More prominence for the Customizer and a consolidation of all of those items into the same admin menu rather than spread out between Appearance and Settings seems like a relatively quick win. Or even duplicating menu items. Would also *love* to find a better color picker – Koop and I have discussed this in the past.
3. Besides the featured image thing, which I’ve talked about several times, I’ve been contemplating a reworking of the dashboard to become more useful – I think I called it “more mission control, less info center”. Full editor + post formats (would help with quick posting photos, much like the iOS app does) for Quick Press, better/prettier information about basic site/post metrics (including all other public post types), etc. This would also hopefully roll in the nice things the welcome panel does – a guide to getting things done for first-timers, as well as quickly enabling power users to get where they usually need to go. For instance, I could see items like “Edit the home page” for a site that has a static front page assigned.
Chelsea 12:08 am on June 29, 2012 Permalink
1) I think the “Settings” issue can be expanded to an overall organization/delineation between “Settings” and “Appearance” (as @lessbloat mentioned in an earlier user test writeup). It might be worth thinking about doing user tests specific to figuring out which part of the admin users generally look for things like “Site Title” “Header” “Language” etc. Its something I see a lot of people struggle with when setting up a new WordPress site.
2) I second @helenyhou on making the Customiser more prominent. It could also go a long way with giving insight on how to address the problem I mentioned in #1.
3) I would prioritize rehashing the media flow/making it easier to create media-heavy content above everything else. Its what we get criticised for the most and I think its the biggest thing other CMSs have over WordPress.
lessbloat 4:29 am on June 29, 2012 Permalink
1) As I alluded to in my disclaimer, the area that I’m most interested in focusing (at the moment) is in gathering more data around who’s using WordPress, and what they’re doing with it. For example, I’d like to figure out the percentage of people who download WP that:
I’m also very interested in figuring out a way to create a benchmark of sorts and a way to chart our peaks and valleys within the app (moments of joy/pain) over time. If we can figure out a way to do that, we can then prioritize the elimination of the biggest valleys, and work on flows that get users to moments of joy quicker and more frequently.
2) A couple of things:
3) Honestly, these all sound exciting to me. I’ve tried to list them in order (based on impact/time) specifically with the goal of improving this discovery cycle. After we’ve got a few patches pulled together, I’ll apply them to my little testing environment, and rerun this scenario-set with 2 additional users.
I’ll also be working towards getting my hands on some more data. I’ll post more about that to this P2 in the coming weeks.
Shane Pearlman 3:43 pm on June 29, 2012 Permalink
For #1, I’m deeply fascinated with that question. While .com may have some success in beginning to build those analytics, I have no clue how to proceed on .org driven sites besides surveys. With such a vast ecosystem, how do you get strong data about user roles and behaviors. For example, I’m deeply curious how many users on average have admin level access (even if they don’t need it) vs people in editorial / authorship roles. It truly affects their experience on the back end.
@lessbloat, how were you thinking of exploring?
lessbloat 4:23 pm on June 29, 2012 Permalink
For the first part of #1, I’d love to start off by running a series of small (one-question) KissInsights polls after people click the download button on http://wordpress.org/download/. It would look something like this: http://cl.ly/0n0y1v3N3Y1m1L0D1X1S
After we get some data, I’ll summarize the results on this P2.
The second part (figuring out a way to measure peaks and valleys over time) is going to take some noodling. I’m all ears if you’ve got any ideas.
Shane Pearlman 6:43 pm on July 3, 2012 Permalink
+1 for the kiss insight poll. That is a great idea.
lessbloat 7:34 pm on July 3, 2012 Permalink
With that said, it’s still self-selecting. I have no idea at this point what percentage of users get their WP install from WP.org. I’d also venture to say that those users fall on the outskirts of WP’s 30 million users (on the more advanced side). But, it’s a start, and any data is better than no data IMO, as long as you acknowledge that it’s a bit tainted.
Though @Jane did have a great idea about adding the UI Advisory signup form on the forums, maybe we could get a few KissInsignts tests in there?