Plugin Directory User Testing – Round 1

With the advent of the new Plugin Directory, I wanted to get some basic user testing in place to help guide UX decisions. Props to @designsimply for helping me with the tasks, and to @tellyworth for making sure functionality was up-to-par for testing.

In this test, I wanted to observe the user’s ability to easily find a plugin, and verify which meta information helped the user select the best plugin for their use.


  1. Find an SEO plugin you think would be the best solution for your website, and talk through the reasons why you’d pick that one
  2. What would be the next step you would take to install the plugin on your site? [Verbal Response]
  3. How was your experience finding this plugin? [Verbal Response]
  4. Locate the Plugin Search Field and search for a plugin that will add an image slider to your website.
  5. Are you able to make an informed decision from the search results page? Why, or why not? [Verbal Response]
  6. If you haven’t already, click on the plugin that you like best to view the details.
  7. Looking at this Plugin Detail page, how do you feel about the content layout and organization? Is there anything you might improve? Is there anything that is unclear? [Verbal Response]
  8. What are the top 3-4 things on this page that influence your decisions when selecting a plugin? [Verbal Response]
  9. Please share your overall experience with this process. Was there anything you’d like to see improved? Was there anything you thought particularly helpful? [Verbal Response]


User Testing Round 1


  1. [1:48] After browsing, user decides to search for ‘SEO’
  2. [2:07] User looks at ratings as a good indicator for which plugin is best b/c other people tried it out and were satisfied.
  3. [3:17] User still doesn’t seem to have enough information to choose a plugin directly from search results page.
  4. [4:21] User clicked to check negative reviews and ended up at older layout for plugins reviews.
  5. [6:11] User is highly influenced by ratings above all else.
  6. [7:14] User checks “last updated” as well at this point.
  7. [8:33] User would download plugin and figure out how to install it. He didn’t see any instructions in the ‘description’.
  8. [9:09] User uses ‘search’ as an option when he’s “stuck”. He prefers browsing first, and then later searches.
  9. [10:23] User found the Plugin Search quite easily but then gets confused seeing another search field above it in the header. He expected it to be in top right corner.
  10. [12:21] Not enough info on search results page to make an informed decision.
  11. [15:09] User says layout is pretty typical on the Detail page.
  12. [16:10] Bringing negative reviews closer to the top would be helpful for this user.
  13. [16:30] Top 3-4 things this user relies on for selecting a plugin are: 1. Ratings, 2. Last updated, 3. Description/Feature list, 4. Maybe FAQs?
  14. [18:31] Overall experience was “fine”. Easy to find plugins. Providing more meaningful descriptions would be helpful b/c they all kinda looked the same in the short descriptions.

#design, #plugin-directory, #user-testing

Next plugin directory chat 2016-11-02

We’ll skip this week and make the next #plugin-directory chat after the DST change in many locations: 2016-11-02 22:00:00 UTC. Agenda TBD.

Plugin directory chat on 2016-10-19

We’ll have a quick chat about the #plugin-directory at 2016-10-20 00:00 UTC in #meta.

No fixed agenda, just some brief updates.


Plugin directory chat on 2016-10-12

The weekly plugin directory chat will be at 2016-10-13 00:00 UTC, in #meta on Slack.

This week we’ll chat about user testing goals, and the new tickets and progress since the last chat.



Plugin directory chat on 2016-10-05

The next plugin directory chat will be at 2016-10-06 00:00 UTC, in #meta on Slack. Here’s a very broad agenda for now:

  • Review the current milestone.
  • Plan the next milestone.

Suggestions for specific agenda items are welcome – we have plenty of time before then. See you in chat.



Plugin Directory Chat Agenda

Agenda for this week’s meeting on September 1, 2016 at 00:00 UTC:

We’ll be meeting in #meta on Slack. If you have anything to propose to add to the agenda, please leave a comment below.

See you in the chat!



Initial Analysis of Plugin Search Logs

In preparing to do some relevancy testing of the new plugin search here is some data analysis of the search logs from the existing plugin search. This data and the search logs should help us improve the search query more methodically, but it may also be of use when thinking about the overall search user experience.

Aggregated Stats

  • 53 days of aggregated, anonymized search records with 4,952,788 total searches and 805,488 unique searches. This excludes API searches(*)
  • Almost 100k searches a day on average.
  • Focusing on en_US: 499,489 unique searches from 3,929,003 total searches
  • Top 100 en_US terms: 944,243 searches (24% of the total)
  • Top 1000 en_US terms: 1,834,942 total searches (46.7% of the total)
  • Bottom 400k en_US terms: 479,604 searches (12.2% of total)
  • 320,390 en_US terms occurred once in 53 days (8.1% of total)
  • 472,500 en_US terms occurred ten times or less 839,123 total (21% of total)
  • 492,500 en_US terms occur less than once a day 1271786 total (32% of total)
  • 245,151 of 472,500 search queries pass aspell (though aspell is limited, there are lots of fun spelling mistakes “composor”, “conditon”, “taxanomy”, “produkt”)
  • Here is a random set of queries that only occurred once in the 53 days to get a sense of what the long tail of queries looks like.

Non-English data is a little hard to draw conclusions from because there aren’t currently any search indices for non-English languages. So the top searches in other languages look about the same as English. About 20% of all searches are not in English. Here are the search counts by language. This likely undercounts the demand since search in non-English languages almost certainly don’t work well.

(*) A very large volume of searches come through the API, but a lot of these appear to be bots or direct lookups of an exact plugin name rather than organic search results. These should be looked at separately and should maybe not use the same search query.

Some Takeaways from the Data

Hopefully from this data it is clear why we can’t just focus on the top 100 queries. Doing so ignores probably 75% of our users. Even top 1000 is less than 50% of all searches. So in testing out our search quality at the very least we should look at top 1000. Ideally we’ll also do some random sampling of the other 50% of searches and find ways to address spelling mistakes and other edge cases that affect a very large percentage of users. Similarly, adding better support for non-English languages could help as many users as improving the top 100 queries.


Plugin Directory Chat Summary (8/25)

This is a summary of the Plugin Directory chat from August 25. (Slack log)

Attendees: @mapk, @webdevmattcrom, @kevinwhoffman, @Ipstenu, @diddledan, @jcastaneda, @hugobaeta, @obenland


  • Status of M7
    Three tickets have been closed so far. #1828 has a mockup now, as discussed last week, and already got some feedback on it. It should be a good base to test and iterate on. @ocean90 is on well deserved vacation after 4.6 so #1691 is still on standby, while @dd32 was at a conference and probably didn’t have much time to look at #1579. #1839 has a final version of the email, so @Otto42 should be able to close that fairly soon.

    One of the three fixed tickets was #1810, that significantly improves the display of the FAQ section. Additional suggestions voiced in the chat were making questions linkable and not using all caps on them. For more feedback please comment on the ticket so @mapk has a chance to respond to it.

  • Review community feedback
    We decided to postpone discussions of the remaining non-urgent items until @mapk and @dd32 can participate again and give their input next week.
  • Open Floor
    @webdevmattcrom created a mockup for plugin cards on index/archive pages. Some of the things that were discussed was potentially removing the description switch position with the new information to make it more cohesive, different view points about whether the icons provide enough context, and how more context can be added.

The next meeting is on Thursday September 1, 00:00 UTC.


Plugin Directory Chat Agenda

Agenda for this week’s meeting on August 25, 2016 at 00:00 UTC:

  • Status update for Milestone 7, due next week.
  • Review remaining feedback items from Open Beta.
  • Open Floor.

We’ll be meeting in #meta on Slack.

If you have anything to propose to add to the agenda, please leave a comment below.

See you in the chat!



Plugin Directory Meeting Summary (8/18)

This is a summary of the Plugin Directory chat from August 18. (Slack log)

Attendees: @SergeyBiryukov, @mapk, @webdevmattcrom, @kevinwhoffman, @deblynprado, @Otto42, @Ipstenu, @dd32, @jcastaneda, @obenland


  • Review M6
    Five tickets were closed. Favorites have a UI now, which @dd32 had largely done earlier. We decided last week to postpone #1691-meta to Milestone 7 so @ocean90 has more availability to give advise on how to move forward there. For #1579 @dd32 committed @tellyworth’s patch.
  • Planning M7
    #1579 and #1691 were taken over from last milestone. After last weeks review of feedback items, #1810 and #1828 were identified as important tickets needed to move forward, so those were moved into M7 as well. #1810 is pretty far already, likely be fixed soon. #1828 will need some more attention, namely a decision on how screenshots should be displayed by next weeks meeting (@mapk). Additionally #1839 and #1841 were added to the milestone since they should be fairly easy to fix.
  • Open Floor
    @webdevmattcrom is working with @mapk on adding additional info to search results: Mockup.

The next meeting is on Thursday August 25, 00:00 UTC.