Video: Site Building Research Q&A

Last week, @jarahsames and I hosted a live video session in which we walked through the process, methodology, and results of the site building research—then answered lots of questions!

For those who weren’t able to make it, here’s a video and a transcript.

Transcript:

Have slides ’cause of course I do. And I’m going to try to share them. If I can figure out how to do that using technology. So give me a quarter of a second to figure that out. Had this figured out earlier today but it was two hours ago and now I know nothing. Okay, so that should show a screen which isn’t very helpful And then I believe if I do that Does that work for everyone? Can everyone see a giant blue screen?

Yep.

Cool, okay, so I can also see a giant blue screen which is why there is a giant blue thing in reflection behind me. Okay so we put together a little walk-through of the site building study sort of as an attempt to demystify a lot of the process what we did, how we did it, why we did it and how we can be involved in research in the future. That’s a lot to fit in in a short period of time along with questions so we’re just gonna do our best and try to sort of clarify a few of those things that may be a bit more opaque. We will have questions at the end but you can also ask questions at any point if something isn’t super clear just sort of hop in, ask us a question, it’ll be very casual and we’re gonna kinda just discuss through these slides that I made last night. Does that seem good?

Hm, yeah.

Okay. So first of all, let’s just talk about a bit about why research is important. Why its necessary to do user research. In short, we want to be able to understand and empathize with end users. Because its really easy to think that everybody who uses the software that we build is just like us. But that’s often not the case. We’re often for instance a lot more technologically capable than a lot of users. And especially with a project like WordPress where its 30% of the internet. That’s a huge variety of different kinds of people who aren’t necessarily going to have the exact same needs and desires as we do. The research allows us to talk to those users and get better understanding of what they need from the software that we build. And those users aren’t necessarily going to be represented at Word Camps. They’re not always going to be the people who are really vocal in Track or Get Hub. A lot of WordPress users don’t really know what an open sourceOpen Source Open Source denotes software for which the original source code is made freely available and may be redistributed and modified. Open Source **must be** delivered via a licensing model, see GPL. project is or what that means. They just know Word processes the thing that makes their website go. So, back in I think October, we started talking about doing this site building study to sort of set up for Phase two so that we had a baseline understanding of how end users think about building a website. So he started it by talking about things that we wanted to learn and the goals for the project. And we did this by doing some interviews with project stakeholders asking a bunch of questions about what went wrong, what went right, what is the vision for phase two to be sure that we were on the right track with our goals. And from that, we came up with three goals. The first was to understand how people currently build websites using WordPress or other tools. Sort of get a lay of the land, a basic understanding if where we’re at. The second was to learn more about the mental models that users have related to building a website. A mental model is basically how a user thinks about or believes that a system works. And those are really important to get right because if a user’s mental model doesn’t match the UIUI UI is an acronym for User Interface - the layout of the page the user interacts with. Think ‘how are they doing that’ and less about what they are doing. that you have for an interaction that yields frustration. But when you can align what you build with a user’s mental model, that can make a design feel really intuitive, easy to use. So we wanted to learn about people’s mental models. And a third goal was basically to find pain points and areas of friction which is just a fancy way of saying to find what sucks, what we can make better, where there are opportunities. And our first goal was to identify terms used in relation to the visual structure of the website. What do people call a site headerHeader The header of your site is typically the first thing people will experience. The masthead or header art located across the top of your page is part of the look and feel of your website. It can influence a visitor’s opinion about your content and you/ your organization’s brand. It may also look different on different screen sizes.? What do they call a navigation menuNavigation Menu A theme feature introduced with Version 3.0. WordPress includes an easy to use mechanism for giving various control options to get users to click from one place to another on a site.? We didn’t accomplish this goal because we didn’t really talk to enough people. And people use a whole lot of different terminology. So this one didn’t really happen. But that would may be worth looking into later. So those were our goals. Once we sort of settled on those then we drafted a research web. There’s a template for this that’s available in Dropbox paper where I’m busy making a bunch of sort of like toolkit resources. So if you want to write your own research plan, you can. They are a great way of clarifying your thinking and ensuring that you’ve considered everything in the research which is often times there’s quite a lot to consider. So the first step in there was to decide on a methodology. How are we gonna do the research? How are we gonna answer those questions we had? And very broadly speaking, most user research falls into one of two categories. It’s either explorative or evaluative. And generally speaking, explorative stuff can speak broader in scope. It tends to be more about understanding a problem. Understanding and empathizing with users. And it often happens at the beginning of a project. So this is what we use to here ’cause we’re at the beginning of phase two. We wanted to learn about how people build websites. We didn’t have something specific that we wanted to evaluate like we didn’t have a mock up that we wanted to pass. We just wanted to find more about how people thought about building websites. So that meant we wanted to go with sort of loosely structured interviews for our methodology. It also meant that its a but more vague, a bit more touchy feely, you don’t really know what you’re looking for when you start which can make it very difficult to communicate because its not as well-defined as something like usability test against. So that’s where we started. The next thing we wanted to do was go and choose the participants that we wanted to talk to. This is really difficult because again 30% of the internet is so many different people and we don’t really have a great understanding of all the different types of people who use WordPress. We went really simple with this and we used a sort of approach known as the Extremes and the Mainstreams which is obviously gonna work because it rhymes and the idea there is the majority of the people that you talk to are the mainstreams, the sort of average user. And again I put that in air quotes ’cause that’s very difficult to define with something like WordPress. But for our purposes we decided to focus on maybe people who haven’t spent a lot of time building a website or they felt a little uncomfortable with it. And our extremes were our power users. People who’ve been using WordPress for 10, 15 years. People who are very involved and then people who build sites for others. Ended up talking to a lot of people who build sites for others. Once we decided on that, that sort of leads really easily into our recruiting screener which is the survey that you use to filterFilter Filters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output. people and figure out which of these categories do they fit into. So then we write a script. Good way to do this is to workshop all the questions you have and just literally write down all the questions. There will inevitably be lots and then you form them into a script. Most interview scripts follow this pattern where you have an intro and some icebreakers and then you drill down into more detailed questions. For this, we talked a lot about their experiences with websites and their goals for their sites. Then we moved into a practical exercise which is a bit unusual for the exploratory research but it actually turned out to be one of the most interesting parts. So we had people show us how they used their website whether they used WordPress of something else. And we just sort of made it up on the fly. So people would show us how they WordPress or they’d show us how they build a site for somebody else or which plug-ins they had installed. It turned out to be really interesting actually. The other thing we did was dealing with logistics. This is where research gets really scattered because you need to cover all of these different things but there’s lots of different moving parts. So you wanna figure out how recruit users. Who’s going to be doing the research? How you handle consent because you need to be sure that you have consent forms from all of your participants. Because we are doing it in an open source space as well. We wanted to be sure that we respected user privacy while also allowing members of the community to participate in the research which meant we needed consent forms for everybody. We talked about how to communicate results of which of this is an eventual part and what the timeline would be. What tasks we needed to get done? Lots and lots and lots of lists. Does that make sense so far? I’m seeing mostly nodding heads so I’m just going to keep steaming along. I do see a chat. Oh wonderful! I can work with that, all right. Thanks Mark. This is how it went down when we actually did it. So the first trick is to make sure that users show up. That’s often times a job in it of itself. We started by using a service called I have recruitment.io written down here but I’m pretty sure its respondent.io isn’t it?

Yeah, its respondent.io

I do that every time.

Anyone looking to get recruitment their organization where you can set up your criteria and have screener questions and they find people for you. And one thing that is nice about them in particular is that you don’t pay for the service until they actually recruit and the participant shows up to do whatever

I don’t know that you’ve had any luck with them. I know people have. I had very mixed results when I used them for this study. Because I ended up with a lot of developers who told me they were beginners technically which I’m not sure was entirely on the level. And then I had a lot of the same people showing up. I did one screener for advanced users and one screener for beginner users and I got a lot of the same users in both batches. So I think I probably used it wrong. And I’d like to revisit that at some point. But we ended up working instead with a contract recruiter who worked part time at Automattic and we were able to use her. She was a delight. She found people, she made sure they showed up, she took care of all the consent and nonsense. All we had to do was show up which was totally magical. So we mostly collaborated in a private slackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. channel. Again that’s to preserve user’s privacy while still allowing for us to discuss them In a way that’s not super public. So for the discussion around the sessions themselves, that would happen in slack in the private slack channel. And then we just used Zoom for the sessions. A moderator would be visible and everybody else face muted so that participants didn’t feel like they were doing a panel exam or anything like that. And we just had a conversation that lasted about an hour and loosely followed the script. Especially for explorative research like this. You don’t tend to follow the script word for word ’cause that would be really awkward. So its more like a conversation that goes back and forth and you pick up on thing that seem interesting. Somebody might mention something like oh I tried putting a widgetWidget A WordPress Widget is a small block that performs a specific function. You can add these widgets in sidebars also known as widget-ready areas on your web page. WordPress widgets were originally created to provide a simple and easy-to-use way of giving design and structure control of the WordPress theme to the user. in my header but it didn’t work. So you gotta maybe follow up on that and dig deeper. We had people taking notes for every single session. Often times multiple people which meant there were lots of notes. And we also did a lot of back chatter in the private slack chatroom. When the session was over, we did an unstructured debrief session that was sometimes two minutes and then sometimes half an hour just discussing overall insights and takeaways. You can learn a lot just from observing one session. There’s a lot to takeaway there. So once that was all done, we have so many notes. We did, I think, 17 sessions in total. And we had at least one set of notes for everyone, every session, often times two or three and then we also had the slack log so there were additional insights in there. So basically we just had a lot of data. And at that point we needed to figure out what on Earth to do with all this data. So, if my computer would work. Oh, this is what we did with that data.

Hey Sara, just to jump in real quick so part of the exploratory research so if you’re setting up a type of research and you’re thinking of things that are more exploratory where you’re asking a lot of open ended questions, this is what we call qualitative research where we’re seeking quantitative information so you can see I don’t get much sleep due to children as quantitative. We have something very specific that we can capture in that It takes a lot more sorting and a lot more organization for figuring out how to find themes and how to find patterns in the data. However, sometimes that is the most rewarding, awarding both types of research because you’re getting a more robust set of answers than say a survey that you only have four choices and someone has to select that so I would just introduce the air table and so I’ll have you talk a little bit more about how you set up the data points and the coding and things that for folks who do not use air table this was amazing qualitative research.

Yeah that’s a good note about qualitative and quantitative Qualitative is really like its sort of touchy feely. Its so tempting to boil everything down to numbers ’cause then you just have easily crunchable data but with this kind of data, as you can see here like our data points are things like oh I look for free services first. That’s not really something that’s quantifiable but its useful to know, well its probably useful to know. So yeah, this is my giant air table table. It is so big that it is over the limits, the size limits. And I need to delete eight records. So basically we took these notes and the debrief, all the tacks that we had in big long pieces and we broke it down to individual pieces so each little insight was a piece of data that got coded here. So I want to expand my business, that’s a piece of data. And we classify them with codes. As we started working through all of these individual insights we’d give each one a code. You can see money is showing up a lot here. ‘Cause people have lots of thoughts about that and this is also the section of the research where we’re talking about some of these questions. Time shows up a lot. So we have this huge database with all of these codes and I can go through it at any point and look. Okay, what do people think about menus? So I can find all of the insights that are related to that. Air table is kind of magical for this because its like a database or a spreadsheet sort of mashed together. I don’t know how many people have used it but it gives you these configurable views so you can stack and arrange things in different ways. I found it very helpful for picking out insights from a large data set like this. So yeah, we coded everything. Why is my computer being completely bananas? Okay, we coded everything We classified the types of data as well into certain so we had quotes, we had stories which we never actually managed to do anything with which is kind okay like a bummer. Pain points basically areas where people were in pain. General insights, goals, tasks, philosophies, emotions, and vocabulary. So we have this big huge database full of insights. Its searchable and at this point really hard to quantify exactly what we could do at this point because I feel like a lot of it was reading through the data and picking out patterns which is just like a very intuitive kind of thing and its difficult to quantify coding helps and looking through the data reconfiguring it, it helped. Buts its mostly about looking through and looking for patterns. I don’t know a good way of making that more explicit. But, so this is obviously kind of a lot of work In total we talked to 17 different participants over seven days and we had 25 different people involved either as facilitators, as researchers, as note-takers, as observers, as analysts going through the giant table. So lots of people all of whom have been really fantastic. So I just want to throw a thank you slide in there. So that’s kind of how it worked. I think I’m already way past time. So lets jump right into looking at some of the results. These results are also shared on make/design I will share the very long link to it shortly. But let’s look really quickly at the key findings that came out of this. The first one was technological change presents cognitive challenges. This seems really obvious but saw it both in experienced users and beginners. We actually had a GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ drinking game that didn’t actually involve any drinking but it was more, we didn’t ask about Gutenberg at all in the script but it always came up and you can kinda tell after awhile when it was coming so we wait to see how long into the session somebody would independently bring it up and what their feelings were. Their feelings were never neutral. They usually had strong feelings one way or the other. And it was always interesting to see what it was. But for most people they feel whether they’re involved in technology on a daily basis or not they feel that technology moves really quickly and there’s always something new to learn and a lot of them really struggled with fear of learning something new without good reason to. They needed the reason to make that investment into learning something new otherwise it was just another thing on their plate. Its not exciting. Which kind of leads to second finding which was that people don’t think that building a website is fun. Sometimes because we’re involved more closely with building websites we think building a website is fun. You’re making something. But the majority of users really don’t, its an obstacle, its a chore. The site isn’t the end result, its just what they’re doing to get to their goal which is actually like way over here. So while we have more patience with maybe things that aren’t as elegant or easy ’cause they could be. A lot of users don’t. A third finding was that WordPress’ theming system and this is a really controversial finding that a lot of people were not crazy about. The way that WordPress’ theming system doesn’t match the way that people think about building a site. That came up a lot of times and the key cause of confusion there was that style and layout aren’t always intertwined. The way that the theming system works they are so people really struggle. A lot of people told us they’d spend hours picking a theme. A lot of people had one theme that they knew and they used constantly because they that it worked. But a lot of people really struggled to find a theme that gave them both the functionality and the layout that they wanted as well as the styling that they wanted. So that was a major pain point that we uncovered. The other finding was that people don’t have a predefined vision of what a site would look like, should look like. I started asking this ’cause I’m a designer so if I build a site I can see it in my head before I build it. I feel that way about well that’s everything actually. And I often assume that other people are the same way that they have a vision in their head of what something needs to look like before it happens. But most of the users that we talked to told us they didn’t have that at all. I think we had one who’s like yeah no absolutely. But most users including the users that build sites for others told us they didn’t have an idea and they tended to sort of browse so a lot of time would be spent browsing through theme directories to find something that gets close and then they would play around by sort of trial and error to see how close they can get to something that they wanted even if they didn’t have a clear vision of what that was.

Sara, to add to that, I was gonna say to add to that, that being said, people might not have visually known what they wanted but they knew their content. They knew what they wanted to write. They knew the types of content so a lot of times they would look at the out of box experience and say okay I know I need this, this, and that. There’s how much of that is covered in perhaps the coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. experience. Here’s what’s added by this thing, by this thing, or this thing. So it was interesting we hear often that the content first approach, that really was. They were thinking of content first. They weren’t like what visually looks cool. Oh, this looks great now I’m gonna write a website about this type of around this theme ’cause the them looks cool so that was another way to think of that key finding number four. People know their content they just don’t have that visual of I’m gonna have it a homepage, an About Us page, and this that and the other so just wanted to add that little caveat there, Sara.

Yeah, that’s actually a way better I realize I don’t do that either and I definitely should. It’s nice to see that users do that. It kind of ties to, I spent like eight years working with clients and they tended to say a lot of things like I don’t know what I want but I know it when I see it which definitely drove me completely bananas. But I think that’s a really, really common stance for people they will recognize something that they want when they see it but before they see it they can’t clearly express what they were looking for. And part of that is definitely they have an idea of what content they have and what they’re trying to express but not necessarily how that needs to be expressed visually which is often times very different. Yeah, so those are sort of the the key findings we have. There are a lot more findings There’s recommendations for future product direction and we also profiled the three segments that we identified which were site builders, people who buy old sites for others, bloggers, people who write blogs, and small businesses. Those were the categories that we saw people falling into as we interviewed them. So we have profiles more in depth on each of the segments. And we have a more in depth analysis of the themes that we uncovered. There’s a series of five very long blog posts on make.wordpress.orgWordPress.org The community site where WordPress code is created and shared by the users. This is where you can download the source code for WordPress core, plugins and themes as well as the central location for community conversations and organization. https://wordpress.org//design/ this big long link It starts there and then there are four other posts that go into these findings in more depths. Is that somebody sharing? Its not somebody sharing. I can share that link if its helpful in slack later. I wanted to give a brief plug for future research efforts and then we can ask more questions. So we have more research efforts ramping up now. These, sorry, these ones are usability testing. So they’re more evaluative. The nice thing about usability testing is its a bit more straight forward. You can tell if something’s working or not. We probably won’t need to make a giant air table, I hope And so right now this week I’m currently setting up usability testing sessions for a proposed design for the navigation menu blockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. and those start tomorrow. I know some of you here are getting calendar invitations If you want calendar invitations, let me know. I’m sending out lots. Those will be before the session the Zoom link will be shared in the private channel so you can join in. There’s also a make design post that goes into a bit more detail about what we’re doing exactly. But basically I will invite you to the sessions. The easiest way to get started is to just join as a silent participant which means again your face mute and you can just watch and see how it goes you can also take notes if you’re feeling really brave you can moderate a session, it’s not that hard, I swear. There’s lots of ways to get involved. Again, more note takers is really helpful. So if you want to get involved with that, that should be happening both this week and next week. I have seven scheduled for this week so if you’ve been dying to watch usability session, now is your chance. And I haven’t entirely determined what we’re going to do afterwards but the analysis will be a lot less intense so if that’s something you’re interested in as well, we’ll also be doing some form of analysis. But I think its gonna be a lot shorter this time and no five long posts of thousands and thousands of words. If you’re interested in doing your own research, that’s also great. I want to really encourage people to do their own research. So one of the things that I’ve been working on is developing some sort of toolkit templates and stuff. So I have a research plan template and bunch of guides for note takers and moderators and debrief sessions. There’s all kinds of resources. So if you are ever at any point interested in running your own research there’s a slack channel, it is research. It’s a good place to join in and ask questions and see how it all works. So there are lots of people there who are interested in research and will be delighted to help you on your way to do more research. That’s basically my plug. And that is all for my slides. So, I’m going to stop sharing and everything disappears. So, do we have questions? Does that help to clarify some of what we did?

Really great work the both of you. That was amazing the amount of data that you both sifted through and pulled together into something that was much less textual abundant in these posts and broken out so nicely so that everybody can just follow along and really understand so thank you very much for that. I do have a question. These bank of people, maybe you addressed it and I might’ve missed it. All these people that we have been talking to are we able to communicate with them again in the future or use them again in other tests?

So, kind of. I don’t actually have direct contact information for those people. They came from a bank of Jetpack users who were recruited for Automattic research and because we worked with Lilidat she basically controlled access to that database. She gave many first names and some information and that was usually how we divided it. That is what I’m trying to do for the next sessions that I’m currently recruiting for as well. Last names are, and I’ve talked to a lawyer about this, but last names count as private, identifying information and so we want to limit that as much as possible like limit the number of people that have access to it. It’s kind of one of the weird challenges about doing research in an open sourcey kinda way ’cause I desperately want everything to be transparent and open but I also don’t want to violate anyone’s privacy or accidentally run a mock GPR or something like that. So that’s an ongoing challenge and my hope is that at some point we will have some sort of database of users that we can recruit from. Woo Commerce has this. They have a great covert design feedback, the Woo Design Feedback Group I think. And they got like five or six people a month signing up propose nothing. And basically they have this pool of Word recruited leads that they can then contact with research opportunities That’s like the distant vision dream. The current dream is we don’t have that. And starting it up involves some, its a bigger project that will require some coordination and be really certain that we do it right in terms of legality, and in terms of privacy, in terms of making it not gate-keepered. So the answer is no. But its something I’ve spent a lot of time thinking about. And hopefully that will get easier and better in the future.

Looks like there are some, there is a question in the chat

I see a question that says what would you change about the process if you could? Thank you Jammy.

You know one thing that was interesting about the process is that with the first time that I had personally collaborated with a larger group of people for research and that was great. So its interesting to say what would would you change about the process simply because it was sort of like we were creating the process as we went and so in some ways I had a lot of feedback in other ways I think it was perfect ’cause what we gained out of that just from learning what to do next and how to approach it next time with the research it had been wonderful and the framework that Dan’s creating is great too. So, I know its a little bit of a cop out answer but I think that we were really I think if I said what would I change about the process well that we would have a framework set up at the beginning versus making up framework as we were going as we were doing it but that’s what we have for next time. So its hard to complain about that when you have it set up really well for the next time we do this.

I feel like so I definitely agree I want the framework to be like two years ahead of where it is right now. If that could be true. I think, sometimes the thing that I really struggle with was doing research. There’s so many different things happening all over the place and things that you need to pay attention to and do. So having more defined process and existing things to use so have script, template. Having to do less on checklists. I think more checklists would be really helpful but definitely ironing out more parts of the process would make things a lot easier.

Sara, I’m gonna

Typically

Oh, sorry.

No, no go ahead.

When you said checklist, I actually do have one. Sara and I both lamented that we didn’t really have a project manager, that we were really doing a lot of the project management of it which is fine however for me personally, that’s not my skillset. I thin project managers are really great at what they do and they have a knack for it. So there were a lot of times that those checklists just to have someone say here’s a timeline, or here to do and not having to use that side of the brain power to also be coordinating that would’ve been really helpful. Just, I remember you and I and Sarah talking about that often so when you were saying the checklist it reminded me of that so my apologies for interrupting but a dedicated project manager just like a dedicated person was with recruitment because recruitment takes so much time those two things are amazing. That’s like red carpet for researchers.

Definitely agreed on the project manager, definitely agreed on the recruitment person. Basically I just want to show up and talk to somebody and then write things down but there’s a lot of like work you need to do in advance and planning and checklists and calendars. Right now I’m doing recruiting on my own and oh my God its so nice having somebody do it for you. So definitely there’s some areas where we can improve process and part of that I think is going to be getting more people involved. It’s actually when we did this research I kind of wasn’t expecting to have a lot of participation from the community. I thought it would just be us going at it and doing everything and we actually have a lot of people interested in joining in to help which was really, really, really helpful especially at the end where we just had all this tables and tables of stuff to go through. So I think more people involved will help. One of the other things I realized is that opening up that process means that there’s sort of a bigger timeline for everything. So I was trying to close things on Make Design and somebody at one point mentioned to me that she couldn’t keep up with the rate at which I was posting because its not her full time job to keep up with the stuff that I’m posting. She’s doing this outside of her full time job and so she said it was nice to see this happening but it was happening a bit fast to keep up. So I think that’s challenging to say ’cause it took us a long time to do this research. Anyway, but I think more time and definitely however much time you think its gonna take multiply it by like three.

Thought I’d read another question from Dan up in chat. Why were the reactions to way WordPress does themes controversial? Was the word you used. Which to Dan seems unsurprising. However, sounds like he’d like to hear a little bit more detail and I know that was more in your warehouse of those findings ’cause you talk so particular researcher so I’ll have you answer that. Or participant not researcher.

Sorry. So I think you’re right to a lot of people. It maybe depends on how you use WordPress and what your experience with WordPress has been. Because for people who are involved in the open source project, even there there’s just a really wide range of types of experiences that people have had. So, people who’ve been doing a lot of maybe like theming support tend to know the theming system is broken. Its not working for people because they heard a lot of people complaining about it. But I think I would need to dig into this a bit more to give a really good answer. But I’m gonna try to give a medium answer instead. I think, so this is something that we saw both in the research when we talked to site builders who’d often been using WordPress for like 10 or 15 years. WordPress has been around for 15 years, right? They’ve been using WordPress for a really long time and they’ve learned all the ins and outs of the system They know how certain things work in WordPress even if its not immediately intuitive they spent time and invested time in learning that model and they know it really well and they’re able to use it really well And I think when you’re in that head space it becomes very difficult to see how those things can be confusing for somebody who is coming at it with more of a beginner mind.

Sure, yeah.

Because you’re not a beginner. And we also saw a lot of the people who, and this was actually really fascinating, see they’d often buy one of the multifunctional, big, huge themes. I don’t know the names of any of them off the top of my head. But a lot of people Sorry?

Avada I think is one of them.

Yeah, so they’re really intense and somebody showed us her press tasks for going through one of these themes and it was ridiculously confusing for me ’cause I hadn’t seen that before and its very, there’s like a bunch of settings on one page and then you go to a different screen entirely to see how those settings impact your site. So for me approaching it was like more of a beginner brain I was like oh my goodness that’s terrifying how do you ever use this? But because she invested a lot of time in learning this model of building websites she was like no, this absolutely makes sense. This is the easiest way possible to make a website. And I think that’s a lot of what we’re seeing when people are like the theming system totally makes sense. Nothing’s broken. But that’s sort of a conflict that WordPress is dealing with right now. How do you make things both satisfy the needs of these power users who’ve been using it for 10 years and love WordPress and think its the greatest thing in the world but also make it accessible to users who don’t have that depth of experience and understanding and that aren’t invested in learning it.

So in the tests, there were different themes that were used? Did the users select their own theme or you threw Avada at them or something else over here?

We kind of made it up as we went. It sounds very unsciency

No.

Sort of depended on, we didn’t have like a specific task or anything like that. We just wanted to see how people used their website. So we had some people show us their website on Squarespace which was interesting they sort of talked through the things they would like and they didn’t like. I think I broke one guy’s website accidentally. But he fixed it afterwards. I went to email him but then I checked and it was fixed. But we’d have a lot of people show us stuff they were working on. So there’s this one guy who he was writing this big long music book and he had drafted the post for the music book in WordPress all as one big long post. But he drafted them, he was gonna move to medium, and he drafted them the way he thought they would look on medium and he showed us his site and he was like yes, this is what it would like in medium except its gonna look nicer. That wasreally interesting. It was less sort of testing and more peering over somebody’s shoulder while they worked with their websites. Sometimes it didn’t work out too well. Like sometimes we sort of got stuck and wasn’t allowed to explore but sometimes you’d look and somebody has like six notifications up and they had six to eight different themes or plug-ins that they didn’t know what they for. It was really interesting just watching them In kind of areas where there were maybe patching over a pain point in the software which I can’t think of any examples of

So it looks like there is, starting to get to the top of the hour here. I see there’s one more question that maybe we can do. How can we not set and forget this research? What can we do to keep it central the work we do? I think one thing certainly is to keep referring back to it and to also expand it. I know there’s other research studies happening but for example one of the early stakeholder interviews that we had. We asked how the current experience was people? And what that looks like and how they like it or not like it I think in one year time we can have that exhaust team so they’d go out and see the differences in what people see. So that’s one part. And the other part with the site builders study in particular I think there’s lots of ways to keep bringing it up, expanding it, and taking little bits like we couldn’t find vocabulary was one of the goals that we had. How do people talk about news and say navigation or header or picture or things like that. You could certainly expand upon. This was exciting research and I agree I would not like for it to be set and forget. Sarah your thoughts on other ways we can keep it going.

That’s the big concern of mine too. My hope is that doing a walk-through will give it a little bit of a boost now. Having put together slides, I’m now realizing that we need to store it somewhere more findable than Make.wordpress.org/design/19/whatever So making the results more findable. Trying to refer back to us. I think that’s something we’ll have to have a bigger conversation about because I don’t know yet

And I think giving a talk at Word Camp you asked about the fact right now is that at the time of this recording that the talks are open not enrollment, I can’t think of my words right now but you can apply right now. The application is open. So that we be another good way to do this.

Cool.

Think that kind of wraps up our time. So any last thoughts before I stop the recording?

I think I’m good. I think I’ve chattered enough.

Cool. Well thank you for everyone who showed up at this again to Make.wordpress.org/design site is a great place to look at research. We also have a research channel in slack. We wanna make this as open as possible and so we love people’s time, thoughts, ideas, all the things, come join.