Summary

The discussion dives into the details of the Annual WordPress Survey. During the conversation, Executive Director of WordPress Josepha Haden Chomphosy (chanthaboune) sheds light on the survey’s origin, objectives, development, usage, and the distribution of the survey data. Multiple participants provided feedback on the survey’s format, questions, and data use. There was a strong emphasis on the importance of data for decision-making, with suggestions for future surveys to be more inclusive, accessible, and community-driven.

Participants

Listed for clarity are participant’s SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. handles, with full names linked to WordPress.orgWordPress.org The community site where WordPress code is created and shared by the users. This is where you can download the source code for WordPress core, plugins and themes as well as the central location for community conversations and organization. https://wordpress.org/ profiles

• sereedmedia – Sé Reed
• chanthaboune – Josepha Haden Chomphosy
• gusas – Gus Austin
• justlevine – Dovid Levine
• patriciabt – Patricia BT
• aurooba – Aurooba Ahmed
• courtney – Courtney Robertson
• Zack Katz – Zack Katz
• peterwilsoncc – Peter Wilson

Key Takeaways

Background

• The Annual WordPress Survey was initially for State of the WordState of the Word This is the annual report given by Matt Mullenweg, founder of WordPress at WordCamp US. It looks at what we’ve done, what we’re doing, and the future of WordPress. https://wordpress.tv/tag/state-of-the-word/. data collection, but its goals have evolved over time. Currently, its main focus is on contributors and the WordPress ecosystem.

Survey Goals

• Initially, the survey was for data collection for the State of the Word (SotW).
• The survey’s objectives have transitioned to gauge ecosystem health and engagement, and it now helps inform initiatives and focus areas for WordPress.
• The input from the survey shapes the direction and strategy of WordPress projects.

Survey Development

• chanthaboune states that the survey has been developed independently of the community. It was created by Jen Mylo, refined by a career researcher, and now is trialed as a census by Dan Soschin.
• Long-term, chanthaboune aims to embed a concise, census-style widgetWidget A WordPress Widget is a small block that performs a specific function. You can add these widgets in sidebars also known as widget-ready areas on your web page. WordPress widgets were originally created to provide a simple and easy-to-use way of giving design and structure control of the WordPress theme to the user. in WordPress CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. for users to denote their roles and functions.

Survey Data Distribution

• The survey results are distributed on WordPress.org/news.
• Chanthaboune has opted not to share the raw data to avoid potential issues related to privacy and individual identification.
• There is a comprehensive slide deck available that provides the data but not in its raw form.

Future Community Participation

• Chanthaboune does not currently see a clear path to community involvement in developing the survey.
• The distinction between work done within the Make WordPress community and external work was emphasized.
• Chanthaboune notes that while “the ownership of the process lies with Automattic and therefore so does the responsibility,” they are open to feedback.

Open Q&A

• Chanthaboune raised the idea of potentially discontinuing the annual survey.
• Participants expressed the value of survey data, suggesting adopting models similar to State of CSSCSS CSS is an acronym for cascading style sheets. This is what controls the design or look and feel of a site./JS and other renowned surveys.

Detailed Summary

Conversation in Slack begins here: https://wordpress.slack.com/archives/C0GKJ7TFA/p1696950347923519

Introduction

sereedmedia welcomes chanthaboune and begins the discussion about the Annual Survey.

chanthaboune shares their professional history, highlighting their role as the Executive Director of WordPress since 2019, their preceding involvement with the Community team, receiving sponsorship from 2015 on, and their earlier marketing roles.

1. What are the goals of the Annual WordPress Survey?

chanthaboune delineates the origin and progression of the Annual WordPress Survey:

•     Initially devised for State of the Word (SotW) data collection.
•     Conceived by Jen Mylo, a former sponsored WordPress contributor.
•     Between 2013-2015, added questions concerning contributor experiences.
•     A dip in 2018 responses instigated a revamp by a specialized contractor.
•     Objectives transitioned from SotW to gauging ecosystem health and engagement.
•     The survey became a tool primarily for chanthaboune to sense sentiments regarding WordPress projects, notably GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/.
•     The present survey goals are intended to be predominantly about contributors and the ecosystem.

chanthaboune clarifies the survey’s objectives, citing their comments in the 2022 survey post:

“My goal is that the input received via the survey helps inform initiatives and focus areas for WordPress in the near term, along with other signals, such as conversations in the community, the Making WordPress blogs, and events. Such data play a key role in shaping the direction and strategy of the project and measuring progress in focus areas.”

chanthaboune expands on that, stating that the survey “helps the entire WordPress ecosystem understand its users and itself, more completely. It also informs the program goals and strategy of WordPress.org contributor teams.”

sereedmedia asks how the survey is currently used to inform the program goals and strategy. chanthaboune provides a recent example: “from 2018 on, I kept tabs on reported hours to create sites compared to reported complexity (via plugins, themes, and overall customization of installations). The complexity reported was increasing while reported time-to-launch was decreasing. This indicated to me that WordPress was becoming easier to use (a net benefit for us all).”

chanthaboune confirms there is no strategy document for the survey and no measurable goals expressed such as % increase in people taking it.

chanthaboune states “It might be worth noting that this year I asked for something pretty dramatically different, because we’ve struggled so much to get a good response rate in the past few years.”

patriciabt asks if the primary objective was changed this year to focus on response rates, how did that translate to what questions were included/cut, and what would next year’s iterative process look like when it’s assumedly not ‘something pretty dramatically different’?

chanthaboune responds “I looked at questions with static responses or that showed very clear trends in one direction or another. I prioritized questions where it seemed possible we could make some big gains with small changes.”

justlevine voices challenges in designating “roles and functions” in the survey and proposes modifications for greater inclusivity, such as an “other” section. chanthaboune acknowledges justlevine’s feedback.

Open Questions

• “How [do] the [survey] goals relate to the survey itself?” – justlevine
• “How are the team[s] involved before the survey for eventual specific questions?” – patriciabt

2. How is the survey currently developed?

chanthaboune reiterates that “this survey has always been developed independent of the community itself.” The survey was created by Jen Mylo (former sponsored contributor), then it was refined by a career researcher, and now it’s being trialed as a census of sorts by Dan Soschin (current sponsored contributor to the WordPress marketing team). chanthaboune stated that, other than Dan’s involvement, “this was not driven by marketing at all.”

sereedmedia requests details on the census-style approach. chanthaboune shares their long-term goal to embed a concise, census-style widget in WordPress Core where users could then easily denote their association with the project, roles, and functions.

chanthaboune expresses the “annual, mass gathering of data hasn’t really continued to be as fruitful as it once was. But I still do want to have an understanding of how we collectively identify.” This year’s shorter version of the survey was intended to confirm that length of the survey was the reason for less participation.

justlevine mentions the survey was not just shorter, but emphasized the removal of specific questions related to “contributors and the ecosystem” and those regarding the Editor. chanthaboune points to the contributor section and proposes the possibility of having separate surveys for such specific topics in the future, especially if the response rate for the current format improves. Regarding the editor questions, chanthaboune explains that there was a consistent trend of decreasing usage of the Classic-only feature, which influenced their decision to remove certain questions to shorten the survey. 

chanthaboune asks what questions that were providing value were omitted this year. justlevine responds with details on questions regarding the Editor that were not included.

3. How is the survey data currently distributed?

chanthaboune states that the survey is distributed on WordPress.org/news.

sereedmedia references a 2022 post which only provides “key takeaways” and not the entire dataset. They seek clarification about the data processing and distribution methods.

chanthaboune discloses their decision to not share the raw data due to concerns about pinpointing individuals based on the responses. They emphasize not wanting to participate in naming or shaming. However, they point out that a slide deck exists that provides comprehensive data, though not in raw form.

sereedmedia requests more information on the concerns in sharing raw numerical data that do not contain narrative responses.

chanthaboune mentions that previously a contractor was involved in processing the data. 

sereedmedia emphasizes the implications for the community as third-party contractors are outside of the “contributor” frame, and they have to be paid for directly, which then become “hidden costs” of the project that aren’t documented for the community at large.

chanthaboune shares that Automattic paid for the contractor services in the past. They further share that currently, the non-contractor plan involves initial analysis by Dan, followed by a secondary analysis by chanthaboune.

justlevine asks if Automattic, as the sponsor to the contractor, had access to the raw data. chanthaboune clarifies that aside from themself and former sponsored contributor Andrea Middleton, no one else had access.

peterwilsoncc discusses concerns about maintaining respondent anonymity when sharing raw survey data. They express appreciation for the data being processed independently and bring attention to potential issues related to data privacy.

justlevine appreciates the external processing of survey data and suggests establishing a transparent process for data release and feedback extraction, fostering transparency and accountability.

Open Questions

• “How is the data that is shared selected?” – sereedmedia
• “What are the concerns about sharing the raw aggregated numerical data, i.e. not the narrative responses?” – sereedmedia

4. In the future, how can the Marketing team and Make WordPress community participate in the survey development?

chanthaboune notes a perceived “othering” of Automattic employees in the conversation and suspect it is a concern unrelated to the survey itself. justlevine clarifies there is no intention on their part to “other” Automattic employees and suggests adding a data policy to future surveys to address concerns about raw data sharing.

sereedmedia clarifies their position, emphasizing a distinction between work done within the Make WordPress structure and work done externally. They further elaborate, indicating they differentiate based on where the work takes place, not who does it. chanthaboune asks for a clearer understanding of why such distinction matters and provides examples of various activities that might be affected by this distinction, such as podcasting. sereedmedia explains that their comments were in the context of how the MakeWP community can participate in survey development and the distinction matters because the survey currently takes place outside the MakeWP contributor structure.

chanthaboune responds that “In the case of the survey, the ownership of the process lies with Automattic and therefore so does the responsibility”. They emphasize that they are open to feedback. 

chanthaboune reiterates that “how can folks be more involved in the development of this? I’m not sure that there’s a clear path forward on that for me.”

“When the survey first came to exist, it was both time based and mission critical. While the goals and purpose have changed, for me it still remains time based and mission critical. I was reminded last week that we have a lot of volunteers here (which is exactly what we want!) but that it means I/WordPress/anyone shouldn’t always assume that we have full access to their intended time.”

coachbirgit inquires about the potential for more specific surveys besides the annual one. chanthaboune expresses a wish for a census survey and opt-in series for participants. sereedmedia expresses optimism about realizing chanthaboune’s survey vision. chanthaboune replies that they are “not sure” about that. 

sereedmedia asks for clarification about the challenges chanthaboune perceives. chanthaboune responds “the question here, if I’m tracking correctly is twofold:”

1. Question: What prevents us from having surveys managed entirely by the community?
   Answer: “The survey as it has been is a massive undertaking. It has taken upwards of 100 hours to manage and analyze it every year and takes specialized knowledge to code and cross reference it to past years. Unlike some other administrative tasks in the project that require a heavy lift, there is no “alternative economy” to gain currency in, either. You don’t wind up with more cred in Core or more connections in your own WordPress network.  And as is the case with every task or role I’m asked to add to our set of available contributions, it behooves me to balance the necessity against the effort. I never want to add work for work’s sake, so I have to ask myself whether I would rather Marketing amplify the new tutorials on Learn or wordsmith a survey that shows declining interest. Even if we set aside the very real existence of HR-level feedback about regular, every day contributors it’s hard for me to ask unsponsored contributors to take on that amount of work. As a side note, I do see that we’ve got teams where that’s used as an excuse to abuse people who are sponsored, which I have to grapple with daily. Please, no one read into my comments any approval to treat each other abusively.”
   
2. Question: What prevents us from being able to have multiple surveys?
   Answer: “On the one hand, the time estimates for the surveys come in to play here as well (which doesn’t really mention translations—in to, out of, and follow up for errors in understanding). But if we imagine a future where we have a low-effort census and then any number of team-driven surveys (where the processes are smoother than a river stone)…it’s my experience that every team relies on the larger “WordPress” entity to raise awareness. Even if those surveys are opt-in, there is always a threshold at which the repetition causes people to ignore what’s there. Which ultimately runs the risk of surveying just ourselves, which is part of the problem that we’re worried about with the declining response rate anyway.”

justlevine suggests posting a draft of the survey or a post-mortem for feedback to bridge any gaps. chanthaboune agrees that a post-mortem will be essential.

patriciabt suggests allowing teams to create their own surveys for specific concerns.

justlevine asks about the primary objective behind the “slimmed down” survey, if that will be the same goal next year, and how the actual survey questions met/can meet these goals on the future. chanthaboune explains “I looked at questions with static responses or that showed very clear trends in one direction or another. I prioritized questions where it seemed possible we could make some big gains with small changes.”

coachbirgit raises concerns about the survey questions guiding participants toward positive outcomes and asks if the MakeWP community can review questions beforehand. They also ask about missing “teams” and “working groups” from the survey.

chanthaboune clarifies that working groups aren’t considered teams and seeks confirmation if the missing team refers to “Security”. coachbirgit confirms Security was a missing team, and expands that it would be “nice to have a generic option to state that one participated in working groups besides the “official” teams – maybe a free form field”

chanthaboune states “there was no intent to hide anything or curate a specific outcome other than to test whether the length of the past survey was the issue with the response rate.”

Courtney voices some concerns about the survey, touching on potential privacy issues, especially regarding age-related questions. They also presented broader inquiries about the survey’s format, clarity, and reach, and pondered on ways to make the raw data more accessible while respecting respondent privacy. 

Responding to Courtney’s privacy concerns about age specificity, chanthaboune expresses that they do not see it as a privacy breach. On the topic of response rates, they share their concerns, mentioning the declining participation in recent years and speculating if the survey’s extensive length might be a factor. 

chanthaboune also discussed the challenges tied to multiple surveys, emphasizing the potential risks of survey fatigue among respondents. patriciabt questions the relevance of survey fatigue if the surveys are opt-in and sought clarity on how specific questions could be tailored for a general audience, especially regarding distinct teams or working groups.

chanthaboune shared their experience that team-specific initiatives often come with an expectation from the broader WordPress community for widespread promotion. They emphasized the potential risk of respondents growing indifferent to continuous survey requests.

Open Questions

• “In the future, how can the Marketing team and Make WordPress community participate in the survey development?” – sereedmedia
• “Is there any way the MakeWP community can review the questions before the census gets published?” – coachbirgit
• “Any way to address parts of the form that could be improved next time?” – Courtney
• “And bigger still, how can we increase response rates that do include diversity in locales, scales of economy, and other factors? What can we do more specifically to increase response rate ahead of next year?” – Courtney
• “Could the raw data per answer minus open-ended questions be made public?” – Courtney
• “It’s still vague to me on how we could ask a general audience specific questions for a given team or working group.” – patriciabt

5. Open Q&A

chanthaboune poses a question about the potential discontinuation of the annual survey. 

justlevine expresses disappointment at the idea of discontinuing the survey and emphasizes the value of data, advocating for a survey model akin to the State of CSS/JS. They stress that even limited data can be valuable and play a role in decision-making.

aurooba shares their experience with the current survey format, citing accessibilityAccessibility Accessibility (commonly shortened to a11y) refers to the design of products, devices, services, or environments for people with disabilities. The concept of accessible design ensures both “direct access” (i.e. unassisted) and “indirect access” meaning compatibility with a person’s assistive technology (for example, computer screen readers). (https://en.wikipedia.org/wiki/Accessibility) issues and unusual question choices. They suggest improvements and greater community involvement while endorsing the adoption of a survey format similar to the State of CSS/JS.

chanthaboune inquires if people have been utilizing the survey results over time. justlevine responds affirmatively, highlighting the survey’s role in guiding decisions for developers, such as “how to prioritize adding new feature support, or what repo labels I should pay extra attention to” and “to determine whether it’s time to ‘learn that new skill’”

gusaus raises the question of potential overlap between post-summit discussions and the annual survey. chanthaboune clarifies the distinctions between the Summit and the annual survey, highlighting their different focuses and the types of questions each seeks to address.

Courtney and aurooba express interest in the State of CSS/JS survey model, discussing its creation and evolution. Courtney offers to connect chanthaboune with OpenJS Executive Director. Zack Katz mentions that there is an annual State of Laravel survey as well.

Additional surveys mentioned:

• https://2022.stateofcss.com/en-US/explorer/
• https://www.stateofdeveloperrelations.com/2023devrelreport
• https://www.zend.com/resources/2023-php-landscape-report
• https://stateoflaravel.com/results

Thank you to @aurooba and @courane01 for reviewing and contributing to this post.