Aditya Kane 1:17 pm on August 19, 2019
Tags: proposal ( 8 ), tools ( 2 )

Proposal: Speaker feedback tool

In the past, the only ways we have gathered feedback from WordCampWordCamp WordCamps are casual, locally-organized conferences covering everything related to WordPress. They're one of the places where the WordPress community comes together to teach one another what they’ve learned throughout the year and share the joy. Learn more. attendees about a speaker session was to use a survey/poll or collect feedback at the event via paper forms.

There are a few problems with these approaches:

Surveys are sent out after the event and therefore don’t usually have good response rates.
The more time that has passed since an attendee has seen a session, the less detail they might remember, which makes the feedback less precise.
The default survey does not collect very much detail about session content and presentation delivery.
Feedback shared in hard copy isn’t easy to share with speakers (so they can grow their skills) or track (so the organizers can compare year to year).

This is a proposal that we build a special speaker-feedback tool to collect attendee feedback that solves those problems.

Goals of the tool:

Collect feedback for individual sessions during the event.
Provide easy access to feedback to the WordCamp organizers and speakers.

Where could it live?

The feedback tool could be accessed at an easy-to-remember URLURL A specific web address of a website or web page on the Internet, such as a website’s URL www.wordpress.org, like <year.cityname.wordcamp.org/feedback>
The schedule page could have a ‘Feedback’ button on each displayed session.

Possible requirements worth some discussion:

Because anonymous feedback is more likely to include abuse, should the feedback tool require users to be logged in?
Should there be an automated way to report abuse to the Community Team?
Would it be helpful for organizers to be able to edit the text-based feedback, so as to remove abuse, slang terms, confusing content, and/or to correct spellings before sharing with speakers?
Should there be a way for the feedback to be made public, and if so, should it show up anywhere other than in the comments on each individual Session?
Should there be a way to export the feedback, and should feedback be included in a requested privacy export?

Feedback Formats

Feedback could be given in a few different ways – either on their own or as a combination:

Emojis

Pros: Simple, standardized way of showing how an attendee felt about a talk. Encourages positive feedback.
Cons: Could come at the expense if useful critical feedback.

Ratings

Pros: Can be provided quickly. Usually allows for more accurate sentiment toward speaker sessions.
Cons: Can be easily skewed either way. Lower ratings without proper feedback are not very useful.

Free text

Pros: Would encourage attendees to be more thoughtful. Should provide more actionable feedback for speakers.
Cons: Some attendees will not be willing to provide more lengthy feedback, or they may take a longer time to submit it.

Mockups

Here are some very early speculative mockups thanks to @karmatosed

Version One

Version Two

Possible Future Additions

Feedback content could be added as “testimonials” for sessions
Allow speakers to add feedback to their WordPress.orgWordPress.org The community site where WordPress code is created and shared by the users. This is where you can download the source code for WordPress core, plugins and themes as well as the central location for community conversations and organization. https://wordpress.org/ profiles
Add feedback to WordPress.tv sessions
Recommend other talks to attend (or watch on WordPress.tv) after giving feedback

Questions and Feedback

What formats of feedback should we provide (emojis, ratings, text, etc.)?
Do we encourage only positive feedback?
Should responders be logged into WordPress.org in order to leave feedback?

#proposal #tools

imath 5:30 pm on August 19, 2019
Hi @adityakane,

Thanks a lot for your great work about designing a tool to get feedbacks about WordCampWordCamp WordCamps are casual, locally-organized conferences covering everything related to WordPress. They're one of the places where the WordPress community comes together to teach one another what they’ve learned throughout the year and share the joy. Learn more. talks. I think it’s a very sensitive topic and I’d recommend to make sure to avoid the downsides of the review tool of the Plugins Directory.

1.a) I think it should be very simple by giving just 2 choices:
- I liked ❤️
- It could have been better 🤔
1.b) I think the interface should be dynamic: it’s difficult to figure out how the great mockups behave. But I’d recommend: 1 page, a tool to select the talk, the 2 buttons and only one text field appearing regarding the button clicked:
- if ❤️ then What did you liked the most ?
- else Any advice to share with the speaker ?
1.c) Once submitted, another talk should load or there should be a text to invite the reviewer to select another talk into the dropdown.

2) We should encourage very constructive feedbacks, if “it could have been better” then share the how. I disliked with no explanations is not a feedback imho. There should be a mention about the fact speakers are volunteering and that reviews should be very friendly: “Please imagine you are a speaker when you write your review”

3) You can use the shortcodeShortcode A shortcode is a placeholder used within a WordPress post, page, or widget to insert a form or function generated by a plugin in a specific location on your site. that prompts for the e-mail [camptix_private] to make sure the reviewer registered a seat/livestream pass.

I think this feature should be a double opt-in
- one by the speaker: I’m ok to receive feedbacks from attendees
- some other ones by the attendee:
  - I’m ok you use my personal info to save my feedback (it would probably be better if encrypted or at least hashed if you need “unique ids”),
  - I’m ok you use my review as a testimonial,
  - I’m ok to be contacted in to have more explanations about my feedback if needed.
Text shouldn’t be displayed, only the number of likes, eg: 89 ❤️ out of 100 reviews.

WordCamp organizers should be able to review texts before they are sent to the speakers to prevent potential strong words.

Should there be a way to export the feedback, and should feedback be included in a requested privacy export?

Absolutely, and it should be possible for the attendee to request for data to be deleted.

Best wishes to this feature 😉
- Andrea Middleton 9:44 pm on August 19, 2019
  
  Interesting, it didn’t occur to me to have just two options (great/not-great) for feedback, which would then branch out to more detail.
  
  On this topic:
  
  I think this feature should be a double opt-in — one by the speaker: I’m ok to receive feedbacks from attendees
  
  In that case, would you then make it impossible for the tool to collect feedback about a certain speaker, or just have the tool inform the organizers that this speaker prefers not to receive feedback?
  
  I think the follow-up questions to attendees are excellent. Thanks for your great suggestions @imath — your feedback is always so valuable!
  - imath 6:19 am on August 20, 2019
    
    Thanks Andrea 😊
    
    About the speaker opt-in: even if I believe most of them would be interested about having feedbacks about their talk, I think leaving them the option is safer because of the rating part of the feature. They should be able to benefit from a removal of their talk from reviewable ones or some sort of private feedbacks for their talk.
    
    My fear is about having the ratings displayed on their WordPress.orgWordPress.org The community site where WordPress code is created and shared by the users. This is where you can download the source code for WordPress core, plugins and themes as well as the central location for community conversations and organization. https://wordpress.org/ profile page without leaving them the choice. It’s very sensitive imho, and I can imagine someone who would never talk again after a bad review.
    - Hugh Lashbrooke 11:43 am on August 20, 2019
      
      It could always be built for all speakers to get feedback, but they can each choose whether it gets displayed on their profile or not. It’s more valuable to have the data, even if it isn’t displayed, so that would be a reasonable compromise I think.
      - imath 9:24 pm on August 21, 2019
        
        👍
    - Andrea Middleton 7:52 pm on August 20, 2019
      
      I agree that public ratings for speakers would be a mess. I’m not really interested in pursuing that idea right now, maybe ever.
      
      But I do think that organizers need a way to assess whether a session was well-received or not. If the speaker doesn’t want to know what kind of feedback organizers got about a session, I think we can leave it to the organizers what they want to do about that.
      
      It’s possible most complex part of this new tool, will be helping organizers decide what feedback to share with speakers, and what not to share. There are plenty of people who start speaking at a WordCampWordCamp WordCamps are casual, locally-organized conferences covering everything related to WordPress. They're one of the places where the WordPress community comes together to teach one another what they’ve learned throughout the year and share the joy. Learn more. with a very rough, imperfect presentation… and then go on to grow their skills and become very accomplished public speakers. I want organizers to have feedback, but also know how to use it judiciously — which isn’t a problem we’ll really be able to solve with code.
- Tammie Lister 11:59 am on August 20, 2019
  
  I see the value in a binary choice, but I do wonder if using an emoji scale similar to Facebook reactions, avoids the problems a binary scale is set to solve.
  
  Regarding dynamic, the idea was to do that so totally agree. I also totally agree the copy should encourage the feedback to be constructive.
  - Andrea Middleton 5:10 pm on August 21, 2019
    I think the emoji in the mockups are confusing, @karmatosed — they seem to have a lot of overlap or could be interpreted differently based on culture/background.
    
    I see the value in a binary choice, but I do wonder if using an emoji scale similar to Facebook reactions, avoids the problems a binary scale is set to solve.
    
    You’re not thinking of the exact same scale as FB (thumbs up, heart, laugh, wow, cry, and anger), are you? Those seem a little more emotion-driven than would be appropriate for session feedback.
    
    I wonder if we might want to use a tag system instead of a more elaborate scale/rating system… Pre-load some tags but let people create their own as well. Preloaded tags could include:
    
    positives
    
    informative
    
    inspiring
    
    engaging
    
    funny
    
    thought-provoking (but a more concise word)
    
    critical
    
    confusing
    
    dull
    
    misinformed
    
    hard to understand (but a more concise term)
    
    And then provide some boxes for actual sentences, like the mockup has.
    - Tammie Lister 5:11 pm on August 21, 2019
      
      I am thinking of that but more being inspired to have a range over directly doing that scale, it’s way too emotive to be useful for this case.
    - Ian Dunn 6:48 pm on August 27, 2019
      
      emoji… they seem to have a lot of overlap or could be interpreted differently based on culture/background.
      
      I think that’s true of most emoji; they’re similar to icons, which led to awful UX when they’re not accompanied by text.
- Ian Dunn 6:52 pm on August 27, 2019
  
  the interface should be dynamic
  Once submitted… select another talk into the dropdown.
  
  I love those ideas!
  
  “I disliked with no explanations is not a feedback imho.”
  mention… speakers are volunteering and that reviews should be very friendly
  
  100% agree on both fronts.
  
  For encouraging attendees to write constructive feedback, it might help to display the speaker’s avatarAvatar An avatar is an image or illustration that specifically refers to a character that represents an online user. It’s usually a square box that appears next to the user’s name. on the form, and say something like, “Please remember that there’s another human being on the other side of this process, and your words will affect them. Be honest, but kind.”.
mrwweb 9:07 pm on August 19, 2019
I love this idea! If done correctly, it has the possibility to greatly improve speakers at WordCamps in the long term through good feedback shared both with speakers and organizers.
1. What format? I believe ratings (probably stars) are more universally understood. They could be accompanied by text labels defining a 1 star rating (“Needs Improvement”) and 5 star (“Excellent”). If emojis are used, there should be a visible and translatable label for each to avoid confusion. The emojis themselves should probably be translatable too, as I assume many emojis have different meanings across cultures (for instance, does “💯” translate? This is general downside to emojis.
2. Only positive feedback? No. Only constructive feedback? Yes. I would be in favor of not allowing a rating be submitted unless it has met some minimum free text threshold (100 characters?). If the goal is improving sessions and speaker selection (my stated goal), then a rating without feedback is not valuable. @imath‘s suggestion of contextual questions based on the selected rating seems like a good one.
3. Require users be logged in? Definitely.
More thoughts on privacy/anonymity:
- I would not share feedback (ratings or text) publicly at all. The goal should be to encourage thoughtful, constructive, honest feedback. Many folks might rightly tone down comments if they think it might negatively impact the public image of a person.
- Adding an opt-in for allowing anonymous quoting of feedback seems reasonable so speakers and organizers can quote positive testimonials for future promotion.
- I like the idea of organizers being able to edit feedback so it can be salvaged. I think a revision log would be critical if this is allowed.
- I think that would also suggest that all reviews should run through moderation before being made available to the speaker.
- I would NOT share identifying details with the speakers. Identifying information should only be used for purposes of preventing abuse or following up with reviewers if they use the form to report some type of concern, question, or misconduct. Telling reviewers that their name won’t be shared with the speaker will encourage more honest feedback.
- I think it would be fine for speakers to opt-in to whether or not they want to see the feedback, but I don’t think a speaker should be able to opt-out of organizers collecting feedback on their talk. Feedback won’t just help speakers but speaker organizers.
- Given that feedback could be abusive or that it might be used to report abusive behavior, I would lean against allowing feedback to be deleted at the user’s request. I don’t know the legality of that, but it seems like organizers have a reasonable expectation that all feedback should be archived so it can be addressed at any time in the future.
I don’t think exporting is necessary. I would just show all feedback directly on the session page to any organizer or speaker(s) of that session when logged in. As mentioned above, I would anonymize all feedback when viewed by the speaker(s).

A quick feature addition request: I would make sure that there are direct links to sharing feedback on a specific talk that speakers can share in their slidedecks. That could be something as simple as year.city.wordcamp.org/feedback/123 where 123 is the post_id of the session.
- Andrea Middleton 10:02 pm on August 19, 2019
  
  I would NOT share identifying details with the speakers. Identifying information should only be used for purposes of preventing abuse or following up with reviewers if they use the form to report some type of concern, question, or misconduct. Telling reviewers that their name won’t be shared with the speaker will encourage more honest feedback.
  
  I assume you think it would be ok to share identifying details if the feedback-giver selected a “ok to use my feedback as a testimonial” option, but I wanted to check to make sure I was right about that. What do you think?
  
  I would make sure that there are direct links to sharing feedback on a specific talk that speakers can share in their slidedecks. That could be something as simple as year.city.wordcamp.org/feedback/123 where 123 is the post_id of the session.
  
  Ah, that’s a really solid idea @mrwweb — thanks! I think it’d have to be numerical, but presumably the numbers wouldn’t be that hard to find.
  - mrwweb 10:07 pm on August 19, 2019
    
    I assume you think it would be ok to share identifying details if the feedback-giver selected a “ok to use my feedback as a testimonial” option, but I wanted to check to make sure I was right about that. What do you think?
    
    I actually wasn’t assuming that. I was thinking testimonials would be anonymously attributed.
    
    If the name were to be made visible, I think I’d want two different opt-ins. Here’s a really rough first pass.
    
    [ ] The speaker, WordCampWordCamp WordCamps are casual, locally-organized conferences covering everything related to WordPress. They're one of the places where the WordPress community comes together to teach one another what they’ve learned throughout the year and share the joy. Learn more. {City} Organizers, and WordCamp.org can anonymously quote my feedback in public promotional materials.
    [ ] Share my name and contact information with the speaker. They may contact me or publicly share my feedback with attribution.
    Note: Your name and contact information will always be shared with WordCamp {City} organizers.
    - Andrea Middleton 10:11 pm on August 19, 2019
      
      This feedback form is getting more and more detailed as we go! laugh-cry-laugh All valid points!
      - mrwweb 10:53 pm on August 19, 2019
        
        A very valid point. My responses were in light of the requests for input on how to handle those features, but my actual preference would be that feedback is intended to be private and all feedback is anonymous for speakers and NOT anonymous for organizers.
        
        If the purpose of this feature is to improve speaking and speaker selection, testimonials don’t really scream out as a coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. need. With general surveys and usually tweets galore, do we really need to consider testimonials at all? I would actually vote no.
        
        Decisions not consent checkboxes? 😜
        
        Revised feedback notice with no checkboxes!:
        
        Feedback is anonymous shared with speakers. Feedback is personally attributed when viewed by WordCampWordCamp WordCamps are casual, locally-organized conferences covering everything related to WordPress. They're one of the places where the WordPress community comes together to teach one another what they’ve learned throughout the year and share the joy. Learn more. {City} organizers.
        
        Andrea Middleton 11:00 pm on August 19, 2019
        
        I wasn’t trying to criticize, just observing our natural tendency to build more complex tools. 🙂
        
        If the purpose of this feature is to improve speaking and speaker selection, testimonials don’t really scream out as a coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. need. With general surveys and usually tweets galore, do we really need to consider testimonials at all? I would actually vote no.
        
        Good point. Testimonials would be a nice-to-have but something people could get elsewhere. I’d love it if this tool could be so simple and easy to use that it exponentially increases the amount of feedback we get about, and, speakers.
        
        Feedback is anonymous when shared with speakers. Feedback is personally attributed when viewed by WordCampWordCamp WordCamps are casual, locally-organized conferences covering everything related to WordPress. They're one of the places where the WordPress community comes together to teach one another what they’ve learned throughout the year and share the joy. Learn more. {City} organizers.
        
        Elegant!
- Ian Dunn 8:37 pm on August 27, 2019
  
  ratings (probably stars) are more universally understood
  
  That’s a great point, I’d strongly prefer those over emoji.
  
  If the goal is improving sessions and speaker selection (my stated goal), then a rating without feedback is not valuable
  
  I think that’s the right goal, and one that was implied in the proposal, but which is worth making explicit.
  
  I 100% agree that ratings alone are not useful to meet our goal.
  
  year.city.wordcamp.org/feedback/123
  
  That’s a great idea! the numerical part could just be the session’s post ID.
David Bisset 12:21 am on August 20, 2019

Emojis remind me of the mytalk.rocks that I came up with. What a consequence. LOL.

Monitoring this discussion because it is helping me in determining how to enhance things. So far very good points. 👍🏻
- Andrea Middleton 7:54 pm on August 20, 2019
  
  Your mytalk.rocks product definitely got people talking about this more, @dimensionmedia! Thanks for bringing your skills and ideas to the WordCampWordCamp WordCamps are casual, locally-organized conferences covering everything related to WordPress. They're one of the places where the WordPress community comes together to teach one another what they’ve learned throughout the year and share the joy. Learn more. program; I really value the work you do here. 🙂 <3
Andrea Middleton 5:44 pm on August 21, 2019
I think we’ve needed a better tool for collecting attendee feedback on sessions for a long time, and I’d love to see this feature added to WordCampWordCamp WordCamps are casual, locally-organized conferences covering everything related to WordPress. They're one of the places where the WordPress community comes together to teach one another what they’ve learned throughout the year and share the joy. Learn more. sites. I think this tool has the potential to help WordCamp speakers grow as public speakers much more quickly than they currently do, and to better understand what works well with an audience and what does not work well. As I mentioned in a comment above, I think WordCamp organizers will probably benefit from some direction about how to use the data wisely. It’s important not to over-reactReact React is a JavaScript library that makes it easy to reason about, construct, and maintain stateless and stateful user interfaces. https://reactjs.org/. to attendee criticism, and to avoid discouraging speakers who are just getting started.

1. What formats of feedback should we provide (emojis, ratings, text, etc.)?

I don’t like the range of emoji in the mockup, but I do think emoji makes the feedback tool more engaging. I like @imath‘s idea of a binary choice to start, due to its simplicity. However, I know I like to give constructive feedback inside of positive feedback, so if the only option after “hearting” a session would be to heap on more praise, I’d be very frustrated with this tool.

2. Do we encourage only positive feedback?

I don’t think so. For me, the main benefits of this tool are:
1. to help speakers understand how the audience experienced their talk (so they can develop their skills and give more effective/engaging WordCamp talks in the future)
2. to help organizers understand what sessions were well-received by attendees, and what sessions were not successful (so they can make better-informed decisions in future WordCamps)
Restricting the feedback to only positive responses… will make both of those benefits harder to get at.

3. Should responders be logged into WordPress.orgWordPress.org The community site where WordPress code is created and shared by the users. This is where you can download the source code for WordPress core, plugins and themes as well as the central location for community conversations and organization. https://wordpress.org/ in order to leave feedback?

I’m of two minds about this. On one hand, logging into WordPress.org on a mobile device… can be a pain, and that pain can be a barrier. On the other hand, I think we’d get higher-quality data by requiring user to log in. I just don’t know whether reducing the flow of data, in order to get higher-quality data, is a worthwhile sacrifice.
- imath 9:33 pm on August 21, 2019
  
  @andreamiddleton about logging into WordPress.orgWordPress.org The community site where WordPress code is created and shared by the users. This is where you can download the source code for WordPress core, plugins and themes as well as the central location for community conversations and organization. https://wordpress.org/, I think it can be a barrier to give feedbacks for some users: the one who are not registered yet.
  
  We should be able to use the Camptix APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. or the private shortcodeShortcode A shortcode is a placeholder used within a WordPress post, page, or widget to insert a form or function generated by a plugin in a specific location on your site. to simply use the email the attendee used when buying his ticket or livestream pass.
  - Ian Dunn 8:44 pm on August 27, 2019
    
    I agree, it will be even harder for the 50%+ of attendees who don’t yet have an account.
    
    I don’t think it’s worth it, but we can always add it later if we find that the quality of the feedback is lower than we expect.
Ian Dunn 9:24 pm on August 27, 2019

There are a few problems…

I 100% agree, great job outlining them!

The schedule page could have a ‘Feedback’ button on each displayed session.

It’d be great to put it on individual sessions too.

automated way to report abuse?

I don’t imaging this being a common problem, so it seems overkill to build a feature for it in v1.

I do think might be worth it to let organizers read/edit the feedback before sharing it with speakers, though. We could keep it hidden from the speaker initially, but then once the organizers have approved it, send the speaker an email with a link to view it. It might be even better to just include all the feedback in the email itself.

If we start there, and discover that there’s still a problem, then we can always do more. But if we start out building things that we don’t know are necessary, we can never get that time back, and there are other projects that we’ll have sacrificed that would probably have made a bigger impact.

Should there be a way to export the feedback?

I don’t think thats an important feature, and definitely not part of an MVPMinimum Viable Product "A minimum viable product (MVP) is a product with just enough features to satisfy early customers, and to provide feedback for future product development." - WikiPedia, but we can always add it later if there’s a compelling reason. I can’t really think of any compelling use cases at the moment, though.

should feedback be included in a requested privacy export?

If we require attendees to log in, then yes. I think it should all be anonymous, though, as discussed in other threads.

Regardless though, I don’t think it’d fit in the speaker’s export, since they didn’t generate it.
Tammie Lister 4:06 pm on August 29, 2019
This is some great feedback. I wanted to try and pull out some next design steps from this based on all the insights.
- Make the interface dynamic and show how that flow would work.
- Once you have commented, nudge to do more.
- Review wording in mocks to include constructive feedback.
- Explore a scales (I personally would say emojis but we should see what others look like and decided as collective)
  - Emoji positives / critical.
  - Star rating.
  - Binary choices with follow up questions.
Anything I am missing people would love to see mocked up?

I think the next steps would be to plot out a flow of how this would work and the next iteration on mock. I will do that and post up here in a few days.
Sam Suresh 11:25 am on September 5, 2019

This is a very good addition. Thank you for the initiative.

Login using WordPress account is bad idea. 75% of users don’t have that.

Using user email is a good option.
Sallie Goetsch 4:14 pm on October 10, 2019
I love the idea of a speaker feedback form, and equally love the idea of extending it to meetups. While Meetup.com does (or did; they keep removing useful features) have a way for attendees to rate and comment on meetups, I’d rather have something more like this.

I helped organize a networking group for about 10 years and we collected print feedback forms at the end of every meeting. They were always short and they went through various iterations, including 1-5 ratings, a net promoter score for the organization as a whole, and an attempt to get people to describe something they learned that they can take action on. (Also an attempt to get suggestions for future programs.)

Given what we know about how WordCamps are structured, we might want to ask some specific questions that are useful to speakers, conditional on the type of presentation. Just riffing off the top of my head here, but

Session Format (Radio buttons)
- Presentation
- Lightning Talk
- Panel
- Workshop
- Q & A
- Networking or Social
For a presentation, include either good/bad or 1-5 scale (with the option of N/A) on things like:
- Slides or visual aids were valuable
- Speaker provided access to code examples or other hard-to-see information
- Speaker explained the subject clearly
- Presentation was the right length for the session
- Speaker left time for questions
For a workshop, you might have questions about the exercises and time allotted.

Some questions that apply to multiple formats:
- Content was what I expected from the description
- Content was appropriate for conference track / audience
- Speaker/moderator made the objectives of the session clear
- Session accomplished its objectives
- I learned something I can apply right away
- Session was educational, not promotional
- I am a beginner/intermediate/advanced/ninja at WordPress (terminology could be better)
- Session was appropriate for my level of WP experience
I’m having too many ideas about this and don’t have time to go down any of those rabbit holes, but I do think that the more specific the questions, the more useful the answers. Of course, it might require some consensus about what makes a good WordCampWordCamp WordCamps are casual, locally-organized conferences covering everything related to WordPress. They're one of the places where the WordPress community comes together to teach one another what they’ve learned throughout the year and share the joy. Learn more. or MeetupMeetup All local/regional gatherings that are officially a part of the WordPress world but are not WordCamps are organized through https://www.meetup.com/. A meetup is typically a chance for local WordPress users to get together and share new ideas and seek help from one another. Searching for ‘WordPress’ on meetup.com will help you find options in your area. session to actually create the questions.