Hugh Lashbrooke 1:25 pm on October 22, 2019
Tags: proposal ( 7 ), speaker feedback, tools ( 2 )

Proposal Update: Speaker Feedback Tool

We recently posted a proposal for a speaker feedback tool to be baked into WordCampWordCamp WordCamps are casual, locally-organized conferences covering everything related to WordPress. They're one of the places where the WordPress community comes together to teach one another what they’ve learned throughout the year and share the joy. Learn more. sites. The proposal included some ideas and mockups, with a call for feedback, suggestions and questions. Thanks to @adityakane, @imath, @andreamiddleton, @hlashbrooke, @karmatosed, @iandunn, @mrwweb, @dimensionmedia, @samsuresh and @wpfangirl for their input on that post.

Based on the feedback from that post, we have some more refined mockups to share with you, courtesy of @karmatosed. In addition to that, we are looking for input on the data storage method here, so check out the mockups and details below.

Front-end views

Initial view before selecting a talk to give feedback on:

Full form view after selecting a talk:

Dashboard views

The following mockups are based on the idea that we would store the feedback in a new custom post typeCustom Post Type WordPress can hold and display many different types of content. A single item of such a content is generally called a post, although post is also a specific post type. Custom Post Types gives your site the ability to have templated posts, to simplify the concept. with each feedback item being a separate post in that type.

List table view of feedback items in the dashboard:

Single view of the feedback post in the dashboard:

Considerations

The main decision that needs to be made at this stage is regarding how the feedback data will be stored. The three options are:

#1 Custom post type

This would appear as shown in the mockups above. Feedback would be stored in individual posts within a new custom post type.

Pros:

Uses existing WordPress APIs, so no need for custom data structures
Easy to extend with further features at a later date
Familiar UIUI UI is an acronym for User Interface - the layout of the page the user interacts with. Think ‘how are they doing that’ and less about what they are doing.

Cons:

Some WordCamp sites could get thousands of feedback items for a single event, this could slow everything down and make for a very tedious UI to look through for feedback items.
Individual feedback items would require a click through to a new page in order to view them

#2 Comments

This would involve storing the feedback as comments on the session post using a custom comment type and comment metaMeta Meta is a term that refers to the inside workings of a group. For us, this is the team that works on internal WordPress sites like WordCamp Central and Make WordPress..

Pros:

Uses existing WordPress APIs, so no need for custom data structures
Easy to extend with further features at a later date
Familiar UI (this would use the edit-comments.php template in the dashboard
Dashboard comments view allows for full feedback content to be viewed in the list table, without a new page load
Feedback is effectively a comment, so this would be a logical way of storing the data

Cons:

Custom comment types and comment meta have historically been tricky to work with (although I think this has been largely fixed in recent CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. releases)

#3 Custom database table

This would involve writing a custom data structure in custom database tables – the exact data structure and dashboard UI would still need to be planned out.

Pros:

Flexibility of building things exactly how we want it, in the most performant way possible
UI can be as optimised as possible

Cons:

Lots of development hours for planning and building
Dashboard UI would likely be unfamiliar and less predictable
Custom database tables on a large multisiteMultisite Multisite is a WordPress feature which allows users to create a network of sites on a single WordPress installation. Available since WordPress version 3.0, Multisite is a continuation of WPMU or WordPress Multiuser project. WordPress MultiUser project was discontinued and its features were included into WordPress core.https://codex.wordpress.org/Create_A_Network. instance can be unpredictable

Questions and Feedback

With all of that in mind, please comment with your thoughts on the following:

Which data structure do you think would be a good fit for session feedback?
Is there anything further that you feel should be included in the feedback form for attendees?

#proposal, #speaker-feedback, #tools

Ian Dunn 2:45 pm on October 22, 2019

I think these are great iterations on the previous version, kudos!

I still worry that we’re not asking the right questions, though. I think the unstated goal of the project is to help speakers improve their presentations — not just to collect feedback for the sake of collecting feedback — but we’re not directly asking attendees how the presentation could have been improved.

How do the questions, “What did you learn?”, “Was the talk what you expected?”, and “Do you have any other feedback?” get us closer to the goal of helping speakers improve?

IMO it’d be better to just ask something like, “What was good about the presentation?” and “What are some ways that the presentation could be improved?”.
- Hugh Lashbrooke 7:48 am on October 23, 2019
  
  That’s fair. The questions would be easy to change as we build this of course, but it would be good to finalise that before we start building. We want to strike a balance between asking people to give loads of information, and making it quick and easy to submit, so that’s something we can definitely improve.
  - Ian Dunn 4:43 pm on October 23, 2019
    
    I really like @mrwweb‘s idea:
    
    In my own presentation surveys, I use the format Keep/Tweak/Delete as in “What is one good thing you would keep from this presentation”, etc. It sometimes discourages broad feedback, but it also is specific and even makes sense to answer in a textbox vs a more intimidating text-area.
    
    I think that’d be a great way to directly ask for the kind of info we’re looking for.
    
    I also like idea of using textboxes. Personally I hate them because I’m so verbose and highly-value context and detail, but I also realize that I’m a bit neurotic, and most people would find a textarea less onerous 🙂
    
    I don’t think we should set a character limit, though, in case someone wants to type something more detailed.
Ian Dunn 2:46 pm on October 22, 2019

I’m curious why the You are leaving feedback as @foo… bit was added?

~99% of attendees will already be logged out, and probably 60%+ won’t have a w.org account to begin with. Of the ~40% that do have an account, I’d guess that ~95% of them only have account.

The project has always discouraged “company” accounts in favor of real human accounts too, so having multiple accounts seems like an extreme edge case, and not one that I think we want to encourage.
- Hugh Lashbrooke 7:49 am on October 23, 2019
  
  I’m happy to remove that – agree with your reasoning.
Ian Dunn 2:47 pm on October 22, 2019

What’s the use case for exporting the feedback to a CSV?
- Hugh Lashbrooke 7:50 am on October 23, 2019
  
  It could make it easier to share feedback with the speakers. A better solution would be to allow them access to it directly rather than needing a manual export.
  - Timi Wahalahti 8:09 am on October 23, 2019
    
    Providing direct access would mean some sort of authentication and new type of UIUI UI is an acronym for User Interface - the layout of the page the user interacts with. Think ‘how are they doing that’ and less about what they are doing. for that purpose. So CSV sounds like the best option for now.
    - Ian Dunn 4:13 pm on October 23, 2019
      
      Doh, yeah, sharing w/ the speakers is a critical feature 🙂
      
      It sounds like exporting to CSV, then separating per speaker, then sending an email to each speaker, etc would be a lot of manual work.
      
      We could do something more automated, like give the speaker’s w.org account a subscriber role on the site when their username is added to the speaker post, and then just have a small has_cap callback that allow them to view that post in wp-admin (including comments), but not edit it. That’d give them the same UIUI UI is an acronym for User Interface - the layout of the page the user interacts with. Think ‘how are they doing that’ and less about what they are doing. that admins would use, so no need to write anything additional for that.
      
      We could also give them access to edit the post as well, if that’d be useful for making them responsible for adding their bio, etc? I don’t think they should have access to edit the comments, though, since that feels like trusting a bit too much.
Ian Dunn 3:00 pm on October 22, 2019

I don’t have any serious technical concerns about comments vs a CPT; I think UIUI UI is an acronym for User Interface - the layout of the page the user interacts with. Think ‘how are they doing that’ and less about what they are doing. should be the main driver, and I think comments would be a much, much better UI.

Even if we grouped the posts by their presentation, they’d still be split across pagination, and you’d have to open a new tab for each one, or click back and forth dozens of times, instead of just scrolling and seeing everything in context.
- Hugh Lashbrooke 7:50 am on October 23, 2019
  
  I agree with comments being the best option in terms of UIUI UI is an acronym for User Interface - the layout of the page the user interacts with. Think ‘how are they doing that’ and less about what they are doing., and I agree that it’s the best way forward here.
- Timi Wahalahti 8:13 am on October 23, 2019
  
  +1 for comments.
  
  But. I think we should do some kind of simple statics page for an average rating in addition to that. Maybe add a new column to session CPT where that is displayed?
  - Ian Dunn 4:36 pm on October 23, 2019
    
    Another option would be a public report like we have for t-shirt sizes. We definitely wouldn’t want to share the rating for individual speakers, but could share the average for each camp.
    
    It’d also be nice to track the year-over-year average for each camp, to see if this helps them improve. Also tracking the year-over-year average for all camps combined would be interesting.
mrwweb 10:00 pm on October 22, 2019

I’m glad to see this work moving forward. A few thoughts…

First, off I’m in agreement with @iandunn both that “the unstated goal of the project is to help speakers improve their presentations” and that these questions don’t feel as targeted as they could (I think the expectation matching question is likely the most concretely useful for both speakers and organizers of the bunch). In my own presentation surveys, I use the format Keep/Tweak/Delete as in “What is one good thing you would keep from this presentation”, etc. It sometimes discourages broad feedback, but it also is specific and even make sense to answer in a textbox vs a more intimidating text-are.

Related, there was quite a bit of discussion on the original post over whether naked ratings without any comments should be allowed. It seemed to me that the general consensus was that some written feedback should be required, but the mockup here doesn’t appear to do that. I would again voice my support for requiring more than just a star rating.

Finally, the original post stated:

The feedback tool could be accessed at an easy-to-remember URLURL A specific web address of a website or web page on the Internet, such as a website’s URL www.wordpress.org, like <year.cityname.wordcamp.org/feedback>

Looking at this design, I realize that I just blindly accepted that project assumption, but I think it would really make sense to show this form on each individual session’s post. This would almost certainly be a place that people both expect to find a feedback form and also where a speaker can easily instruct audience members to go to provide feedback. That has the further advantage of placing the full title and description next to the feedback form to help the audience remember how the talk was framed and what it covered. I could imagine the /feedback/ page also existing as shown, but it really feels like it should go on each presentation’s page…

…which leads me to my answer to question #1 and my answer of “comments”. @iandunn is much more qualified to answer this question than I, but it sure feels like this feedback walks like a comment and talks like a comment, so it’s probably a comment. It feels like any other choice only make sense if there is a significant downside to comment metaMeta Meta is a term that refers to the inside workings of a group. For us, this is the team that works on internal WordPress sites like WordCamp Central and Make WordPress. that has yet to be identified.
- Hugh Lashbrooke 7:53 am on October 23, 2019
  
  It seemed to me that the general consensus was that some written feedback should be required, but the mockup here doesn’t appear to do that.
  
  I’m personally somewhat undecided about that as we don’t want to make giving feedback too onerous for the attendees. I think a good middle ground is making one of the questions mandatory, but leaving the rest as optional.
  
  I think it would really make sense to show this form on each individual session’s post.
  
  That seems sound to me – as you say, it’s a logical place to go looking for a feedback form. I’m sure we could include the form on the session post, as well as on a separate /feedback page – it would be the same code blockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. used in both places.
  - Ian Dunn 4:04 pm on October 23, 2019
    
    +1
    
    It’d be nice to skip the “select a presentation” step when it’s on a session page, which shouldn’t be hard to do.
  - mrwweb 4:09 pm on October 23, 2019
    
    I think a good middle ground is making one of the questions mandatory, but leaving the rest as optional.
    
    I like the sound of that.
  - Ian Dunn 7:01 pm on October 23, 2019
    
    That sounds good to me too. If we use the keep/tweak/delete format, then the “delete” question would be an obvious choice for the required one. It’d have to be conditional, though, so only required if they rate less than 3 or 4 stars.
    - Ian Dunn 7:03 pm on October 23, 2019
      
      🤔, actually, I guess “tweak” could make sense too, especially for a 3 star review. Maybe we make it so that one of those two questions be completed, but not both?
Ian Dunn 4:52 pm on October 23, 2019

If we go w/ comments, I’m wondering how we should treat the questions, since there’ll be multiple.

Should we have a separate comment for each answer, or combine all the answers into a single comment?

The former is more structured, in case we ever want to do some kind of data processing that would be easier if they were separate (like word clouds, or some fancy-pants AI blah blah blah).

The latter would probably be a better UXUX UX is an acronym for User Experience - the way the user uses the UI. Think ‘what they are doing’ and less about how they do it. for reading, though. We could make a consistent format, like:

What’s one good thing you’d keep in this presentation?

The structure of the presentation was organized well, it helped me track what they said as they moved between concepts.

What’s one thing you’d tweak to improve the presentation?

The font was so small, I couldn’t read the words from the back of the room.

What’s one unhelpful thing you’d delete from the presentation?

The company logo on every slide felt crass, and made me suspicious of their motives.

I’m leaning towards the latter, but am curious what others think.
- mrwweb 6:48 pm on October 23, 2019
  My gut reaction is to merge them too, but I can think of a few reasons not to, and it makes me think keeping them separate is the way to go in the end.
  - If the export-to-CSV feature happens, outputting each answer in an individual column would be nicer than one giant field of all comments. (I think this is the most realistic near-term “data processing” scenario.)
  - If keeping these out of the default comment field prevents them from being edited, that’s relevant one way or the other. (I know some folks thought comments should be editable by site admins, but I question whether anything other than comment status should be editable given the need to maintain a record of the original comments.)
  - If one question feels like the “primary” comment, keeping it as the sole content in the comment body field feels like the way to go.
imath 3:41 pm on October 25, 2019

Hi !

This is shaping great! Many thanks to the people involved into this project.

Using comments seems the best way to store these feedbacks. I guess the rating would be a comment metaMeta Meta is a term that refers to the inside workings of a group. For us, this is the team that works on internal WordPress sites like WordCamp Central and Make WordPress. and the answers would be in the comment content field. There’s probably a way to use some structured markup to be able to pull out specific answers from this field (like how the BlockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. Editor separates blocks with htmlHTML HTML is an acronym for Hyper Text Markup Language. It is a markup language that is used in the development of web pages and websites. comments).

I agree with @iandunn about adding (or replacing an existing question) in favor of « How could the talk be improved? »
- Ian Dunn 3:51 pm on October 26, 2019
  
  Yeah, using comments ala GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ sounds like a good way to add some structure around the data, but still keep it all in a single comment content field 🙂
Mariano Perez 11:36 am on November 7, 2019

It looks great. I agree with @sippis that she could use some simple statistics to help in the analysis.
Miriam Goldman 8:41 pm on November 7, 2019

If we’re talking about scalability, then definitely comments is the way to go. It is a valid point with the CPT. As for the questions – they sound good! Throwing in a +1 here. 🙂
Francesca Marano 8:44 pm on November 7, 2019

I’ve got nothing valuable to add here, except for +1000000000 on having the tool!
Birgit Pauli-Haack 9:33 pm on November 7, 2019

Just back from WordCampWordCamp WordCamps are casual, locally-organized conferences covering everything related to WordPress. They're one of the places where the WordPress community comes together to teach one another what they’ve learned throughout the year and share the joy. Learn more. US, I had a few additional thoughts:
1) it would be great if a speaker could see their rating in a single place like at their profile (not publicly just a view for the logged in speaker) , there are quite a few speakers that not only speak at their local WordCamp, admittedly probably only couple hundred but still…

2) It would be interesting reading for speaker wranglers, when vetting of future WordCamps. A central place for lookup would be awesome for interested speaker wranglers.

If those considerations would be added to the roadmap they might change probably the answer on how the data should be stored. I would think a custom table might introduce a broader reach of the information.
- Timi Wahalahti 6:44 am on November 8, 2019
  
  It would be interesting reading for speaker wranglers, when vetting of future WordCamps. A central place for lookup would be awesome for interested speaker wranglers.
  
  I disagree a little here. Having one database for speaker wranglers to check what kind of feedback the speaker has gotten and how they “performed” might lead to a situation, where the “best” and “most reliable” speakers do get selected. In my opinion, the coolest thing in WordCamps is that everyone gets a chance to speak.
  
  Maybe my doubt is unnecessary, as most of the WordCamps I know do make the speaker selection with anonymised data and only after that check the diversity of those speakers to make some adjustments.
- mrwweb 12:34 am on November 9, 2019
  
  I agree with @sippis that access to past speaker ratings should be limited to the speaker, that year’s local Speaker Wranglers (and I guess anyone else with access to the camp site), and presumably WordCamp CentralWordCamp Central Website for all WordCamp activities globally. https://central.wordcamp.org includes a list of upcoming and past camp with links to each.. We want speakers to feel motivated to improve, and helping people not get bogged down by a first less-than-5-star talk is very much in that spirit. I also believe that all Speaker Wranglers should be taking proactive steps to help speakers improve like setting strong expectations, using a detailed application, reviewing slides, and mentoring speaker for those that want it. (Basically, I think the motivation to share that data can be achieved in other ways.)
  
  I don’t know if this is a real risk, but I could imagine a scenario in which a centralized speaker database could play out like when people discovered that passengers of ride share companies were rated by their drivers and they didn’t know it. Obviously, this would have more intent for transparency up front, but… we all know that 100% of people closely read every word of every email they receive 🤣 (joke).