Hugh Lashbrooke 1:25 pm on October 22, 2019
Tags: proposal ( 7 ), speaker feedback, tools ( 2 )

Proposal Update: Speaker Feedback Tool

We recently posted a proposal for a speaker feedback tool to be baked into WordCampWordCamp WordCamps are casual, locally-organized conferences covering everything related to WordPress. They're one of the places where the WordPress community comes together to teach one another what they’ve learned throughout the year and share the joy. Learn more. sites. The proposal included some ideas and mockups, with a call for feedback, suggestions and questions. Thanks to @adityakane, @imath, @andreamiddleton, @hlashbrooke, @karmatosed, @iandunn, @mrwweb, @dimensionmedia, @samsuresh and @wpfangirl for their input on that post.

Based on the feedback from that post, we have some more refined mockups to share with you, courtesy of @karmatosed. In addition to that, we are looking for input on the data storage method here, so check out the mockups and details below.

Front-end views

Initial view before selecting a talk to give feedback on:

Full form view after selecting a talk:

Dashboard views

The following mockups are based on the idea that we would store the feedback in a new custom post typeCustom Post Type WordPress can hold and display many different types of content. A single item of such a content is generally called a post, although post is also a specific post type. Custom Post Types gives your site the ability to have templated posts, to simplify the concept. with each feedback item being a separate post in that type.

List table view of feedback items in the dashboard:

Single view of the feedback post in the dashboard:

Considerations

The main decision that needs to be made at this stage is regarding how the feedback data will be stored. The three options are:

#1 Custom post type

This would appear as shown in the mockups above. Feedback would be stored in individual posts within a new custom post type.

Pros:

Uses existing WordPress APIs, so no need for custom data structures
Easy to extend with further features at a later date
Familiar UIUI UI is an acronym for User Interface - the layout of the page the user interacts with. Think ‘how are they doing that’ and less about what they are doing.

Cons:

Some WordCamp sites could get thousands of feedback items for a single event, this could slow everything down and make for a very tedious UI to look through for feedback items.
Individual feedback items would require a click through to a new page in order to view them

#2 Comments

This would involve storing the feedback as comments on the session post using a custom comment type and comment metaMeta Meta is a term that refers to the inside workings of a group. For us, this is the team that works on internal WordPress sites like WordCamp Central and Make WordPress..

Pros:

Uses existing WordPress APIs, so no need for custom data structures
Easy to extend with further features at a later date
Familiar UI (this would use the edit-comments.php template in the dashboard
Dashboard comments view allows for full feedback content to be viewed in the list table, without a new page load
Feedback is effectively a comment, so this would be a logical way of storing the data

Cons:

Custom comment types and comment meta have historically been tricky to work with (although I think this has been largely fixed in recent CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. releases)

#3 Custom database table

This would involve writing a custom data structure in custom database tables – the exact data structure and dashboard UI would still need to be planned out.

Pros:

Flexibility of building things exactly how we want it, in the most performant way possible
UI can be as optimised as possible

Cons:

Lots of development hours for planning and building
Dashboard UI would likely be unfamiliar and less predictable
Custom database tables on a large multisiteMultisite Multisite is a WordPress feature which allows users to create a network of sites on a single WordPress installation. Available since WordPress version 3.0, Multisite is a continuation of WPMU or WordPress Multiuser project. WordPress MultiUser project was discontinued and its features were included into WordPress core.https://codex.wordpress.org/Create_A_Network. instance can be unpredictable

Questions and Feedback

With all of that in mind, please comment with your thoughts on the following:

Which data structure do you think would be a good fit for session feedback?
Is there anything further that you feel should be included in the feedback form for attendees?

#proposal, #speaker-feedback, #tools

Ian Dunn 2:45 pm on October 22, 2019

I think these are great iterations on the previous version, kudos!

I still worry that we’re not asking the right questions, though. I think the unstated goal of the project is to help speakers improve their presentations — not just to collect feedback for the sake of collecting feedback — but we’re not directly asking attendees how the presentation could have been improved.

How do the questions, “What did you learn?”, “Was the talk what you expected?”, and “Do you have any other feedback?” get us closer to the goal of helping speakers improve?

IMO it’d be better to just ask something like, “What was good about the presentation?” and “What are some ways that the presentation could be improved?”.
- Hugh Lashbrooke 7:48 am on October 23, 2019
  
  That’s fair. The questions would be easy to change as we build this of course, but it would be good to finalise that before we start building. We want to strike a balance between asking people to give loads of information, and making it quick and easy to submit, so that’s something we can definitely improve.
  - Ian Dunn 4:43 pm on October 23, 2019
    
    I really like @mrwweb‘s idea:
    
    In my own presentation surveys, I use the format Keep/Tweak/Delete as in “What is one good thing you would keep from this presentation”, etc. It sometimes discourages broad feedback, but it also is specific and even makes sense to answer in a textbox vs a more intimidating text-area.
    
    I think that’d be a great way to directly ask for the kind of info we’re looking for.
    
    I also like idea of using textboxes. Personally I hate them because I’m so verbose and highly-value context and detail, but I also realize that I’m a bit neurotic, and most people would find a textarea less onerous 🙂
    
    I don’t think we should set a character limit, though, in case someone wants to type something more detailed.
Ian Dunn 2:46 pm on October 22, 2019

I’m curious why the You are leaving feedback as @foo… bit was added?

~99% of attendees will already be logged out, and probably 60%+ won’t have a w.org account to begin with. Of the ~40% that do have an account, I’d guess that ~95% of them only have account.

The project has always discouraged “company” accounts in favor of real human accounts too, so having multiple accounts seems like an extreme edge case, and not one that I think we want to encourage.
- Hugh Lashbrooke 7:49 am on October 23, 2019
  
  I’m happy to remove that – agree with your reasoning.
Ian Dunn 2:47 pm on October 22, 2019

What’s the use case for exporting the feedback to a CSV?
- Hugh Lashbrooke 7:50 am on October 23, 2019
  
  It could make it easier to share feedback with the speakers. A better solution would be to allow them access to it directly rather than needing a manual export.
  - Timi Wahalahti 8:09 am on October 23, 2019
    
    Providing direct access would mean some sort of authentication and new type of UIUI UI is an acronym for User Interface - the layout of the page the user interacts with. Think ‘how are they doing that’ and less about what they are doing. for that purpose. So CSV sounds like the best option for now.
    - Ian Dunn 4:13 pm on October 23, 2019
      
      Doh, yeah, sharing w/ the speakers is a critical feature 🙂
      
      It sounds like exporting to CSV, then separating per speaker, then sending an email to each speaker, etc would be a lot of manual work.
      
      We could do something more automated, like give the speaker’s w.org account a subscriber role on the site when their username is added to the speaker post, and then just have a small has_cap callback that allow them to view that post in wp-admin (including comments), but not edit it. That’d give them the same UIUI UI is an acronym for User Interface - the layout of the page the user interacts with. Think ‘how are they doing that’ and less about what they are doing. that admins would use, so no need to write anything additional for that.
      
      We could also give them access to edit the post as well, if that’d be useful for making them responsible for adding their bio, etc? I don’t think they should have access to edit the comments, though, since that feels like trusting a bit too much.
    - Dion hulse 1:29 am on December 5, 2019
      
      Just to touch on this – I think the best option would be to share a signed/tokenised link that allows Speakers to view their feedbacks. No edit, no admin access, no requirement for an account, just a privately shared link.
      
      I’m thinking a hybrid between Camptix Tickets and Camptix Attendance, a unique url that can be shared with the speaker that just lists everything out for them – Talk description, any links, and an anonymous set of Q/A reviews.
      - Hugh Lashbrooke 12:58 pm on December 5, 2019
        
        I like that!
      - Ian Dunn 12:30 am on December 13, 2019
        
        TL;DR: Yeah, I think that’d be be a good option.
        
        I was initially hesitant because I just built one of those for Five for the Future, and it felt a lot more complicated than it appears on the surface.
        
        We can get rid of a lot of that complexity if the token doesn’t need to expire, though. That’d also improve the UXUX UX is an acronym for User Experience - the way the user uses the UI. Think ‘what they are doing’ and less about how they do it.. The downside is that it’d make it much less secure.
        
        🤔
        
        In this situation that feels like it’s probably an acceptable tradeoff, though.
        
        We’d still need to send nocache headers on the page. We probably don’t need to encrypt the token at rest.
        
        Regardless, I don’t think we should build one interface in wp-admin for organizers, and then duplicate everything on the front end for speakers. So maybe we’d do them both on the front end?
Ian Dunn 3:00 pm on October 22, 2019

I don’t have any serious technical concerns about comments vs a CPT; I think UIUI UI is an acronym for User Interface - the layout of the page the user interacts with. Think ‘how are they doing that’ and less about what they are doing. should be the main driver, and I think comments would be a much, much better UI.

Even if we grouped the posts by their presentation, they’d still be split across pagination, and you’d have to open a new tab for each one, or click back and forth dozens of times, instead of just scrolling and seeing everything in context.
- Hugh Lashbrooke 7:50 am on October 23, 2019
  
  I agree with comments being the best option in terms of UIUI UI is an acronym for User Interface - the layout of the page the user interacts with. Think ‘how are they doing that’ and less about what they are doing., and I agree that it’s the best way forward here.
- Timi Wahalahti 8:13 am on October 23, 2019
  
  +1 for comments.
  
  But. I think we should do some kind of simple statics page for an average rating in addition to that. Maybe add a new column to session CPT where that is displayed?
  - Ian Dunn 4:36 pm on October 23, 2019
    
    Another option would be a public report like we have for t-shirt sizes. We definitely wouldn’t want to share the rating for individual speakers, but could share the average for each camp.
    
    It’d also be nice to track the year-over-year average for each camp, to see if this helps them improve. Also tracking the year-over-year average for all camps combined would be interesting.
mrwweb 10:00 pm on October 22, 2019

I’m glad to see this work moving forward. A few thoughts…

First, off I’m in agreement with @iandunn both that “the unstated goal of the project is to help speakers improve their presentations” and that these questions don’t feel as targeted as they could (I think the expectation matching question is likely the most concretely useful for both speakers and organizers of the bunch). In my own presentation surveys, I use the format Keep/Tweak/Delete as in “What is one good thing you would keep from this presentation”, etc. It sometimes discourages broad feedback, but it also is specific and even make sense to answer in a textbox vs a more intimidating text-are.

Related, there was quite a bit of discussion on the original post over whether naked ratings without any comments should be allowed. It seemed to me that the general consensus was that some written feedback should be required, but the mockup here doesn’t appear to do that. I would again voice my support for requiring more than just a star rating.

Finally, the original post stated:

The feedback tool could be accessed at an easy-to-remember URLURL A specific web address of a website or web page on the Internet, such as a website’s URL www.wordpress.org, like <year.cityname.wordcamp.org/feedback>

Looking at this design, I realize that I just blindly accepted that project assumption, but I think it would really make sense to show this form on each individual session’s post. This would almost certainly be a place that people both expect to find a feedback form and also where a speaker can easily instruct audience members to go to provide feedback. That has the further advantage of placing the full title and description next to the feedback form to help the audience remember how the talk was framed and what it covered. I could imagine the /feedback/ page also existing as shown, but it really feels like it should go on each presentation’s page…

…which leads me to my answer to question #1 and my answer of “comments”. @iandunn is much more qualified to answer this question than I, but it sure feels like this feedback walks like a comment and talks like a comment, so it’s probably a comment. It feels like any other choice only make sense if there is a significant downside to comment metaMeta Meta is a term that refers to the inside workings of a group. For us, this is the team that works on internal WordPress sites like WordCamp Central and Make WordPress. that has yet to be identified.
- Hugh Lashbrooke 7:53 am on October 23, 2019
  
  It seemed to me that the general consensus was that some written feedback should be required, but the mockup here doesn’t appear to do that.
  
  I’m personally somewhat undecided about that as we don’t want to make giving feedback too onerous for the attendees. I think a good middle ground is making one of the questions mandatory, but leaving the rest as optional.
  
  I think it would really make sense to show this form on each individual session’s post.
  
  That seems sound to me – as you say, it’s a logical place to go looking for a feedback form. I’m sure we could include the form on the session post, as well as on a separate /feedback page – it would be the same code blockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. used in both places.
  - Ian Dunn 4:04 pm on October 23, 2019
    
    +1
    
    It’d be nice to skip the “select a presentation” step when it’s on a session page, which shouldn’t be hard to do.
  - mrwweb 4:09 pm on October 23, 2019
    
    I think a good middle ground is making one of the questions mandatory, but leaving the rest as optional.
    
    I like the sound of that.
  - Ian Dunn 7:01 pm on October 23, 2019
    
    That sounds good to me too. If we use the keep/tweak/delete format, then the “delete” question would be an obvious choice for the required one. It’d have to be conditional, though, so only required if they rate less than 3 or 4 stars.
    - Ian Dunn 7:03 pm on October 23, 2019
      
      🤔, actually, I guess “tweak” could make sense too, especially for a 3 star review. Maybe we make it so that one of those two questions be completed, but not both?
Ian Dunn 4:52 pm on October 23, 2019

If we go w/ comments, I’m wondering how we should treat the questions, since there’ll be multiple.

Should we have a separate comment for each answer, or combine all the answers into a single comment?

The former is more structured, in case we ever want to do some kind of data processing that would be easier if they were separate (like word clouds, or some fancy-pants AI blah blah blah).

The latter would probably be a better UXUX UX is an acronym for User Experience - the way the user uses the UI. Think ‘what they are doing’ and less about how they do it. for reading, though. We could make a consistent format, like:

What’s one good thing you’d keep in this presentation?

The structure of the presentation was organized well, it helped me track what they said as they moved between concepts.

What’s one thing you’d tweak to improve the presentation?

The font was so small, I couldn’t read the words from the back of the room.

What’s one unhelpful thing you’d delete from the presentation?

The company logo on every slide felt crass, and made me suspicious of their motives.

I’m leaning towards the latter, but am curious what others think.
- mrwweb 6:48 pm on October 23, 2019
  My gut reaction is to merge them too, but I can think of a few reasons not to, and it makes me think keeping them separate is the way to go in the end.
  - If the export-to-CSV feature happens, outputting each answer in an individual column would be nicer than one giant field of all comments. (I think this is the most realistic near-term “data processing” scenario.)
  - If keeping these out of the default comment field prevents them from being edited, that’s relevant one way or the other. (I know some folks thought comments should be editable by site admins, but I question whether anything other than comment status should be editable given the need to maintain a record of the original comments.)
  - If one question feels like the “primary” comment, keeping it as the sole content in the comment body field feels like the way to go.
  - Ian Dunn 11:24 pm on December 12, 2019
    
    Yeah, those are really good points. I think I’m leaning more that way now.
- Dion hulse 1:32 am on December 5, 2019
  
  Combining them, but also storing the answers individually as comment metaMeta Meta is a term that refers to the inside workings of a group. For us, this is the team that works on internal WordPress sites like WordCamp Central and Make WordPress. for future processing isn’t out of the question. The minimal extra data storage and duplication would be well offset by future sanity for a v2 or bulk processing of all feedbacks over all wordcamps.
  - Ian Dunn 11:25 pm on December 12, 2019
    
    Yeah, that’s a really good point. Although, if we’ve got them stored in comment_meta, it’d also be trivial to simulate them being stored in comment_content via a filterFilter Filters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output. 🤔
imath 3:41 pm on October 25, 2019

Hi !

This is shaping great! Many thanks to the people involved into this project.

Using comments seems the best way to store these feedbacks. I guess the rating would be a comment metaMeta Meta is a term that refers to the inside workings of a group. For us, this is the team that works on internal WordPress sites like WordCamp Central and Make WordPress. and the answers would be in the comment content field. There’s probably a way to use some structured markup to be able to pull out specific answers from this field (like how the BlockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. Editor separates blocks with htmlHTML HTML is an acronym for Hyper Text Markup Language. It is a markup language that is used in the development of web pages and websites. comments).

I agree with @iandunn about adding (or replacing an existing question) in favor of « How could the talk be improved? »
- Ian Dunn 3:51 pm on October 26, 2019
  
  Yeah, using comments ala GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ sounds like a good way to add some structure around the data, but still keep it all in a single comment content field 🙂
Mariano Perez 11:36 am on November 7, 2019

It looks great. I agree with @sippis that she could use some simple statistics to help in the analysis.
Miriam Goldman 8:41 pm on November 7, 2019

If we’re talking about scalability, then definitely comments is the way to go. It is a valid point with the CPT. As for the questions – they sound good! Throwing in a +1 here. 🙂
Francesca Marano 8:44 pm on November 7, 2019

I’ve got nothing valuable to add here, except for +1000000000 on having the tool!
Birgit Pauli-Haack 9:33 pm on November 7, 2019

Just back from WordCampWordCamp WordCamps are casual, locally-organized conferences covering everything related to WordPress. They're one of the places where the WordPress community comes together to teach one another what they’ve learned throughout the year and share the joy. Learn more. US, I had a few additional thoughts:
1) it would be great if a speaker could see their rating in a single place like at their profile (not publicly just a view for the logged in speaker) , there are quite a few speakers that not only speak at their local WordCamp, admittedly probably only couple hundred but still…

2) It would be interesting reading for speaker wranglers, when vetting of future WordCamps. A central place for lookup would be awesome for interested speaker wranglers.

If those considerations would be added to the roadmap they might change probably the answer on how the data should be stored. I would think a custom table might introduce a broader reach of the information.
- Timi Wahalahti 6:44 am on November 8, 2019
  
  It would be interesting reading for speaker wranglers, when vetting of future WordCamps. A central place for lookup would be awesome for interested speaker wranglers.
  
  I disagree a little here. Having one database for speaker wranglers to check what kind of feedback the speaker has gotten and how they “performed” might lead to a situation, where the “best” and “most reliable” speakers do get selected. In my opinion, the coolest thing in WordCamps is that everyone gets a chance to speak.
  
  Maybe my doubt is unnecessary, as most of the WordCamps I know do make the speaker selection with anonymised data and only after that check the diversity of those speakers to make some adjustments.
  - Ian Dunn 11:32 pm on December 12, 2019
    
    +1
- mrwweb 12:34 am on November 9, 2019
  
  I agree with @sippis that access to past speaker ratings should be limited to the speaker, that year’s local Speaker Wranglers (and I guess anyone else with access to the camp site), and presumably WordCamp CentralWordCamp Central Website for all WordCamp activities globally. https://central.wordcamp.org includes a list of upcoming and past camp with links to each.. We want speakers to feel motivated to improve, and helping people not get bogged down by a first less-than-5-star talk is very much in that spirit. I also believe that all Speaker Wranglers should be taking proactive steps to help speakers improve like setting strong expectations, using a detailed application, reviewing slides, and mentoring speaker for those that want it. (Basically, I think the motivation to share that data can be achieved in other ways.)
  
  I don’t know if this is a real risk, but I could imagine a scenario in which a centralized speaker database could play out like when people discovered that passengers of ride share companies were rated by their drivers and they didn’t know it. Obviously, this would have more intent for transparency up front, but… we all know that 100% of people closely read every word of every email they receive 🤣 (joke).
Mel Choyce-Dwan 8:12 pm on December 11, 2019

If the idea is to base this off commenting, it might be faster/easier to use a list table:
- Ian Dunn 12:08 am on December 13, 2019
  I like that idea. Are you picturing that as just using wp-admin/edit-comments.php, or a custom screen, or displayed as part of the post editor?
  
  I think I’m leaning towards building a custom screen.
  
  edit-comments.php
  - I think we’d need a way to filterFilter Filters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output. by speaker, since the primary use case will be looking at all of the comments for a specific speaker, rather than wanting to look at the comments for all speakers intermixed with each other. Maybe a dropdown list of speaker names with a SelectWoo-style progressive enhancement, so you can search it.
  - We’d also need to add the Export button. Probably a bit tricker than it’d be on a custom screen, but not a big deal.
  Custom screen
  
  I’m guessing this would just be a stripped-down version of edit-comments.php, with a list table, way to filter by speaker, and export button.
  
  I think we’ll also need a way to filter comments by the speaker here. I don’t see a @wordpress/component thats similar to SelectWoo, but we could build a custom one. If that turns out to be too much trouble, we could just use SelectWoo.
  
  GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/
  
  The Classic editor has a metaboxMetabox A post metabox is a draggable box shown on the post editing screen. Its purpose is to allow the user to select or enter information in addition to the main post content. This information should be related to the post in some way. to view comments, but it doesn’t look like Gutenberg does, and I don’t see any GitHubGitHub GitHub is a website that offers online implementation of git repositories that can can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/ issues about future plans for it.
  
  That might be an interesting interface to consider. In some ways, the post itself feels like a natural place to view comments on the post, but we’ve also mostly trained users to read them on the front end, or edit-comments.php, so it could also be unexpected to add them there now.
  
  I don’t think the front end really works well in this situation, though.
  
  In Gutenberg, maybe we could add a button to the top toolbar, and clicking on that would switch to a “comment mode”, similar to how you can switch between the “Visual” and “Code” modes.
  
  That still feels pretty awkward, though 🤷🏻‍♂️
  - Ian Dunn 12:20 am on December 13, 2019
    
    Er, maybe the front end is also an option, since we might need to do that for speakers anyway.
    
    As long it is very obvious to admins and the speaker that the comments are private, and only being shown to them because they’re authenticated, then that could work.
    
    I think it’d be better to avoid building one interface in wp-admin for organizers, and a separate one on the front end of speakers.
    - Ian Dunn 12:54 am on December 13, 2019
      
      I don’t think we’d need an Export button, since each Speaker post could instead have a button to email the speaker a link to view their feedback.