The Test Team helps manage testing and triage across the WordPress ecosystem. They focus on user testing of the editing experience and WordPress dashboard, replicating and documenting bug reports, and supporting a culture of review and triage across the project.
- Triage Tags
- The Flow of Flow Patrol
Triage of posts on make/flowFlow Flow is the path of screens and interactions taken to accomplish a task. It’s an experience vector. Flow is also a feeling. It’s being unselfconscious and in the zone. Flow is what happens when difficulties are removed and you are freed to pursue an activity without forming intentions. You just do it.
Flow is the actual user experience, in many ways. If you like, you can think of flow as a really comprehensive set of user stories. When you think about user flow, you’re thinking about exactly how a user will perform the tasks allowed by your product.Flow and Context is done with tags.
TagTag Tag is one of the pre-defined taxonomies in WordPress. Users can add tags to their WordPress posts along with categories. However, while a category may cover a broad range of topics, tags are smaller in scope and focused to specific topics. Think of them as keywords used for topics discussed in a particular post. make/flow posts with the relevant core component, iOS component, or Android component. The component is the most important triage tag. When tagging a visual record that crosses components, tag all exercised components. Tagging by component allows component teams to follow the make/flow posts relevant to their team. For example, the Editor component team follows their tag page and feed. Components that are exercised by many flows will be tagged more often. This is a feature.
Here are some often used components. For the complete list, see the core components page.
- network-and-sites – We are inconsistent with this component. Use “multisiteMultisite Multisite is a WordPress feature which allows users to create a network of sites on a single WordPress installation. Available since WordPress version 3.0, Multisite is a continuation of WPMU or WordPress Multiuser project. WordPress MultiUser project was discontinued and its features were included into WordPress core.https://codex.wordpress.org/Create_A_Network.” instead of “network-and-sites” when tagging.
The device tags specify form factor. These are mainly about screen size.
Platform is the operating system.
Web interfaces are served from coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. WordPress to a web browser. All posts relating to our web interfaces are tagged with web. Posts relating to the iOSiOS The operating system used on iPhones and iPads. or Android applications are tagged app. If a post documents a flow that traverses both web interfaces and an application (such as moving from app to web to use a feature not yet in the app), use both tags. The web tag is used inconsistently. Consider it optional. The app tag is consistently used and is not optional.
For web interfaces, indicate the browser used. This is not necessary for apps.
The iOS and Android apps offer betaBeta A pre-release of software that is given out to a large group of users to trial under real conditions. Beta versions have gone through alpha testing in-house and are generally fairly close in look, feel and function to the final product; however, design changes often occur as part of the process. releases in addition to the production releases available through the app stores. Core WordPress offers nightly builds. When using pre-release software, use the beta tag, otherwise use production. We haven’t used these consistently. If you don’t add them, welcome to the club. Development Phase might be retired.
To make finding visual records taken against a certain release easier, include the version number of the interface being tested as a tag. Using only the major releaseMajor Release A set of releases or versions having the same major version number may be collectively referred to as “X.Y” -- for example version 5.2.x to refer to versions 5.2, 5.2.1, and all other versions in the 5.2. (five dot two dot) branch of that software. Major Releases often are the introduction of new major features and functionality. number is sufficient.
- 4.3 – The current production/stable version of core WordPress.
- 4.4 – The current development version of core WordPress.
- 5.2 – The current production/app store version of the WordPress iOS app.
- 5.4 – The current beta version of the WordPress iOS app.
- 4.2 – The current production/app store version of the WordPress Android app.
The workflow tags are analogous to the workflow tags used in core tracTrac Trac is the place where contributors create issues for bugs or feature requests much like GitHub.https://core.trac.wordpress.org/., with a few additions. We haven’t used these consistently either. This needs to be hashed out. needs-ticket and needs-captions are the most frequently used.
- needs-ticket – Posts that report bugs need to be cross-referenced with trac tickets.
- needs-captions – Galleries need to be captioned.
Spotting trends and patterns is fun and an important part of flow patrolFlow patrol Flow patrol is the regular exercise and monitoring of important flows. Issues found while on flow patrol are kibbled and ticketed. Continuous flow patrol encourages use of our own software and increases awareness of what our users are experiencing day to day. Flow patrol duties are outlined in the flow handbook.. For example, iOS devices often expose scroll bleed bugs in our modals. Tag scroll bleedScroll bleed Sometimes when scrolling a modal such as the media modal on touch devices, the scrolling bleeds through to the page beneath the modal. Instead of scrolling the modal, the page beneath scrolls. Sometimes the scrolling nests and sometimes the page beneath grabs scrolling all for itself. Android doesn't usually have this problem, but iOS triggers it often.https://make.wordpress.org/test/tag/scroll-bleed/ related posts with scroll-bleed. Here are some common patterns.
- Scroll bleed: scroll-bleed
- Keyboard flyup: keyboard-flyup
- Rogue cursor: rogue-cursor
- Boxed input scrolling: boxed-input-scrolling
- Desktop bias: desktop-bias
- Forked flow (open in new window): forked-flow
- Treshold friction: threshold-friction
- Anxiety flow: anxiety-flow
For more trends and patterns, see the glossary.
make/flow is a venue for documenting flow, reporting bugs, and discussing usability through screenshots, galleries, and screencasts. As we go about our daily publishing with WP, we document our experience of WP, with WP. This recursive dogfooding is the basis of flow patrol.
The simplest way to document your experience is to publish visual bug reports. A misaligned button label on the Posts screen, for example, is easily captured in a screenshot and published to make/flow. Or, in the case of “Posts list: double entry in post list after publishing, iOS app beta, iPhone 6+“, a doubled entry in the posts list of the iOS app is captured. The title provides a short description of the problem, headline style, followed by some platform, device, and version context. The body of the post is one screenshot with some explanatory text of the problem. That’s all a visual bug report needs to be. We want a low barrier, especially for those reporting phone bugs from phones. Our example visual bug report was published using the iOS app. The bug was found in the iOS app and then reported with the iOS app. This recursive dogfooding is playfully referred to as kibbling. Since visual bug reports are the simplest means of feedback in this visual kibblingKibbling Kibbling is the process of dogfooding a flow and publishing visual records and narratives of that dogfooding. The make/flow blog is an example of a kibbling blog. Since we are publishing these “kibbles” with the software we are dogfooding, the act of kibbling further dogfoods the software. Kibbling encourages more dogfooding which encourages more kibbling and so on. process, they’re sometimes known as kibbles. A visual bug report isn’t limited to reporting defects. A visual bug report can be a visual opinion or anecdote. Kibbles, all.
If you want to triage your own visual bug report, add tags from the Triage Tags lists at the top of this page. The tags for our “Posts list: double entry in post list after publishing” example are app, beta, ios, phablet, posts-list, and visual-bug-report. This bug is evidenced on the posts list screen, so posts-list is the component tag. Consult the iOS app’s list of labels on GitHub for a suitable label. Lowercase that label and replace any spaces with dashes to turn a label into a component tag. “Posts List” becomes posts-list. When turning components and labels into tags, follow the rules WordPress uses to sanitize titles for urls.
With the components established, make sure the title of the post is correctly prefixed with component names. In this example, Posts list is the component so the title prefix is “Posts list:”.
The app tag is from the “Application or Web” list. Since the bug is in our iOS application rather than in a web view in Safari, our visual bug report is tagged with app.
beta is the Development Phase. The iOS app offers beta builds and official app store builds. This bug was found in a beta build.
An iPhone 6+ is considered to be a phablet sized device, so the phablet tag is used from the Device list.
Since this post is a visual bug report, it is tagged visual-bug-report.
Don’t let these tags stand in the way of publishing. If you don’t have time or patience for them at the moment of publishing, don’t worry about them. The Flow Patrol team will add them later.
Now that this visual bug report is tagged, it can be consumed in a number of ways. Someone who wants to follow all make/flow posts relating to the iOS and Android apps can visit this link: https://make.wordpress.org/test/tag/app/ Tag intersections can be used to get all posts relating to the iOS app or the Android app. https://make.wordpress.org/test/tag/ios+app/ https://make.wordpress.org/test/tag/android+app/ Append feed/ to any WordPress url to turn that url into a feed. https://make.wordpress.org/test/tag/ios+app/feed/ make/flow is a visual zeitgeist of our usability. Hopefully, this set of tags and their intersections will be lightweight enough to use from a phone and sufficient to helping us consume and understand this attempt at continuous recursive dogfooding and visual survey. If nothing else, we’re using what me make.
After tagging, the next triage phase is creating a ticket. Continuing with our “Double entry in post list after publishing” example, search the iOS app’s open tickets over on Github. If a ticket already exists for this bug, comment on the ticket with a link to the make/flow post. The post will provide another data point to developers and document the bug with a screenshot. If this bug has not already been reported, create a new issue. Label the ticket with the appropriate component. Our example make/flow post is tagged posts-list, which corresponds to the Posts List label. On the ticket, include a link to the make/flow post in the description. On the make/flow post, add a link to the ticket in a comment and tag the post with the ticket number. This cross-links the post and the ticket.
Visual bug reports usually deal with one particular screen. A visual record (vizrec) follows a flow across screens, snapshotting each screen and interaction. Vizrecs are most often expressed as captioned galleries. Captions provide narrative that glues the screenshots together. Captions also editorialize. Express your user experience in the captions.
To tag a visual record, follow the visual bug report tagging process for each screenshot in the record. Create tickets as needed for every issue noted in the visual record. Cross-link the tickets with the visual record via comments. Visual records often cross multiple components. Add a tag for each component. Finally, tag all visual records with visual-record. We’ve been very inconsistent with using the visual-record tag. Since so many posts are visual records, it has become an untagged default.
For the visual record “Customize, Menus: menu customizer, iPhone 6+“, the tags are 32733, customize, ios, keyboard-flyup, menus, phablet, and visual-record. “customize” and “menus” are the relevant components, which are also used to prefix the post title. “ios” is the platform. “phablet” is the device. “keyboard-flyup” is a common usability bug evidenced in the vizrec. “32733” is the number of a ticket opened for a visual bug shown in the vizrec. #32733 is cross-linked with the make/flow post via a comment on the ticket and a comment on the post. All make/flow posts related to 32733 are accessible via make/flow/tag/32733.
A visual surveyVisual survey A visual survey is a collection of screenshots for a like set of screens or interfaces, such as all list table screens or all toolbar incarnations across various devices. These screens are not presented in the context of a particular flow.https://make.wordpress.org/test/tag/visual-survey/ is a collection of screenshots for a like set of screens or interfaces, such as all list table screens or all toolbar incarnations across various devices. These screens are captured as captioned galleries the way visual records are but are not presented in the context of a particular flow.
When an existing feature changes or is replaced, a flow comparisonFlow comparison When an existing feature changes or is replaced, a flow comparison visual record is made to compare flow through the old interface with flow through the new interface. Seeing two flows side-by-side helps determine whether the new interface actually flows better.https://make.wordpress.org/test/tag/flow-comparison/ visual record is made to compare flow through the old interface with flow through the new interface. Seeing two flows side-by-side helps determine whether the new interface actually flows better.
A visual historyVisual history A visual history shows the evolution of an interface over time and across releases. shows the evolution of an interface over time and across releases.