Building a User Feedback Channel for Kids
Product & Team: Messenger Kids, Integrity
My Role: Product Manager, Product Designer
About the Project
The following case study is a project that I completed as part of the Messenger Kids Integrity team to create a kid-friendly user feedback channel within the Messenger Kids app. I led this project as both the Product Manager and Product Designer and was involved during the entire product development cycle, from ideation to launch.
What is Report a Problem?
The in-app feedback channel used across the Facebook family of apps for users to report technical issues.
Benefits of In-App User Feedback
In-app user feedback ('report a problem') is strongly preferred over external sources of user feedback, such as App Reviews or Help Center Reports, because it...
Is the highest volume source of user feedback
Contains valuable user logs and device information
Allows the team to set up automated, real-time monitoring of user feedback
Messenger Kids doesn't have an in-app feedback channel for users to report technical issues, resulting in low user feedback volume and quality. This hinders the team’s ability to identify, investigate, and resolve user issues. Improving user feedback is crucial for the team to ship a high-quality product that users love.
By the Numbers
Messenger report per user rate is 14x the Messenger Kids report per user rate.
100% of Messenger Kids feedback is submitted from external sources, so all feedback is text-based only and doesn't include any user or device logs.
Kids are experiencing issues and kids want to report technical issues about the Messenger Kids app, however, there is no convenient way for them to do so.
Messenger Kids’ app review per user rate is 23x Messenger
App reviews about technical issues are much more prevalent for Messenger Kids than for other Facebook Apps
To solve this problem, we need to make reporting issues easier for kids by building a user feedback channel in the Messenger Kids app for users to report technical issues.
How might we build a kid-friendly, legally compliant in-app feedback channel that captures the quantity and quality of user feedback needed for our team to better resolve user issues?
Increase user feedback and volume and improve user feedback quality to make it easier for our product team to identify, investigate, and resolve top user issues.
Number of user feedback reports, number of user and device logs associated with user feedback reports, time it takes to identify and resolve user issues.
Initial User Research
We conducted interviews with 6 kids (ages 8-11) to better understand kid comprehension of 'report a problem' and guide our approach to building this feedback channel.
All kids understood the difference between ‘report a person’ and ‘report a problem’. Additionally, some kids had prior experience reporting in apps like Roblox and Minecraft.
I conducted a competitive analysis to understand the prevalence of feedback channels in kids apps as well as any specific kid-friendly features.
Almost all kids apps built for slightly older kids from ages 9-13 had feedback channels, while apps for ages 6-9 had simpler UI / no feedback channel.
All feedback channels utilized free-form text, while some had multi-step flows that also contained selectable issue categories.
A small number of apps implemented parent gating with a code or challenge question to access the feedback channel, while most apps allowed kids to access the feature and report.
This is a sensitive project that involves collection of kid data. With this in mind, before jumping into the designs, I started by meeting with all relevant product and non-product partners to understand constraints, other design considerations, and get an idea of what is and isn't possible...
Legal & Privacy: limit data collection to only what is necessary and follow all regulations regarding data retention.
Limited Resources: the Integrity team only has 2 engineers, so we need to utilize as much of the existing feedback channel infra and UI as much as possible.
Personal Information: with free-form text feedback there is a risk that kids could submit personal information (i.e. phone number, address, etc.), which goes against kid data collection regulations.
Incorrect Feedback Channel: there is a chance that kids could submit 'report a person' reports through the 'report a problem' feedback channel.
Kid-Friendly Design: Design with kid usability and comprehension in mind. All of the feedback channels across the Facebook Family of Apps are designed for Adults, so we need to rethink what can be improved to tailor to kid needs.
After identifying the main Constraints, Risk Areas, and Challenges, I began putting together explorations and identified 4 different directions that we could pursue for the reporting flow.
In analyzing these 4 options we determined that the parent reporting and issue categories only options would not provide reports with enough information to help the team identify and solve user issues: 1) parent reporting from outside of the Messenger Kids app doesn't contain user logs or device info; 2) issue categories only doesn't provide us with the detail and granularity required to identify and understand the user issue being reported.
Therefore, we decided on the in-app 'free-form text only' and 'issue categories and text' options as the optimal solutions for helping us achieve our goal of identifying and resolving user-reported issues.
Choosing a Design Direction
We had external users evaluate and compare various reporting prototypes to gauge comfort, comprehension, and preference and provide us with directional insights for our final design. These were our findings:
Most kids expected to find ‘report a problem’ in the settings / profile page
The icons and naming conventions for the reporting channels could be improved to increase comprehension
The text only option performed the best as selectable categories and multiple steps created confusion and friction
Based on user research findings, the direction we chose to move forward with is the Free-Form Text Only feedback channel flow.
Updating Visual Style
The visual style of the reporting surfaces was out-dated and didn't match the rest of the app. As a result, these experiences felt external to the app and out of place. To make these experiences feel more approachable, trustworthy, and native to the app, we updated both the report a person and report a problem visual styles to match the rest of the Messenger Kids app.
In the design of the entry-point I kept kid-comprehension and risk areas top of mind. To address these areas I made the following design decisions:
Single entry-point on the ‘Your Info’ page to limit chances of accidental reports and increase the quality of reports received
Added ‘Report a Person’ entry-point above ‘Something Isn’t Working’ to help ensure integrity reports go through correct channel
Worked with content to update naming conventions and icons to enhance comprehension & emphasize ‘Report a Person’
Feedback Input Flow Design
Similar to the design of the entry-point, I made sure to take into account the requirements gathered in the previous design stages, which is reflected in the following design decisions:
Worked with content to provide intuitive phrasing and add an example of what to report to aid kid comprehension
Added option to ‘Report a Person’ if kid wants to submit an integrity report
Provided a disclaimer in the free-form text area to tell users not submit personal information
After designing high fidelity prototypes, aligning with product partners, and getting all of the necessary approvals, this is the design I proposed, ready to be tested with users...
We took various measures to address the risk areas associated with free-form text. However, before the full rollout, to ensure the safety of users, we decided to run an experiment to determine the efficacy of these measures to prevent integrity reports and personal information being reported through ‘Flytrap’.
Although uncommon, we identified a handful of reports about sensitive issues or that contained personal info - we also saw a large number of off-topic reports.
Solution to Address Remaining Risk Areas
To address these risk areas, the team is considering a feature whereby the child’s reports are routed to their parent who reviews & determines whether or not their child’s report needs to be escalated to MK. This solution would...
Alleviate legal risk associated with reports containing personal information due to parental consent and submission of feedback
Allow parents to take action on integrity reports, greatly reducing the likelihood and danger of kids sending in reports via the wrong channel and mitigating legal risk due to parental consent and submission of feedback
Improve the quality of feedback and reduce noise, with parents taking action to resolve some issues their kids are having and not submitting irrelevant reports
Weighing Pros and Cons of This Solution
The main drawbacks to this solution are that it would reduce feedback volume and require more engineering work. However, this solution ensures that we are building a long-term solution that avoids potential legal issues for the team and won't require any future fixes. Additionally, since the current state of user feedback is 0 for Messenger Kids, we decided that this feature is worth it to build as user feedback is a top priority for the team and vital for our ability to build a high quality product that users love.
We Launched an on-platform survey of parents, reporting on their expectations of the role of parents vs Messenger Kids in monitoring their child’s activity and their receptiveness to involvement in reviewing their child’s reports.
Parents were very receptive to reviewing their child’s reports on first and deciding whether to escalate to Facebook.
Kid Experience Final Design
After design, experimentation, and iteration, this is the final design we landed on for the kid experience of submitting a report about a technical issue...
Parent Experience Final Design
Additionally, this is the final parental involvement experience that we landed on for parents to review their kids' reports about technical issues and decide whether to send to Facebook...
The launch of the feedback channel in MK was successful and achieved our initial goals, increasing daily report volume by more than 6x (from 387 to 2,438 reports per day), adding 40+ user and device logs to reports, and enabling the team to set up real-time automated monitoring.
Opportunities for Optimization
The drop-off in engagement at each stage is an opportunity to further increase the volume of user feedback we receive about Messenger Kids. For future improvements, we should understand why parents aren't engaging with the report at each stage of the pipeline, and implement solutions to address any shortcomings that the current feedback channel has.
Add additional entry points in Facebook and Messenger for parents to get really accurate, specific, and segmented user feedback no matter where the issue is occurring. Getting app-specific feedback is extremely helpful in understanding the issue.
Potentially add a fork in the feedback button to send positive reports to app store review and issues and negative reports to internal feedback channel.
What have you learned from this project?
You shouldn't take shortcuts and choose a design direction just for the sake of convenience for the team. It's important to create designs that last and will bring long-term value to the user and the company. If you cut corners that will likely just create extra work and headaches in the future, wasting valuable time and resources.
During this project, there was a lot of uncertainty and back and forth with regard to legal risks. This resulted in fluctuating designs and wasted time. From this project, I learned the importance of anticipating risk areas and the ability to identify launch-blocking hurdles early on.