Privacy Manager

Designing the survey

I made use of closed-ended, categorical questions on Google Forms so that I can more easily normalize and visualize the data.

• 12 questions total (not too many questions to consider response rate)

Sample of questions with embedded screenshots of the design:

• What does this permission allow the app to do?
• Where is your permission data going?
• Where is your permission data for this specific purpose going?

I released the survey through Mechanical Turk for global reach. We distributed the screening survey capped at 300 respondents. Of these 300, I sent the full survey to 100 qualified respondents (109 total qualified) and received 93 responses back.

Respondents struggle to identify why apps need access to a permission

Respondents find it difficult to understand where their data is being sent to when a purpose behind the requested permission isn't given

Planning participatory design session

I split the participatory design activities into two parts:

1) Using pens and printouts of design mockups of app settings and global settings and progressively showing one design at a time, having participants take me through the designs and talk about what might come next so that I can better understand their expectations. I had participants freely mark up the printouts - highlighting confusing parts or helpful key parts.

2) Applying flexible modeling kit activity through artifacts to understand what participants prioritize across these different moving parts. I gave them paper UI components and blank pieces of paper to have them present their ideas physically.

Participatory design activities in action

I included one designer when co-creating with the participant, and one developer and one designer in the note taking process. During the second part activity, I presented the previously marked up mockups as a point of reference for participants to refer to so that they can immediately recall what parts were helpful/not helpful while engaging in the creating process.

Analyzing artifacts and data sources

During the analysis, I color-coded all my data across participants based on attitudinal and behavioral data captured through video recordings, raw notes, quotes, transcription, and artifacts/photographs.

Collaborative synthesis process

For the synthesis, I led affinity diagramming sessions with designers to think about the opportunities that emerged from the themes. 7 clusters emerged - having 2 opportunity clusters based on gaps we haven’t addressed yet, 3 need-based clusters based on how participants prioritize certain information over others, and 2 problem clusters based on pain points.

Pain points to address

From the synthesis, I identified key pain points to address:

• Learning curve for Privacy Manager is relatively high for all particpants due to lack of text descriptors for what each purpose name and third party library name means
• Missing context on how purposes and permissions are related has most participants asking for clarification
• Low discoverability of the collapse/expand menu has most of the participants fustrated because the ability to control access to permissions based on purpose wasn't clear

Design opportunities

Based on pain points and key user needs, I proposed design opportunities to help users feel more confident in making their privacy decisions:

• Consider increasing the discoverability of third party use controls
• Refer back to familiar Android UI design patterns to match users' expectations on seeing a list of apps under each purposes in global settings
• Consider placing more emphasis on purpose level controls separated based on internal and third party use control as most participants didn't find value in categorizing permissions based on whether it is uncommon or common for an app category

Kicking-off design sprint

This led us to generating design concepts for next iterations of app and global settings flow. I facilitated ideation sessions to brainstorm different design solutions that incorporate the key focus areas.

Uncovering new questions

I realized that we know a lot about what people don’t understand about the designs, but the team has been lacking research on Android users’ needs.

• How are they currently configuring their privacy settings?
• What hacks are they using to get around limitations of the current experience?

Planning interview guide

I planned for a two part interview:

1) Starting with open-ended questions to gauge into their current experiences and attitudes
2) Using picture cards as a way to provide a tangible anchor point for participants to use to help recall stories.

Sample of context questions about their experiences, attitudes, and problems:

• Describe what comes to your mind when you think of your phone privacy settings. Why do you think that?
• Walk me through a time when you faced a problem while configuring your phone privacy settings. How do you currently solve this problem?

Sample of questions for debriefing:

• If you have 3 wishes to make your experience of configuring phone privacy settings better, what would they be?

Interviews in action

After pilot sessions, I conducted interviews alongside two designers who engaged in the note taking process and observed the session first-hand to better understand the participants’ struggles.

Analyzing interviews

During the analysis, I watched video recorded conversations to capture key quotes/moments and coded through transcribed data. These codes were clustered into themes: emotions/feelings, current ways of setting privacy, pain points/struggles, use of other information sources, and needs.

I clustered the codes into themes so that I can discuss about emergent themes with the team.

Creating models to communicate research insights

To better communicate my findings, I created a model to highlight key mental model components like installing an app, granting and revoking permissions. This allowed the designers and developers to have a shared understanding of key design opportunities and start to build out concepts based on users' mental model.

Kick-off conversations

I wanted the team to start thinking - How can we help people adopt this new way of controlling their privacy by making people’s intentions the focal point of the design?

App Settings Design Concepts

The two design concepts for app settings are shown below. They have similar content but are rendered differently through different design patterns and intentions by providing the point of entries for controls at different places. I wrote the intent under each concept with the designers so that we can all be in alignment of what makes each concept unique and how it addresses user needs in different ways.

Changes

• Granular purpose-level controls and third party-level controls are presented earlier on to ensure higher discoverability and quicker access
• Text descriptors for all permissions, purposes, and third party libraries have been added
• Permissions are no longer categorized based on uncommon vs. common but now based on app internal use vs. third party use

Global Settings Design Concepts

The two design concepts for global settings are displayed below. Concept A highlights the ability to control across multiple apps collectively based on purposes and concept B leads to individual app settings from the global settings based on a list of apps requesting for the permission.

Changes

• Text descriptors for all permissions, purposes, and third party libraries have been added
• List of apps are shown below each purposes to align with familiar Android UI patterns

Planning Concept Tests

I planned for a 3 part concept test:

1) Checking in on the core problems/limitations that users face when configuring phone privacy settings before showing the concepts, and learning about their existing solutions
2) Briefly describing and showing the concept to seek reactions
3) Comparing the different parts of each design concepts

Sample of introductory questions:

• What are the current limitations that you experience when configuring your phone privacy settings?
• How do you currently go about solving this problem, if at all?

Sample of follow-up questions when introducing concepts:

Let’s say when you check your privacy settings on your phone, this is a rough idea that appears and allows you to control your privacy settings for a particular app.

• You previously mentioned the problem about XYZ. How do you think this idea is helpful or not helpful in solving the limitations that you face? Why or why not?

Sample of questions for comparison between concepts:

• Which parts of each idea would you combine to create a new, better version? (Giving the freedom to draw on a blank paper)

Concept Tests in Action

I ran concept tests with one designer who took notes and recorded the session so that we can loop in other teammates who weren’t there to observe first-hand. I printed out the design concepts on paper so that participants can mark them up freely with a pen and move it around.

Analyzing Data from Concept Tests

During the analysis, I play backed recorded sessions to capture key quotes and collect thoughts on what participants didn’t understand or found confusing/not useful and what they preferred or matched with their expectations.

I clustered the notes into high-level themes: emotions, pain points, opportunities, and thoughts/attitudes and supported with sub-themes of what works and what doesn’t work for each concept and of design opportunities for different parts of each concept.

Key Insights

Key Insights

Privacy Overview Graph Design Concepts

The two design concepts for privacy overview graphs are shown below. They have similar content but are rendered differently through different design patterns and intentions by putting more focus on either app categories or data types.

Recommendation Page Design Concepts

The two design concepts for the recommendations page are displayed below. I actually helped prototype Concept A using Figma which highlights the ability to view apps that are collecting data in the background and view recommendations grounded on users’ previously configured settings for other apps in the same app category. Concept B focuses on quick access to revoking permissions across multiple apps based on inactivity or access to sensitive data types.

Concept Tests in Action

Analyzing Data from Concept Tests

I play backed recorded sessions to capture key quotes and collect thoughts on what participants didn’t understand or found confusing/not useful in the red bucket and what they preferred or found useful in the green bucket.

Key Insights

Key Insights

Creating Flow Diagrams

While designers are iterating on the designs from previous concept tests, I created flow diagrams to identify missing experiences and opportunities based on current state of flows. Based on interviews and concept tests, I noticed that most participants expressed the desire to be notified when sensitive data are accessed and when these permissions are left unchecked. I also wanted to plan out for cases when users choose to override manually configured multiple sub-options (purpose level controls) by controlling a parent option (permission level control). By leveraging heuristics like helping users to recover from errors and supporting recognition over recall, I mapped out possible points of entries for notifications and dialogs to surface and reflected these points onto the flow diagram so that I can use it as a boundary artifact when discussing new avenues for design opportunities with designers/developers.

Usability Testing with Prototype on Figma

Based on recommendations, the designers have incorporated the feedback into the Figma prototype for usability testing with users. We used the Figma Mirror app as a way to preview the prototype live on a Google Pixel 2 XL device.

Determining what UX metrics to use to achieve the goals

Observational metrics:

• Time-on-task
• Task success rate
• Error rate

Post-test session questionnaires with ratings and multiple choice:

• Ease of use and perceived difficulty (following each task with evaluative questions facilitated through 7 point scale)
• User satisfaction rating (5 point scale)
• NPS
• UMUX-LITE (shortened version of SUS)

Aligning what success means for the team

Task-level:

• All tasks should take no longer than 3 minutes
• All tasks should have at least 70% success rate
• Error rate should be at most 25% for all tasks
• Average ease of use should be at least 5 for all tasks

Test-level:

• Satisfaction ratings across all users should be at least a 4
• Average NPS score should be at least 0
• Converted average SUS score should be at least 68 (average)

Our Hypothesis:

• Users will find it easy to understand when controlling for internal/third party use in global settings and individual app settings. We will have demonstrated this if our tasks for app settings and global settings flow meet the task-level success criteria above
• Users will be satisfied with the experience and find Privacy Manager useful and easy to use. We will have demonstrated this if our test-level metrics meet the criteria above.

Planning for Moderated Usability Tests

Creating realistic scenarios:

You come across a new software update on your Android and notice a change in your phone privacy settings labeled as ‘Privacy Manager’.

Sample of goal-oriented tasks:

1) You want to know how often your camera was accessed yesterday. How would you go about doing that?
2) You recently downloaded XYZ app and want to stop third parties from accessing your approximate location to show you targeted ads based on location. How would you do that?
3) You now want to make sure that all apps will stop accessing your approximate location for advertisement. Find out which apps are accessing your approximate location for ads and stop them from doing so.

Moderated Usability Tests in Action

Two designers and one developer sat in on sessions and took part in note taking and controlling Figma Mirror when needed.

Analyzing data from Usability Tests

I play backed recorded sessions and captured notes on spreadsheets with columns like observed tasks, where the issue occured, description of issue or key quotes, persistence/impact/frequency of issues, and screenshots of participants’ key gestures/facial expressions. Each team member who attended the sessions read through his or her notes and cross referenced with my notes on the spreadsheet.

Tracking metrics

Click to expand on the graphs

Our average NPS score came out to be 20, which was 10% more than what we expected!

Our converted average SUS score came out to be 73.82 on the 68th percentile, which was 5.82% more than what we expected!

Pairing quantitative data with qualitative data to get the full view

Key Pain Points

App settings:

• Few participants are still struggling to understand what the difference between internal use and third party use is initially due to third party names

Global settings:

• Most users ended up having to manually check for particular permissions (location) under each of the apps individually, which they found annoying, as they thought they were unable to control across multiple apps at once
• Few users were confused about the controls as there weren’t enough indication to show that they can set the controls for them to act on
• Some users are surprised to see a list of apps under purposes as they’re accustomed to the current Android patterns and failed to recognize that they’re able to access individual app settings from the global settings page

Recommendation:

• Few users wanted to see recommendations linked with trusted external sources as they seek for more information on how to protect their privacy and feel like they lack knowledge to do so

Graph:

• Most users had difficulty with finding where frequently accessed data is - they had no idea that the graph served for that purpose initially

• Most users expected to see more in-depth information by tapping into individual bar graphs as the graph was not sufficient to view frequency access in detail

Design Recommendations

Distilled findings into design recommendations. Based on suggestions, key design changes have been made. Final design prototype link here

App settings:

• Provide a more accessible point of entry for users to access third party library controls by replacing the advanced button as this made some users feel as if it was unapproachable/not relevant for them
• Consider using helper text that directs the user’s attention to the granular controls, guiding their attention to the controls of off/ask/on and communicating the value of what “ask” does
• Enable users to search against multiple criteria in the app menu by promoting consistent categorization of apps across both app settings and graph by using app categories shown on graph

Global settings:

• Consider increasing point of entries to global settings as some users were expecting to find this capability from the list of apps in app menu and want the flexibility to control across all apps without having to go into each app individually
• Support the journey of first-time use for new users who are often unsure on how to control permissions across all apps by leveraging an onboarding flow that exposes them to different features

• Consider clearly communicating the value of override status when individual apps have different settings than what is globally applied across all apps.
• Give clear feedback to users on what actions they can take by having a visual indicator of 1) what the user can expand to and giving 2) a clear indication of where the user is when they tap into individual app settings to avoid confusion

Recommendation:

• Guide users with resources that they can refer to so that they can learn more about how to protect their data (like referring to PrivacyGrade)

Graph:

• Provide users the flexibility to toggle between monthly and weekly view so that the graph is more readable and less cluttered
• Provide more clarity on what the list of apps shown below the graph are, as some users were unsure whether those were a comprehensive list of all apps that used their data or a shortlist of apps, and enable users to directly control for these apps

• Leverage the list of apps shown below the graph when users tap into individual bar graphs to provide them a more in-depth layer of information as most users wanted to see more information on demand after viewing the first layer of information (frequency of data access) and know what classifies as an unusual activity