How comprehensive user research revealed what engineers really needed from AI and changed our product roadmap
Due to confidentiality, respect of privacy policies, and enterprise compliance, some visuals are recreated or simplified for portfolio purposes. A detailed version is available upon request.
CONTEXT
Role & Impact
UX Research Lead | 3 months, Aug 2024 - Oct 2024 | Houston, TX
Challenge: After the initial launch of our AI assistant, the product was being designed in the dark, building features based on assumptions rather than evidence.
Outcome: Led the platform's first structured user research, uncovering critical trust gaps and workflow misalignments that directly shaped our next several design sprints and influenced product strategy.
THE PROBLEM
We Were Building Features Without Understanding How Engineers Actually Used Them
Our AI assistant had 1,600+ active users, but we lacked systematic understanding of how they used it, what frustrated them, or where opportunities existed. In safety-critical environments, unused features represent missed opportunities to prevent incidents.
The Stakes:
Building features based on assumptions, not evidence
No visibility into trust barriers or workflow gaps
Roadmap decisions lacked user validation
About to invest in new capabilities without validating existing ones
RESEARCH APPROACH
I Designed a Mixed-Methods Approach to Uncover Both Usability Issues and Strategic Opportunities
Objectives:
Identify usability friction across 17 core features
Understand workflow integration patterns
Explore unmet needs and expansion opportunities
Participants:
11 engineers across disciplines, experience levels, and time zones
Early career to 20+ year SME veterans
Global representation (Houston, AGT, Sunbury)
Methodology:
Heuristic evaluation using Nielsen's principles
11 moderated remote usability tests (~45 min each)
Contextual inquiry observing real-world usage
ANALYSIS
Turning Raw Interview Data Into Actionable Product Strategy
After 11 interviews, I created systematic frameworks to spot patterns and prioritize insights:
Rose, Bud, Thorn Summary - What's working, opportunities, and pain points
User Profile Matrix - Behavioral patterns by role, region, and experience
Prioritized Issues List - Recommendations mapped by impact and effort
Thematic Clustering - Grouped into: discoverability, trust, workflow integration, expansion
If you wanted to rename a chat, how would you do that?
Participant Name
Body of notes
Participant Name
Body of notes
Participant Name
Body of notes
Participant Name
Body of notes
Participant Name
Body of notes
Participant Name
Body of notes
Participant Name
Body of notes
KEY INSIGHTS
Engineers Could Use Features, They Just Couldn't Find Them
6 Insights:
BEHAVIORAL PATTERNS
Observing Real Workflows Revealed Deeper Behavioral Patterns
Primary Use: Verification, not ideation - Engineers were using the assistant to confirm known facts rather than exploring abstract ideas.
Trust Concerns: Users were worried AI might lag behind process updates and document versions. They wanted to ensure data was up to date.
Workflow Complexity: Multi-step workflows needed simplification to reduce cognitive load.
Scale Desires: Users expected bulk actions, configurable interfaces, and personalization.
Onboarding Gap: Users consistently requested walkthroughs and contextual tips to make the product easier to adopt, especially for first-time users.
IMPACT
Research Identified Clear Wins and Critical Gaps That Shaped Our Roadmap
What Worked:
Significantly faster than legacy workflows
High satisfaction when features were discoverable
Strong excitement about document upload
Critical Gaps:
Feature discovery is the biggest adoption barrier
Trust and accuracy need work
Workflows under-supported for deeper research
Voice of the User:
"This is like a million times faster than finding the source myself."
"Creating the RACI charts—it did the work for us. I didn’t have to slog through documents."
"Saved me a day's worth of work—I did it in an hour."
WHY THIS MATTERED
The Most Valuable Insights Came from Challenging Our Assumptions
The Blue Text Revelation: I flagged blue text as a problem in my heuristic evaluation based on UX principles and accessibility standards. But user interviews revealed it was actually valuable because it mirrored internal documentation norms.
Learning: Users need clarity and control, not just uniformity.
Speed vs. Accuracy: The team assumed users prioritized speed. 82% preferred accuracy even if slower. This redirected our technical priorities from performance optimization to reliability.
The Real Value: Shifted the team from designing in isolation to building based on real-time user needs.
TRANSFORMING INTO DESIGN
This project is apart of a larger system of work
This research directly informed the design strategy detailed in my AI Assistant for Safety-Critical Engineering case study. The insights discovered here regarding trust, transparency, and workflow integration, directly informed the product's foundational design principles and feature roadmap.
See how I translated these research findings into shipped features like Workspaces, Agentic Tools, and transparency-first interfaces in the companion design case study.