Voice cloning is increasingly used for content creation. Yet current systems remain complete black boxes.
How can we break this black box and make this creative workflow more intuitive & human-centered?
2026 PRODUCT DESIGN
AI UX
Voice AI
App development
Overview
CLIENT
Salesforce AI
Time
Jan - June 2026
Team
2 Researchers, 2 Designers
Role
Product Designer
Tools
Git, Claude, Vercel
My Role
This project was University of Washington's final capstone project sponsored by Salesforce AI team based in San Francisco.
As product designer of the team, I led the product prototyping and interaction design from concept to final experience.
My biggest contributions were:
Setting up the initial Git repository and Vercel deployment workflow.
Creating the branded design system.
Leading the design and implementation of the voice editing experience.
Solution
We built an AI Playground that unifies recording, evaluation, and fine-tuning into a single workflow.
*This video demonstrates a sample workflow. For the full experience, try the our product!
Outcomes
$ 10K
API funding secured through our proposal
14 +
validated the system across 2 research rounds
5 of 7
rated 4/5 Likert on control, guidance & ease of use.
Validated by experts from
Special thanks to Julie Zhuying, Senior UX Manager at Google, and Behzod Sirjani, Chief of Staff at Vercel.
Key features
Overview
4-step workflow
Voice cloning is still a very professional area. For general users to be able to use it with agency, we structured our approach into 4 steps.
Step 1. Recording
Users starts recording their voice.
Step 2. Preview
User gets two distinct options with clear guideslines.
Step 3. Content Selection
User & model share a baseline for contextual editing.
Step 4. Editing
Editing with pinpointing tools.
Feature 1.
Actionable and guided recording
A quality voice clone starts from the recording process. Clear feedback and progress provide actionable feedback.
Realtime Voice Coverage
Users receive real time feedback on each voice parameter analyzed by the voice model API.
Feature 2
Structured clone evaluation
When there's only one clone, it's hard to tell whether its good or bad. To support decision making, we provide a structured comparison.
Feature 3
Global and granular fine tuning in context
By choosing a content type, users sets the starting point. This creates a shared baseline for both user and the system, enabling context-aware presets and editing controls in the playground.
Natural speech controls
Pauses and filler words are fully editable, giving users fine-grained control over pacing and delivery.
Feature 4
Pinpoint editing
Users can pinpoint the parts that sound "off." Without any technical voice attributes, system translates high-level intent into precise model edits.
Iteration as collaboration
AI explains how it interprets each request. This makes every edit transparent and collaborative.
Context
This project was sponsored by Salesforce AI, where voice cloning is an emerging area for B2B applications.
Our research explored how voice cloning could become more interpretable and controllable for users without deep AI expertise.
Problem
Voice cloning is becoming increasingly common for content creation.
However, despite advances in model quality, the cloning process remains largely a black box.
Research process
We analyzed voice cloning platform workflows and conducted co-design sessions with experienced voice cloning and TTS users.
Co-design Interviews
Made blank page, simplified prototype with hume model for co-design with N=7 participants
What already exists
Current platforms such as Hume AI and ElevenLabs offer polished, user-friendly experiences.
However, their workflows are primarily transactional. Users can't understand how it was generated.
Initial findings
Across 7 participants, multiple themes emerged that shaped our iterations.
Input
Users believed output quality depends on the quality of their recordings.
Gap
The prototype gave them no guidance on what "good" looked like.
Training
Users wanted to understand what was being captured in their clone.
Gap
The system offered no visibility into what was being learned or why.
Output
Without context, editing lacked direction. Users who spotted errors tended to re-record rather than refine.
Gap
The system didn't allow finer control over filler words, accent, or emotion.
Design iterations
Challenge 1.
Providing real control without adding complexity
The real challenge wasn't giving users enough controls; it was keeping them in the flow.
I explored sliders and a 2D matrix dial, but each added a layer users had to think through before they could act. The goal was to match how users actually experience voice.
Challenge 2.
Bridging word-level intent with API constraints
Users wanted to fix a specific word or phrase, but the API only operates at the sentence levels. The design challenge was making a sentence-level system feel word-precise.
Solution: Pin-based editing
To solve this, I designed the pin system to feel word-precise while actually regenerating the whole sentence with targeted parameter overrides. I used the UX pattern of dropping a pin to flag or comment something.
Challenge 3.
When & how to surface pinpoint fixes
Usability issues
A comment-style UI confused some users. It felt like leaving feedback rather than making an edit. The "Pin edit" label added confusion.
Position was tricky. Adding it to the control panel felt disconnected from where users were actually looking.
Users want to know before how adjusting each control would affect the voice output.
Solution: Contextual positions
I designed a contextual action bar surfaced directly on text selection: bringing Edit with AI, Pause, and Filler words inline with the problem.
Contextual action bar
The edit lives where the issue is.
Edit queue feedback
Users can see how their requests are being sent to the system.
Clear & intentional UX writing
Copies were rewritten so users knew exactly what each control would do.
How AI was used
The workflow
Figma Make → human refinement → shared spec → parallel build
For our first prototype, we used Figma Make to generate a minimal, skeleton-level flow to test with users. It helped us speed up by drawing on existing UI patterns from voice cloning tools like Hume and ElevenLabs.
When the wireframes were ready, I took the lead in the initial git set up and deployment by integrating Hume's voice clone API.
AI-generated UI still needed design oversight. Error states, minimum recording criteria, loading behaviors, and sample sentence copy were all intentional designer decisions.
Collaborating in vibecode
Turning constraints into a shared spec
We collaborated directly in code. We used Claude to map our technical limitations, discussing what's possible and what we wanted to build based on the user testing insights. Then, we created a shared claude.md file, keeping design and code aligned.
Impact
Showcased at UW's HCDE capstone showcase in June 2026, we introduced the system to 100+ visitors, with 30+ people trying it firsthand and sharing positive responses.
Reflections
Using AI wisely, owning the craft
Building a working AI prototype meant designing at the system level. From day 1, Soyun & I treated feasibility like a PMs, discussing model limitations and technical constraints before designing.
Make every choice intentional
Every interaction was intentional. I grounded decisions in user feedback, resulting in a more engaging product.
Weekly critiques to validate
Intentional design crafts!
Unforgettable 6 months!
A strong capstone matters, but the people matter more. I'm grateful for working alongside teammates who were thoughtful and deeply collaborative!
