User research for AI features

Why standard user research lies

"Would you find an AI summary useful?" gets ~95% yes in interviews. Then the feature ships and 8% of users use it weekly. The gap is not user dishonesty — it's that abstract usefulness is easy to imagine; concrete use requires a specific moment of need that's hard to discover in an interview.

Three practices that get past the lying:

1. Wizard-of-Oz prototypes. Manually produce the AI output (a researcher fakes it via copy-paste) and put it in a real workflow. Watch whether users actually use it.
2. Diary studies. Ship a low-friction prototype to 20 users for 2 weeks. Track usage by hour, not by survey response.
3. Refusal acceptability. Specifically test: when the model refuses or gives a partial answer, do users tolerate it or churn? The refusal experience is often the make-or-break — and surveys never catch it.