Why the same AI earns wildly different reliance from different people - and what product teams can do about it.
Your 10-seconds read: Trust in AI is not a binary product metric; it is a dynamic cognitive attitude. While engineers track model accuracy, user reliance is governed by psychological, cultural, and organisational factors. To build safe and sticky AI products, product leaders must shift from maximising raw usage to designing for calibrated reliance - mitigating both misuse (over-trust) and disuse (distrust).
Trust in AI has become a buzzword. Scientists have studied trust in automation for decades, but consumer and B2B AI products gave it a commercial twist. Too little trust means low adoption, drop offs, and poor ROI. Too much trust leads to costly automation mistakes and the bill includes court hearings, global scandals, and, occasionally, people's lives.
Here's a fun statistic from KPMG's 2025 study.
Only 46% of people say they're willing to trust AI systems. Meanwhile, 66% of workers admit they use AI output without “evaluating accuracy”, and 56% have made mistakes because of AI. So: we don't trust it, we use it anyway, and we don't check. If that sounds like a toxic relationship, it's because it is one. [ref]
Scientists in automation and human-AI interaction stay cautiously pessimistic:
Trust, mental workload, and risk can influence automation use, but interactions between factors and large individual differences make prediction of automation use difficult.” [ref].
Product managers, meanwhile, often treat trust issues as a binary switch (trust = good, no trust - bad, bad, bad) and bet on polishing the model and frictionless UX with a sprinkle of transparency to prevent drop-offs high in the funnel.
Time to play a bad cop. Trust is a complex, dynamic cognitive attitude, and getting the level wrong creates legal, financial, and reputational liabilities for users and for the companies that build the product.
Trust is an attitude. Reliance is what you can actually see.
The good news is that the conversation is shifting from maximizing trust to calibrated reliance. Two ideas matter here:
Trust is an attitude, mostly invisible to the product eye. Reliance is an observable, concrete action you can (and should) track to know whether a user accepts, rejects, or modifies an AI's output [ref].
Calibration is the right goal. It means aligning trust with warranted reliance: lean on the AI when it improves decisions, and qualify or withhold reliance when it doesn't. [ref]
In plain English: you don't want users who trust the AI more. You want users who trust it accurately.
Careful, though. Trust and reliance are cousins, not twins. People rely on tools they don't trust when they are under time pressure, have no better alternative, or their manager tracks usage. Others say they trust a tool and never open it. And "trust" doesn't always point at the tool itself. It can land on the model, on the product wrapped around it, or on the company behind both. You can like the output and still not want your data anywhere near whoever built it.
So any AI product tends to fall into one of two abysses: algorithm aversion - the reluctance to trust or rely on AI or algorithmic advisors [ref], and algorithm appreciation, when people prefer AI over human judgment [ref].
It leaves us with two attitudes and related behaviors:
When trust is miscalibrated, both modes could be a cost.
“Appropriate for what, exactly?”
Wait, you’ll say. What does it mean “appropriate for a specific situation”? Shouldn't that be "appropriate for model performance"? Latency, hallucination rate, citation coverage, and ground truth are what engineers build and PMs track. Focus on what you can control.
Fair point, and accuracy does matter a lot. A 2021 analysis [ref] found reliability to be the strongest single predictor of trust (we’ll come back to it in another article). But it's not the only driver - human psychology is way too complex.
“People might show extreme behaviors (misuse or disuse) that are not necessarily correlated with the model performance and accuracy. ”
In other words, reliance isn’t always correlated with accuracy. If you ever run any A/B tests on the same model, you might have found that changes in voice and tone, or even visual nuances might make users feel and behave differently.
In one experiment (N=404), having the model express uncertainty in the first person ("I'm not sure, but…") lowered users' confidence in the system and their agreement with its answers, and raised accuracy. [ref]
In Stanford's work on sycophancy (N=1604), participants rated sycophantic responses as higher quality, trusted the sycophantic model more, and were more willing to use it again. [ref]
So if model accuracy isn't the only driver, what is? Back to basics.
Enter the 3P model
The classic People-Process-Product model is an oldie but a goodie: three interconnected pillars of sustainable organizational growth. Product is what you offer to the market. People include not only employees and leaders but customers and the wider net of stakeholders. Process covers the management systems, workflows, and standards that get work done with minimal waste and friction. Over-invest in one pillar and the whole thing eventually tips over.
I bet you can already see how to twist this for AI products. We’ll do exactly that, one pillar at a time:
Trust and reliance aren’t determined by the AI alone - it emerges from people interacting with a product in a particular process and context.
Pillar 1. People: individual, social, and cultural context
Trust and reliance are mediated by the humans who build, operate, and experience AI. This is where psychological and environmental forces come into play across three key groups of factors: individual, cultural and demographic ones, plus the social and organizational environments around them.
1.1. Individual dynamics.
Big Picture: Expertise drives trust in AI and sharpens error detection, because experts know exactly where the system is likely to fail.
As we already know, not everyone has the same level of trust and reliance on AI. Each user brings their own heritage: what they know, how sure they are of themselves, how well they can tell when the AI is wrong, and how they and their immediate social circles feel about AI.
Let’s start with ability based factors like domain expertise, overall level of competency in the AI system, and the level of task understanding.
All of this has been shown to have a significant effect on the level of trust in AI [ref]. A 2021 analysis of trust in AI by Kaplan et al. found that users who understood the AI system and were competent with it trusted it far more (d = 1.02; for reference, above .8 counts as large), and experience at the task helped too (d = .47). Makes sense: the more you know, the better you can tell when the thing deserves your trust. The opposite is also true: when users lack the skills to inspect the output, they are more likely to skip scrutiny. In other words, the people least equipped to catch an error may also be the least inclined to look for one.
So if you're designing for high-stakes decisions (medical, financial, legal) and expect junior specialists among your users, don't count on their critical thinking to show up on its own. Build it into the flow. For example, ask them to commit to their own answer before seeing the AI's, or surface the evidence behind a claim at the moment of decision. We’ll come back to these tools in the upcoming articles.
Personality and personal situation play a smaller part (d = .25 in Kaplan’s study) but still bring us some interesting findings. Lonelier participants might trust AI less, more innovative ones trust it more, and extroversion pulled in both directions. [ref] Please, treat that as a hypothesis and no, don't add a loneliness question to your screener.
1.2. Cultural & demographic dynamics.
Big picture: Trust doesn’t travel well: the same AI model can earn very different levels of trust in different markets and across different demographics: make sure to split your data and conduct research with a varied group of users.
Culture
Power distance, uncertainty avoidance, and broad population-level trust gaps can influence how people perceive AI or automated / robotic systems. Time to dust off the good old Hofstede bible [ref]. Quite a bunch of studies highlight regional differences: for example, German participants appeared to trust AI more than Japanese participants [ref]. You might also discover that design treatments that build trust in high uncertainty-avoidance or collectivist cultures (like detailed validation logs) can produce opposite reactions in other regions (e.g. causing friction or skepticism).
So if you roll out globally, make sure to account for possible regional fluctuations in trust and reliance. For a quick regional read, KPMG’s 47-country study is a good start, though it measures general willingness to trust AI, not reliance on your product - don’t forget to treat it as hypotheses, re-examine your personas, and test locally.
Gender
Gender is another significant trust predictor. Across most of the global surveys men consistently report higher levels of trust, optimism, and acceptance of AI systems than women. KPMG Study found that 46% of men report high willingness to trust AI systems, compared to 35% of women. One hypothesis worth testing is familiarity with how AI algorithms operate (which, as we saw, goes with higher trust).
At the same time women consistently report higher concern regarding ethical risks, data privacy, bias, and potential job displacement caused by AI systems. Whatever the appropriate mix of reasons is, make sure your research panel doesn’t skew male or female participants, otherwise your trust and reliance estimates may be off.
Age
Trust and confidence in AI decline as participant age increases. Thus, Gen Z & Millennials demonstrate the highest levels of trust (averaging 55% plus willingness to trust/accept AI). Older Adults / Baby Boomers show the lowest trust scores, with a majority reporting skepticism, hesitation, or anxiety around AI, especially in the high-stakes environments (such as healthcare and HR) [ref], [ref].
Education level
Users with a university degree tend to report higher trust (52% acceptance globally) and better recognition of where AI sits in the tools they use daily. Those who don't have a university degree normally report more scepticism, which is plausibly driven by uncertainty about how AI systems operate and decide. [ref], [ref].
A caution before you build personas and research strategy out of this. Age, education, and gender might be stand-ins for exposure and familiarity, so these factors might be hard to pull apart in any one survey. Treat demographic splits as hints about where to look, not as an immediate segmentation strategy.
1.3. Social & Organizational dynamics.
Big picture: People decide whether to trust AI partly by looking sideways, at their peers and their leaders so that acceptance calibration might be a part of a bigger team dynamic.
Humans are social animals and we tend to lean on our peers’ opinions and norms in high stakes or novel situations. Many are still not sure how to address AI that feels like a very ambiguous terra incognita, and tend to check what their immediate social connections (like colleagues and friends) say about the new technology, what the group norm is.
Peer validation
Some studies show that users who have low direct experience or low domain expertise, often look for a helping hand in assessing whether they should trust AI or not. Here peer validation acts as a primary proxy for trustworthiness. If respected teammates endorse an AI assistant, individual skepticism drops significantly. [ref]
Leadership signals
Similarly, leadership endorsement might also serve as a positive institutional signal. When leaders actively use AI and openly share both its utility and limitations, baseline trust across engineering and product teams increases.
Psychological safety
Overall organizational trust also might be an adoption gatekeeper: if employees suspect management uses AI metrics punitively, they evaluate the technology more negatively and adoption might plummet into disuse [ref].
The evidence on how strong these factors can be in defining trust and reliance is mixed. The most plausible reading I can offer is that social signals matter less than reliability and performance in the long run, but can swing early rollout and the initial calibration.
Product manager To-Do list
(1) Know your user
☑️ Avoid reporting aggregate, sitewide averages that flatten critical behavioral variance. Ensure that product dashboards include default splits across:
Demographic & regional cohorts: thus you can slice usage, disuse and misuse, and retention data across demographics and regional/cultural markets. This detects disparities in baseline trust, privacy sensitivities, and tolerance for automated actions. Always look for correlations in user type, behaviors and product decisions - for example, high uncertainty-avoidance cohorts might be exhibiting higher drop-offs when confirmation steps are missing.
Where possible, implement user tiers (juniors vs. seniors / novices vs. experts). As we’ve seen, less experienced users are significantly more vulnerable to the fluency trap and sycophancy bias accepting AI outputs without verification. Connect override rates and inspection time to tenure/expertise to ensure junior cohorts aren't absorbing automation errors.
☑️ Make tiered workflow scoping a norm defining different interaction pathways based on expertise tiers.For a junior user you might want to provide stepped verification, source-grounding checks, or forced pre-commitment, while granting experienced senior operators low-friction override controls.
(2) Product and marketing strategy
☑️ Design for secondary consumers: Take into account people that the main AI users share its outputs with (colleagues, clients, managers, etc) and understand their role in the output acceptance and overall product adoption. If they have a say, you might need to design specifically for these downstream users and introduce into the AI output the features they care about ( e.g. visible lineage metadata, data watermarks, verification badges or similar).
☑️ Make market-specific rollout audits a norm. When expanding globally, avoid assuming that adoption patterns in one market will replicate elsewhere and budget for local validation. Track regional baseline trust and reliance disparities to catch significant cultural divergences early and dig deeper to understand what causes them (model behavior? design patterns?)
☑️ Where possible, plan the social side of rollout: carefully select early champions, peer examples, and a safe way to collect feedback that doesn’t involve a punishment.
(3) Data and feedback loops
☑️ Ensure your product is data-ready and you have a full picture on performance. Data readiness means logging not just basic clicks, but user intervention points (edits, manual rewrites, overrides, aborts) and - on the other side - system uncertainty values and latency at the moment of interaction.
☑️ Don’t rely only on explicit sentiment surveys (thumbs up, CSAT) and look for / build alternative feedback channels. For example, in the enterprise environment peer gossip is difficult to catch and measure but you can turn it into transparent feedback by building in-product team error-transparency logs or shared annotation widgets so that users can publicly flag prompt quirks, and track bug fixes.
☑️ Catch misuse vs disuse early. Track interaction logs to monitor the misuse-to-disuse ratio and split it across different user cohorts (juniors vs seniors, different regions, demographics) or at the team level. It will help to identify user cohorts or the whole business units that either accept outputs without inspection (misuse) or abandon capable tools (disuse) and follow up with a qualitative dive in to understand the reasons behind it.
Product leadership To-Do list
☑️ Stop treating raw usage as the success metric. If you want your product to be successful in the long term, you need to shift executive KPIs away from binary adoption, DAU, drop offs or thumbs-up CSAT toward trust calibration and decision quality.
Remember, unverified overreliance in high-stakes workflows is more of an emerging operational liability rather than a growth victory. Ask for calibration-aware metrics in reviews to understand how helpful your product really is.
Empower PMs and designers to introduce intentional cognitive friction where necessary without penalizing them for minor conversion drop-offs. Everyone in the org needs to understand that friction at critical decision points mitigates costly legal, financial, and safety failures.
☑️ Push for understanding the why’s behind measured behaviors
Don’t rest on the assumption that data and telemetry alone tell the whole story. Fund qualitative user research as an always-on operational pipeline running in parallel with telemetry to capture human cognitive friction, cultural nuances, unarticulated frustration, and trust erosion that you will never see on a dashboard.
When expanding to new markets, ensure teams get a budget for local validation to ensure successful rollout.
User Researcher’s To-Do list
(1) Recruiting
☑️ When running research, ensure stratified sampling across expertise tiers: as we’ve seen, domain competency and system understanding are the strongest predictors of trust, recruit a representative mix of users across all expertise tiers - especially juniors, novices, and non-power users.
Guard against panel skew by recruiting across demographics that exhibit established trust gaps:
Gender: Remember that men report higher baseline optimism and familiarity, whereas women report greater concerns regarding ethical risk and data privacy.
Age & overall digital literacy: if your prospected audience is mixed, include both younger cohorts (Gen Z/Millennials) that exhibit higher baseline acceptance, and older cohorts that might show more pronounced skepticism and anxiety in high-stakes contexts.
Education: Higher formal education correlates with greater awareness of AI integration and higher baseline acceptance
☑️ If rolling out in culturally different regions, conduct cross-cultural baseline benchmarking across target deployment regions. Design treatments that build trust in high uncertainty-avoidance or collectivist cultures (e.g. granular validation logs) can produce opposite reactions in other regions (e.g. causing friction or skepticism), making local verification essential.
(2) Methods
☑️ Introduce error-injection technique into your usability testing: instead of testing only standard "happy paths", intentionally seed hallucinations or errors into prototypes*. Observe whether participants blindly accept the output or actively audit and correct it to build your overreliance hypothesis (that you can later check on the real usage data).
* To make testing feel more realistic, you can also use the Wizard of Oz technique: have participants interact with what they believe is an autonomous AI agent, while a teammate secretly controls the responses.
☑️ Don’t test a separate task - build an E2E view. Not all tasks in a workflow are even. Map out the user's end-to-end task flow, cataloguing decision points, related cognitive workload, and time pressure. You might discover that at some points users tend to drop vigilance more often and adopt uncritical reliance shortcuts.
☑️ Secondary user testing. In some cases, people consume AI outputs created by someone else (think of a manager reading an AI-summarised brief from a junior IC). If this is your case, expand your usability studies to the downstream recipients to evaluate whether they can accurately assess AI output credibility and whether lineage metadata or AI watermarks change their reliance.
☑️ If you will be lucky to get access, explore peer chatter & informal gossip: use contextual inquiry to get a sneak peek into informal team communication networks to understand how peer validation and informal word-of-mouth influence feature adoption.
☑️ Embrace longitudinal studies. Don’t invest all your resources into 30-minute single-task lab sessions. Trust, acceptance, and reliance evolve over time and are easily disrupted by model updates and social dynamics. Use multi-week diary studies and async interviews combined with telemetry logs to capture how model changes, habituation, delegation scope, and recovery after errors unfold in practice.
Data Scientist’s To-Do list
(1) Tracking apples and oranges separately
☑️ As we’ve seen, trust and reliance are two different measures. Trust is more qualitative and sentimental, while reliance is a real behavior reflected in product metrics. Make sure you track them separately and look for correlations.
To measure trust, you can use standardized surveys like the Trust in Automation Scale (TIAS) or its short version, S-TIAS. You can also gather momentary ratings - like a quick thumbs up or down - right after a user gets an AI recommendation or finishes a task.
Reliance is tracked through:
acceptance/rejection rates (how often a user follows or ignores the AI's advice),
for some flows (like medical diagnosing decisions, financial decisions) you might use weight on advice (WoA) that quantifies how much change an AI's recommendation causes in a user's final decision (you would need to compare the user's initial independent judgment to their final decision after seeing the AI's output).
Over-reliance and under-reliance.
(2) Tracking demographic & cultural dynamics
☑️ If you need to remember one single thing from all this lengthy write up, it would be - don’t evaluate product performance or test sets in aggregate. Ensure that data is ready for an easy audience slicing across demographic groups (gender, age, region, socioeconomic tiers, digital literacy) and expertise level to identify differences in adoption, reliance, drop off rates and behavioral nuances.
☑️ Evaluating user biases
Yes, those can actually be tracked! You can set up evals that test for sycophancy bias and verbosity bias and track whether user engagement and preference ratings decouple from factual accuracy. Thus you will ensure that optimization algorithms do not reward agreeable or elaborate hallucinations at the expense of correctness and long-term reliance.
Of course, human psychology and team dynamics don't exist in a vacuum - they are constantly reacting to what the software itself signals and executes. Knowing who is doing the trusting is vital, but what product features, interaction styles, and safety boundaries actually trigger or destroy that trust? In our next piece, Beyond accuracy and transparency: the real product factors driving human-AI trust we dive deep into the engineering and UX mechanics of reliability, the transparency paradox, and why designing for calibrated trust requires more than just a slick interface.
