The Andon Cord: Designing Humans Into AI Systems

In the 1960s, Toyota made a decision that seemed counterintuitive for a manufacturer trying to maximize output. They gave every worker on the assembly line the ability to stop the entire line by pulling a cord. Any worker who spotted a problem could halt production immediately, trigger a response from supervisors, and prevent a defect from moving further down the chain.

The assumption in mass manufacturing had been that lines should run continuously and humans should fit around them. Toyota inverted that. Workers weren’t a variable to be managed around by the system. They were designed into it as the primary quality control mechanism, with real visibility and real agency over what happened. Quality improved dramatically. So did worker engagement. The humans were now part of the architecture of the system, and it made things better.

Many teams building systems with AI-driven automation today are making the same mistake as Toyota’s competitors were making then. The operating assumption seems, again, to be that systems should run, and humans should fit around them (whether they ultimately stay in the loop or not). The irony is that we have decades of evidence showing why that’s wrong, from factory floors to flight decks, and we’re repeating it anyway.

In any business AI system, humans aren’t a fallback or a stopgap. They’re a design element, as fundamental as the data model or the workflow logic. The question isn’t whether to include them. It’s how to design their role thoughtfully, and how that role should evolve as the system earns trust.

Trust doesn’t transfer, it builds

Research on human relationships suggests it takes roughly 200 hours of shared experience before people consider someone a close friend. The mechanism matters: small interactions, accumulated over time, that build familiarity and predictability. You can’t shortcut it. You can’t just tell someone to trust you.

Systems work the same way. People build relationships with software. A well-supported model in human-computer interaction research, the Technology Acceptance Model, identifies trust as a key moderator of whether people actually use systems that are available to them. Usefulness alone isn’t enough. People need to feel confident in what the system is doing before they’ll rely on it.

The evidence on human-algorithm relationships

The research here is more specific than most product teams realize.

A 2015 study by Dietvorst, Simmons, and Massey in the Journal of Experimental Psychology found that people readily abandon algorithms after seeing them make a single mistake, even when the algorithm demonstrably outperforms human judgment overall. They called this “algorithm aversion.” A follow-up paper, “Overcoming Algorithm Aversion”, produced a striking finding: when people were allowed to modify the algorithm’s output, even slightly, aversion dropped dramatically. The ability to adjust the result, even without changing it much in practice, was enough to maintain trust through errors.

There’s also a well-documented phenomenon called automation bias: the tendency to over-rely on automated systems and stop monitoring them critically. Removing meaningful human involvement can paradoxically degrade performance as well as hurt adoption. People who are kept in the loop stay sharper. People who are reduced to rubber-stamping become less effective monitors over time. The aviation industry has learned this painfully, most famously in cases where highly automated flight systems handed back control unexpectedly and pilots, out of the loop for too long, couldn’t respond effectively. We’re starting to see the same pattern in AI-driven systems.

Meaningful human agency turns out to be good for outcomes, not just adoption.

What this looks like in practice

Over the past fifteen years, I’ve worked on a number of AI/ML systems that strategically bring humans into the loop. Here are a couple of practical examples of how to think about this dynamic in different industry contexts and over the adoption curve.

At Versive, we were building a machine learning system to detect hackers inside corporate networks. Our customers were SOC operators: skeptical, experienced, and very aware of what happened when security systems either cried wolf or missed things entirely. We couldn’t just surface alerts and expect them to act.

So we showed our work. We adapted a “scrollytelling” approach, borrowed from long-form data journalism, to walk operators through exactly how the system built its case. Not just “this is suspicious” but “here’s the sequence of events, here’s why each one matters, here’s the full trail.” The goal was to let people evaluate the reasoning, not just accept the conclusion. It worked. Operators who understood how the system thought started trusting what it found.

At eSentire, we moved further along the automation curve. The approach shifted toward empowerment across a broader range of cybersecurity signals. Our focus was automating the low-value repetitive detections, and surfacing the high-priority alerts with full context for human investigation when needed. Operators stayed in the driver’s seat on the decisions that mattered. And every action, whether taken by the system or the operator, was logged. The paper trail was always available.

At Moxiworks, in the realm of real estate sales and marketing, we have a different driver: less about surfacing and managing risk, and more about presenting and actioning appealing opportunities. In this case, our customers don’t have to act, they want to.

So our focus becomes making the best AI-driven suggestions we can based on data from around the system. We explain why they’re interesting and provide supporting details. Then we give people choices about how to move forward, ranging from full automation, to a simple confirmation, all the way to full manual execution. Most people choose the automated path, but the important thing is that they don’t have to.

The overall point is that rather than designing human touchpoints into individual workflows after the fact, it’s important to build the entire system around them. That means designing people into the system from onboarding forward, as the organizing principle. The goal is a genuinely intelligent platform where human agency is part of the architecture.

The role evolves, the need doesn’t

Here’s what I’ve noticed across these projects: the reason people need to be in the loop changes over time, but the need itself doesn’t disappear.

Early on, human involvement is functionally necessary. The system is new, its track record is thin, and there’s real value in having people review and approve before consequential actions are taken. Humans are doing real work in the system.

Later, as trust accumulates, the role becomes more about control and legibility. People don’t need to approve every action, but they want admin controls, audit logs, visibility into what’s running. They want to know they could intervene if something went wrong, even if they never do. The system has earned autonomy, but people still want the keys.

Skip either phase and you’re in trouble. Move to full automation before trust is built, and you’ll see resistance and workarounds. Remove visibility and control after trust is established, and you’ll erode what you built. And when you need a human to be there, they’ll be so disconnected that they won’t be ready to intervene.

A note on growing AI familiarity

There’s a reasonable counterargument here: as people use Claude, Copilot, and similar tools in their daily work, won’t general familiarity with AI reduce the trust-building burden over time? Probably, at the margins. Category trust is real, and teams already comfortable with AI tools do come to new workflows with lower initial resistance.

But category trust and system trust aren’t the same thing. Trusting an AI assistant for personal productivity is different from trusting an automated workflow making consequential business decisions. The stakes are different, and trust has to be earned in that specific context. And if anything, experienced AI users tend to have higher expectations for transparency and control, not lower ones. They know what can go wrong. The button to press isn’t a concession to skeptics. It’s what good design looks like regardless of how AI-savvy your users are.

The design principle

Build humans into the workflow deliberately, and design for how that role evolves.

That means starting with transparency: show how the system thinks, not just what it concludes. It means creating checkpoints that give people meaningful agency, not just the illusion of it. It means preserving audit trails and controls even when they’re rarely used, because their presence matters to the relationship.

And it means having a plan for how human involvement shifts as the system earns trust, rather than treating automation as the finish line and human steps as the obstacles in the way.

The companies that get this right see better adoption and build systems that improve over time, because the humans in the loop are engaged rather than alienated, contributing rather than routing around.

Pull the cord. It’s not a brake on the system. It’s what makes it work.