We Tried Replacing Employees with AI It Broke

In mid-2025, McDonald's AI-powered hiring platform McHire built on Paradox.ai's system was found to have a test administrator account secured with the default credentials '123456/123456' and no multi-factor authentication. Security researchers accessed data linked to approximately 64,000 applicants. The exposed information included full chat transcripts from the 'Olivia' hiring chatbot and responses to personality assessment questions. McDonald's and Paradox.ai both issued statements. No social security information was accessed. But the incident illustrated a consistent pattern in enterprise AI deployment: AI systems are often treated as products to be turned on, not infrastructure to be secured.

A federal court in California allowed a nationwide class action to proceed in May 2025 against Workday, alleging that its AI-driven resume screening system discriminated against older applicants. The lead plaintiff a Black male jobseeker over 40 applied to more than 100 jobs through companies using Workday's AI screening and was automatically rejected every time. One rejection arrived at 1:50 AM, less than an hour after he applied. No human could have reviewed his application. The lawsuit alleges the AI 'baked in' existing biases by training on historical hiring data that reflected discriminatory human decisions. Thousands of companies use Workday's tools to screen candidates with no human in the loop.

When Jack Dorsey cut 4,000 employees in February 2026 and attributed the decision to AI capability advances, he acknowledged in his own memo the risk of 'moving too fast' and said the company had built in 'flexibility to account for mistakes.' Within weeks of the announcement, Block began quietly rehiring some of the employees it had just cut. At least four former workers returned to the company. One departure was acknowledged by Block leadership as a 'clerical error.' Another was described as 'operational necessity.' The speed of the reversal in some cases less than two weeks suggests the decision to cut was not as precisely calibrated to AI capability as the public framing implied.

In a widely cited 2024 case that set legal precedent, Air Canada's AI chatbot told a customer that bereavement fare discounts could be applied retroactively a policy that did not exist. When the customer attempted to claim the discount after booking, Air Canada refused, arguing the chatbot had made an error and the company was not bound by it. A Canadian civil tribunal ruled against Air Canada: the company was responsible for information provided by its own AI system, whether or not a human authorized it. The ruling established that AI chatbots are not separate legal entities. Their outputs are the company's outputs. The liability follows.

• High-stakes, judgment-dependent contexts: every significant AI failure involves a domain where the cost of error is high hiring decisions, customer commitments, security of personal data, clinical assessments
• No human in the loop at the point of impact: the failures occur when AI makes or communicates decisions without a human review step before consequences are delivered
• Training data that encodes past discrimination: AI hiring, lending, and housing tools consistently reproduce the biases in their training data not because of malicious intent, but because historical data reflects historical discrimination
• Security treated as an afterthought: AI deployments frequently inherit the security posture of consumer products rather than enterprise infrastructure default credentials, no MFA, no access reviews
• Speed of deployment outpacing governance: the timeline from 'pilot' to 'production affecting real people' compresses in AI in ways that do not happen with traditional software, and governance structures have not kept pace

MIT's 2025 review of over 300 AI implementations found that the biggest failures were not technical. They were organizational: weak controls, unclear ownership, and misplaced trust in AI systems that had not been stress-tested in production conditions.The companies that have made AI work are the ones that defined a specific business problem, assigned a named owner, invested in data quality, maintained human review for high-consequence decisions, and measured outcomes against the baseline. The companies that failed are the ones that deployed first and governed later. Given that AI systems can make thousands of decisions per minute, 'governing later' can mean thousands of errors before the problem is detected.