🔴 REAL INCIDENT: NYC MyCity business chatbot — inaccurate legal guidance documented March–April 2024; remained publicly accessible
What Happened
New York City built an AI chatbot to help small businesses navigate rules, incentives, and compliance.
Then journalists asked it ordinary questions—and the city's own agent told them to break the law.
On March 29, 2024, The Markup (copublished with Documented and THE CITY) reported that the Microsoft Azure–powered MyCity business chatbot was giving incomplete and, in worst cases, "dangerously inaccurate" answers on housing policy, worker rights, and entrepreneur rules. The bot presented itself as trained on official NYC Business information. Users had little reason to treat it as a parlor trick.
Days later, Mayor Eric Adams acknowledged problems at a press conference. The city updated disclaimers. It did not take the bot down. As of The Markup's April 2 follow-up—and contemporaneous AP coverage—the chatbot remained on the official site, still capable of returning false guidance even after the scandal was public.
An official agent. Authoritative branding. Illegal advice. Still online.
That is the horror show.
Who Ran It / What Broke
Who ran it: New York City's MyCity initiative under the Adams administration, with the business chatbot positioned as a front door to guidance from more than 2,000 NYC Business web pages (per the city's launch framing). The Markup reported the stack as powered by Microsoft's Azure AI services. A Microsoft spokesperson declined to comment to The Markup. City statements framed the tool as a pilot that had already helped "thousands" with timely answers while disclosing that it might produce incorrect, harmful, or biased content.
What broke: Grounding, evaluation, and kill-switch discipline—not the press release.
The Markup's testing catalogued concrete failures. Examples reported:
- Source-of-income / housing vouchers: Asked whether buildings must accept Section 8 or rental assistance, the bot repeatedly said no. In NYC, source-of-income discrimination by landlords is generally illegal (with a narrow exception for certain small owner-occupied buildings). Housing advocates called the misinformation fundamental.
- Tips: The bot said employers can take a cut of workers' tips—wrong under applicable tip protections (with limited tip-credit nuances the bot did not responsibly navigate).
- Cashless businesses: It said restaurants could go cash-free with no NYC rule requiring cash acceptance. NYC had passed a law requiring businesses to accept cash.
- Lockouts / rent rules: Housing experts testing after Markup's alert found the bot saying lockouts were legal and that there were "no restrictions" on residential rent—contradicting tenant protections and rent-stabilization reality.
- Other misses: Scheduling-notice rules, funeral-price disclosure (FTC Funeral Rule), and similar errors; problems also appeared in other languages.
Answers were inconsistent. Sometimes the bot contradicted itself across identical prompts. Sometimes it offered no verification links. A small disclaimer about possibly incorrect content sat beside branding that screamed "official."
Andrew Rigie of the NYC Hospitality Alliance told The Markup a business owner had already flagged inaccuracies. Rosalind Black of Legal Services NYC said that if the chatbot could not be accurate and responsible, it should be taken down.
April 2024: Acknowledge, Soften, Leave It Up
After the March 29 story, the city quietly strengthened on-page warnings: beta product; inaccurate or incomplete answers possible; double-check links; do not use responses as legal or professional advice.
Mayor Adams, at an April 2 press conference covered by AP and others, acknowledged the bot was "wrong in some areas" and said the team would fix it. He defended keeping a public pilot in the "real environment" to iron out kinks—lab perfection, he argued, was not how technology ships.
Ingrid Lewis-Martin, then Chief Advisor to the Mayor, compared early errors to early MapQuest: messy, then better.
Meanwhile The Markup and THE CITY re-tested. False answers continued. In one pointed exchange, reporters asked whether the bot could be used for professional business advice—the page said no; the bot said yes.
City officials had previously stressed trust. On a March panel recording obtained by THE CITY, Small Business Services Commissioner Kevin D. Kim said government could not afford an Air Canada–style chatbot liability moment: "We cannot be in a situation where we lose that kind of trust even one time." The Markup story landed weeks later.
Trust lost. Bot still serving.
Why "Just a Pilot" Is Not a Control Plane
Pilots are fine in sandboxes. This agent sat on an official government URL, trained—users were told—on official business pages, answering compliance questions for people who cannot afford wrong answers.
Three failures stacked:
1. Authority without verification. "Official NYC Business information" plus weak, easy-to-miss caveats.
2. No hard stop on high-risk topics. Housing discrimination, wage/tip rules, and consumer protections are not cute hallucination demos. They are liability and harm vectors.
3. No meaningful offline switch after public failure. Disclaimers got louder; the agent kept talking—including, in testing, contradicting its own disclaimer.
Air Canada's bereavement-fare chatbot episode already taught a private-sector lesson: organizations can be held to what their bots say. NYC's episode taught a public-sector rhyme: when the issuer of the rules runs the agent that misstates the rules, the blast radius is civic, not just contractual.
The Governance Gap
Every customer-facing AI agent that answers policy, legal, or compliance questions needs the same ops basics the MyCity rollout under-weighted:
- Grounding with citations that actually appear—and refusal when sources are missing.
- Eval suites for known-illegal advice before and after launch (voucher discrimination, tip theft, cashless bans, lockouts).
- Human escalation paths for regulated topics; agents should not freestyle statutory guidance.
- A real kill switch owned by someone who can pull the page when journalists—or your own red team—prove systematic failure.
- Monitoring for contradiction between UI disclaimers and model answers.
Soft-sell, hard truth: observability without the ability to stop the agent is a spectator sport. The city watched the incident in public and chose continuity over containment.
Takeaway
NYC's MyCity chatbot was supposed to reduce friction for small businesses. Instead it demonstrated how fast an "official" AI agent can manufacture illegal advice at scale—and how institutional pride in being first can delay the only control that mattered: turning it off until it was safe.
If your agent speaks with your brand's authority on rules people must follow, wrong answers are not a beta quirk. They are an operational incident. Treat them like one—detection, containment, correction—before the screenshots do it for you.