What goes wrong
The Markup reported on 29 March 2024 that New York City's official business chatbot told employers they could take a cut of workers' tips, that no regulations required notice of schedule changes, and that restaurants could refuse cash; the article says each was wrong. The bot was built to help and still invented rules, and an assistant for staff questions can do the same about pay or leave.
Make every answer quotable
A safe design only answers from the handbook, shows the section, and says plainly when a question is not covered. If it cannot quote, it should not answer. Test it by checking that the section quoted really contains the answer, not by asking whether the answer sounds right.
- Quote a section or say not covered.
- Name the person to ask.
- Say it is automated before the first answer.
Write the refusal list first
List the kinds of question that must go to a person: anything about one individual, pay, discipline, health, grievances and legal questions. The data protection regulator's guidance on employment records has separate sections on how much personal information to hold and on sickness, injury and occupational health records, which shows why that pile is handled with care. Test those questions first, and treat an answer to any of them as a failed test.
Keep the handbook current
An assistant repeats whatever the handbook says. If the handbook contradicts itself or names a rule that has gone, the assistant will quote the wrong one with confidence. Check the handbook for contradictions and expired dates before the assistant is built, and again after every change.
What this guide does not cover
It is not HR or legal advice and does not say what your policies should be. The paid job builds one assistant limited to your handbook and tests it on 100 questions you write, 40 of which are for it to refuse or hand over and 40 of which stay with you until the test run. It never answers about one person.
Sources and limits
- The Markup: NYC's AI chatbot tells businesses to break the law Checked 2026-10-12.
- Published 29 March 2024: the city's MyCity chatbot told employers they could take a cut of workers' tips, said no regulations required notice of schedule changes and said restaurants could refuse cash; the article says each of these answers was wrong.
- ICO: employment practices, keeping employment records Checked 2026-10-12.
- The guidance has sections on how much personal information can be held, how long workers' information can be kept, and how to handle sickness, injury and occupational health records.
- A banner says that, because of the Data (Use and Access) Act, the guidance is under review and may change.