New York City's AI chatbot faces backlash for dispensing questionable advice to businesses, risking legal violations. Despite concerns, the city, defended by Mayor Eric Adams, opts to maintain the chatbot on its website.
NEW YORK, NEW YORK - MARCH 01: New York City Mayor Eric Adams attends a memorial for the 30th anniversary of the killing of teenager Ari Halberstam on the Brooklyn Bridge on March 01, 2024, in New York City. (Photo: Spencer Platt/Getty Images)
Controversy Surrounding AI Chatbot
New York City's AI chatbot has stirred controversy by providing questionable advice to businesses, including suggestions that could lead to legal violations. Despite criticism and concerns about the accuracy of its responses, the city has decided to keep the chatbot operational on its website.
Mayor Eric Adams defended this choice, acknowledging the inaccuracies in some areas while maintaining the tool's availability. Launched as a resource hub for business owners in October, the chatbot offers algorithmically generated responses to queries navigating the city's bureaucratic processes.
While accompanied by disclaimers warning of potential inaccuracies and affirming that its responses do not constitute legal advice, the Associated Press reported that the chatbot's utility remains debatable.
The AI chatbot provided erroneous responses to inquiries, inaccurately suggesting that employers have the right to terminate employees who report sexual harassment, conceal pregnancies, or refuse to alter their hairstyle.
Furthermore, it contradicted the city's waste management initiatives by claiming that businesses can dispose of their trash in black garbage bags and are exempt from composting requirements.
In some instances, the chatbot's responses were ludicrous. For example, when asked about serving cheese that a rodent had gnawed on, it replied affirmatively, stating that it was permissible to serve the cheese to customers despite the rodent bites.
However, it did emphasize the importance of evaluating the extent of damage caused by the rodent and informing customers about the situation.
A representative from Microsoft, the provider of the bot's underlying technology through its Azure AI services, confirmed ongoing collaboration with city officials to enhance the service's accuracy and ensure alignment with official city documentation.
Mayor Adams, a Democrat, addressed concerns during a press conference on Tuesday. He portrayed the bot's issues as typical hurdles encountered in refining new technology. Stoyanovich criticized this response as "risky and negligent."
Navigating the Pitfalls of Large Language Models
Concerns among scientists persist regarding the limitations of large language models like ChatGPT, which draw from vast internet text collections and often produce inaccurate or nonsensical responses.
Despite the widespread attention garnered by successful chatbots like ChatGPT, private companies have introduced their versions, leading to varying outcomes.
For instance, Air Canada recently faced a court order to refund a customer due to misinformation provided by its chatbot regarding the airline's refund policy. Similarly, TurboTax and H&R Block have encountered backlash for chatbots offering erroneous tax-preparation guidance.
Suresh Venkatasubramanian, director of the Center for Technological Responsibility, Reimagination, and Redesign at Brown University, says New York City's chatbot debacle offers valuable lessons for municipalities.
Venkatasubramanian stressed the importance of cities contemplating the rationale behind implementing chatbots and identifying the specific issues they aim to address.
He emphasized that if chatbots substitute human interaction, they may compromise accountability without providing significant benefits in return.
