The Gen AI Divide report from the MIT NANDA initiative reveals a major lack of ROI on gen AI projects
A recent report from the Massachusetts Institute of Technology (MIT)’s Networked Agents and Decentralized AI (NANDA) put forward some numbers that threw cold water on red-hot AI hype: Despite an estimated $30-$40 billion of enterprise investments into generative AI, only 5% of organizations are seeing a return on their money. The gap between that five percent and the 95% of organizations who are getting zero return is so stark that the MIT researchers dubbed it the “Gen AI Divide.”
While those few successful integrated AI pilots “are extracting millions in value, while the vast majority remain stuck with no measurable [profit and loss] impact. This divide does not seem to be driven by model quality or regulation, but seems to be determined by approach,” the researchers concluded.
The report was based on a combination of interviews, surveys and analysis of 300 public implementations of generative AI. Adnan Masood, chief AI architect at UST, wrote on Medium that the report was “possibly the most candid snapshot I’ve seen of where generative AI is actually moving the needle — and where it isn’t.”
Here are seven major conclusions from the Gen AI Divide report:
–AI adoption is high, but disruption is low. The MIT researchers categorized this in terms of structural change in sectors adopting AI — and seven out of nine sectors showed low disruption, meaning that there aren’t new market leaders displacing incumbents, major changes in customer behavior or disrupted business models. The two exceptions to this were in the tech sector, and media.
One mid-market manufacturing COO was quoted in the Gen AI Divide report as saying: “The hype on LinkedIn says everything has changed, but in our operations, nothing fundamental has shifted. We’re processing some contracts faster, but that’s all that has changed.”
–Big enterprises are leading the AI charge — without much to show for it so far. The report says that companies with more than $100 million in annual revenue have the most generative AI pilots and the most staff dedicated to AI-related efforts. But they also have the lowest rates of converting pilots into scaled projects; mid-market companies moved faster.
–Few custom AI tools reach deployment. Adoption isn’t necessarily the problem; people are using gen AI. In fact, the report concluded that “generic tools like ChatGPT are widely used,” but it also found that “custom solutions stall due to integration complexity and lack of fit with existing workflows.” In fact, the gen AI divide report found that only 5% of custom AI tools for enterprises reach the production stage. Mostly, that has resulted in internal AI chatbots that are easy for companies to try, but that they “fail in critical workflows due to lack of memory and customization.”
–“Shadow” AI use dominates — and has pros and cons. “Shadow” AI use is when employees turn to public gen AI tools like ChatGPT or Copilot to help them with daily tasks, even if there are enterprise-grade AI alternatives available.
“Our research uncovered a thriving ‘shadow AI economy’ where employees use personal ChatGPT accounts, Claude subscriptions, and other consumer tools to automate significant portions of their jobs, often without IT knowledge or approval. The scale is remarkable,” the MIT researchers wrote in the Gen AI Divide report. “While only 40% of companies say they purchased an official LLM subscription, workers from over 90% of the companies we surveyed reported regular use of personal AI tools for work tasks. In fact, almost every single person used an LLM in some form for their work.”
Users described the public tools as being flexible and having immediate usefulness — they liked the outputs from ChatGPT, for example, while describing enterprise AI solutions as “brittle, overengineered, or misaligned with actual workflows.” But at the same time, the very users who used public AI chatbots didn’t trust them for mission-critical work because the bot didn’t remember, and couldn’t learn, important things in order to produce accurate, complex work.
One lawyer who used ChatGPT told the researchers: “It’s excellent for brainstorming and first drafts, but it doesn’t retain knowledge of client preferences or learn from previous edits. It repeats the same mistakes and requires extensive context input for each session. For high-stakes work, I need a system that accumulates knowledge and improves over time.”
-Integration is key, but so is data separation and protection. The report highlighted another contradiction of AI implementation: People want AI systems that remember, learn and improve; that work with their existing data systems and tools; but that also still keep sensitive data safe. Anonymous quotes from survey participants illustrated this tension, with one participant saying, “If it doesn’t plug into Salesforce or our internal systems, no one’s going to use it,” while another said bluntly, “I can’t risk client data mixing with someone else’s model, even if the vendor says it’s fine.”
–Humans are preferred over AI for complex, long-term tasks. It may not be a surprise that when it comes to the potential uses for AI, humans already preferred AI for tasks like drafting emails or “basic analysis,” but chose other humans by a nine-to-one margin for anything “anything complex or long-term.”
“The dividing line isn’t intelligence,” the researchers claimed. “It’s memory, adaptability, and learning capability — the exact characteristics that separate the two sides of the GenAI Divide.”
The solution? Agentic AI (which NANDA happens to focus on) that can “maintain persistent memory, learn from interactions, and can autonomously orchestrate complex workflows.” Successful AI providers are focusing on “narrow but high-value use cases” where “domain fluency and workflow integration matter more than flashy UX.”
–Workforce impacts aren’t material yet, for most sectors. For businesses which did see gains from AI, those improvements are so far coming “without material workforce reduction,” the report found (with some caveats). The most valuable changes were in back-office automation, and AI tools helped improve personal productivity and sped up work, but didn’t spur structural change in human teams. “ROI emerged from reduced external spend, eliminating [business process outsourcing] contracts, cutting agency fees, and replacing expensive consultants with AI-powered internal capabilities,” the paper found.
Successful AI implementations meant measurable decreases in external costs “while slightly decreasing internal headcount.” This was true across verticals like healthcare, energy and advanced industries. However, in tech and media, hiring was expected to drop within 24 months.
The researchers also concluded, based on conversations with procurement officers, that the next 18 months will be crucial in terms of enterprises solidifying their AI vendor relationships and integration, to the point that they will be “nearly impossible to unwind.”
They concluded: “The next wave of adoption will be won not by the flashiest models, but by the systems that learn and remember and/or by systems that are custom built for a specific process.”
A pdf of the Gen AI Divide report is available here.
