Editor’s note: I’m in the habit of bookmarking on LinkedIn, books, magazines, movies, newspapers, and records, things I think are insightful and interesting. What I’m not in the habit of doing is ever revisiting those insightful, interesting bits of commentary and doing anything with them that would benefit anyone other than myself. This weekly column is an effort to correct that.
It’s no secret that getting gen AI right in an enterprise context is hard. Why? Because transitioning from point solutions that drive individual productivity to a system-level solution that’s integrated into potentially brittle workflows is hard; because siloed data hides interdependencies that make the machine work; because organizational inertia is real; and because without business clarity and top-down change management, transformation in general doesn’t work. Nonetheless, the pressure to go do AI is real and businesses of all types are busy experimenting and running pilots. But moving from pilot to production is tricky. A July paper from MIT Media Lab’s Project NANDA put a number to it — 95% of enterprise gen AI projects fail as measured by return.
There’s a simple read here: 100% of ill-conceived experiments or pilots fail, so maybe 95% of these pilots are ill-conceived. But that’s a bit cynical and a bit reductive. And because this paper came out against the backdrop of more macro discussion around whether we’re currently in an AI bubble, it’s worth unpacking. The report authors tallied $30 billion to $40 billion in enterprise gen AI investment yielding “outcomes…so starkly divided across both buyers (enterprises, mid-market, SMBs) and builders (startups, vendors, consultancies) that we call it the Gen AI Divide…This divide does not seem to be driven by model quality or regulation, but seems to be determined by approach.”
So what’s the fundamental problem here? The MIT folks see it as learning. “Most gen AI systems do not retain feedback, adapt to context, or improve over time. A small group of vendors and buyers are achieving faster progress by addressing these limitations directly. Buyers who succeed demand process-specific customization and evaluate tools based on business outcomes rather than software benchmarks. They expect systems that integrate with existing processes and improve over time.”
This week I’ve talked to about a half dozen people about this report — and more broadly about AI — and a couple things stand out. Here’s one of them: rather than hand-wringing about the 95% failure rate, examine the 5% and learn from what they’ve gotten right. So let’s do that. Spoiler alert: it has to do with understanding your business — its core assets and values as well as its limitations — and assigning measurable return when asking why a problem lends itself to a gen AI solution before burning money on figuring out how to do it.
Consider Dell Technologies COO Jeff Clarke who laid out the tech giant’s approach to gen AI during a keynote earlier this year at the company’s flagship event in Las Vegas. “We were pretty horrified when we started,” Clarke said. The company had more than 900 “AI projects” within the company, and was grappling with suboptimal data governance and a general lack of business clarity and purpose.
Clarke said step one was to lay out the underlying structure to guide Dell’s internal AI ambitions. That includes defining an AI data architecture and building an enterprise data mesh to connect relevant data. “Processes had to be simplified, standardized and automated. It became very clear to us that if you apply AI to shitty process, you get a shitty answer faster.”
How to get gen AI right
Next, Clarke explained, the AI strategy and attendant use cases had to align with the company’s core interests. And, finally, there had to be committed, meaningful ROI. “Unless you were willing to sign up for real dollars, real efficiency and productivity, we were not going to fund it.” For more from Clarke on how exactly Dell is deriving value from gen AI, read this research note. Suffice to say, he left the audience with five thoughts:
- “It’s really time to get busy…The threat is existential…If you haven’t started, you’re behind.”
- “There is no one-size-fits-all approach.”
- “Many of you have the power, cooling and space in your existing data centers already.”
- “You don’t need the latest models, you don’t need the latest GPUs, to get started.”
- “There’s a compelling ROI out there for the right use cases inside your organizations.”
What Clarke lays bare, and what I’ve heard from other people, seems obvious; in one conversation I believe I described it as “the kind of things you’d learn in the first couple months of an MBA program.” Have a goal, understand that technological transformation and organizational transformation are a joined pair, remember you can’t improve what you can’t measure, etc…
So what is it about the lure of AI that makes business leaders of all stripes abandon the basics and throw first principles thinking out the window? It’s, as the report authors made clear: “The GenAI Divide is not permanent, but crossing it requires fundamentally different choices about technology, partnerships, and organizational design.” But remember that although pilot purgatory is real, this dramatic failure rate isn’t inescapable. Don’t forget the basics and study what the 5% are getting right.
