In 1987, Nobel Prize-winning economist Robert Solow quipped that we could see the computer age everywhere but in productivity statistics. In other words, claims not only about what computerization could do for work but also about what it was actually doing were everywhere, but there wasn’t any evidence that it was doing what was claimed.
Does this sound familiar? If not, think back before ChatGPT to driverless trucks and the claim in the late 2010s that soon—by 2019 many consultant reports claimed—human drivers would be obsolete. Instead, we had a big shortfall in the number of truck drivers.
Now, almost hourly, we hear new claims about how AI—more specifically large language models (LLMs) like ChatGPT and now the broader category of Generative AI—are already transforming everything. When I look for actual evidence, all I can find are accounts saying that it is being used here or there to replace workers. In just a paragraph, I promise to report an actual experience of use.
But first, why do we hear these overblown claims? The main reason is that they come mostly from people who build the tools, and they are reporting what they think the tools could do. That is a claim that says, in theory here is what they are capable of doing. The tools are much cooler than what they had been using before. But they are not thinking about what will happen in practice. The many reasons why driverless trucks did not take off, for example, begin with the fact that they require huge upfront investments, changes in government regulations, a way to perform tasks that IT could not do (e.g., pumping gas), figuring out insurance liability, and so forth. The second reason for overblown claims is that consultant and industry reports are not designed to be right—they are designed to get attention. It is easier to do that with extreme claims. Once the claims become obviously inaccurate, they take them down off their websites, and no one checks to point out how wrong they were.
Onto the example: We followed the effort to automate a simple white-collar task common in business, which was the initial step in sorting mail. In this case, it was processing insurance claims: Opening the envelope, figuring out what is in it, sorting the document into the appropriate category, and transferring the information on the document into the company’s own system. Roughly 44 employees performed this task, but quality was poor, turnover was high, and the problems of managing a sharp increase in claims/documents seemed insurmountable.
They found one of the large language models could do this pretty well, just as the AI advocates had said. What they also found was that it cost more to do it with AI than with their current employees. In fact, it was a lot more. In short, while possible, not worth doing.
They eventually came up with a solution that, in good operations research fashion, used the cheapest AI tools to do part of the work and then more expensive ones to handle the harder tasks. It took an additional six AI-related new staff the better part of a year to get it to work: $500,000 in initial consulting and AI assistance, $200,000 every month going toward AI costs to use it. We thought these tools were free, right?
Further, the idea that we can simply drop an AI tool into our organization is a huge and persistent myth. These tools have to be adapted by us to each different task. That requires lots of data—millions of observations. For the insurance claims example, the company would have to show: this is what a claim form looks like, this is what an appeal form looks like, etc. That learning has to be updated every time a new claim form is used, every time anything changes. Without that data, it is not possible to use these tools.
The company also discovered that they still needed employees to solve common but quirky problems: missing data, handwriting that AI can’t read, and so forth. So headcount remained the same, but the company was able to take on three times as much work, and quality improved. The full story is here. Sneak it into the pile of documents for your CEO and board members who are pushing to introduce AI as a way to cut headcount.
What do we conclude about AI’s current ability to transform work? A big conclusion here is that it takes a huge amount of resources to do this. It is not possible to drop an AI tool into your workplace and expect it to do anything. The upfront cost is also huge. A second conclusion, then, is that it might well not pay off unless the work you are trying to cut is itself expensive. Because it is so costly, it is unlikely to pay off in any sensible ROI calculation if the workers you are trying to replace are low paid.
The third conclusion defines the difference between people who are in tech and people who are running organizations. The goal of running organizations is to have efficient, low-cost operations. It isn’t to automate or use cool technology per se. Yes, these tools may get cheaper, and it may get easier to “train” them. But if you are running an organization, your response should be, “call me when that happens.”
Peter Cappelli
George W. Taylor Professor of Management
Director – Center for Human Resources for the The Wharton School
