Interestingly, the study also revealed that the customer-service workers with the least amount of skill were the ones who experienced the most significant improvements over the course of a year.
A new study by researchers at Stanford University and the Massachusetts Institute of Technology has found that customer service workers at a Fortune 500 software firm who used generative artificial intelligence tools were 14% more productive on average than those who did not use the tools. The least-skilled workers benefited the most from using the AI tools, according to the study, which tracked the performance of over 5,000 customer support agents over the course of a year. The agents were divided into groups, with some given access to the AI tools while others were not.
The research is the first to measure the impact of generative AI tools on productivity outside of the laboratory. Previous studies had only tested the tools in isolated writing tasks in small-scale laboratory settings or benchmarked their capabilities against tasks in fields such as law and medicine. According to Erik Brynjolfsson, the director of the Digital Economy Lab at the Stanford Institute for Human-Centered AI and one of the study’s co-authors, the results of some of these earlier experiments showed the potential of large-language models in the workplace, but until they are tested in the real world, their impact remains mostly speculative.
Brynjolfsson, alongside MIT researchers Danielle Li and Lindsey Raymond, tracked the speed and success of the customer support agents in solving clients’ problems. The AI tools were trained on a large set of successful customer service conversations. The name of the company, which specializes in enterprise software for small and medium-sized US businesses, was not disclosed in the report. The study highlights the potential for generative AI tools to increase productivity in customer service, with the least-skilled workers benefiting the most.
“Having people use it for over a year in this company, you get a much better sense of how that translates into real-world productivity,” Brynjolfsson, one of the study’s co-authors, said in an interview. “As far as I know, this is the first time it’s been done in a real-world setting.”
According to Erik Brynjolfsson, one of the study’s co-authors, the research marks the first time the impact of generative AI tools on work has been measured outside of the lab. The study is significant because until now, the impact of AI tools on productivity has largely been speculative. The researchers tracked the performance of more than 5,000 customer support agents based primarily in the Philippines, across key metrics such as how quickly and successfully workers were able to solve clients’ problems.
One of the study’s key findings was that novice workers benefitted the most from the AI tools. With the assistance of AI, the firm’s least-skilled workers were able to get their work done 35% faster. New workers’ performance also improved much more rapidly with the assistance of AI than without. The research suggests that the boost in low-skilled workers’ productivity and performance may come, in part, from the way that AI tools can absorb the tacit knowledge that makes the firm’s top performers excel and then disseminate that knowledge to less-skilled or experienced workers through AI-generated suggested responses.
Overall, the study provides important insights into how generative AI tools can improve productivity in the workplace. The researchers say that the results of the study suggest that AI can help to level the playing field for workers by providing them with the tools they need to excel in their jobs. They note, however, that there are still many questions that need to be answered about the impact of AI on work, and that more research is needed to fully understand the potential benefits and risks of this technology.
The introduction of artificial intelligence into the workplace may not always lead to better outcomes for the most skilled workers. According to the study, highly skilled workers, such as top customer service agents, saw little to no improvement in their work as they were already performing at the same level as the AI recommendations. In some cases, the prompts from AI may have even acted as a distraction.
However, the study also suggests that if AI ultimately narrows the gap between low- and high-skilled workers, companies may need to rethink their compensation strategies. Currently, high-skilled workers may be inadvertently providing the data that feeds into AI without being adequately compensated for their efforts. This raises weighty policy questions about how companies should recognize the value of their star employees’ data and how they should be compensated for it.
Forward-thinking companies should recognize the tacit knowledge and skills of their star employees as these are likely to form the basis of the AI tools that power the rest of the organization. The study suggests that companies will need to adopt incentive and reward systems that acknowledge the value of these workers, even if their performance with any given customer is not demonstrably better than their peers. By recognizing the expertise of their top performers, companies can ensure that the skill is amplified and multiplied throughout the organization, changing the whole company.
It’s worth noting that this study is just the beginning of the exploration of generative AI, and much more research is needed before any concrete conclusions can be drawn. However, as AI continues to develop and integrate into workplaces, it is clear that companies will need to adapt their compensation strategies to ensure that their most skilled workers are recognized and adequately compensated for their contributions.