When generalists are better than specialists and viceversa
What’s the best way to boost creativity on your team? There’s really no simple answer. Even the research is split on the best approach to take.
One view is that the key to creative breakthroughs is being able to combine or leverage different areas of expertise. After all, every innovation somehow recombines or reimagines things that already exist.
Many studies have found that the best ideas emerge from combining insights from fields that don’t seem connected. For example, Charles Babbage’s invention of computational machines powered by punch cards, the foundation of modern computers, was inspired by Babbage’s knowledge of the silk-weaving industry, which used cards with holes to create patterns in the silk fabric. Similarly, Henry Ford’s revolutionary idea of the car manufacturing assembly line was inspired by Singer sawing machines and meatpacking plants.
Based on this thinking, you might try to make your team more creative by encouraging employees to explore new fields or by hiring more generalists, people who have a variety of experience and expertise. They can connect dots where others don’t see a link.
But other studies have found that there are costs to generalizing. As the saying goes, jacks of all trades are masters of none. This line of research argues that specialists, with their deeper understanding of subject matter, can better spot and seize on emerging opportunities. For example, researchers Sarah Kaplan and Keyvan Vakili found that recombining ideas from one domain of specialization, as opposed to multiple domains, led to more novel innovations in the area of nanotubes. Specialists may also have an easier time collaborating because it’s clearer how the work should be split up.
These points would suggest you’re better off hiring employees who have very deep expertise in an important area or encouraging your employees to become true specialists in whatever they do.
There’s considerable evidence supporting both sides, so we reason that both are probably right. But there must be certain circumstances under which generalists shine and others under which specialists do. In a forthcoming paper in Administrative Science Quarterly, we studied what these are.
We theorized that the benefits of being a generalist are strongest in fields with a slower pace of change. In these fields (think oil and gas, mining), it might be harder for specialists to come up with new ideas and identify new opportunities, while generalists may be able to find inspiration from other areas. We also theorized that the situation flips for fields with a faster pace of change. In this case (think of quickly evolving fields such as quantum computers and gene editing), generalists may struggle to stay up to date, while specialists can more easily make sense of new technical developments and opportunities as they arise.
To test this, we needed to study an area where some fields experienced a sudden shift of pace while other fields remained stable. This is exactly what happened in theoretical mathematics after the collapse of the Soviet Union. In the 1980s Soviet mathematicians were largely ahead of their Western colleagues in some fields of theoretical mathematics (integral equations, for example) but not in others (commutative rings and algebras). As the Soviet Union collapsed, a large store of scientific advances suddenly became available to Western mathematicians. This increased the pace of change in fields where the Soviet Union was ahead of the West.
Theoretical mathematics is also a field that allowed us to distinguish between specialists and generalists. For example, the Italian Fields Medal winner Enrico Bombieri is known for bringing together insights from widely differing areas of mathematics. In contrast, French Fields Medal winner Laurent Schwartz spent most of his career working on distributions.
We compared mathematicians working in fields that experienced rapid change (mainly subfields of mathematical analysis, such as integral equations, partial differential equations, and Fourier analysis) with those working in less affected fields (mainly subfields of algebra and geometry, such as rings and algebras and combinatorics). We then tracked the performance of over 4,000 mathematicians across the period 1980–2000 — 10 years before and 10 years after the collapse of the Soviet Union — using an extensive data set of publication and citation data provided by the American Mathematical Society.
We quantified the specialization level of mathematicians based on their academic publications. We defined specialists as researchers who published in only one domain of theoretical mathematics and generalists as researchers who published in several domains. We measured changes in the creative output of generalists and specialists, in faster-paced and slower-paced fields, by the number of academic papers they published. Since publications vary in quality, we also adjusted for the number of citations publications later received.
Of course, theoretical mathematics is a unique setting, but it allowed us to precisely measure how an increase in the pace of change impacts the creative performance of specialists and generalists. The results from our analyses confirm our theory.
The Soviet collapse altered the pace of change differently for different fields of theoretical math. In fields that didn’t change much (slower-paced fields), specialists used to be slightly less productive than generalists, but this gap widened significantly after the collapse. Our regression estimates a 22% decrease in specialists’ citation-weighted publications per year, relative to generalists in the same slow-paced fields. (This number refers to average change in the productivity gap between the decade preceding and the decade following the Soviet collapse, and is obtained after controlling for changes in publication trends over time and for various individual characteristics such as age, gender, and education.)
In fields that were most affected (faster-paced fields), the opposite happened. Whereas before the collapse specialists were slightly more productive than generalists, after the collapse this gap widened, because specialists’ productivity increased while generalists’ decreased. More specifically, our regression estimates that specialists ended up publishing 83% more citation-weighted papers than generalists in the 10 years after the collapse, relative to the 10 years before.
Not only did generalists in faster-paced areas perform worse than specialists, but they also performed worse than generalists in slower-paced areas. Before the collapse, the productivity of generalists from the two areas was very similar. After the collapse, generalists in faster-paced areas decreased their productivity and published 37% fewer citation-weighted papers than generalists in slower-paced areas.
In other words, generalists appear to be relatively successful as long as the pace of change is not too rapid, but their productivity decreases when the pace of change increases. At the same time, specialists appear to perform better when the pace of change accelerates.
Our study suggests that two types of capabilities can improve creative performance. One is the ability to connect ideas across subject areas. This isn’t easy, as ideas that are successful in one area might not be successful in another, but it can pay off, especially if the field you’re working in is relatively stagnant. Another is the ability to efficiently build on progress in your field and seize opportunities emerging at the frontier, which may be particularly valuable when things are progressing quickly.
There is no one-size-fit-all strategy to promote creativity. But our study suggests that leaders should assess how many specialists and generalists they have on their teams. If the pace of change is slow, teams will likely benefit from employing generalists, who can challenge the industry’s taken-for-granted assumptions and bring in new ideas. If the pace of change is rapid, teams will benefit from specialists, who are more likely to help the team innovate.