This is a crosspost from Jonathan Dursi, R&D computing at scale. See the original post here.
As research computing and data becomes more complex and diverse, we need more professional services firms and fewer utilties
(Note: This post is adapted from #127 of the Research Computing Teams Newsletter)
I get to talk with a lot of research computing and data teams - software, data, and systems. Sometimes in these conversations it’s pretty clear that some teams, or the team and their funder, or a team and I, are talking a bit past each other. And that’s usually because they or we are (currently) operating with very different mental models of how they operate.
Some research computing and data teams are operating as Utilities, and see the world through that lens; a growing number are operating as Professional Services Firms. Others are moving from one to the other, and are at different places along that very abrupt transition. Some kinds of groups (like bioinformatics cores) are much more likely to already be operating in service mode, while others (like research compute infrastructure teams) are more likely to still think of themselves as utilities. It varies from place to place, though, depending on local conditions. But they’re very different models!
Utilities, like power companies or garbage collection or municipal potable water, were really the only sensible role models for the first decades of research computing and data teams. Those teams were entirely about operating large equipment purchased from vendors. Costs were mostly a big capital expense. Everyone who needed the utility needed the same thing - undifferentiated flops and bytes, or 60Hz 120VAC. Because everyone needed the same thing, economies of scale led to natural monopolies; the most reasonable provision model was for the local jurisdiction/institution to own or control a single operator. Differentiation or strategy, or gaining new customers, weren’t meaningful discussion topics. The only thing that really makes a difference is scale, which leads to mergers. Innovation happens slowly, top-down, at the industry-wide scale and usually from the vendors (“hey, did you hear about those new gas compressors Dyneco announced?”), and diffuses outwards. Employees take pride in and the organization values operational skill and things ticking along smoothly. Customers value reliability. The only thing that matters for any individual operator is to operate effectively and to provide the standard service with the right amount of cost: high enough to absorb the available subsidy, low enough to not go broke. If a customer needs something other than what the utility provides, rather than that being a market opportunity, it’s either an inconvenience or an irrelevance. The power company or the water utility or the old phone monopoly just doesn’t serve that need.
Professional Service Firms — say engineering firms, or architects, or consultancies — are very different beasts. They might very well have significant capital investment in specialized equipment, but their main selling point and their biggest cost is expertise. Competing for and retaining that expertise, and developing that expertise in house and amongst their clients, are principal concerns. As part of a “full-service” offering they they likely have some fairly standard services they offer at the low end, where operating cost and efficiency is vital. But what the organization values, and the employees enjoy, is at the high-touch end — getting deeply involved with the client work, and being as much a collaborator or partner or “trusted advisor” as a service provider. Different clients want very different things, and that high-touch high-expertise work is specialized and labour intensive, so the firms themselves need a clear focus; they can’t meet all needs. Clients can go elsewhere, so there is redundancy and competition, but less than you’d think at a distance. In civil engineering a geotechnical firm is complementary, not competing, with one that specializes in water resource engineering.
As in the rest of our lives, in research computing we need to have utilities. As research data management matures, institutional or regional data depositories become mature and “enterprise” enough to become utilities, likely run by IT or the Library. Teaching or CI/CD or MLOps resources for data science or software development are likely best served by this model. The closer the operations are to standard, something that can be run by IT, the more likely it is to be a utility. But one has to be careful. Utilies are commodoties: they tend to get merged together wherever feasible, since scale matters and it’s all undifferentiated commodity provision.
As research computing becomes broader and faster changing and more diverse, we need more and more professional services firms, too; nimble groups specialized to particular needs and ready to adapt as those needs change. As even infrastructure is becoming less one-size-fits-all, and methods for making use of computing and data for diverse fields grow more complex and expertise intensive, the preconditions for the utility model are met in fewer situations than used to be.
A lot of research computing teams are interested in providing something more like professional services, but were created in the Utility model, and are stuck there by their funders. The institutional or external funders still have this very specific (and to their mind time tested and successful) operating model in their plans. Utilities are funded very differently than professional services firms. At utility scale, it doesn’t make sense to outsource things, or develop non-standard services (who wants non-standard power coming into their house!) Funders requirements on eligible expenses may focus almost entirely on the capital spend, and not on operating funding that’s needed to make effective use of the capital, or to be more agile in how services are delivered.
Even those teams who aren’t being held back by funders and who want to make the switch to professional services from their original utility model find it a hard transition. There’s no obvious, incremental path to go from providing a standard, stable commodity to changing, specialized, bundles of expertise. Utilities operate very differently from professional services firms. They value different things. The models for staff growth are different. So they have to be managed quiet differently, and there’s no clear path internally from A to B.
Besides funding, and internal considerations, utilities and professional services firms are also percieved and valued by their clients very differently. Utilities’ existing customers don’t want change, and new customers aren’t yet interested in getting advanced app software development suggestions from what they perceive to still be the mobile telephony provider.
But research computing and data is changing, increasingly quickly, and the utility approach only meets a piece of these growing needs. Navigating the transition isn’t going to be easy, for RCD teams, leaders, or funders; but expressing it clearly and talking about it more will maybe mean we’re not talking past each other so often.