Stop asking engineers how many machines they need

Estimated reading time: 8 minutes

Key takeaways:

• Reasonable buffers, stacked across hundreds of teams, inflate total demand massively and untraceably.
• Ask product teams about growth, not machine counts, then translate that automatically into hardware.
• Tying forecasts to durable products and not projects cuts the process from months to about a month.

---

Every large technology company eventually faces the same challenge. When the company is still small, with maybe a few hundred engineers and a handful of services, capacity planning is a simple exercise. A few teams make an educated guess, someone plugs it into a spreadsheet, and leadership approves the budget. 

At hyperscale, with thousands of services and products, hundreds of teams, and billions of daily requests, this approach doesn’t scale. The exercise that used to take weeks now takes three to four months. Teams are still guessing. Leadership is confused and finance is frustrated.

Even after spending so much time and resources, no one is able to answer “how much of this projected hardware growth can we actually tie back to business growth?” The cost of getting it wrong could run into millions of dollars.

Why forecasting breaks at scale

The problem was not that teams did not understand their future growth. Product teams had a strong understanding of their upcoming launches, expected traffic, queries per second (QPS), data growth, and storage needs.

However, they did not know how that demand translated into machines. Infrastructure teams understood hardware, but they did not have the same visibility into every product’s roadmap.

Without a standard way to bridge this gap, every team created its own assumptions and added buffers for uncertainty. Individually, those buffers looked reasonable, But when forecasts from hundreds of teams were added together, they inflated the total demand dramatically.

The process also carried old hardware decisions into every new planning cycle. Teams forecasted the same hardware Stock Keeping Units (SKUs) because those were familiar, not necessarily because they were still the best fit. Over time, this created unnecessary hardware variations and prevented us from using more standardized, cost-effective purchases.

Finally, the forecasts were difficult to connect with existing capacity, actual utilization, historical accuracy, or the products driving the growth. Finance received a large number but could not clearly understand where it came from. Leadership asked teams to reduce it, teams defended their buffers, and the entire exercise turned into a multi-month negotiation.

The solution was not yet another incremental improvement to the capacity planning process but rather to build forecasting as a platform capability. Centralizing the data came first. The other capabilities evolved, often in parallel, over several planning cycles and multiple years.

5 capabilities for forecasting infrastructure demand

1. Centralize the data before you add intelligence

The first step is shockingly simple yet effective: you just need to put everything in one place. Replace your spreadsheets with a single, centralized forecasting registry that sits on top of a relational database. The first version of our centralized system was nothing but a glorified spreadsheet.

It might sound trivial but centralizing data delivered immediate benefits for us by solving many fundamental problems. The forecast became queryable and trackable, and for the first time, it could be traced all the way from the platform back to the submitter. Data aggregation became simple and instant. Questions like “how much hardware is this org requesting compared to last year?” went from weeks of manual data collection to minutes. This centralized platform worked as a foundation for everything that came after. 

Do not try to solve everything in the first iteration. Centralize first. Get adoption. Add sophistication later. The temptation is to launch the smartest version in the first attempt, but asking people to change where they work and how they think in the same cycle is how these programs die.

2. Ask for workload intent, not hardware

Centralization exposed a deeper issue: the product teams that submitted forecasts usually had the strongest understanding of business growth, product launches, and technology demand, but they were not infrastructure experts. They did not manage servers, storage, or networking on a day-to-day basis. Infrastructure teams, on the other hand, operated in a language geared around hardware SKUs, capacity units, and platform constraints. 

We bridged this gap by building system specific calculators that automatically converted business growth into hardware units. Instead of asking teams “how many machines do you need?” the system asked questions familiar to product teams, such as expected traffic growth, data ingestion, retention, replication, job volume, reads, writes, or storage growth. When you ask questions in their language, the conversation shifts entirely. A product team can confidently project 10% traffic growth based on specific initiatives. 

Workload-aware calculators then translated those inputs into compute, memory, storage, and network demand. A stateless service could take inputs in terms of QPS growth and project corresponding hardware growth while a streaming platform could use ingestion rate, retention period, and replication factor to forecast a completely different hardware profile. Each workload type used a tailored capacity model based on its CPU, memory, storage, and traffic profile.

3. Dependency-aware forecasting

Early versions treated each forecast as an independent entity, making it extremely hard to correlate and deduplicate demand across related services. If an entry-point service grew, the downstream services and data platforms behind it would likewise receive additional work.

We implemented dependency-aware forecasting so a single business projection could cascade through the service graph. We started with stateless services, where traffic relationships were easier to observe, and expanded to stateful platforms as the data and models matured. 

For stateful systems, the problem was harder and took several cycles to solve. The relationship between upstream traffic and stateful resource consumption was indirect, delayed, and often non-linear. A request might pass through ten services before writing to a database through an asynchronous batch job.

Solving this required combining deployment metadata, data flow lineage, queue consumer mappings, and domain-specific attribution models. It took multiple years, but the result was that a single traffic projection could automatically generate forecasts across the full stack.

4. Attach forecasts to products, not projects

Early forecasts were associated with projects because that matched the annual planning process. But projects are temporary. They start, finish, get renamed, and get duplicated. They were a weak foundation for long-term accountability.

Hence, we shifted to products, which persist, have owners and lifecycles, and carry business accountability. By integrating with the organization’s centralized product registry, every hardware request now ties to a specific product. A single team owning ten products can itemize capacity across all of them in one workflow: Product A needs more storage, Product B needs more compute, Product C is being deprecated so capacity can be returned.

Leaders could see how fast a product’s infrastructure cost was growing and compare that growth with its business trajectory. Teams could still group several product requests under one initiative, but the lineage remained connected to a durable owner and lifecycle. 

5. Improve efficiency

Once teams trusted the forecasting system, forecast submissions surged. The bottleneck shifted from collecting accurate forecasts to ensuring they were cost-effective. So we connected the platform to the existing fleet data: current compute allocation, historical utilization, and each team’s past forecast accuracy.

Now, when a team asks for capacity, the system already knows its current allocation, actual usage, and how much capacity sat idle the previous year. Idle capacity was automatically deducted from the team’s future forecast.

We also moved from machine counts toward common capacity units such as CPU cores, memory, and storage units. As the demand model was no longer tied to a specific hardware generation, conversion logic could evolve underneath it.

At the infrastructure layer, we can serve those units from whatever hardware generation makes sense, shift capacity between teams as workloads scale up and down, and let one team borrow a neighbor’s idle capacity for a quarter and hand it back. Those are levers we did not have when every request was for a specific hardware SKU, and they allowed us to run forecasting continuously rather than through a rigid annual cycle.

At maturity, the forecasting system operates continuously as a control plane, not as an annual exercise. Forecasts are living artifacts. Organic growth models update as data arrives. Conversion logic evolves as hardware platforms change. Infrastructure teams adjust utilization targets and allocation policies without service teams changing how they express intent.

What I would carry to another organization

Across multiple cycles, this transformation reduced a three-to-four-month exercise to roughly one month. It also helped identify and avoid hundreds of millions of dollars’ in planned capacity through reduced overprovisioning, hardware simplification, and reuse of existing supply. This figure represented capacity removed during the review phase, not realized cash savings.

Finance got a number it could trace to specific products, organizations, and business growth assumptions. Engineers liked it, not because we mandated it, but because it replaced weeks of spreadsheet work and manual sizing with a self-service platform.

For leaders facing a similar problem, I would carry forward six principles:

1. Centralize the data on a platform before adding intelligence.
2. Ask product teams for workload intent, not machine counts.
3. Let domain owners define and maintain the hardware conversion logic.
4. Model downstream demand explicitly rather than relying on defensive buffers.
5. Connect every request to a durable product and accountable owner.
6. Close the loop by comparing forecasts with actual usage and available capacity.

The hardest part was not building the calculators or dependency logic. It was earning enough trust for teams to stop using their existing methods and private spreadsheets and rely on a shared system. We succeeded by delivering value incrementally, keeping early versions understandable, leaving experts in control of the assumptions and logic for their domains, and making adoption easier through clear documentation and responsive support.

At that point, we were no longer collecting numbers from teams once a year. We had a system that continuously translated business plans into infrastructure needs.

Get articles like this in your inbox

Choose your LeadDev newsletters to subscribe to.