A Primer to Batch Sizes
In software development we talk a lot about batch sizes, so let’s talk at a super high level about what batch size is, and what economic laws govern the right size of your batches.
Batch size is an optimization problem, and one that has its roots outside of software. Ford Whitman Harris was an engineer in the early 1900’s and worked in manufacturing. He spent a lot of time trying to understand how to approach problems like “how much stock should we keep on hand?” and “how many parts should we manufacture at once?” The result of his work is called the Economic Lot Size equation, which was introduced in 1913:
Transaction cost is the cost of sending a batch, while holding cost is the cost of not sending it. Confused?
Let’s say that you operate a knitting store and you buy yarn in bulk and sell it to hobbyists. Yarn has a high fixed cost to ship - it costs a lot whether you order 1 spool or 1,000 spools, so ideally you want to do a few bulk orders a year to minimize your shipping costs. But you can’t predict which color will be hot this season, so you can’t stock up on those colors in advance.
You make your first order for the year, and it turns out that reds are the hot hue of the season. You sell out of crimson fairly quickly. What do you do?
You could order more crimson right now - but you have to pay the full shipping cost for just this one color (this is the transaction cost)
You could just go without for a while, but you will lose some sales when someone comes in and can’t find the crimson they wanted (this is the holding cost).
You decide that you can’t afford to buy just crimson, so you decide to accept some lost sales. Then burgundy sells out, and you have a new decision - do you order the two colors, or lose sales from two colors? Then cherry sells out, and then maroon. Finally you decide to go ahead and order a restock of the four colors.
Was this the right decision?
Let’s break down a few principles in play to find out:
The more items you batch together, the lower the fixed cost
The fixed cost itself is fixed, but adding more items into your order can spread that cost around more. In a high fixed cost world, it’s best to create large batches to spread out the costs.
The equation for the Transaction (fixed) cost is:
And so as the number of items that you order together increases (the Items Per Batch), then the average transaction cost per item decreases.
If you wait to batch more items then you will incur more holding costs
Once we sell out of crimson, we are going to start having customers who can’t find the color that they want - and we will start to lose sales. The longer we wait to re-order, then more sales we are going to lose. If we wait twice as long to re-order, we will lose twice as much in crimson sales.
The equation to figure out your average holding cost per item is:
(We divide by two because the average item in the batch has to wait for half the time. the first color may be losing sales for four weeks until you re-order, but the last color to sell out loses almost no sales)
The longer you wait (Items Per Batch) the most holding cost you will incur.
These two costs are competing with each other
Larger batches decrease the average fixed cost per item, while increasing the average holding cost per item. Figuring out the optimal batch size is a U-curve optimization problem.
We can distill down the above equations to figure out how to calculate the optimal batch size for re-ordering yarn:
Solving for our example above, we can see that ideally we should re-order once we sell out of seven colors.
What if we could reduce our transaction cost?
Let’s say that we find a new supplier of yarn in a nearby town, and switching to this supplier will reduce our fixed shipping costs in half. How would that impact our decision for the optimal re-order batch size?
It turns out that if we can cut our fixed transaction costs in half, then it becomes economically optimal to re-order after just five colors sell out.
So how does this relate to software development?
When it comes to deploying software, the fixed cost is the cost of releasing finished code to production. From a development perspective that includes:
The cost of review
The cost of merging
The cost of testing
The cost of deploying
And so on…
Decades ago, when I was a developer, we released very large batches. We would define a “release” and build many new features into it, then spend a few weeks testing and patching those features, and then deploy the release into production (typically over a weekend because it required taking the servers down).
We created large batches because our fixed costs were so high to deploy that it didn’t make sense to do smaller releases (except in emergency hot-patch scenarios).
There were holding costs involved with that approach; we would build useful features in the first few weeks of the release schedule that would not go live for months. Those features could help close sales or reduce churn - but they wouldn’t be able to do so for months until we had a full release ready.
Over the last two decades, the field of software development has advanced considerably. Advances like robust (and performant) automated test suites, CI/CD tooling, dark launching and rolling deployments have reduced the fixed cost of releasing software to a tiny fraction of what it was.
These reductions in the fixed cost of deployment have enabled companies to simultaneously reduce their batch size as well. Today many organizations not only ship every pull request (PR), but they have also managed to reduce the size of those code changes to ship larger projects in multiple stages.
But even in today’s environment, where most organizations can deploy code multiple times per day, this formula holds true. Inevitably when there are challenges in the value pipeline that increase the work it takes to deploy (for example non-performant tests start to take too long to run, creating a backlog of code to merge; or challenges in teams approving each other’s code quickly enough), you will inevitably see developer respond to these slowdowns by trying to squeeze a bit more change into every pull request.