Why your CI pipeline is slow, and why it is probably not the tests
Queue wait, toolchain setup, and caches that quietly do nothing usually cost more than the test suite. How to profile a pipeline stage by stage, and the fixes that took one from nine minutes to under three.
When a team says CI is slow, they usually mean the tests are slow, and they usually mean it without having measured. On the pipelines we have profiled, the tests were rarely the biggest stage. The minutes hide in queueing, toolchain setup, and caches that quietly do nothing. This is how to find them, in the order that tends to pay off.
Measure every stage before touching anything
Take the job that gates pull requests and break its wall time into stages: time waiting for a runner, time to check out and install the toolchain, dependency install, static analysis, database setup, tests, and anything that runs after. GitHub Actions gives you per-step timing on every run; pull a couple of weeks of it rather than one run, because the slow stages vary by time of day.
On a Laravel monorepo with about 38 runs every weekday, the gating job took about nine minutes at the median. The complaint was the queue. The measurements showed the queue was the first of four problems, and not the largest.
Stage one: the queue
Hosted runners take time to appear. On that pipeline the first 65 to 80 seconds of every run was dead time before a single command executed. Multiply by 38 runs a day and it is close to four hours a week spent waiting for a virtual machine, before anyone counts the context switch.
The fix is a runner that is already there. If you have a Kubernetes cluster, GitHub's Actions Runner Controller puts runners in it, and a warm pool of pre-registered idle runners takes start latency from over a minute to about two seconds. Schedule the pool for business hours and scale it to zero on weekends; a pool like that costs on the order of $150 a month.
One warning if the cluster uses Karpenter. The usual trick for warm capacity, low-priority balloon pods that a real job preempts, does not work: Karpenter provisions a fresh node faster than the scheduler evicts the balloon, so the job lands on the cold node every time. Warm capacity has to be real runners, scheduled and paid for.
Stage two: toolchain setup
Every run that installs PHP extensions, Node, a coverage driver, and database clients from scratch is repeating work that never changes. Bake the toolchain into the runner image. On the pipeline above the image carried three PHP versions with extensions, Node 22, opcache, the coverage driver, and the database clients, so the per-run setup step fast-paths instead of downloading.
Stage three: caches that do nothing
This is the one to check first, because it is the cheapest fix and the most common. A static-analysis or build tool with a cache is only fast if the cache survives between runs. Many tools pick a cache location that lives inside the job, and under a fresh runner that cache is thrown away every time while the log cheerfully reports a cache hit on nothing.
Rector was taking 170 seconds per run on that pipeline. Under GitHub Actions it silently selected an in-memory cache that died with the job. Forcing the on-disk cache onto a shared mount took the stage to 7 seconds. One flag, 163 seconds a run, and nobody had noticed because nothing was erroring.
Check the same thing for dependency caches, PHPStan or Psalm, ESLint, webpack or Vite, and Docker layer caches. For each one, confirm the cache directory is restored before the tool runs and saved after, and look at its size. A cache that is always a few kilobytes is not a cache.
Stage four: the test database
A test suite that runs against a database on disk is paying for durability it does not need. Run MySQL or Postgres in memory on tmpfs with durability disabled for the job's lifetime. Then run the suite in parallel: most frameworks support it, and the constraint is usually runner size rather than the suite. The test stage on that pipeline went from 104 seconds to 47 with tmpfs and eight parallel processes on a larger runner class. A later pass onto 16-vCPU runners cut a grown test stage from 347 seconds to 179.
What the numbers looked like afterwards
Median run time went from nine minutes to 2 minutes 44 seconds, a 70 percent cut, with the queue wait at 2 seconds. That is about 70 hours of engineer wait time a month reclaimed, roughly $72,000 a year at the client's loaded salary assumption, against under $2,000 a year in runner cost. The pool pays for itself if it recovers seven percent of the dead time.
Just as important, the run became compute-bound. Tests, database start, and code style were what remained, so the next improvements belonged to the application team, and the platform team could stop.
Keep it that way
Pipelines regrow. Put a small dashboard on the runner fleet: job latency against pool size, throughput by repository, and a demand heatmap by hour. Then the pool size is a tuning decision with data, and a slow stage shows up as a trend rather than a complaint six months later.
The full story
The stage-by-stage numbers, the runner image, and the warm-pool design are in the CI pipeline case study.
Is your team waiting on CI?
We'll profile your slowest pipeline stage by stage and show you where the minutes are going before recommending a change.
