CI feels slower: separate time spent waiting for a runner from time spent running

Two clocks

When people say the pipeline is slow they usually mean the time from pushing to seeing a result. That time is the sum of two different things. Waiting is the time a job spends queued before a runner picks it up. Running is the time the job's steps take once it has started. A change that speeds up one does nothing for the other, and the fixes belong to different people: waiting is mostly about plan and runner capacity, running is mostly about the pipeline's own steps.

Why a pipeline can wait with no slow step

Hosted platforms cap how many jobs can run at once, and the cap depends on the plan. GitHub, for example, publishes different concurrent-job limits for its plans, and a job waiting for a self-hosted runner is cancelled after 24 hours. A pipeline that grows from three jobs to thirty, or a matrix that multiplies them, can start queueing when the cap is reached even though no step has got slower. If your team started working in parallel on several pull requests, the same effect appears. This is why a time budget that only watches total duration can misdiagnose a capacity problem as a code problem.

What makes the running part longer

The usual contributors are the install step when a cache stops hitting, a build that now compiles every package in a monorepo, a test suite that grew, a container image that rebuilds from scratch, and uploads of large outputs. A cache is only an optimisation: on GitHub it expires after seven days unused, so a pipeline that runs rarely can slow down with no code change, and on GitLab a runner-local cache helps only when the next job lands on the same runner. Look at the duration of each step across several runs before deciding which one grew.

• Install and cache behaviour
• Builds of packages the change did not touch
• Test suite growth
• Image rebuilds without layer reuse
• Large uploads

Read a typical run and a slow run

Take the last twenty successful runs of the pipeline on the default branch. Sort their durations. The typical run is the middle: with twenty values, the average of the tenth and eleventh. The slow run is the one near the ninetieth percentile: with twenty values sorted from shortest to longest, the eighteenth. Do this separately for waiting and for running if your platform's run data gives you a creation time, a start time and an end time for each job; check what those timestamps mean on your platform before you rely on them, because not every reference defines them. A budget set on the typical run, with the slow run reported beside it, is more honest than one number.

Which fix belongs to which cause

Waiting: reduce the number of jobs, stagger them, or change the plan or runner capacity, which is your decision. Install time: repair the cache. Unneeded builds: detect affected packages. Test time: parallelism or test selection, which is a separate piece of work. A monthly time-budget service keeps watching all of these and reports waiting and running separately, escalating anything that sits outside your repository. It costs £245 a month for one pipeline, untested. It checks the run history once a week, investigates every overrun and includes one small fix pull request a month; larger fixes are the separate one-off jobs at their own prices. It does not promise a duration or a response time.