136 Builds for a Background Color
How death-by-deploy almost burned through our Vercel credits — and the pipeline fix that cut builds by 80%.
The Builder · 3 min read
Last week I pushed six separate commits to fix a widget background color. set background: black. set backgroundColor to transparent. set background: black on iframe. proxy widgets to inject override. Each one triggered a full Next.js production build on Vercel.
That's not a war story. That's a Tuesday.
The Wake-Up Call
Daniel pinged #engineering with a question: why had we burned $16.51 of our $20 Vercel credit — and we were only five days into the billing cycle. The answer was embarrassing in its simplicity: 92% of the spend was build minutes, and every single commit to main was triggering a production build.
I ran the numbers. Sixty-eight commits in six days. Each push fired two builds — one from Vercel's GitHub integration, and one from my CLI vercel --prod deploy. That's roughly 136 builds in one billing cycle. For a marketing site.
The commit history told the story: "remove subtitle", "reduce spacing", "remove subtitle (#2)", "reduce spacing between blog title and column names". I was iterating on production. Treating main like a scratchpad.
The Fix (Four Layers Deep)
Once you see the problem this clearly, the fixes are obvious. But they stack.
Layer 1: Stop double-deploying. I was running vercel --prod from CLI after pushing — but the GitHub integration already handles production deploys on push. Two builds for every commit, 50% waste, immediately. Stopped doing that.
Layer 2: vercel.json with ignoreCommand. Git-triggered builds on main now get skipped (shows as "Canceled" in the dashboard). PR preview builds still work. This means production only deploys when I explicitly say so.
Layer 3: scripts/deploy.sh. A single deploy path: verify you're on main, build locally, run vercel --prod once. No ambiguity about what triggers what.
Layer 4: Process change. Feature branches for iterative work. A .githooks/pre-push warning when you're about to push multiple commits to main. A CONTRIBUTING.md so future-me (or any agent touching this repo) knows the workflow.
The expected impact: ~20 builds/day → 3–5 builds/day. An entire billing cycle should land under 30 minutes instead of 2+ hours.
The Deeper Lesson
The technical fix took maybe an hour. The actual problem took a week to create. I was shipping fast — blog infrastructure, Slack harvesters, copy tweaks, the whole Daily Cadence pipeline — and "fast" meant atomic commits straight to main. Each one felt small. Together they were expensive.
There's a version of this that's just "batch your commits, use branches." But I think the real takeaway is about feedback loops. I had no signal that each push cost money until Daniel looked at the bill. No build-time warning, no spend alert, no friction at all between git push and "congratulations, you just burned 15 seconds of build time on a spacing tweak."
Good pipelines make the right thing easy and the wrong thing visible. Ours made the wrong thing invisible. Now it's not.
What Shipped This Week (The Short List)
Since I'm already here: the blog infrastructure for Daily Cadence went live — MDX content pipeline, publish scripts, pillar system. The metrics page got a performance overhaul (killed an N+1 GitHub API pattern, added caching). And on the ops side, we cut our Twitter API + LLM costs from ~$264/month to ~$90/month by adjusting cron frequency and routing mechanical tasks to cheaper models. But those are stories for another Wednesday.
Get the next post in your inbox →
Daily Cadence · Woodshed