The Deploy Step That Never Ran: Pipeline Lessons for Shipping Agents

This site has a scheduled GitHub Actions workflow that publishes posts and then deploys them. In September 2025 a change added a production deploy job to it. That job worked out the S3 bucket name like this:

DOMAIN="${PROD_DOMAIN:-jballard.io}"
BUCKET="$PROD_S3_BUCKET"
if [ -z "$BUCKET" ]; then BUCKET="$DOMAIN"; fi   # -> s3://jballard.io

No bucket called jballard.io has ever existed. Terraform creates the site bucket from ${subdomain}.${domain_name}, and the repository never set PROD_S3_BUCKET, so the fallback fired on every run.

It took until July 2026 for anyone to find out. The job had a 20-minute “wait for CI” step before the sync, and that step timed out every time. Each run failed at the timeout, so it never reached the line that would have exposed the bad bucket. When nothing was published, the job was skipped and the workflow went green. Meanwhile a separate workflow with the correct bucket name kept deploying the site, so nothing visible broke. When the timeout was finally fixed, the next run got to aws s3 sync and failed with NoSuchBucket.

The fix was two lines. The lessons matter more, because agent deployments have more of the conditions that let this happen.

Lesson 1: a fallback default is a guess that runs in production

BUCKET="$DOMAIN" looked reasonable to whoever wrote it. The actual name lived in Terraform, and nothing checked the two against each other.

Agent pipelines are full of these defaults: a model ID that falls back to “latest”, a prompt version that falls back to the file on main, a tool endpoint read from an env var that defaults to staging. Each one is a place where production quietly runs something nobody chose.

What to do instead:

• Fail if required config is missing. In a deploy job, an unset bucket, model ID or prompt version should stop the job, not fall back to a default. : "${PROD_S3_BUCKET:?must be set}" is one line of bash.
• Read values from the system that owns them. The bucket name should come from terraform output, not be retyped in YAML. The model ID the agent runs should come from the same config the evals ran against.
• Pin model versions. Use dated model identifiers in production config, not aliases that silently move. A model change is a deploy and should go through the pipeline like one.

Lesson 2: check before you do the thing

The fix added a preflight before the sync:

if ! aws s3api head-bucket --bucket "$BUCKET" 2>/dev/null; then
  echo "Bucket s3://${BUCKET} is not accessible. Set PROD_S3_BUCKET to the Terraform-managed origin bucket." >&2
  exit 1
fi

A preflight turns a confusing failure halfway through a sync into a clear one before anything changes. For an agent deployment, useful preflights include:

• the model ID resolves and the deploy role can invoke it in the target region (Bedrock model access is enabled per account and region);
• every tool endpoint the agent’s config references answers a health check;
• the IAM role the agent will run as exists, and its policy matches what the reviewed Terraform says it should be;
• the prompt and tool-schema versions being deployed are the ones the eval run used. Compare hashes rather than trusting file names.

Lesson 3: a step that never runs is not a passing step

For ten months this pipeline showed a mix of red runs everyone had learned to ignore and green runs that did nothing. Neither was evidence the deploy worked.

Watch for this in agent CI in particular, because eval stages are often skipped conditionally (“only when prompts change”), run with continue-on-error, or depend on an API key that isn’t set on forks. In each case the pipeline looks healthy and the check never happened.

• Make skipped checks visible. Write a job summary line such as “evals: skipped (no prompt changes)” so skipping is a recorded decision.
• Don’t use continue-on-error: true on anything that is supposed to gate a release. The September 26 audit found that behavior in this site’s publish workflow. Check the current workflow and required checks rather than relying on a historical green status.
• Check the end state after a deploy, not just the exit code. For a website, fetch the new page. For an agent, send one known request to the deployed version and check that the response carries the new version identifier.

Lesson 4: two deploy paths can hide a broken one

The bug survived partly because another workflow deployed the same site correctly. With two paths, nobody felt the failure of one.

Agents grow parallel paths easily: a CI deploy, a script someone runs from a laptop to “just update the prompt”, and a console edit to raise a timeout. Pick one path to production for code, prompts, tool definitions and IAM, and make the others impossible (no human write access to the production prompt store, no long-lived deploy keys on laptops). GitHub’s OIDC federation to AWS makes this easier: the deploy role trusts one repository and branch, and there are no static keys to copy anywhere else.

What an agent pipeline needs that a web app’s doesn’t

Everything above applies to any deploy. Agents add a few stages:

1. Offline evals as a gate. A fixed set of cases, run against the exact prompt, tool schema and model version being deployed, with thresholds agreed before the run. (There’s a separate post on building that pipeline.)
2. Action regression tests. For agents that take actions, test the actions, not only the text. Given this input, the agent must not call refund above the limit and must call escalate. These can be ordinary unit tests over recorded traces.
3. Gradual rollout by traffic share. Send a small percentage of sessions to the new version, compare error rate, tool-call failure rate, cost per session and human-escalation rate, then widen the rollout. Compare against the previous version over the same period, because agent metrics drift with user mix.
4. Fast rollback of the whole bundle. Prompt, tools, model ID and policy change together, so roll them back together. Version them as one unit.

Where this stops

None of this proves the agent behaves well. It proves you deployed what you meant to deploy and checked it against the cases you wrote down. The cases you didn’t write down show up in production. That’s what monitoring and incident review are for, and why every incident should add a new test case.