Sign in Subscribe

How to Safely Roll Back a Failed Kubernetes Deployment

How to Safely Roll Back a Failed Kubernetes Deployment
How to Safely Roll Back a Failed Kubernetes Deployment

A deployment can complete successfully in Kubernetes while still breaking the application. Pods may be Ready even as requests fail, a feature flag changes behavior, or an incompatible schema makes the previous version unsafe to restore. Conversely, a rollout can stall before the faulty version receives much traffic. These are different incidents and require different recovery decisions.

The essential question is not merely which command rolls back a Deployment? It is which system currently owns the release, which previous state is known to work, and what parts of the incident can that state actually reverse?

Establish the blast radius before changing anything

Start by finding the affected namespace, Deployment, release owner and start time of the regression. Check application error rate, latency, a representative business operation and whether existing Pods still serve requests. Record the current image digest and deployment revision while evidence is available. Avoid making a new deployment simply because the rollout looks slow.

For a conventional Deployment named api in namespace production, an initial read-only inspection could be:

kubectl -n production rollout status deployment/api --timeout=30s
kubectl -n production describe deployment api
kubectl -n production get deployment api -o wide
kubectl -n production get rs -l app=api
kubectl -n production get pods -l app=api -o wide
kubectl -n production get events --sort-by=.metadata.creationTimestamp

The selector app=api is illustrative: use labels actually present on the workload. A timeout from rollout status is not itself proof that the desired release is bad; scheduling capacity, image pulls and probe settings can also delay progression. The official Kubernetes debugging guidance explains how events, container state and logs clarify those failure modes.

Then inspect container logs for relevant Pods and compare them with application telemetry. Kubernetes observability guidance distinguishes operational signals from the controller's own status. An increase in payment failures, for example, may matter even if every Pod reports Ready. This is an illustrative failure mode, not a claim about a particular incident.

Distinguish an unhealthy rollout from an unhealthy release

There are two operationally distinct failure paths. In the first, the new ReplicaSet cannot converge because Pods are Pending, images are unavailable, containers crash or readiness never succeeds. In the second, the new Pods converge and receive traffic, but users experience failures. The first path is usually visible in Deployment conditions and Pod events. The second may only be visible through application metrics, traces, user support signals or business-specific probes.

The decision to roll back should therefore depend on evidence of harm and reversibility, not solely on a red or green controller condition. Before acting, confirm whether the incident started after the release, whether another shared dependency failed at the same time, and whether restoring the previous application binary would actually remove the suspected fault.

For services with asynchronous consumers, look beyond request metrics. A release can continue returning successful HTTP responses while message retries, dead-letter volume or scheduled-job failures increase. Treat health as a property of the whole critical transaction, not just the public endpoint.

Know what a Deployment rollback can restore

Kubernetes Deployment documentation specifies that a revision is created when the Pod template changes. The retained ReplicaSets provide the history used by kubectl rollout undo. Restoring a previous revision therefore restores the older Pod template: typically its container image references, environment declarations, labels and other template fields.

It does not restore an entire application environment to a prior timestamp. Scaling a Deployment does not itself create a Pod-template revision. Changes to separate Services, ConfigMaps, Secrets, Ingresses, databases and third-party systems are outside the native undo operation. If the incident was triggered by one of those changes, rolling back an image may leave the fault intact.

The standard rolling-update task documents kubectl rollout history, kubectl rollout undo and targeted revision selection. It also explains that revisionHistoryLimit controls how many old ReplicaSets remain available; the documented default is ten. If history has been pruned, a previous release may need to be reconstructed from immutable manifests and pinned image digests rather than from native revision history.

Choose the target rather than blindly selecting the previous revision

Inspect the revisions:

kubectl -n production rollout history deployment/api
kubectl -n production rollout history deployment/api --revision=12

Revision 12 is an example; substitute a revision present in your cluster. Identify the last release that actually passed production validation. The immediately preceding revision may also be faulty if two deployments occurred during one incident. Read the revision's Pod template and match its image digest and configuration references to your release records.

The CHANGE-CAUSE field in history can be empty. Kubernetes does not automatically populate the corresponding annotation, so an empty value is not evidence that no change happened. Build release provenance separately from CI metadata, Git commits, image digests and chart versions.

Check whether the Deployment is paused. A paused Deployment cannot be rolled back until resumed, as the controller documentation notes. Do not routinely pause a running rollout as an emergency stop without checking the consequence: pausing affects application of new template updates, not the existing Pods, and a rollback will require resuming.

Capture what is about to change

Before a production mutation, save a copy of the live Deployment and its recent events in the incident workspace. Preserve your change-control record and redact credentials, environment variables and other potentially sensitive material before sharing it.

kubectl -n production get deployment api -o yaml > api-before-rollback.yaml
kubectl -n production rollout history deployment/api
kubectl -n production get rs -l app=api -o wide

This is evidence preservation, not a full backup. It does not capture database contents, external services, secrets or full cluster state. Store it in an access-controlled location and record the intended target revision, operator and incident timestamp.

If your release process relies on image tags rather than pinned digests, double-check the actual image references of the chosen ReplicaSet. A mutable tag such as latest is not a reliable historical artifact: a future image pull may retrieve different bytes. A safer deployment supply chain keeps immutable digests associated with signed builds or release attestations, while ensuring the chosen digest remains available from the registry.

Execute a native rollback deliberately

Once the incident owner has confirmed the target revision and any GitOps controller is not competing with the change, the direct command is:

kubectl -n production rollout undo deployment/api --to-revision=12
kubectl -n production rollout status deployment/api --timeout=5m
kubectl -n production get deployment api
kubectl -n production get rs -l app=api

A plain kubectl rollout undo deployment/api targets the previous revision. Prefer explicit --to-revision when multiple emergency changes may have occurred. The rollback triggers normal Deployment reconciliation, so Kubernetes may need to create or scale Pods and terminate others; it is not an instantaneous traffic switch.

If the command reports that a Deployment is paused, inspect the reason before issuing kubectl rollout resume. If the expected revision is absent, stop and use a reproducible release artifact rather than inventing a revision number. If readiness fails after rolling back, the previous binary may no longer be compatible with the current environment.

Do not interpret progressDeadlineSeconds as an automatic rollback facility. Kubernetes marks a stalled Deployment with Progressing=False and ProgressDeadlineExceeded after the configured deadline, but the Deployment controller continues retrying. The deployment update guide describes the condition; initiating recovery is a separate operator or higher-level automation decision.

When the rollback itself stalls

A rollback can fail because a previously working image is no longer pullable, nodes no longer have enough spare resources, Pod admission policies changed, or startup behavior now depends on a newer external interface. Inspect the new ReplicaSet and Pod events instead of immediately chaining further rollbacks. Verify registry access, image availability, namespace quota, service accounts and updated admission constraints.

If an old ReplicaSet is absent because revision history was pruned, the recovery route becomes a controlled redeployment of a known-good manifest or Helm/GitOps revision. Reconstruct it from the release system; do not assume that manually specifying an old tag recreates the former Pod template.

Avoid simultaneously scaling, reverting, restarting and editing resources without recording which action caused which effect. Those overlapping mutations make causality harder to establish and may leave the declared source out of sync with production. Designate one incident operator to authorize each step and keep a timestamped log of decisions.

Verify service recovery, not only rollout completion

A successful rollout status means the controller sees the target rollout as complete. It does not establish that downstream dependencies work or that the application meets its service objectives. Use a time-bounded observation window appropriate to the workload.

At minimum, verify the intended image or Pod-template revision is serving, available replicas are stable, application error rates and tail latency have recovered, critical synthetic requests pass, and failed background work is not accumulating. Compare pre-incident, incident and recovery windows so that a quiet period with no traffic is not mistaken for success.

Understand the role of readiness, liveness and startup probes. Readiness removes failing Pods from Service traffic; liveness can restart an unhealthy process; startup probes allow extra initialization time. A shallow readiness check can report success while important application functions remain broken. Conversely, a badly tuned probe can prevent an otherwise healthy image from becoming available.

The incident remains open if the cluster looks healthy but business or data health is still impaired. Record the recovery decision and arrange a clean reconciliation of desired state, rather than leaving an undocumented manual override.

Account for surge capacity and graceful termination

The default RollingUpdate strategy gradually replaces Pods. Its maxUnavailable and maxSurge values control how many Pods may be missing and how many extra Pods can be created. Kubernetes documents default values of 25% for each, with different rounding rules for percentages. These are mechanics, not a guarantee of zero user-visible disruption.

With limited headroom, extra Pods can remain Pending; with too much allowed unavailability, a service can drop below its useful capacity. Inspect scheduling events and current replica distribution before changing these parameters under pressure. For critical paths, test the planned rollout and rollback settings under realistic load in a non-production environment.

Graceful shutdown also matters. During Pod termination, application handlers need time to stop accepting new work and finish in-flight requests, within the configured termination-grace period. The Pod lifecycle documentation covers termination behavior and lifecycle hooks. Rollback safety depends partly on connection draining, idempotency, queue consumers and compatible old/new versions during overlap.

A worked availability configuration

For a stateless API with multiple replicas and enough cluster capacity, the following illustrates a conservative rollout shape. It is a configuration example, not a universal default or a guarantee of zero downtime:

spec:
  replicas: 4
  revisionHistoryLimit: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    spec:
      terminationGracePeriodSeconds: 40
      containers:
        - name: api
          image: registry.example.com/api@sha256:REPLACE_WITH_REAL_DIGEST
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 5
            failureThreshold: 2

Replace the illustrative image digest placeholder with a real digest before deploying. The example assumes the service has an implementation-specific /ready endpoint and enough quota to schedule an extra Pod. A strict maxUnavailable: 0 may stall a rollout if surge capacity is absent; this is an intentional availability-versus-progress trade-off. Termination grace must be matched to real request and job duration, and the readiness endpoint must test useful dependencies without flapping during transient failures.

For long-lived connections, event consumers or stateful workloads, additional shutdown coordination may be required. A Pod that becomes unready is not necessarily finished processing work. Application design needs to tolerate brief overlap between versions, and a retry after an interrupted request should be safe.

If the release was installed with Helm

Helm tracks release revisions, which are not interchangeable with Kubernetes Deployment revision numbers. Start with:

helm -n production history api
helm -n production status api

The official Helm history command shows release revisions. After reviewing the target and any hooks, use the supported Helm rollback operation:

helm -n production rollback api 7 --wait --timeout 5m

Here 7 is only an example release revision. Confirm the installed Helm version supports the selected flags. A Helm rollback can restore the resource manifests and values represented by a previous release, but it still cannot rewind external database writes or undo side effects of arbitrary hooks. Review hooks, jobs and compatibility implications before triggering them.

For future releases, Helm exposes upgrade failure-handling options; their names and behavior can vary by Helm major version. Check the versioned upgrade reference against the binary used in CI rather than copying a flag from a different release series.

If Argo CD or Flux owns the workload

A direct Kubernetes undo is liable to conflict with a continuously reconciled desired state. The recovery path should primarily change the authoritative declaration: revert the offending Git commit or pin the last verified manifest and image version, then let the controller reconcile that target.

The Argo CD automated-sync documentation explicitly states that application rollback cannot be performed while automated sync is enabled. It also explains that self-healing may restore divergence from Git. If an emergency manual operation is unavoidable, coordinate the controller's sync policy with the team, understand ApplicationSet ownership, and follow it by a permanent desired-state correction.

Flux HelmRelease documentation defines upgrade remediation including rollback, retries and actions after the final failure. These mechanisms are useful when Helm detects a failed upgrade, but they do not establish that a deployment was functionally correct after becoming Ready. Inspect controller conditions and reconcile the source artifact, not just the live Pods.

When automation is the wrong first move

Automated rollback can be valuable for obvious deployment failures: image pull errors, missing readiness, or a canary that crosses a carefully designed error budget threshold. It is less reliable for a release that modifies persistent state, affects several independently deployed services or changes request semantics. A mechanically successful return to the previous container may still violate the new data contract.

Set explicit boundaries for automatic recovery. For example, a team may authorize automatic abort for a canary that has not yet run a database migration, but require human approval once a destructive migration or irreversible queue transformation has started. This is a proposed policy pattern, not a standard imposed by Kubernetes. Record the conditions in the delivery configuration and test them.

When both old and new Pods can be active during a rolling update, compatibility must work in both directions. That includes the shape of database rows, API versions, serialized messages and cached content. A safe application release is designed to survive that temporary mixed-version state.

Recognize cases where an image rollback is unsafe

Consider a new version that migrated database columns, changed persisted event formats, rotated a shared credential, or deployed a protocol incompatible with an older worker. Rolling back its binary may turn one error into widespread data corruption or message-processing failures. Such examples are hypothetical but represent concrete compatibility questions that must be answered before reverting.

A Deployment also does not version the contents of an external ConfigMap. Kubernetes ConfigMap documentation explains that environment-variable values do not update in a running Pod without restart, while mounted ConfigMap volumes are updated eventually, subject to propagation behavior and exceptions. Native undo does not by itself restore the previous ConfigMap data.

For migrations, favor an expand-and-contract sequence: introduce backwards-compatible fields, deploy code able to read both shapes, migrate data separately, and remove the old shape only after the rollback window closes. Where migration is destructive or irreversible, choose a tested forward fix, feature disablement or controlled restoration plan rather than claiming that rollout undo is sufficient.

Make the recovery decision explicit

Use three separate questions when selecting a response:

  • Can an old application version safely run against current data? If yes and its artifact is available, revert the release through the owning system. If not, consider a forward fix, traffic isolation or application feature disablement.
  • Is the defect limited to one Deployment? If the release also changed ingress routing, a shared Secret or other services, coordinate those resources as a single change set.
  • How will success be observed? Specify a handful of customer-visible signals and a watch period. A rollback without a measurable acceptance criterion is only a configuration change.

Prefer reversible migrations and feature flags as independent mitigation mechanisms, but do not assume all feature flags are instantly safe: flag clients may cache values, and toggling a flag cannot reverse state already written. Document those limits before treating a flag as a rollback alternative.

Design the next rollback before the next incident

Maintain immutable build artifacts and their provenance, known-good manifests, an appropriate revision-history limit, release ownership records and audited emergency permissions. Define who can stop automated promotion, who can authorize a database restore, and which service indicators must recover before closing the incident.

Progressive delivery can limit exposure to a bad release. Argo Rollouts analysis supports metric-driven analysis and abort decisions for canary and blue-green rollouts; its rollback-window feature can speed a return to selected earlier ReplicaSets. These are available mechanisms, not proof of improved reliability in every environment. Faulty metrics, weak health checks or incompatible data migrations can defeat any automated safety gate.

The practical rollback runbook is short enough to rehearse: identify release owner; capture current state; decide whether rollback is compatible; select a verified target; stop conflicting deployment automation; apply the appropriate recovery operation; validate both controller and user-facing signals; restore Git and incident records to a coherent state.

A deployment is recovered when the application behaves correctly and the system of record agrees with the running workload. A green Kubernetes rollout status is necessary evidence for many workloads, but never the entire definition of recovery.