Zero-downtime app development: deployments and rollback

Developing an application without downtime for users means running deployments and production changes so that availability is never interrupted and data consistency is never lost, while operational risk stays under control: (1) a zero-downtime deployment strategy; (2) API and migration version compatibility; (3) monitoring and automatic rollbacks.

Quick facts

• The least risky production changes come from separating the release of code from the activation of features.
• A lack of backward compatibility in APIs and data schemas causes outages more often than the deployment mechanism itself.
• The contingency plan must cover rolling back the version, switching off the feature and reversible data migrations.

Keeping the service running during application development most often comes down to limiting the “blast radius” of changes. Stability improves when modifications reach production in small batches and are observed through technical metrics.

• Gradually routing traffic to the new version, and controlling the state of sessions and connections.
• Automatically pausing the deployment when error, latency or resource saturation thresholds are exceeded.
• Designing data changes as a sequence of reversible steps instead of a single “all at once” migration.

Developing an application that is live in production is like working on a living system: users make transactions, the system processes events and the data is already in circulation. Downtime is no longer just a technical problem, because it affects revenue and reputation and, in many industries, SLA requirements too. In practice, zero-downtime delivery of changes rests on three pillars: a safe release strategy, backward compatibility and fast feedback from observability. Risk management also becomes key: rollback must be possible in minutes, not hours, and data migrations must allow for a return to the previous state. Only the combination of processes, architecture and operational discipline makes it possible to modernise an application without downtime that users can feel.

Zero-downtime deployments: patterns and edge conditions

A zero-downtime deployment keeps at least two versions of the system, or two infrastructure layers, running in parallel so that traffic can be switched over without losing availability. The best results come from matching the strategy to the type of application, the nature of the traffic and the way state is stored.

The most common approaches are blue-green and canary. Blue-green makes rollback simpler, because the old version stays ready to take over traffic and the switch comes down to changing the routing. Canary reduces risk, because the new version serves a small percentage of traffic and its share is increased only as error and response-time metrics allow. In systems that are sensitive to sessions and connections, connection “draining” can also matter, so that operations in progress are not interrupted.

Edge conditions include application state and protocol compatibility. If state is held in instance memory (for example sessions), you need a mechanism for sharing it or to move it to an external store. Otherwise a traffic switch can cause logouts, duplicated orders or idempotency errors. Stability also improves when you limit how much changes at once and keep the ability to restart services quickly.

If switching traffic requires a full restart of the stateful layer, the most likely explanation is that the deployment strategy ignores dependencies on sessions or long-lived connections.

Feature flags and separating release from activation

Separating the release of code from the activation of features lets you bring changes into production without an immediate impact on users. This model makes risk easier to control, because a feature can be switched on selectively and switched off without rolling back the whole version.

Feature flags are worth designing as a mechanism with clear rules: who manages the flags, how long they may exist and how they are removed. Flags left in place permanently create technical debt, and conflicting configurations can produce errors that are hard to reproduce. Good practice includes limiting the number of flags on critical paths, versioning the configuration and auditing changes, so that it is clear when a switch happened.

Flagging should be tied to observability. Enabling a feature should automatically mark events in logs and metrics, so you can tell a regression caused by the new feature from an infrastructure problem. In applications with a high transaction volume, user segments and percentage ramp-up are useful, because they let you measure the impact on performance and errors in a controlled way.

If a feature is activated without segment limits or a metric threshold, the consequence can be no time to react before errors escalate across the whole user base.

API and contract compatibility: versioning without breaking integrations

Backward compatibility in APIs reduces the risk of downtime caused by old and new components running side by side. During a deployment the system often runs in a mixed mode, so contracts must withstand differences between versions.

The safest approach is to treat the API as a contract: removing fields, changing the meaning of values and other “breaking changes” should be planned as a multi-stage process. New fields and paths are added first, then clients and services are migrated to the new behaviour, and only at the end are the old elements retired. Versioning can take the form of a version in the path, in a header or through schema negotiation, but it always requires contract tests and agreed deprecation dates.

In asynchronous integrations, event schemas and the evolution policy are critical (for example, allowing new fields and keeping the old ones). Making consumers tolerant of unknown fields and maintaining idempotency minimises the effects of retries and delays. For mobile apps and offline devices, client-side backward compatibility matters too, because some users stay on earlier versions for a long time.

A contract test makes it possible to tell a safe change from one that breaks an integration, without increasing the risk of errors.

Data migrations without downtime: the expand/contract approach

Zero-downtime data migrations follow the principle of “compatibility first, tidying up later”, so that the application can run on the old and new schema at the same time. The most commonly used pattern is expand/contract, which minimises the risk of locks and long transactions.

The expand phase adds new structures (columns, tables, indexes) in a way that does not disrupt current operations, and introduces code that can write data in the new format while still reading the old one. In practice you may need “dual write” or a mapping layer, as well as a backfill run in batches with a load limit. Only after stability is confirmed are reads switched to the new schema.

The contract phase removes the old elements after an observation period and once you are sure that no consumers still use the former fields. The most common problems are a backfill that is not idempotent, migrations that are too broad and run in a single transaction, and no way back. A safe data rollback usually means being able to run the old code on the new data, or keeping the data in a format that both versions can interpret.

If a migration runs longer than the maintenance window, the most likely explanation is that the backfill has no load limits or was not split into reversible stages.

Observability and alert thresholds: metrics, logs, traces

Observability is a precondition for running an application without downtime, because it lets you detect a regression quickly after a change and stop the deployment safely. It is most effective when technical metrics are linked to service quality signals.

The foundation is three pillars: metrics, logs and distributed traces. Metrics should cover errors (for example the share of 5xx responses), latency (p95/p99), resource saturation (CPU, memory, connection pool) and data layer performance (query time, locks). Logs must be structured and carry correlation between services, so that a single error can be traced from the request to the datastore. Traces are especially useful in microservice architectures, where a regression can come from one slow dependency call.

Alert thresholds should reflect the risk policy. For a canary deployment, individual “gates” are practical: if the error rate rises above a set threshold or p95 goes beyond the latency budget, the deployment is paused automatically. Anomaly detection is also needed, because a regression is sometimes visible only in segments with a particular traffic profile. Synthetic availability tests and monitoring of critical paths are a valuable addition.

If the granularity of signals does not increase after a release, the consequence can be that a regression is identified too late, before the rollback procedure kicks in.

Runbook, rollback and incident drills

An emergency procedure shortens the duration of an incident, because it shows how to stop the escalation and restore stability quickly. Good runbooks describe decisions and thresholds, not just a list of commands, which makes it easier to act under time pressure.

Rollback should be treated as the first scenario, not a last resort. Technically, it means being able to switch traffic back to the previous version, switch a feature off with a flag and halt migration processes. Rolling back an application is usually fast, whereas rolling back data is hard; the contingency plan should therefore assume reversible migrations and backward compatibility. In event-driven systems, queue management matters too: delays and retries can make an error “come back” after things have apparently calmed down.

Incident drills reveal gaps in procedures: outdated steps, missing permissions, dependence on individual people and gaps in monitoring. Simulations should cover typical classes of problems: database overload, a rise in API errors and incorrect configuration. Organisationally, what matters is clear roles, a communication channel, escalation criteria and the moment at which the incident is declared over.

If a runbook does not include thresholds for stopping a deployment, the most likely outcome is a prolonged incident caused by ambiguous rollback decisions.

Which sources are better: vendor documentation or community reports?

Vendor documentation is usually more verifiable, because it describes the formal contract of operation and versioning, whereas community reports can be quicker to flag practical failures and workarounds. The selection criteria should take into account the format (specification, RFC, release notes), reproducibility (reproduction steps, environment) and trust signals (author, review process, link to a release). The highest quality comes from comparing both types of source and choosing those that let you confirm a claim with a test or a metric.

Pre-deployment checks and risk: a sample decision matrix

A repeatable pre-deployment checklist simplifies risk assessment and reduces the number of errors caused by haste. It is most valuable when the checks are framed as simple “ready/not ready” criteria that can be automated.

Checks should cover: changes to API contracts, the data migration plan, operational requirements and observability thresholds. For applications where downtime is costly, risk should be tied to the “blast radius”: what percentage of traffic the change affects, whether it touches payment or login paths, and whether it touches the data layer. Also important are the ability to switch a feature off without a rollback and the compatibility of mobile client versions, which do not update evenly.

In practice, a check means documenting one failure scenario and the response: which metrics will show it, who makes the rollback decision and what the maximum acceptable time of degradation is. Such a record protects against “deployment optimism”, where the lack of hard thresholds leads to decisions being dragged out. A well-organised process combines technology and organisation, because even a correct deployment can trigger an incident if it is misconfigured.

If a change affects login or payments, the consequence must be a more restrictive canary scope and lower error thresholds than for changes in non-critical areas.

When maintaining deployment quality, it can help to review projects such as Tacho App, because it lets you assess availability and load-resilience requirements in apps used intensively in day-to-day operations.

In public administration and training environments, stability at peak hours and control over content publishing are important, as illustrated by the Tarnów City Office training platform, a system where interruptions affect the continuity of processes.

In sales projects, payment and cart paths carry particular weight, and the scope of changes can be analysed in line with the practices used for the Pan Drwal store, where short risk windows must be limited in advance.

Frequently asked questions

What most often causes downtime when updating an application?

Downtime most often comes from incompatible changes to the data schema or API contracts, which prevent versions from running in parallel. Configuration errors and the lack of thresholds for automatically halting a deployment are also common causes.

Is zero-downtime development possible without microservices?

Yes, because zero downtime depends mainly on the deployment strategy, compatibility and observability, not on the architectural style. A monolith with properly designed migrations and traffic switching can keep availability at a high level.

What are the minimum elements of a rollback plan?

A minimum rollback plan covers switching traffic to the previous version, the ability to switch features off and a procedure for stopping migration processes. For data, you need steps that let you run the old version of the application without losing consistency.

How can you reduce risk during a database migration?

Risk falls with the expand/contract approach, a backfill run in small batches and keeping read compatibility across both versions. It is also critical to avoid long blocking transactions and to ensure the steps are reversible.

Which metrics detect a regression after a deployment fastest?

Error and latency metrics work fastest, especially error rate and p95/p99, supplemented by resource saturation indicators. In distributed systems, traces matter too, because they point to the bottleneck in a dependency.

Summary

Zero-downtime application development rests on a deployment strategy that allows versions to run in parallel, and on the controlled activation of features. Changes to data and contracts generate the greatest risk, so they need to be phased and backward compatible. Observability and automatic deployment-halting thresholds shorten the time to detect a regression. Runbooks and incident drills bring order to decisions and improve the time to restore stability.

__________

Sources

• Google Site Reliability Engineering: the SRE book / Google / 2016
• Microsoft Azure Architecture Center: blue-green deployment and canary release / Microsoft / 2023
• Cloud Native Computing Foundation: Observability and Monitoring / CNCF / 2022
• Martin Fowler: Feature Toggles / 2017
• Amazon Web Services: Building and Operating Resilient Systems / AWS / 2020