System design
System Design Beyond the Diagram: Building for Failure, Scale, and Change
A good architecture explains what happens when a request is repeated, a dependency slows down, or the business changes its mind. Here is how to make those decisions visible before production makes them urgent.
Start with a promise a user can recognize
A system design discussion can become a shopping list surprisingly quickly. Someone draws an API gateway, another person adds a queue, and the database gains a replica before anybody has explained what the customer is trying to finish. The picture looks sophisticated. The behavior remains uncertain. I prefer to begin with a sentence that a product owner and an on-call engineer would interpret the same way.
Consider a fictional service that lets business customers export account activity. A user requests a file, returns later, and downloads the result. The important promise is that the service produces the right account's data, makes progress visible, and does not create an uncontrolled number of duplicate exports when the user clicks twice. This example is an architectural exercise, not a claim about a particular employer's production system.
Before choosing infrastructure, I would ask what belongs in the file, which permissions apply, how recent the data must be, and how long the download remains valid. Those answers affect the storage model, authorization checks, processing schedule, and user interface. They also expose disagreements cheaply. Discovering that two teams mean different things by “current account activity” during a meeting is far less expensive than discovering it after customers have received inconsistent exports.
Separate the states that fail independently
An HTTP success response is an awkward substitute for a workflow model. The request can be accepted while generation has not started. Generation can complete while notification fails. A notification can arrive after the customer loses access. Treating all of those events as a single completed flag forces users and support engineers to guess which fact the application is actually reporting.
For the export service, I would keep a durable operation record with an identifier, tenant, input version, requested time, processing state, artifact location, and failure classification. Notification delivery would have its own state. An application can then say “your file is ready; the email is delayed” instead of pretending that a useful artifact does not exist. The operation record becomes the starting point for both the status page and investigation.
Write the valid transitions beside the diagram. Can a cancelled request become available? What happens when an artifact expires during a download? Can generation be retried after an input correction? If a transition requires an operator, name the permission and evidence that operator needs. A state model earns its place when it prevents an unsafe action or explains a confusing result, rather than merely adding attractive boxes to documentation.
Make retries preserve meaning
Imagine that the service stores an export request and then the connection closes before the client receives a response. The client cannot tell whether the request was accepted. Repeating the request is reasonable. Creating a new expensive export every time is usually not. I would give the submission a stable request identifier and define what repeating that identifier means within a tenant and a retention window.
The identifier cannot be a magic annotation pasted onto a controller. The persistence layer needs a uniqueness rule or another atomic mechanism so two concurrent submissions cannot both become the first request. The service must also distinguish a repeated request with the same input from reuse of the identifier for different input. Returning the previous operation can be correct in the first case and dangerously misleading in the second.
AWS's discussion of retries emphasizes two relevant limits: retries add load, and a timeout does not establish whether a side effect happened. For the export exercise, a bounded retry policy should therefore distinguish a failed computation from an unknown remote outcome. If a notification provider accepted a message but the response was lost, reconciliation may be safer than an immediate new send. The design should explain who owns that ambiguity and when it becomes an operational task.
Sources: AWS Builders Library: timeouts, retries, and backoff with jitter
Scale the constraint, not the drawing
Suppose export requests arrive faster than workers can finish them. Adding workers may help, but only while the underlying database and storage services can support the extra work. If every worker opens many connections and scans a large table, a larger worker fleet can make the entire application less responsive. Queue depth alone does not reveal which resource is limiting useful throughput.
I would measure arrival rate, completion rate, oldest pending operation, database wait time, connection usage, and the distribution of export sizes. A fictional workload with many tiny requests and a few huge ones might need separate processing lanes or per-tenant concurrency limits. The service can communicate an estimated wait rather than allowing one customer's large export to monopolize every worker. Those are product and fairness decisions as much as infrastructure decisions.
Capacity planning should state assumptions that can be challenged. What fraction of users exports at once? How much data is read for a completed file? What happens during a retry burst? A useful load exercise records the dataset, request mix, machine sizes, warm-up period, and observed failure point. “Handles scale” is too vague to validate. “Remains responsive under this workload, with this bottleneck and this admission limit” gives another engineer something concrete to reproduce and improve.
Design the change path while the system is small
A service that works today can still be expensive to change tomorrow. Imagine introducing a new export status while an older application version is still running. An old reader might reject the new value. A rollback might restart code that assumes a column cannot be empty. Deployment order has become part of the architecture, even if the original diagram never mentioned it.
I would document a compatibility sequence: add the new representation, deploy readers that tolerate both forms, begin writing the new form, verify the results, and remove the old form only after the compatibility window closes. The exact sequence depends on the application. The essential question is whether mixed versions can coexist safely and whether the team can stop between stages without corrupting meaning.
Recovery deserves the same attention. Redeploying an older binary does not undo a bad export sent to a customer or restore deleted source records. For each change, separate code rollback, data repair, and user communication. An operator needs to know which action is safe, which evidence to preserve, and when to stop. A design that explains only forward progress leaves the hardest decisions to the moment when the team has the least time to think.
Make observability answer the next question
A log line saying “export failed” is a beginning, not an explanation. Which operation failed? Which input version was used? Was the failure caused by permission, malformed data, an unavailable dependency, or resource exhaustion? Can the operation be retried safely? If the answer requires searching multiple systems with unrelated identifiers, investigation becomes a reconstruction exercise.
For this design, I would carry the operation identifier through submission, queue processing, storage writes, and notification attempts. Structured events would include stage and reason without exposing exported data. A dashboard would connect customer-visible completion time with internal symptoms. An alert on old pending operations is often more useful than an alert on raw CPU, because it describes a promise the service is failing to keep.
The OpenTelemetry signal model provides common vocabulary for traces, metrics, and logs. The architectural work is deciding which question each signal answers. A trace helps follow one attempt, a metric reveals a pattern across attempts, and a log can preserve a specific decision. Instrumentation should also have cardinality, privacy, and cost limits. Collecting everything indefinitely can produce an expensive system that still does not tell the operator what to do next.
Sources: OpenTelemetry: signals
A small review that changes the design
To make the export exercise concrete, imagine a customer submitting the same request from two browser tabs. The reviewer should be able to follow both attempts to one durable operation or explain why the product intentionally allows two. If the answer changes according to which application instance handles the call, the identity rule has not reached the place where concurrency is resolved. That is a useful finding before any load test begins.
Now let generation finish but delay the notification provider. The customer should still be able to discover the available file through an authorized status lookup. Next revoke access before the download. The completed artifact should not become a permanent permission grant. These three cases connect identity, workflow state, and authorization without requiring a complicated topology. They also reveal whether the API and interface describe the same promise.
I would record the expected outcome and the evidence from each check in the design review. A short table of those observations gives a future maintainer a useful starting point. When a later optimization introduces a cache or a second worker pool, the team can rerun the cases to check that performance work has preserved meaning. The diagram is now supporting an executable understanding of the service instead of serving as a static picture of its components.
Finish with a review somebody else can run
The most revealing design review is often a short exercise. Give a teammate an export that appears stuck and ask them to identify its authoritative state, decide whether repeating the work is safe, and explain what the customer should see. Let the teammate use the documentation and tools without private guidance from the original author. Missing context becomes visible immediately.
I would also test one concurrent submission, one stale permission, one oversized export, and one incompatible deployment sequence. These scenarios are deliberately small. Their value comes from connecting the user contract to implementation evidence. If a check fails, update the design and its operational instructions together. A polished diagram with an incorrect recovery procedure is less useful than a plain document that lets another engineer act safely.
Senior system design is the ability to keep important reasoning available as a product grows. The service may begin as one application and one database. It may later need independent workers or a different storage model. What should survive those changes is a clear explanation of the promise, the authority, the failure boundaries, and the evidence needed to change the system responsibly. That is the architecture I want a diagram to support.
Key takeaways
- Define the user-visible promise before choosing infrastructure.
- Give retries stable identity and make unknown outcomes visible.
- Measure the actual constraint and document mixed-version deployment behavior.
- Ask another engineer to operate the design using only its public contracts and runbook.