Why I Wrote the Book
I wrote Practical System Design: Building Reliable Systems Through Production Failures to explore why systems that appear correct during design reviews fail under real production conditions. Drawing on my experience in research, robotics, cloud infrastructure, high-traffic consumer platforms, and financial services, I examine fourteen incident-based case studies covering database connection pools, retries, slow dependencies, caching, containers, autoscaling, replication, observability, circuit breakers, rate limits, load shedding, and deployment strategies. Through these examples, I aim to help engineers understand how workloads, latency, concurrency, and resource limits affect system behavior and use measurable evidence to identify failure causes and design more reliable systems.
Learning to Ask Better Questions
Production work changed the questions I asked during system design. Choosing a database, cache, messaging system, or cloud service matters, but the component name alone does not tell us whether a design can support its workload.
One example in the book begins with a service handling about 400 requests per second while each request occupies a database connection for roughly 70 milliseconds. That workload requires about 28 concurrent connections. At 560 requests per second and 105 milliseconds of connection occupancy, the requirement increases to about 59. A pool configured for 40 connections now has requests waiting even though the database technology and application architecture have not changed.
That calculation is simple, but it changes the investigation. Instead of saying the service is under high load, an engineer can identify the constrained resource, measure why demand for it increased, and evaluate whether a proposed correction changes the condition that produced the failure. I wanted the book to develop that habit of reasoning.
What COSMOS Contributed
My years at the COSMOS Research Center under Prof. Agrwal’s supervision played an important role in shaping how I approach engineering and problem-solving. Prof. Agarwal has brought computer science together with social and behavioral research, using large-scale data, network analysis, AI, machine learning, and computational methods to study online behavior and complex digital environments at COSMOS. Prof. Agarwal taught us to develop software that supports research by collecting, processing, and analyzing data, evaluating assumptions, and producing reliable tools for others to use. This experience helped me recognize that research and production engineering are closely connected, as reliability, observability, resource limits, deployment, and reproducibility become essential when researchers depend on software and data pipelines. I also learned the importance of using evidence to investigate problems, test hypotheses, and revise explanations based on observations. These research-driven practices continue to influence how I identify system failures, analyze performance, and develop reliable engineering solutions.
Writing the Book
I did not want to write a catalog of technologies or a collection of architecture diagrams. There are already many good resources for learning what a cache, message queue, load balancer, or circuit breaker is. I was more interested in what happens when those components interact under production conditions.
That meant asking questions such as: If a dependency slows from 80 milliseconds to two seconds, how much concurrency does the caller now require? If requests time out and retry, how many additional attempts reach that dependency? If thousands of requests miss the same cache entry, what reaches the database? If autoscaling adds application instances while the database or external provider has a fixed concurrency ceiling, what did the additional instances actually solve?
The writing process involved repeatedly tracing each incident from a design or configuration decision to a runtime condition, then to a resource or dependency, a measurable change, and finally the customer or operational consequence. I rewrote sections when they explained a concept without helping the reader decide what to measure or change.
The diagrams went through the same process. I wanted a latency trace, connection-pool model, cache flow, or deployment figure to help the reader answer a question. If a diagram only made the page look more technical, it did not belong in the book.
Why This Matters to COSMOS Researchers
This is where I think the book connects most directly with the work being done at COSMOS today. Under Prof. Agarwal’s leadership, the center studies complex, multi-platform information environments and develops computational tools for analysts, policymakers, and researchers. As data volume, sources, models, and AI components increase, the supporting infrastructure directly dictates whether results can be produced, reproduced, and trusted. For example, if a data pipeline hits a slow external API, unrestricted retries can consume workers needed by other sources, creating dataset gaps that look like data anomalies when the real cause is a timeout or exhausted connection pool. Metrics that track collection latency, queue duration, and API errors make these issues far easier to diagnose.
The same principles apply to computational experiments and overall research validity. Changes in package versions or runtime environments can alter analytical outcomes, making containers and controlled image promotion essential for reproducibility, while metrics like queue depth offer far clearer insight into processing bottlenecks than CPU utilization alone. COSMOS has a strong history of turning research into software tools with real social impact. Reliability engineering supports that mission by making technical infrastructure easier to measure, reproduce, and diagnose, giving researchers practical models to reason about these conditions without needing to become site reliability engineers.
What Comes Next
Publishing Practical System Design has strengthened my interest in exploring how AI can support production-risk analysis. I am developing ways for AI agents to review system architectures and operational evidence before a release, identify potential failure modes, and help engineers determine what requires further investigation. I plan to expand my educational work by teaching system design through real-world incidents, encouraging learners to examine measurements, identify system limitations, challenge initial assumptions, and use evidence to determine what happened and how to improve system reliability. These approaches reflect the research-driven problem-solving habits I developed at COSMOS, which continue to shape my engineering work today.