If you work in system administration, software testing, or backend development, you’ve probably encountered unexpected deadlocks that grind critical workflows to a halt. The best way to prepare for and mitigate these costly outages is to intentionally replicate them in a controlled environment, and that’s exactly where knowing how to build seven deadlock scenarios tailored to your tech stack comes in. I’ve spent the past 8 years testing distributed systems and database performance, and I’ve found that building these test deadlocks is the single most effective way to validate your detection and recovery tools before a real outage hits. You don’t need fancy lab equipment to run these tests either—most can be run on a standard laptop with a virtual machine or local database install.
Core Prerequisites Before You Build Seven Deadlock Scenarios
First, make sure you understand the four core Coffman conditions required for any deadlock to occur: mutual exclusion, hold and wait, no preemption, and circular wait. You don’t need to memorize these for testing, but they’ll help you tweak scenarios if your initial test doesn’t trigger a deadlock as expected. When you build seven deadlock scenarios that match your actual production stack, you get far more actionable results than using generic test cases from online tutorials.
Never run these tests on production systems, even if you think you’re running a small, low-impact test. Even a simple row-level database deadlock can lock up critical user workflows if it accidentally runs on a live instance. Always isolate your test environment from any shared storage or network resources. Even small test deadlocks can lock up shared file systems if you’re not careful, and you don’t want to disrupt other team members’ work just to run your tests. If you’re using cloud resources, spin up a temporary VPC with no access to production assets to run all your tests.
7 Deadlock Scenarios You Can Build in Under an Hour Each
Each of these scenarios targets a common real-world deadlock cause, so you can test different parts of your stack and cover most outage risks your team might face. You can adjust timeouts, lock settings, and resource limits to match your exact production configuration for more accurate test results.
- Single OS thread deadlock: Set up two processes that each lock a dedicated system resource, then request the resource held by the other, with no timeout settings enabled. This is the simplest deadlock to replicate for testing basic OS detection tools.
- Database row-level deadlock: Write two SQL transactions that each update a separate row in the same table, then attempt to update the row locked by the other transaction before committing. Most default database setups will detect this automatically, but you can adjust lock timeouts to test slower detection workflows.
- Distributed system network deadlock: Spin up two microservices that each require a response from the other to complete a request, with no circuit breaker rules configured. This mimics the common cross-service deadlocks that take down e-commerce checkout flows during peak traffic.
- File system deadlock: Have two processes each lock a separate file on a shared network drive, then attempt to read the file locked by the other process. This is a common overlooked deadlock for teams that use shared storage for batch processing jobs.
- GPU resource deadlock: Write two small machine learning scripts that each reserve a portion of GPU memory, then request the remaining memory held by the other script to finish training. This is a frequent pain point for data science teams running shared GPU clusters.
- Cache invalidation deadlock: Set up two cache write operations that each lock a separate cache key, then attempt to update the key held by the other operation before writing to the core database. This often slips past testing because cache locks are usually set to expire quickly by default.
- Print spooler deadlock: Configure two print jobs that each reserve access to a separate printer peripheral, then request access to the other printer to complete a dual-sided print job. This is a classic legacy system deadlock that still plagues office network setups.
You don’t have to build all seven at once, of course. Start with the scenarios that align most closely with the systems your team manages. For example, if you work on e-commerce backend systems, start with the database and distributed service deadlocks before moving on to GPU or print spooler tests. I usually recommend teams run through one scenario a week during low-traffic periods to avoid overwhelming testing resources.
Common Mistakes to Avoid When Building Test Deadlocks
First, don’t forget to set a hard kill switch for all your test environments. I’ve made the mistake of skipping this step early in my career, and I ended up having to hard reset my entire work laptop after a test file system deadlock locked up all my local storage. Set a 10-minute auto-reset rule for all test instances so you don’t get stuck cleaning up a mess for hours after your tests are done.
Next, don’t modify default system settings unless you’re intentionally testing edge cases. If you turn off default deadlock detection for your database to test slower recovery, make sure you flip that setting back as soon as your test is complete. Document every change you make to your test environment so you can replicate results accurately later, and so other team members can run the same tests without having to reverse-engineer your setup.
Another common mistake is running multiple deadlock tests at the same time. If you run a database deadlock test and a file system deadlock test in the same environment, you won’t be able to tell which event triggered your detection tool alerts. Run one test at a time, and wait for all resources to clear before starting the next one. You should also clear all logs between tests to avoid mixing up data from different runs.
How to Use Your Test Deadlocks to Improve System Reliability
Building the deadlocks is only half the work. The real value comes from running your existing detection and recovery tools against these test scenarios to find gaps in your workflow. For example, if you run the distributed service deadlock test and your circuit breaker tools don’t trigger within 30 seconds, you know you need to adjust your timeout rules to avoid outages during peak traffic.
Test both automated recovery and manual recovery workflows for every scenario. Most teams only test automated recovery, but if your automated tool fails, your on-call team needs to know how to resolve the deadlock quickly without causing data loss. I usually run each scenario twice: once to test the automated tooling, and once with the automated tools turned off to let the on-call team practice manual resolution.
You can also use these test deadlocks to train new team members on how deadlocks work in real systems, instead of just teaching them the theoretical Coffman conditions in a onboarding presentation. I’ve run these tests with new SRE hires for years, and it cuts down their time to resolve real deadlock outages by roughly 60% in their first 3 months on the job. You can even add custom scenarios that match unique parts of your workflow, such as custom API lock rules or proprietary hardware resources, to make your training even more relevant.
Once you’ve run all your tests and updated your recovery workflows, you can revisit and tweak these scenarios as your tech stack evolves. New tools, service integrations, or database upgrades can introduce new deadlock risks, so running a round of tests after every major update is a smart way to stay ahead of outages. Learning how to build seven deadlock scenarios for your specific stack doesn’t just help you avoid costly downtime—it also gives your team the confidence to resolve unexpected issues quickly when they do pop up. Start with the simplest scenario first, and you’ll have a full test suite built out before you know it.