If you've ever stared at a frozen application, a queue of unprocessed database transactions, or a server that's using less than 10% of its CPU but still can't get work done, you've probably encountered a deadlock. It's one of the most frustrating performance issues for developers and sysadmins, because it doesn't leave obvious error logs, and it usually pops up during peak traffic when you can least afford downtime. Learning how to fix deadlock quickly and permanently can save you hours of after-hours troubleshooting, lost revenue, and frustrated users. This guide skips the overly academic jargon and focuses on real, tested methods you can apply to any environment, from a small side project server to a large enterprise distributed system.
How to Fix Deadlock: Start With Proper Detection for Your Environment
Before you can fix any deadlock, you need to make sure you're actually dealing with a deadlock, not a different issue like a memory leak, network timeout, or infinite loop. Deadlocks have very specific traits that set them apart: two or more processes are each holding a resource the other needs, and neither will release their held resource until they get the one they're waiting for. Zero progress for affected processes even when system resources are free is the biggest tell that you're dealing with a deadlock, not another performance bug.
For operating system-level deadlocks, start with built-in monitoring tools: on Linux, you can use ps aux to look for processes in the 'D' (uninterruptible sleep) state, which usually means they're waiting on a resource like a disk or network lock. You can also use pidstat to track which processes are holding file locks or semaphores. For database deadlocks, almost every modern relational database has built-in deadlock logging. For MySQL and MariaDB, the InnoDB status output will show you exactly which two transactions are deadlocked, what locks they're holding, and what locks they're waiting for. For PostgreSQL, the pglocks view combined with pgstat_activity will show you the same details.
Don't skip this detection step and jump straight to restarting services – if you don't capture the deadlock details when it happens, you'll have no way to prevent it from happening again. I learned this the hard way a few years ago, when I restarted a frozen e-commerce server mid-Black Friday to get it back online fast, but I didn't save the deadlock logs. The same deadlock happened again 4 hours later, and I had to spend twice as long troubleshooting it the second time.
Immediate Deadlock Recovery Methods for Urgent Outages
Once you've confirmed you have a deadlock, your first priority if you're dealing with a user-facing outage is to resolve it as fast as possible, then worry about permanent fixes later. The most common and reliable recovery method is terminating one of the deadlocked processes. You don't want to terminate random processes, though – you need to pick the one that will cause the least damage when it's rolled back.
Prioritize terminating processes that have made the least progress, or that are running non-critical workloads. For example, if you have a deadlock between a customer's checkout request and a nightly inventory report, terminate the inventory report. It's far easier to rerun a report later than it is to recover a lost customer order. For databases, most have automatic deadlock recovery enabled by default, which will kill the transaction with the smallest rollback cost automatically. If you've disabled that feature for some reason, you can manually kill the deadlocked session using the database's KILL command.
A less common recovery method is resource preemption, where you take a lock away from one process and give it to another. This only works for specific use cases, like read-only locks, and you have to make sure the process you're taking the lock from can roll back its changes safely without corrupting data. You should only use preemption if you can't terminate any of the deadlocked processes without critical business impact.
Permanent Deadlock Prevention Techniques to Avoid Recurrence
Once the immediate outage is resolved, you need to put fixes in place to make sure the same deadlock never happens again. All deadlocks are caused by four core conditions, called the Coffman conditions: mutual exclusion, hold and wait, no preemption, and circular wait. You don't need to memorize these, but if you break even one of them, you'll eliminate deadlock risk entirely. The easiest and most effective prevention tactics are listed below:
- Enforce a consistent lock acquisition order across all services and codebases. For example, if you're working with user records, order records, and inventory records, make a rule that every service must lock user records first, then order records, then inventory records, no exceptions. This breaks the circular wait condition entirely, because no process will ever hold a later lock while waiting for an earlier one. I've worked with teams that cut their deadlock incidents by 90% just by implementing this one rule, even if they didn't change any other part of their code.
- Set realistic timeouts for all lock requests. If a process can't acquire the lock it needs within a set timeframe (usually 10 to 30 seconds for user-facing workloads), it should automatically release all locks it's currently holding, wait a short random interval, and retry. This breaks the hold and wait condition, because no process will hold locks indefinitely while waiting for another one. Just make sure your timeouts are long enough for normal operations, so you don't end up with unnecessary failed requests.
- Avoid nested locks wherever possible. If you can rewrite your code to request all required locks upfront, instead of grabbing one lock, doing some work, then grabbing another, you'll eliminate most hold and wait scenarios. This is a great fix for small to mid-sized applications that don't have extremely high concurrency requirements.
- Replace unnecessary exclusive locks with shared locks for read-only operations. If multiple processes just need to read data, not modify it, they can all hold a shared lock at the same time instead of waiting for an exclusive lock. This reduces the number of conflicting lock requests dramatically, and it's a low-effort change for most codebases.
If you're working with a distributed system, you can also use a distributed lock manager with built-in deadlock detection, but that's usually only necessary for very large, high-concurrency environments. For most teams, the four tactics above will eliminate nearly all deadlock risk with minimal effort.
Common Deadlock Fix Mistakes You Should Avoid
While fixing deadlocks, there are a few common mistakes that can make your problem worse, or create new issues you didn't have before. First, don't just restart your entire server or database every time you hit a deadlock. This will resolve the immediate deadlock, but it also kills all other running processes, rolls back all in-progress transactions, and erases all the deadlock logs you need to fix the root cause. It's a temporary band-aid that will cost you more downtime in the long run.
Second, don't increase lock timeouts to try to reduce deadlock frequency. Longer timeouts mean processes hold locks for longer periods of time, which actually increases the chance of conflicting lock requests and deadlocks. Stick to short, reasonable timeouts that match your workload's normal performance. Third, don't disable built-in deadlock detection to save a small amount of performance. The overhead of deadlock detection is usually less than 1% of system resources, and it's well worth it to avoid hours of downtime.
I worked with a client a few years ago that disabled InnoDB's deadlock detector because they thought it was slowing down their checkout flow. A few months later, during their busiest sale of the year, 200+ checkout transactions got deadlocked, and there was no automatic recovery. They had to manually kill every deadlocked transaction, and they lost over $50,000 in revenue during the 3-hour outage. Finally, don't ignore deadlocks that only happen rarely. Even if a deadlock only happens once a month, it's a sign of a flaw in your lock logic, and it will almost certainly happen more often as your traffic and user base grows.
At the end of the day, learning how to fix deadlock doesn't require a graduate degree in computer science. It just requires a structured, calm approach: first, detect the deadlock and capture all the details you can about which processes are involved and what locks they're holding. Second, resolve the immediate outage with targeted recovery, like terminating the lowest-impact deadlocked process. Third, implement permanent prevention measures to break one of the core deadlock conditions, so the issue never comes back. You don't have to implement every prevention tactic at once – start with the low-effort, high-impact fixes like enforcing a consistent lock order and adding lock timeouts, and you'll see most deadlock issues disappear before they impact your users. If you're still dealing with frequent deadlocks, take the time to map out every lock acquisition path in your system. 9 times out of 10, the problem is a small, easily fixed gap in your existing lock rules that you missed during development.