In-Process Synchronization and Where It Stops Working

The Setup Dispatching material to testing in a LIMS

The domain for this whole series is a Laboratory Information Management System. A sample arrives, gets split into aliquots (portions of material in separate tubes), and each laboratory test consumes a defined volume from an aliquot. The transfer of material to testing is the moment that matters: the LIMS must guarantee two invariants, and both are physical: you cannot pipette 500 µL out of a tube that holds 400.

• Consumed volume can never exceed available volume - no negative material.
• One dispatch decision corresponds to exactly one physical transfer - no double dispatch.

The model is deliberately small:

public class Aliquot
{
    public int Id { get; set; }
    public string SampleCode { get; set; } = default!;      // "SER-2026-018842-A1"
    public decimal AvailableVolumeMicroL { get; set; }      // remaining material, in µL
    public AliquotStatus Status { get; set; }               // InStorage, Depleted, Discarded
}

public class TestAssignment
{
    public int Id { get; set; }
    public int AliquotId { get; set; }
    public string TestCode { get; set; } = default!;        // "HBA1C", "TSH", "CRP" ...
    public string InstrumentRunId { get; set; } = default!; // which analyzer run gets the material
    public decimal VolumeConsumedMicroL { get; set; }
}

    

And here is the transfer service, the way it usually gets written on the first pass: read, validate, mutate, save:

public async Task<TransferResult> TransferToTestingAsync(
    int aliquotId, string testCode, decimal requiredVolumeMicroL)
{
    var aliquot = await _db.Aliquots.SingleAsync(a => a.Id == aliquotId);

    if (aliquot.Status != AliquotStatus.InStorage)
        return TransferResult.Rejected("Aliquot not available.");

    if (aliquot.AvailableVolumeMicroL < requiredVolumeMicroL)
        return TransferResult.Rejected("Insufficient material.");     // 1. CHECK

    aliquot.AvailableVolumeMicroL -= requiredVolumeMicroL;            // 2. ACT

    _db.TestAssignments.Add(new TestAssignment
    {
        AliquotId = aliquotId,
        TestCode = testCode,
        InstrumentRunId = _runAllocator.CurrentRunId,
        VolumeConsumedMicroL = requiredVolumeMicroL
    });

    await _db.SaveChangesAsync();                                     // 3. PERSIST
    return TransferResult.Ok();
}

    

Every unit test passes and code review approves it. The bug is not in any single line - it sits in the gap between the check and the write, and it only appears when a second caller runs concurrently.

The Race Making it fail on demand

Two technicians order tests against the same aliquot at the same moment. ASP.NET Core runs both requests in parallel on the thread pool. Instead of waiting for two humans to click at the same time, we can reproduce the race by hammering the endpoint:

// Aliquot 42 holds 900 µL. HBA1C consumes 500 µL.
// Only ONE of these 50 concurrent transfers may succeed.
var responses = await Task.WhenAll(
    Enumerable.Range(0, 50).Select(_ =>
        client.PostAsJsonAsync("/api/aliquots/42/transfers", new
        {
            TestCode = "HBA1C",
            RequiredVolumeMicroL = 500m
        })));

var succeeded = responses.Count(r => r.IsSuccessStatusCode);
Console.WriteLine($"Succeeded: {succeeded}");

    

A typical run against the naive service:

Seven confirmations for material that exists once. The exact number varies per run, which is typical for a race: it is nondeterministic, load-dependent, and never shows up in tests that run requests one at a time. Interleaved, it looks like this:

Note the subtlety at t5: request B doesn't write a negative number - it writes 400, computed from its own stale read. A's decrement is silently overwritten. That is a lost update: the database looks internally consistent even though 1000 µL was dispatched from a tube holding 900. The instrument runs dry anyway.

Fix #1 lock - why the compiler rejects it

The reflex fix is a lock around the whole method body:

private static readonly object _gate = new();

public async Task<TransferResult> TransferToTestingAsync(...)
{
    lock (_gate)
    {
        var aliquot = await _db.Aliquots.SingleAsync(a => a.Id == aliquotId);
        // ...
        // error CS1996: Cannot await in the body of a lock statement
    }
}

    

The compiler rejects this to protect you from a class of deadlocks. Monitor (which lock compiles down to) is thread-affine: the thread that enters must be the thread that exits. But await exists precisely to release the current thread and resume the continuation on whatever thread-pool thread is free. Owner enters on thread 14, continuation lands on thread 23, exit is attempted by a thread that never took the lock. The CLR forbids the whole construct instead.

The usual workaround is the dangerous part:

lock (_gate)
{
    // "fine, I'll just make it synchronous"
    var aliquot = _db.Aliquots.Single(a => a.Id == aliquotId);
    // ...
    _db.SaveChanges();   // sync-over-async in disguise on many providers
}

    

Fix #2 SemaphoreSlim - correct, but global

The async-native gate is SemaphoreSlim(1, 1). WaitAsync queues the continuation, not the thread - nothing blocks while waiting:

private static readonly SemaphoreSlim _gate = new(1, 1);

public async Task<TransferResult> TransferToTestingAsync(
    int aliquotId, string testCode, decimal requiredVolumeMicroL,
    CancellationToken ct = default)
{
    await _gate.WaitAsync(ct);
    try
    {
        var aliquot = await _db.Aliquots.SingleAsync(a => a.Id == aliquotId, ct);

        if (aliquot.Status != AliquotStatus.InStorage)
            return TransferResult.Rejected("Aliquot not available.");

        if (aliquot.AvailableVolumeMicroL < requiredVolumeMicroL)
            return TransferResult.Rejected("Insufficient material.");

        aliquot.AvailableVolumeMicroL -= requiredVolumeMicroL;
        _db.TestAssignments.Add(/* ... */);

        await _db.SaveChangesAsync(ct);
        return TransferResult.Ok();
    }
    finally
    {
        _gate.Release();
    }
}

    

The hammer now behaves:

The result is correct, but it comes at a cost:

• We serialized the entire laboratory. The gate is static: a transfer for aliquot 42 blocks a transfer for aliquot 87031, a serum tube blocks a urine tube, one slow database write queues every technician in the building. Under the same 50-request hammer, throughput drops to one transfer per DB round-trip - the queue is invisible until the day a morning batch of 3,000 orders lands.
• SemaphoreSlim has no owner. Unlike Monitor, nothing stops a stray Release() from a different code path silently raising the count to 2 - at which point the semaphore starts letting two callers in at once. The try/finally around Release() is what keeps the count correct.
• It is not reentrant. If anything inside the critical section calls back into a method that takes the same gate, you deadlock yourself - with no lock-owner diagnostics to tell you who is holding it.

Fix #3 Per-aliquot locking - serialize the tube, not the lab

The invariant lives on one aliquot. Two different aliquots have no shared state, so there is no reason their transfers should queue behind each other. What we want is a lock per key:

public sealed class KeyedAsyncLock
{
    private readonly ConcurrentDictionary<int, SemaphoreSlim> _gates = new();

    public async Task<IDisposable> AcquireAsync(int key, CancellationToken ct = default)
    {
        var gate = _gates.GetOrAdd(key, _ => new SemaphoreSlim(1, 1));
        await gate.WaitAsync(ct);
        return new Releaser(gate);
    }

    private sealed class Releaser(SemaphoreSlim gate) : IDisposable
    {
        private int _disposed;

        public void Dispose()
        {
            // idempotent: double-dispose must not double-release the gate
            if (Interlocked.Exchange(ref _disposed, 1) == 0)
                gate.Release();
        }
    }
}

// usage
using (await _aliquotLock.AcquireAsync(aliquotId, ct))
{
    // read - check - mutate - save, races only with itself
}

    

Now the hammer on aliquot 42 still yields exactly one success, while transfers for every other aliquot proceed in parallel, fully independent. Same correctness, none of the global queue. Two senior-level details hide in those 20 lines:

• The GetOrAdd race is benign, but you should know why. Under contention the value factory can run twice and construct two semaphores, but ConcurrentDictionary guarantees every caller receives the same winning instance; the loser is unreferenced garbage. This is only benign because constructing a SemaphoreSlim is side-effect-free. The moment a factory acquires resources (timers, file handles), this idiom silently leaks them.
• The dictionary only grows. One gate per aliquot ever touched, held forever. For a LIMS producing hundreds of thousands of aliquots a year, that is a genuine slow leak. The classic fix is striped locking - a fixed pool of gates, key hashed onto a stripe:

private readonly SemaphoreSlim[] _stripes =
    Enumerable.Range(0, 64).Select(_ => new SemaphoreSlim(1, 1)).ToArray();

private SemaphoreSlim GateFor(int aliquotId) =>
    _stripes[(aliquotId & 0x7FFFFFFF) % _stripes.Length];

    

Fixed memory and no cleanup problem, at the price of false contention: aliquots 42 and 106 share stripe 42 % 64 and will queue behind each other for no domain reason. With 64 stripes and realistic traffic the collision probability is negligible; the point is that you are now consciously trading memory against contention instead of leaking by accident.

Scale-Out Now deploy it the way production deploys it

Everything above rests on one unstated assumption: every transfer for aliquot 42 passes through this process's memory. Production breaks that assumption routinely:

# instance 1
dotnet run --urls http://localhost:5040

# instance 2 - same code, same per-aliquot locks, same database
dotnet run --urls http://localhost:5041

    

Split the same 50-request hammer across both ports:

Each instance allowed exactly one success, as designed. But there are two processes, each with its own gate for aliquot 42, and the race moved up one layer to the shared database. You don't even need a load balancer to hit this: a rolling deploy runs old and new instances side by side, IIS overlapped recycle does it on a single server, and a background worker touching the same table was never behind your semaphore to begin with.

Takeaways What to keep from Part 1

The material balance lives in the database, so the database has to arbitrate the race. Part 2 does exactly that with the mechanism that should be your default: optimistic concurrency in EF Core - a rowversion token on the aliquot, a DbUpdateConcurrencyException instead of a lost update, and a retry policy that doesn't stampede. The same load test, the same 900 µL, and no locks held.

← Back to the series overview