SSD QoS: Sources of Read Latency Tail

A conceptual notebook entry on why NAND SSD read latency has a long tail, and the specific mechanisms that produce it: die-level queuing collisions, the LDPC decode fallback ladder, and power/thermal-limited operation. This sits downstream of nand_overview.md (NAND op timing, error mechanisms) and ssd_bottleneck_analysis.md (§4 ECC engine, §9 queue depth, §10 power/thermal throttling). Numbers below are illustrative order-of-magnitude figures, not a specific drive's datasheet.

1. Why "Average Latency" Isn't the QoS Metric

Enterprise/data-center SSD QoS specs are written in percentiles — p99, p99.9, p99.99, sometimes p99.999 ("four or five nines") — not averages, because the mean read latency is dominated by the fast path and barely moves when rare-but-slow events occur. A drive can have a beautiful ~50 µs average read latency and still fail an SLA if 1 in 10,000 reads takes 2 ms.

That tail isn't one phenomenon — it's a ladder of qualitatively different fallback mechanisms, each one kicking in only when the previous one isn't enough, each rarer than the last, and each considerably slower than the last:

Stage Trigger Roughly how much slower than baseline
Fast path Normal hard-decision decode succeeds first try, no contention 1x (baseline t_R)
Read-read die conflict Another read already in flight on the same die ~2x
Write suspend Read lands on a die that's mid-program ~3x
Erase suspend Read lands on a die that's mid-erase ~7x
Read retry Hard decode fails at default reference voltage ~5-10x
Soft decode Read retry's best hard-decision attempt still fails ~20-50x
Full RAID/XOR recovery Soft decode still fails (dead page/plane/die) ~100x+

Sections 2-5 walk through the mechanisms behind each row, and §6 the effect of running power/thermally limited, which doesn't add a new rung to this ladder so much as it shifts the whole ladder up.

2. Read-Read Die Conflicts

2.1 Why dies collide

As covered in nand_overview.md §5, the die is the smallest independently-operable NAND unit — it has its own command queue, and (plane-level parallelism aside) can only be doing one operation at a time. If two outstanding host reads happen to target the same die, the second one queues behind the first: instead of paying one t_R (tens of µs), it pays roughly two.

Good FTL address striping (round-robin or hashed die/channel selection across the L2P mapping — ftl_l2p_overview.md §2) spreads sequential and random traffic evenly across dies specifically to keep this collision rate low. But at any given instant, with QD outstanding commands in flight, some collisions are inevitable just from the queue depth exceeding the number of independent dies available to absorb it.

2.2 Modeling it with the binomial distribution

Treat each of the QD outstanding read commands as an independent trial landing uniformly at random on one of N_dies dies (a reasonable first-order model when striping is good and access is not pathologically hot-spotted):

p = 1 / N_dies                                   probability a given command lands on a specific die

P(k hits on that die) = C(QD, k) * p^k * (1-p)^(QD-k)        Binomial(QD, p)

P(no conflict)   = (1-p)^QD
P(exactly 1 hit) = QD * p * (1-p)^(QD-1)
P(conflict, >=2) = 1 - (1-p)^QD - QD*p*(1-p)^(QD-1)

E[# contended dies] = N_dies * P(conflict on a given die)

This is the same shape as the classic birthday-paradox setup (QD "people," N_dies "days") — conflicts show up faster than intuition suggests as QD approaches N_dies, because it only takes any pair landing together.

Worked example — 8 channels x 8 dies/channel = 64 dies:

QD p = 1/64 P(conflict on a given die) E[# contended dies]
8 0.0156 0.64% 0.41
16 0.0156 2.53% 1.62
32 0.0156 8.90% 5.70
64 0.0156 26.42% 16.91

(computed by qos_read_latency_model.py, print_worked_example())

At QD32 on a 64-die drive, nearly 9% of dies are statistically expected to have a conflict at any instant — roughly 1 in 11 outstanding reads is waiting behind another op on the same die — this is why deep queue depths trade average throughput for a fatter conflict-driven tail, and why the levers that fight this are exactly "more independent dies/channels" (ssd_bottleneck_analysis.md §6) and "better address striping," not raw NAND speed. qos_read_latency_model.py's plot_qd_sweep_staircase() isolates this effect on its own: at a fixed 128-die, nominal-power config, sweeping QD from 1 through 128 moves only the read-read-conflict point (from 0% at QD1, where a conflict is impossible, up to ~26% at QD128) — every other stage in this ladder is independent of queue depth.

Limitation: this assumes uniform random targeting. Real workloads have locality — a sequential stream, a hot LBA range, or a small working set can concentrate traffic onto fewer dies than the binomial model assumes, making actual conflict rates worse than predicted; conversely, an FTL that is aware of in-flight commands and actively load-balances new ones away from busy dies can do better than pure-random. Treat the binomial number as a baseline, not a guarantee.

3. Write/Erase Suspend: Reads That Land on a Busy Die

3.1 The other two ways a die can be busy

§2 covered a read landing on a die that's already servicing another read. A die can just as easily be busy with a program (write) or an erase when a read arrives — and on a drive under any real write/GC load, that happens often enough to matter. At a given instant, a die is in exactly one of four states: idle, mid-read, mid-program, or mid-erase.

Unlike a read-read conflict, a read can't simply queue behind an in-flight program or erase and wait it out — t_PROG runs hundreds of µs to ~2ms and t_BERS runs several ms (nand_overview.md §6), and stalling every read behind those would be a QoS disaster. Instead, modern NAND and controllers support program-suspend and erase-suspend: the controller signals the busy die to pause at a safe internal checkpoint, services the read, then resumes the paused write or erase. The cost to the read is just that suspend/resume protocol overhead — write_suspend_time_us / erase_suspend_time_us — not the remaining program or erase time. Erase suspend costs more than write suspend: an erase is a coarser, longer-running operation on the whole block, so reaching a safe pause point takes longer.

3.2 A simple ratio model

Since a die is in exactly one of those four states, and assuming (as in §2.2) that an incoming read's target die is uniformly random:

P(hits a writing die)  = N_dies_writing / N_dies
P(hits an erasing die) = N_dies_erasing / N_dies

Worked example — 64 dies, nominal power, 4 dies concurrently writing and 1 concurrently erasing (a normal write/GC mix):

Cause Busy dies P(hit) Added latency (illustrative)
Write suspend 4 of 64 6.25% +100 µs
Erase suspend 1 of 64 1.56% +300 µs

(computed by qos_read_latency_model.py, write_suspend_hit_probability() / erase_suspend_hit_probability())

Worth noticing: at these die counts, a write-suspend hit (6.25%) is actually more common than a read-read conflict at QD16 (2.53%, from §2.2) — even though each write-suspend event costs more (100 µs added vs. ~50 µs). Contention from ordinary write/GC traffic can matter more to the tail than queue-depth-driven read-read conflicts, depending on the drive's write intensity.

Limitation: same uniform-random-targeting assumption as §2.2, plus an implicit assumption that write/erase scheduling is independent of where reads land. Real GC and write schedulers may deliberately spread program/erase activity across dies to avoid exactly this contention, or may cluster it (e.g., a sustained sequential write stream keeping the same handful of dies busy) in ways that push the actual hit rate either direction from this ratio estimate.

4. The LDPC Decode Ladder: Hard Decode → Read Retry → Soft Decode → RAID Recovery

Every NAND read comes back from the array as raw, noisy voltage-threshold data (raw bit error rate, RBER — see nand_overview.md §9 for the mechanisms that create it: read/program disturb, retention loss, cell-to-cell coupling, wear). The controller's ECC/LDPC engine (ssd_bottleneck_analysis.md §4) exists to correct that noise back to the original data, and it does so through an escalating series of fallbacks, only reaching for the next one when the current one fails.

  1. Hard decode (the fast path). The page is read once at the default reference voltage; each cell is thresholded to a firm 0/1 (a hard decision), and the LDPC decoder runs against that hard-bit codeword. This is fast — effectively just t_R plus inline decode latency — and succeeds whenever RBER is within the code's correction strength, which is the overwhelming majority of reads on a healthy drive.
  2. Read retry (voltage-shift search). If hard decode fails, the controller doesn't give up — the default reference voltage may simply not be well-centered between the voltage distributions anymore (retention loss and read disturb both drift the distributions over time, per nand_overview.md §9). The controller re-reads the same page at one or more alternate reference voltages from a precomputed retry table, hard-decodes again at each step, and stops as soon as one succeeds. Each step costs another full t_R; controllers may try anywhere from a handful up to a few dozen steps before escalating further, so this stage runs from a few hundred µs up depending on how many steps are needed.
  3. Soft decode. If no single reference voltage produces a clean hard-decision decode, the controller switches strategy entirely: instead of one threshold per cell, it reads the page several times at closely-spaced reference voltages around the best point found during retry, and combines the reads into multi-level soft information (log-likelihood ratios, LLRs) per bit — essentially "how confident am I this cell is a 0 vs a 1," not just a firm guess. A soft-decision LDPC decode (belief propagation / min-sum with real LLRs) is significantly more powerful than hard-decision decode for the same code, at the cost of several extra page reads plus meaningfully longer decoder run time — pushing this stage from hundreds of µs into the low milliseconds.
  4. Full RAID/XOR (RAIN) recovery — the last resort. If soft decode still can't recover the page — a truly excessive RBER, or a dead page/plane/die — the drive falls back to its internal redundancy stripe, commonly called RAIN (redundant array of independent NAND): an XOR parity relationship across dies or planes, conceptually the same idea as RAID 5. Recovering the failed page means reading every other member of its protection stripe (N-1 additional full page reads, ideally in parallel across dies, but still bound by the slowest one) and XOR-ing them together to reconstruct the missing data. This is correct but expensive — multiple page-read latencies plus XOR compute, typically landing in the multi-millisecond range — and it usually also triggers a "this block/plane looks unhealthy" signal to wear-leveling/bad-block management, since reaching this stage at all is a sign of a genuinely failing region, not just noise.
Stage Extra NAND reads Approx. added latency (illustrative) Approx. trigger rate on a healthy drive
Hard decode 0 (the original read) ~0 (inline) dominant case
Read retry 1 to ~dozens tens–hundreds of µs ~1e-4 – 1e-5 of reads
Soft decode several (3-7+) hundreds of µs – low ms ~1e-6 – 1e-7 of reads
RAID/XOR recovery N-1 (stripe width) low–high single-digit ms ~1e-8 – 1e-9 of reads

Each rung is both rarer and slower than the one before it — that's precisely what makes the tail of a read-latency distribution look like a rising staircase rather than a smooth curve when plotted (§7).

5. Power-Limited Operation: Shifting the Whole Distribution, Not Just the Tail

ssd_bottleneck_analysis.md §10 models power/thermal throttling as a duty-cycle reduction: Sustained_BW_thermal = Peak_BW x Duty_Cycle_allowed(T, PowerBudget). Under an active power limit — an NVMe power state, a PCIe slot power cap, or thermal throttling from insufficient cooling — the controller can't keep every channel and die active concurrently the way it can at full power. It responds by reducing effective parallelism: fewer channels/dies serviced simultaneously, lower NAND interface clock, or scheduled idle windows to hold average power under budget.

This is a fundamentally different effect than §2-4. Read-read conflicts, write/erase suspend hits, and the decode ladder are rare-event mechanisms — they lengthen the tail (p99.9 and beyond) while leaving the median almost untouched, because most reads never hit them. Power limiting instead reduces the concurrency available to service every read. By Little's Law (ssd_bottleneck_analysis.md §9, IOPS_QD = QD / Latency_per_op), if the same arrival rate now has fewer servers (active dies/channels) to spread across, the average per-op latency rises for the whole population — the entire distribution shifts right, median included, not just the far tail.

These mechanisms also compound rather than stay independent:

  • Fewer effective dies raises the binomial conflict probability. If power limiting means only, say, 32 of 64 dies are kept active at once, that's the same as halving N_dies in the §2 model — p doubles, and conflict probability rises sharply for the same QD, even though nothing about the host's access pattern changed.
  • Fewer effective dies also raises the write/erase-suspend hit rates. The same absolute amount of background write/GC traffic (§3) now lands on a smaller pool of available dies — halving N_dies roughly doubles both P(hits a writing die) and P(hits an erasing die), e.g. from 6.25%/1.56% to 12.5%/3.13% in the worked examples used throughout this doc.
  • Higher temperature raises RBER, feeding more reads into the decode ladder. Retention loss and several other error mechanisms in nand_overview.md §9 accelerate with temperature. A thermally-throttled drive is, almost by definition, running hot — so a larger fraction of reads fail hard decode and spill into read retry, soft decode, or (worst case) RAID recovery than on a cool, unconstrained drive.

The practical takeaway for a QoS spec: latency percentiles measured on an open bench with full airflow and no power cap are not representative of the same drive in a power- or thermally-constrained enclosure. A meaningful QoS number has to be measured (or at least modeled) under the actual power/thermal envelope the drive will run in, since that envelope shifts the baseline and fattens the tail at the same time.

6. Putting a Number to the Whole Picture

Rough numeric picture combining §2-5 into one workload snapshot (illustrative, not a datasheet). P(write suspend) / P(erase suspend) are each stage's own hit probability (§3.2); the rest are the same style of figure used throughout:

Regime Fast-path latency P(read-read conflict) P(write suspend) P(erase suspend) P(read retry) P(soft decode) P(RAID recovery)
Nominal power, QD16, 64 dies ~50 µs ~2.5e-2 ~6.25e-2 ~1.56e-2 ~1e-4 ~1e-6 ~1e-8
Power-limited / thermally throttled ~80 µs ~8.8e-2 ~12.5e-2 ~3.13e-2 ~1e-3 ~1e-5 ~1e-7

(computed by qos_read_latency_model.py, print_stage_table() for NOMINAL_REGIME / POWER_LIMITED_REGIME)

Note both the baseline shifting up and every downstream probability shifting up under the power-limited row — that's the compounding effect from §5, not independent changes. The read-read conflict figure roughly triples here (2.5%→8.8%) even though N_dies only halves — the binomial model is more sensitive to that ratio than a linear scaling, since conflict probability grows faster than 1/N_dies as QD gets closer to N_dies. Write/erase suspend, by contrast, use the simpler linear ratio model from §3.2, so they roughly double exactly as N_dies halves.

7. Visualizing the Tail: Latency vs. Probability

The natural way to plot this ladder is read latency on a log-scaled x-axis (values span ~50 µs to ~10 ms, over two orders of magnitude) against a probability on a log-scaled y-axis (values span ~1 down to ~1e-9, over nine orders of magnitude). Both axes need to be logarithmic here — a linear probability axis would squeeze every rare-but-important event into an invisible sliver near zero, which defeats the point of the chart.

What the y-axis actually is. Rather than plotting each stage's own isolated probability (which, once write/erase suspend are in the mix, doesn't decrease monotonically with latency — a write-suspend hit at 6.25% is more probable than a read-read conflict at 2.53% despite costing more), qos_read_latency_model.py treats all seven causes as mutually exclusive and exhaustive (a simplification — a read could in principle both wait on a die conflict and still fail hard decode afterward, but the model doesn't stack these) and plots a proper tail probability at each stage: P(read latency ≥ this stage's latency), i.e. its own probability plus every rarer/slower stage's. That's what keeps the plotted curve a well-behaved, monotonically non-increasing staircase.

The result is the classic SSD "tail latency staircase": a descending curve from high-probability/low-latency (fast path) down to low-probability/high-latency (RAID recovery), with seven points per curve — fast path, read-read conflict, write suspend, erase suspend, read retry, soft decode, RAID/XOR recovery — and the power-limited series sitting above and to the right of the nominal series at every stage.

qos_read_latency_model.py also has a second chart, plot_qd_sweep_staircase(), that holds power fixed at nominal and instead sweeps queue depth (QD 1/8/32/64/128 at a fixed 128 dies) — since QD only enters the model through the read-read conflict term, that single point fans out across the sweep while every other stage stays exactly fixed, isolating the queue-depth effect on its own.

Sources