Where Did 18.5 Milliseconds Come From

In real-time bidding, a response that misses tmax is worth zero. No matter how sophisticated the bidder, a bid that arrives after the deadline is simply thrown away. So the number I chased was not the average but the tail latency. This post is the record of tracking down one such tail.

Microseconds and milliseconds sat side by side

I kept our own pipeline under sustained overload, with the bid decision and value filtering we were developing both switched on. As I went through the logs, I noticed something interesting. Our decision computation itself finished in microseconds (µs), yet the tail latency had stretched to 18.5 milliseconds (ms) at p99.9.¹

The coexistence of these two numbers was the evidence that pointed to the cause. If the delay had come from the computation itself, the decision time too would have grown into milliseconds — but the decision still finished in microseconds. In other words, what was late was not the decision but the stage after it. Once decided, requests waited their turn in the send queue and were left at risk of expiring there.¹

Not a problem of value, but of time

The usual response runs along two lines. You throttle incoming load with admission control, or you shed low-value requests first with priority-based load shedding to shorten the queue. Filtering out invalid traffic (IVT) belongs to the same family. All three are valid, and all three address a question of value — what to process. Shorten the queue and the tail shrinks with it. But none of them guarantees an upper bound: the moment inflow exceeds even the shedding capacity, the survivors start aging again.

What we faced was not a problem of value but of time. Even a fresh, high-value request becomes useless if it ages past the deadline in the queue. So what we needed was not a filter that looks at value, but a gate that checks how long a request has waited. Time and value are two orthogonal axes. Where value shedding defers what matters less when it can no longer keep up, a time gate cuts what is late regardless of value. In later work we made the value axis its own separate layer as well. Admission control at the entrance shrank the depth of the busiest queue from tens of thousands to hundreds, and as the queue got shorter, so did what the time gate had to discard. That story is for the next post.

We added a time-bound gate — and moved it three times

If the culprit was queue waiting, the remedy was clear: a time-bound gate that will not release any request whose time-in-system has passed an upper bound T. The question was where to put the gate, and we had to move it three times.

First, before the decision. Just before processing, we measured the request’s age (time elapsed since arrival) and dropped it without deciding if it had passed the bound. That was right for saving computation. But this gate binds only the age at the moment of the decision. After the decision, the request aged again in the queue. The very stretch where the 18.5 milliseconds accumulated sat downstream of the gate.

Second, at the point of handoff. We moved it past the decision, to the point where a request is pulled from the queue to be sent out. Now the wait after the decision was included too. Still, a blind spot remained: the residence in the final buffer, where the request waits for physical transmission after being pulled, was not captured. When the bottleneck was exactly that buffer, a request could clear the gate and still age further there.

Third, just before physical transmission. Right before the request actually hits the wire, we measured the total residence time from arrival to that moment. The final buffer stretch that the first two positions had missed now fell inside the bound. The gate now stood behind the last point where a bid’s aging was created.

A guarantee bound by structure

At the third position, the gate becomes a simple proposition. A request that goes out has, by definition, a residence time at or below the bound. A request whose age has passed T has no path out at all. Either it is dropped, or it goes out with a residence time at most T — and there is nothing in between.

This is not a statistical guarantee but a structural one. It is not that the average is good or that things are usually fast; it is that because the gate sits behind the last aging point, it cannot be otherwise. When we replayed the arrival timing of real pilot traffic onto the gated send path, the residence of delivered requests never exceeded the bound (zero violations). The fraction discarded changed with where we set the bound T — that absolute value is specific to that traffic and that environment — but the structure, that an outgoing request is at or below the bound, held regardless of the load regime.²

In one line

The gate must sit behind the last aging point. Put it in front, and it misses the aging that follows. That one line is why we moved the gate three times.

We build a low-latency traffic filtering system that structurally controls tail latency for real-time bidding infrastructure.

If you want to test this gate on your real service traffic with no impact, you can start with a traffic-mirroring-based shadow-mode pilot.