The Dial for Choosing What to Drop

In the previous post, we placed the time-bound gate behind the last point where delay accumulates, and from that we earned a structural guarantee: any request that goes out is, without exception, at or below the residence-time bound. And we ended the first post on this line — “Admission control at the entrance shrank the depth of the busiest queue from tens of thousands to hundreds, and as the queue got shorter, so did what the time gate had to discard.” This post is the story of designing that entrance.

Cutting is not the same as reducing inflow

A time-bound gate cuts late requests whose timeout is imminent. But trimming what has already aged inside the system is a wholly different problem from reducing the load that enters the system in the first place. The gate does clear out expired requests and relieve the queue somewhat, yet it never decides which requests the system takes in.

Still, the two meet at one point. When the queue is shorter, requests wait less; when they wait less, fewer of them age past the bound and get discarded. So on top of the time axis, we decided to add one more layer that sheds load right at the system entrance.

The admission decision must come before parsing

Filtering out low-value requests at the system entrance is called admission control. It is of a piece with priority-based load shedding and with filtering out invalid traffic (IVT). But we had one strict constraint: the admission decision had to be made before a request entered the queue — before we had even parsed the payload. Only then could we block, right at the entrance, the root cause of a crowded queue.

Before parsing, the cost of judging a request’s value has to be extremely cheap. The moment you crack a request open to run a sophisticated value model, that computation becomes another bottleneck. So we took payload size as our value signal. Size is known at the network layer without any parsing, so we can assign a value grade with a single comparison and no extra computation.

On top of that we added logic that adapts to system load. Under normal conditions the threshold is zero, so every request passes; there is nothing to filter, so there is no cost. Once the queue begins to fill, the threshold rises and starts cutting the lowest-priority requests first. To keep the threshold from oscillating on a momentary traffic spike, we gave the entry and release conditions a buffer, so a transient surge is absorbed smoothly.

We hit a cliff

Up to here it looked flawless, exactly as designed. Then we turned this “dial” for real, and an unexpected problem surfaced.

The size-based grade we first designed was a staircase: it climbed one step each time the size doubled. The trouble was that in the traffic data we used for validation, request sizes clustered tightly in one range. As a result, raising the threshold by a single grade sent the drop rate shooting straight up from 33.75% to 77.13%. Turning the dial one notch made 43.37 percentage points of traffic vanish at once. (This extreme cliff figure is specific to that validation data.)

Even when we wanted to “drop only about half the traffic,” there was no intermediate step to do it with. One notch down dropped too little; one notch up dropped too much. The dial’s notches were so coarse that the controller — which has to fine-tune the threshold against load — could not find its target and had no choice but to swing between the two extremes.

We split the steps finer

The fix was to split a single grade into finer pieces. By reading a few more of the high-order bits within the same grade, we can impose a fine ordering among requests. That turns the coarse double-the-size staircase into a much denser set of dial notches.

The key is that this refinement costs almost nothing. Without division or floating point, we just lay a few more bit operations on the size value already in memory. Once the notches were this dense, we could land within 1 percentage point of the “drop half” point that had been out of reach before.

The two axes help each other

We confirmed the effect in an experiment that reproduced overload. When we switched on admission control at the entrance, the depth of the jammed queue dropped comfortably from tens of thousands to hundreds. And as the queue got shorter, the late requests the earlier time-bound gate had to discard on timeout also fell sharply. (The correlation itself is clearly demonstrated, but absolute figures such as queue depth and the size of the drop-rate reduction are specific to this test environment.)

The time axis and the value axis are different yardsticks. The former looks at how delayed a request is; the latter, whether the request is worth processing at all. That the two are orthogonal and yet in perfect synergy — that regulating inflow by value at the entrance also lightens the burden the time axis has to carry — is what we confirmed by measurement. On the normal processing path, the extra computation this admission-control gate introduced was so slight it could not be told apart from the baseline noise.

The limit, stated plainly

One limit, though, has to be stated clearly. “Request size” is only an approximation of value, not the absolute truth. A larger request tends to be higher-value, but there are always exceptions.

To probe this limit, we deliberately built the worst case. We pushed in validation data engineered so that payload size and true value ran inversely (mixing in many small but high-value requests), and everything admission control cut away was one of those small, precious high-value requests. As long as size is the filtering criterion, this structural flaw is unavoidable. That is why, when this system is adopted, we left careful measurement of how strongly payload size and value correlate in real live traffic as a mandatory item for the pilot.

In one line

If the dial you turn has coarse, widely spaced notches, the control system cannot settle and it oscillates. The criterion for shedding traffic must have resolution fine enough to hit the target you actually want.

We build a low-latency traffic filtering system that structurally controls tail latency for real-time bidding infrastructure. Our core filtering method is patent pending.

If you want to test this gate on your real service traffic with no impact, you can start with a traffic-mirroring-based shadow-mode pilot.