HoL blocking of critical resources in HTTP/2

[Kyriakos Zarifis <ky...@gmail.com>]

Hi,

TL;DR:

I've been experimenting with HTTP/2 Push and Prefetch to preposition
content on the client before it requests it, and specifically I want to
evaluate the pitfalls of prepositioning objects aggressively, with regards
to how it could impact the delivery of higher priority objects.

While experimenting with post-onLoad prepositioning mechanisms to optimize
next page navigations (mostly with prefetch/prerender), I noticed that
pending prepositioned objects can considerably delay the next page; if the
next page navigation is triggered while prepositioned objects are in
flight, the next page HTML will be backed behind a considerable part of
them on the server.

From what I can tell, object/stream priorities cannot help since the HoL
blocking happens at the SSL socket where the server can't preempt data that
it has already written.

So I wonder if there could be some logic on the server that controls how
aggressively it writes low-priority frames to the SSL socket, making sure
they are being drained before pushing more and bloating the buffer (and
causing critical objects that are requested soon after to be delayed).



Details:

To evaluate the impact, I tried a simple, extreme scenario where I trigger
the Prefetch (low-priority) of a large image after the onLoad of page A
fires, and soon after it fires and the image is being downloaded I manually
navigate to page B. In this case it takes as long as it takes for the image
to download before the next page starts showing. That was many seconds in
my case since it was a purposefully large (5MB) image and bad connection
(4G, ~200ms from server).

I also tried prefetching 8 copies of the same image, over a fast connection
(cable, 20ms to the server). I navigate to page B while the first few
prefetched images are being downloaded. In this case page B's HTML arrived
900ms after the request was sent. Looking at Chrome's event log, the HTML
(stream weight 256) was indeed prioritized over the prefetched images
(weight 110): the HTML arrives in the middle of the 4th prefetched image -
but there was still more than 1xBDP worth of data that arrived during the
900ms between the HTML request-response.

I believe that these are H2 frames that have been written for sending (but
not yet sent on the wire), where the server can no longer preempt them so
anything new (regardless of priority) can't be sent on the wire before the
buffer is drained.

My hypothesis was that priorities are honored at the application level -
i.e. the HTML frames are interleaved into the socket as soon as the request
arrives (I think I've now verified that from both server and chrome logs) -
but the socket doesn't keep up at draining the previously written low-prio
frames. I've also observed that the HTML's delay is in multiples of RTTs.
So I guess I'm just describing an instance of bufferbloat, which is likely
to happen when many/large low-prio requests arrive at the server before a
critical one (as is the case with prepositioning)

So I think some Head-of-LIne blocking (which H2 is supposed to do away
with) is happening even though priorities are honored in the application
layer. As a result the client receives the critical object (much) later
than it could have.

I considered tweaking the browser to open a second connection for
prefetch/pushed/otherwise low-prio content, but this goes against the
goals/purpose of HTTP/2. I believe that the right place to tweak this would
be the server. I think the application does things right as far as it
knows, i.e. frames leave the application layer in OK priority, but not the
wire.

What do you think would be the right way to fix this (assuming it is indeed
something that needs fixing)? Do you think there could be some additional
logic that controls how aggressively the server writes to the SSL socket,
that checks how much of low-prio data has been drained from the socket
before writing more, to avoid bloating the buffer on the server side with
low-priority frames which more critical frames will have to wait behind
when they become available?

I've started looking at httpd's source code to get a sense of how big of a
change this would be, but since I haven't looked at the code before I
wanted to get your thoughts on whether this makes sense first.

Thanks and sorry for the length