You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by Ryan Schmitt <rs...@apache.org> on 2019/12/12 07:25:30 UTC
Update on HPACK errors
I've discovered a few things about the HPACK errors I'm seeing during
integration testing:
- They are deterministic: they always occur on the
first RegisterStreamConsumer call, which is usually assigned streamId 17 or
so on the h2 connection.
- They are completely unaffected by disabling Huffman encoding.
- The RegisterStreamConsumer call succeeds when retried (automatically, by
the SDK's default retry logic) on a new connection.
I think these facts strongly suggest some sort of client-side corruption of
the connection state, causing the client to eventually write a corrupt
HEADERS frame that the server rejects. I've posted GitHub gists with full
debug logs for the integration tests with Huffman encoding disabled [1] and
enabled [2]. I don't think debug logs are an especially useful format in
this case, so tomorrow I'll work on dumping just the raw binary sequence of
HEADERS frames (for the whole connection) to a dedicated file, which we can
then shove up some other HTTP/2 implementation (starting with Netty
4.1.29.Final).
By the way, the test I'm running is [3], modified to run with my
apache-async-client [4].
[1]
https://gist.githubusercontent.com/rschmitt/bd996852c1bb8b56341051e585a4ed93/raw/c0c8bb6d337de3a881ace9bb52ebd55f4bb29dd3/huffman-disabled.txt
[2]
https://gist.githubusercontent.com/rschmitt/311e8413d88ed7cf8450e844fb74df86/raw/66b559eca0eb9bf5d2192a817881a9efdd3c2c1d/huffman-enabled.txt
[3]
https://github.com/aws/aws-sdk-java-v2/blob/master/services/kinesis/src/it/java/software/amazon/awssdk/services/kinesis/SubscribeToShardIntegrationTest.java
[4] https://github.com/aws/aws-sdk-java-v2/pull/1543
Re: Update on HPACK errors
Posted by Ryan Schmitt <rs...@apache.org>.
I've cracked it. I'll soon send out a PR, which will be self-explanatory.
On Thu, Dec 12, 2019 at 1:21 AM Oleg Kalnichevski <ol...@apache.org> wrote:
> On Wed, 2019-12-11 at 23:25 -0800, Ryan Schmitt wrote:
> > I've discovered a few things about the HPACK errors I'm seeing during
> > integration testing:
> >
> > - They are deterministic: they always occur on the
> > first RegisterStreamConsumer call, which is usually assigned streamId
> > 17 or
> > so on the h2 connection.
> > - They are completely unaffected by disabling Huffman encoding.
> > - The RegisterStreamConsumer call succeeds when retried
> > (automatically, by
> > the SDK's default retry logic) on a new connection.
> >
> > I think these facts strongly suggest some sort of client-side
> > corruption of
> > the connection state, causing the client to eventually write a
> > corrupt
> > HEADERS frame that the server rejects. I've posted GitHub gists with
> > full
> > debug logs for the integration tests with Huffman encoding disabled
> > [1] and
> > enabled [2]. I don't think debug logs are an especially useful format
> > in
> > this case, so tomorrow I'll work on dumping just the raw binary
> > sequence of
> > HEADERS frames (for the whole connection) to a dedicated file, which
> > we can
> > then shove up some other HTTP/2 implementation (starting with Netty
> > 4.1.29.Final).
> >
> > By the way, the test I'm running is [3], modified to run with my
> > apache-async-client [4].
> >
>
> Hi Ryan
>
> I will try to find time and help you with the fix whichever way you
> deem helpful. However I would like to see if I can reproduce the same
> issue with other HTTP/2 servers such Apache HTTPD, Ngnix or Jetty.
>
> Oleg
>
> > [1]
> >
>
> https://gist.githubusercontent.com/rschmitt/bd996852c1bb8b56341051e585a4ed93/raw/c0c8bb6d337de3a881ace9bb52ebd55f4bb29dd3/huffman-disabled.txt
> > [2]
> >
>
> https://gist.githubusercontent.com/rschmitt/311e8413d88ed7cf8450e844fb74df86/raw/66b559eca0eb9bf5d2192a817881a9efdd3c2c1d/huffman-enabled.txt
> > [3]
> >
>
> https://github.com/aws/aws-sdk-java-v2/blob/master/services/kinesis/src/it/java/software/amazon/awssdk/services/kinesis/SubscribeToShardIntegrationTest.java
> > [4] https://github.com/aws/aws-sdk-java-v2/pull/1543
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>
Re: Update on HPACK errors
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2019-12-11 at 23:25 -0800, Ryan Schmitt wrote:
> I've discovered a few things about the HPACK errors I'm seeing during
> integration testing:
>
> - They are deterministic: they always occur on the
> first RegisterStreamConsumer call, which is usually assigned streamId
> 17 or
> so on the h2 connection.
> - They are completely unaffected by disabling Huffman encoding.
> - The RegisterStreamConsumer call succeeds when retried
> (automatically, by
> the SDK's default retry logic) on a new connection.
>
> I think these facts strongly suggest some sort of client-side
> corruption of
> the connection state, causing the client to eventually write a
> corrupt
> HEADERS frame that the server rejects. I've posted GitHub gists with
> full
> debug logs for the integration tests with Huffman encoding disabled
> [1] and
> enabled [2]. I don't think debug logs are an especially useful format
> in
> this case, so tomorrow I'll work on dumping just the raw binary
> sequence of
> HEADERS frames (for the whole connection) to a dedicated file, which
> we can
> then shove up some other HTTP/2 implementation (starting with Netty
> 4.1.29.Final).
>
> By the way, the test I'm running is [3], modified to run with my
> apache-async-client [4].
>
Hi Ryan
I will try to find time and help you with the fix whichever way you
deem helpful. However I would like to see if I can reproduce the same
issue with other HTTP/2 servers such Apache HTTPD, Ngnix or Jetty.
Oleg
> [1]
>
https://gist.githubusercontent.com/rschmitt/bd996852c1bb8b56341051e585a4ed93/raw/c0c8bb6d337de3a881ace9bb52ebd55f4bb29dd3/huffman-disabled.txt
> [2]
>
https://gist.githubusercontent.com/rschmitt/311e8413d88ed7cf8450e844fb74df86/raw/66b559eca0eb9bf5d2192a817881a9efdd3c2c1d/huffman-enabled.txt
> [3]
>
https://github.com/aws/aws-sdk-java-v2/blob/master/services/kinesis/src/it/java/software/amazon/awssdk/services/kinesis/SubscribeToShardIntegrationTest.java
> [4] https://github.com/aws/aws-sdk-java-v2/pull/1543
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org