indicating an error during a streaming response

[Ben Duffield <bd...@palantir.com>]

Hello there,

 

I have a java server (jetty, jersey) which produces a streaming arrow response (via a javax.ws.rs.core.StreamingOutput).

 

The stream of arrow is produced ‘on the fly’ and occasionally something goes wrong once we’ve already begun streaming. In this case we throw an exception which propagates up to jersey, which then closes the response normally (there seems to be no good way to get jersey to hand the exception back up to jetty to cause an abort on the response stream). We’re sending whole arrow batches, so the client just sees a regular arrow stream (i.e. the stream up to and including the last entire arrow row batch we sent) and so has no way to detect that an error occurred.

 

Maybe there is actually a way to get jetty to hang up the connection without proper termination (which the client can then detect) – although this does mean we have no ability to indicate why the error happened – but we have so far been unable to figure out how to do this.

 

Another thought was having some way in a streaming arrow response to indicate this kind of error inline in the stream/be able to ‘poison’ an arrow stream – to indicate to the client that something exceptional happened part way through the streaming response. Maybe some way of sending an EOS with an error flag set.  

 

There are probably other, arguably more elegant ways to handle this kind of thing e.g. if we can assume http2 everywhere something like https://http2.github.io/http2-spec/#StreamErrorHandler but it seems currently the tooling isn’t really there/at any rate it could be nice to have a transport-independent way of indicating an error.

 

I was wondering if anyone has dealt with anything similar to this/has any pointers/thoughts on whether adding to the streaming format makes sense here? 

 

Best,

 

Ben

 

Re: indicating an error during a streaming response

[Wes McKinney <we...@gmail.com>]

hi Ben,

The stream writer and reader classes that exist now are fairly
primitive. We've been discussing a richer RPC protocol using http2 or
GRPC that would permit more nuanced control messages, errors, and
other. This would be a great area for new members of the dev community
to make some valuable contributions. Since Dremio uses Arrow heavily,
there may be some tech that can be repurposed or at least designs
reused to streamline the process.

Bottom line is that this is a use case that we want to be
well-supported out of the box in the Arrow libraries.

Would be great to hear from some others. Copying Nong Li who was
looking into GRPC integration sometime last year.

best,
Wes

On Mon, Feb 12, 2018 at 7:41 PM, Ben Duffield <bd...@palantir.com> wrote:
> Hello there,
>
>
>
> I have a java server (jetty, jersey) which produces a streaming arrow
> response (via a javax.ws.rs.core.StreamingOutput).
>
>
>
> The stream of arrow is produced ‘on the fly’ and occasionally something goes
> wrong once we’ve already begun streaming. In this case we throw an exception
> which propagates up to jersey, which then closes the response normally
> (there seems to be no good way to get jersey to hand the exception back up
> to jetty to cause an abort on the response stream). We’re sending whole
> arrow batches, so the client just sees a regular arrow stream (i.e. the
> stream up to and including the last entire arrow row batch we sent) and so
> has no way to detect that an error occurred.
>
>
>
> Maybe there is actually a way to get jetty to hang up the connection without
> proper termination (which the client can then detect) – although this does
> mean we have no ability to indicate why the error happened – but we have so
> far been unable to figure out how to do this.
>
>
>
> Another thought was having some way in a streaming arrow response to
> indicate this kind of error inline in the stream/be able to ‘poison’ an
> arrow stream – to indicate to the client that something exceptional happened
> part way through the streaming response. Maybe some way of sending an EOS
> with an error flag set.
>
>
>
> There are probably other, arguably more elegant ways to handle this kind of
> thing e.g. if we can assume http2 everywhere something like
> https://http2.github.io/http2-spec/#StreamErrorHandler but it seems
> currently the tooling isn’t really there/at any rate it could be nice to
> have a transport-independent way of indicating an error.
>
>
>
> I was wondering if anyone has dealt with anything similar to this/has any
> pointers/thoughts on whether adding to the streaming format makes sense
> here?
>
>
>
> Best,
>
>
>
> Ben
>
>