Suggesting change on BQ streaming dead-letter

[Carlos Alonso <ca...@mrcalonso.com>]

Hi devs!!

After working on this PR https://github.com/apache/beam/pull/5341 I've
realised that the dead-letter functionality on BQ streaming could be
improved a little bit with by retrying a configurable amount of times and
then, if still failing, return the error via getFailedInserts rather than
throwing an exception as currently happens...

I'd be happy to work on it but before kicking off I'd like to have some of
your feedback...

Thanks!

Re: Suggesting change on BQ streaming dead-letter

[Reuven Lax <re...@google.com>]

I agree .- one reason why the current InsertRetryPolicy takes a Context
object (InsertRetryPolicy.java) is exactly so we can augment that
information with extra information, such as the current retry count. One
tricky issue though is that we can only accurately know the retry count in
the current bundle. If the entire bundle fails (because an exception is
thrown or a worker crashes), then the retry count will probably start from
0 again.

Reuven

On Mon, May 14, 2018 at 1:27 AM Carlos Alonso <ca...@mrcalonso.com> wrote:

> Hi devs!!
>
> After working on this PR https://github.com/apache/beam/pull/5341 I've
> realised that the dead-letter functionality on BQ streaming could be
> improved a little bit with by retrying a configurable amount of times and
> then, if still failing, return the error via getFailedInserts rather than
> throwing an exception as currently happens...
>
> I'd be happy to work on it but before kicking off I'd like to have some of
> your feedback...
>
> Thanks!
>