You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Derek Hao Hu <ph...@gmail.com> on 2017/10/19 21:15:00 UTC
Infinite retry in streaming - is there a workaround?
Hi,
We are trying to use Dataflow in Prod and right now one of our main
concerns is this "infinite retry" behavior which might stall the whole
pipeline.
Right now for all the DoFns we've implemented ourselves we've added some
error handling or exception swallowing mechanism to make sure some bundles
can just fail and we log the exceptions. But we are a bit concerned about
the other Beam native transforms which we can not easily wrap, e.g.
PubSubIO transforms and DatastoreV1 transforms.
A few days ago I asked a specific question in this group about how one can
catch exception in DatastoreV1 transforms and the recommended approach is
to 1) either duplicate the code in the current DatastoreV1 implementation
and swallow the exception instead of throwing or 2) Follow the
implementation of BigQueryIO to add the ability to support custom retry
policy. Both are feasible options but I'm a bit concerned in that doesn't
that mean eventually all Beam native transforms need to implement something
like 2) if we want to use them in Prod?
So in short, I want to know right now what is the recommended approach or
workaround to say, hey, just let this bundle fail and we can process the
rest of the elements instead of just stall the pipeline?
Thanks!
--
Derek Hao Hu
Software Engineer | Snapchat
Snap Inc.
Re: Infinite retry in streaming - is there a workaround?
Posted by Aleksandr <al...@gmail.com>.
Hello Derek,
There no general solution for failing bundle. Some kind of dataflow errors
you can fix using dataflow update feature. Another solution is to catch
exceptions in ParDo function.
25. okt 2017 9:42 PM kirjutas kuupäeval "Griselda Cuevas" <gr...@google.com>:
Hi Derek, yes you can use that mailing list and also the SO channel.
Cheers,
G
> BTW, do you know if there's a Dataflow mailing list for questions like
> this? Would dataflow-feedback be the appropriate mailing list?
>
> Thanks,
>
> Derek
>
> On Wed, Oct 25, 2017 at 10:58 AM, Griselda Cuevas <gr...@google.com> wrote:
>
>> Hi Derek - It sounds like this is a Dataflow specific questions so I'd
>> recommend you also reach out through the Dataflow's Stack Overflow
>> <https://stackoverflow.com/questions/tagged/google-cloud-dataflow>
>> channel. I'm also cc'ing Thomas Groh who might be able to help.
>>
>>
>>
>> On 20 October 2017 at 11:35, Derek Hao Hu <ph...@gmail.com> wrote:
>>
>>> Kindly ping as I'm really curious about this. :p
>>>
>>> Derek
>>>
>>> On Thu, Oct 19, 2017 at 2:15 PM, Derek Hao Hu <ph...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> We are trying to use Dataflow in Prod and right now one of our main
>>>> concerns is this "infinite retry" behavior which might stall the whole
>>>> pipeline.
>>>>
>>>> Right now for all the DoFns we've implemented ourselves we've added
>>>> some error handling or exception swallowing mechanism to make sure some
>>>> bundles can just fail and we log the exceptions. But we are a bit concerned
>>>> about the other Beam native transforms which we can not easily wrap, e.g.
>>>> PubSubIO transforms and DatastoreV1 transforms.
>>>>
>>>> A few days ago I asked a specific question in this group about how one
>>>> can catch exception in DatastoreV1 transforms and the recommended approach
>>>> is to 1) either duplicate the code in the current DatastoreV1
>>>> implementation and swallow the exception instead of throwing or 2) Follow
>>>> the implementation of BigQueryIO to add the ability to support custom retry
>>>> policy. Both are feasible options but I'm a bit concerned in that doesn't
>>>> that mean eventually all Beam native transforms need to implement something
>>>> like 2) if we want to use them in Prod?
>>>>
>>>> So in short, I want to know right now what is the recommended approach
>>>> or workaround to say, hey, just let this bundle fail and we can process the
>>>> rest of the elements instead of just stall the pipeline?
>>>>
>>>> Thanks!
>>>> --
>>>> Derek Hao Hu
>>>>
>>>> Software Engineer | Snapchat
>>>> Snap Inc.
>>>>
>>>
>>>
>>>
>>> --
>>> Derek Hao Hu
>>>
>>> Software Engineer | Snapchat
>>> Snap Inc.
>>>
>>
>>
>
>
> --
> Derek Hao Hu
>
> Software Engineer | Snapchat
> Snap Inc.
>
Re: Infinite retry in streaming - is there a workaround?
Posted by Griselda Cuevas <gr...@google.com>.
Hi Derek, yes you can use that mailing list and also the SO channel.
Cheers,
G
> BTW, do you know if there's a Dataflow mailing list for questions like
> this? Would dataflow-feedback be the appropriate mailing list?
>
> Thanks,
>
> Derek
>
> On Wed, Oct 25, 2017 at 10:58 AM, Griselda Cuevas <gr...@google.com> wrote:
>
>> Hi Derek - It sounds like this is a Dataflow specific questions so I'd
>> recommend you also reach out through the Dataflow's Stack Overflow
>> <https://stackoverflow.com/questions/tagged/google-cloud-dataflow>
>> channel. I'm also cc'ing Thomas Groh who might be able to help.
>>
>>
>>
>> On 20 October 2017 at 11:35, Derek Hao Hu <ph...@gmail.com> wrote:
>>
>>> Kindly ping as I'm really curious about this. :p
>>>
>>> Derek
>>>
>>> On Thu, Oct 19, 2017 at 2:15 PM, Derek Hao Hu <ph...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> We are trying to use Dataflow in Prod and right now one of our main
>>>> concerns is this "infinite retry" behavior which might stall the whole
>>>> pipeline.
>>>>
>>>> Right now for all the DoFns we've implemented ourselves we've added
>>>> some error handling or exception swallowing mechanism to make sure some
>>>> bundles can just fail and we log the exceptions. But we are a bit concerned
>>>> about the other Beam native transforms which we can not easily wrap, e.g.
>>>> PubSubIO transforms and DatastoreV1 transforms.
>>>>
>>>> A few days ago I asked a specific question in this group about how one
>>>> can catch exception in DatastoreV1 transforms and the recommended approach
>>>> is to 1) either duplicate the code in the current DatastoreV1
>>>> implementation and swallow the exception instead of throwing or 2) Follow
>>>> the implementation of BigQueryIO to add the ability to support custom retry
>>>> policy. Both are feasible options but I'm a bit concerned in that doesn't
>>>> that mean eventually all Beam native transforms need to implement something
>>>> like 2) if we want to use them in Prod?
>>>>
>>>> So in short, I want to know right now what is the recommended approach
>>>> or workaround to say, hey, just let this bundle fail and we can process the
>>>> rest of the elements instead of just stall the pipeline?
>>>>
>>>> Thanks!
>>>> --
>>>> Derek Hao Hu
>>>>
>>>> Software Engineer | Snapchat
>>>> Snap Inc.
>>>>
>>>
>>>
>>>
>>> --
>>> Derek Hao Hu
>>>
>>> Software Engineer | Snapchat
>>> Snap Inc.
>>>
>>
>>
>
>
> --
> Derek Hao Hu
>
> Software Engineer | Snapchat
> Snap Inc.
>
Re: Infinite retry in streaming - is there a workaround?
Posted by Derek Hao Hu <ph...@gmail.com>.
Aha, I just realized this is just a Dataflow behavior not a Beam default
behavior. :) Thanks Griselda. I'll post in the SO channel.
BTW, do you know if there's a Dataflow mailing list for questions like
this? Would dataflow-feedback be the appropriate mailing list?
Thanks,
Derek
On Wed, Oct 25, 2017 at 10:58 AM, Griselda Cuevas <gr...@google.com> wrote:
> Hi Derek - It sounds like this is a Dataflow specific questions so I'd
> recommend you also reach out through the Dataflow's Stack Overflow
> <https://stackoverflow.com/questions/tagged/google-cloud-dataflow>
> channel. I'm also cc'ing Thomas Groh who might be able to help.
>
>
>
> On 20 October 2017 at 11:35, Derek Hao Hu <ph...@gmail.com> wrote:
>
>> Kindly ping as I'm really curious about this. :p
>>
>> Derek
>>
>> On Thu, Oct 19, 2017 at 2:15 PM, Derek Hao Hu <ph...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> We are trying to use Dataflow in Prod and right now one of our main
>>> concerns is this "infinite retry" behavior which might stall the whole
>>> pipeline.
>>>
>>> Right now for all the DoFns we've implemented ourselves we've added some
>>> error handling or exception swallowing mechanism to make sure some bundles
>>> can just fail and we log the exceptions. But we are a bit concerned about
>>> the other Beam native transforms which we can not easily wrap, e.g.
>>> PubSubIO transforms and DatastoreV1 transforms.
>>>
>>> A few days ago I asked a specific question in this group about how one
>>> can catch exception in DatastoreV1 transforms and the recommended approach
>>> is to 1) either duplicate the code in the current DatastoreV1
>>> implementation and swallow the exception instead of throwing or 2) Follow
>>> the implementation of BigQueryIO to add the ability to support custom retry
>>> policy. Both are feasible options but I'm a bit concerned in that doesn't
>>> that mean eventually all Beam native transforms need to implement something
>>> like 2) if we want to use them in Prod?
>>>
>>> So in short, I want to know right now what is the recommended approach
>>> or workaround to say, hey, just let this bundle fail and we can process the
>>> rest of the elements instead of just stall the pipeline?
>>>
>>> Thanks!
>>> --
>>> Derek Hao Hu
>>>
>>> Software Engineer | Snapchat
>>> Snap Inc.
>>>
>>
>>
>>
>> --
>> Derek Hao Hu
>>
>> Software Engineer | Snapchat
>> Snap Inc.
>>
>
>
--
Derek Hao Hu
Software Engineer | Snapchat
Snap Inc.
Re: Infinite retry in streaming - is there a workaround?
Posted by Griselda Cuevas <gr...@google.com>.
Hi Derek - It sounds like this is a Dataflow specific questions so I'd
recommend you also reach out through the Dataflow's Stack Overflow
<https://stackoverflow.com/questions/tagged/google-cloud-dataflow> channel.
I'm also cc'ing Thomas Groh who might be able to help.
On 20 October 2017 at 11:35, Derek Hao Hu <ph...@gmail.com> wrote:
> Kindly ping as I'm really curious about this. :p
>
> Derek
>
> On Thu, Oct 19, 2017 at 2:15 PM, Derek Hao Hu <ph...@gmail.com>
> wrote:
>
>> Hi,
>>
>> We are trying to use Dataflow in Prod and right now one of our main
>> concerns is this "infinite retry" behavior which might stall the whole
>> pipeline.
>>
>> Right now for all the DoFns we've implemented ourselves we've added some
>> error handling or exception swallowing mechanism to make sure some bundles
>> can just fail and we log the exceptions. But we are a bit concerned about
>> the other Beam native transforms which we can not easily wrap, e.g.
>> PubSubIO transforms and DatastoreV1 transforms.
>>
>> A few days ago I asked a specific question in this group about how one
>> can catch exception in DatastoreV1 transforms and the recommended approach
>> is to 1) either duplicate the code in the current DatastoreV1
>> implementation and swallow the exception instead of throwing or 2) Follow
>> the implementation of BigQueryIO to add the ability to support custom retry
>> policy. Both are feasible options but I'm a bit concerned in that doesn't
>> that mean eventually all Beam native transforms need to implement something
>> like 2) if we want to use them in Prod?
>>
>> So in short, I want to know right now what is the recommended approach or
>> workaround to say, hey, just let this bundle fail and we can process the
>> rest of the elements instead of just stall the pipeline?
>>
>> Thanks!
>> --
>> Derek Hao Hu
>>
>> Software Engineer | Snapchat
>> Snap Inc.
>>
>
>
>
> --
> Derek Hao Hu
>
> Software Engineer | Snapchat
> Snap Inc.
>
Re: Infinite retry in streaming - is there a workaround?
Posted by Derek Hao Hu <ph...@gmail.com>.
Kindly ping as I'm really curious about this. :p
Derek
On Thu, Oct 19, 2017 at 2:15 PM, Derek Hao Hu <ph...@gmail.com>
wrote:
> Hi,
>
> We are trying to use Dataflow in Prod and right now one of our main
> concerns is this "infinite retry" behavior which might stall the whole
> pipeline.
>
> Right now for all the DoFns we've implemented ourselves we've added some
> error handling or exception swallowing mechanism to make sure some bundles
> can just fail and we log the exceptions. But we are a bit concerned about
> the other Beam native transforms which we can not easily wrap, e.g.
> PubSubIO transforms and DatastoreV1 transforms.
>
> A few days ago I asked a specific question in this group about how one can
> catch exception in DatastoreV1 transforms and the recommended approach is
> to 1) either duplicate the code in the current DatastoreV1 implementation
> and swallow the exception instead of throwing or 2) Follow the
> implementation of BigQueryIO to add the ability to support custom retry
> policy. Both are feasible options but I'm a bit concerned in that doesn't
> that mean eventually all Beam native transforms need to implement something
> like 2) if we want to use them in Prod?
>
> So in short, I want to know right now what is the recommended approach or
> workaround to say, hey, just let this bundle fail and we can process the
> rest of the elements instead of just stall the pipeline?
>
> Thanks!
> --
> Derek Hao Hu
>
> Software Engineer | Snapchat
> Snap Inc.
>
--
Derek Hao Hu
Software Engineer | Snapchat
Snap Inc.