You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by Gayan Weerakutti <ga...@linuxdeveloper.space> on 2021/11/29 05:29:50 UTC

PubsubIO writer error handling

Hi,

We are using PubsubIO.writeMessages() in one of our production 
pipelines. We were wondering what would happen if there's a temporary 
connection failure to Pub/Sub.

Will PubSubIo be able recover? Will the failed messages be replayed?

Really appreciate any insights on this.

-- 
Thanks & regards,

Gayan Weerakutti

linkedin.com/in/gayanweerakutti 
<https://www.linkedin.com/in/gayanweerakutti/>

Re: PubsubIO writer error handling

Posted by Israel Herraiz <ih...@google.com>.

If you are using Cloud Dataflow, and since this would be a streaming
pipeline, any error will be retried indefinitely. So if the write errors
are transitory, the messages will be eventually published in the
destination topic. More info at
https://cloud.google.com/dataflow/docs/resources/faq#how-are-java-exceptions-handled-in-cloud-dataflow

On Mon, 29 Nov 2021 at 06:30, Gayan Weerakutti <ga...@linuxdeveloper.space>
wrote:

> Hi,
>
> We are using PubsubIO.writeMessages() in one of our production pipelines.
> We were wondering what would happen if there's a temporary connection
> failure to Pub/Sub.
>
> Will PubSubIo be able recover? Will the failed messages be replayed?
>
> Really appreciate any insights on this.
> --
> Thanks & regards,
>
> Gayan Weerakutti
>
> linkedin.com/in/gayanweerakutti
> <https://www.linkedin.com/in/gayanweerakutti/>
>
>
>
>
>
>
>
>
>
>
>

Re: PubsubIO writer error handling

Posted by Luke Cwik <lc...@google.com>.

Depending on the runner and how it considers the pipeline a failure but
typically most runners that are executing in a streaming mode (e.g.
Dataflow) will keep retrying until there is a successful publish.

On Sun, Nov 28, 2021 at 9:30 PM Gayan Weerakutti <ga...@linuxdeveloper.space>
wrote:

> Hi,
>
> We are using PubsubIO.writeMessages() in one of our production pipelines.
> We were wondering what would happen if there's a temporary connection
> failure to Pub/Sub.
>
> Will PubSubIo be able recover? Will the failed messages be replayed?
>
> Really appreciate any insights on this.
> --
> Thanks & regards,
>
> Gayan Weerakutti
>
> linkedin.com/in/gayanweerakutti
> <https://www.linkedin.com/in/gayanweerakutti/>
>
>
>
>
>
>
>
>
>
>
>