You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Thiago Chiarato <ch...@gmail.com> on 2020/10/27 12:29:26 UTC
Dataflow Pipeline Stucked While Processing
Hi,
I have a pipeline running on Dataflow that is reading messages (already published) to a Pub/Sub topic and save to BigQuery. I’m running into an issue that after processing tons of millions of messages the pipeline get stuck and stop reading more messages from Pub/Sub. How can I start investigating to find what’s going on?
Re: Dataflow Pipeline Stucked While Processing
Posted by Marcin Kuthan <ma...@gmail.com>.
Deploy exact copy of the job aside, separate subscriptions, buckets and destination datasets. If it will work smoothly it could be an internal dataflow issue.
Have you contacted Google support? They have more tools and metrics for dataflow job investigation than available on GCP console :)
Regards,
Marcin
> Wiadomość napisana przez Sofia’s World <mm...@gmail.com> w dniu 27.10.2020, o godz. 14:54:
>
>
> Right. Try to swap BQ with instead writing to bucket. I am wondering if some of your messages result in invalid bq structure..
> Are you swallowing exception by chance? Have seen similar behavior when writing to Bq.
> With millions of msgs it's hard to debug. Perhaps someone here can suggest how to debug busted writes to Bq?
> Kr
>
>> On Tue, Oct 27, 2020, 1:32 PM Thiago Chiarato <ch...@gmail.com> wrote:
>> Thanks, I didn’t try this yet.
>> It’s just weird to see the throughput decreasing from millions of rows per second to just a few hundred per second.
>> Em 27 de out. de 2020 10:03 -0300, Sofia’s World <mm...@gmail.com> escreveu:
>>> Hey
>>> 2 cents. Would you know which part of pipeline is 'breaking'? If you see your pipeline as.
>>> Source ----->processor ---->sink then you could put it through a unit test. Create some source data and pass it through.
>>> If your processor is fine you could then build a small pipeline which just read from source.
>>> If the issue is writing to be in the sink - most of my problems - you will do similar
>>> Sorry to say something you already know..but I am debugging all my pipelines like this..I found this as the quickest way
>>> Hth
>>>
>>>> On Tue, Oct 27, 2020, 12:30 PM Thiago Chiarato <ch...@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> I have a pipeline running on Dataflow that is reading messages (already published) to a Pub/Sub topic and save to BigQuery. I’m running into an issue that after processing tons of millions of messages the pipeline get stuck and stop reading more messages from Pub/Sub. How can I start investigating to find what’s going on?
Re: Dataflow Pipeline Stucked While Processing
Posted by Sofia’s World <mm...@gmail.com>.
Right. Try to swap BQ with instead writing to bucket. I am wondering if
some of your messages result in invalid bq structure..
Are you swallowing exception by chance? Have seen similar behavior when
writing to Bq.
With millions of msgs it's hard to debug. Perhaps someone here can suggest
how to debug busted writes to Bq?
Kr
On Tue, Oct 27, 2020, 1:32 PM Thiago Chiarato <ch...@gmail.com> wrote:
> Thanks, I didn’t try this yet.
> It’s just weird to see the throughput decreasing from millions of rows per
> second to just a few hundred per second.
> Em 27 de out. de 2020 10:03 -0300, Sofia’s World <mm...@gmail.com>
> escreveu:
>
> Hey
> 2 cents. Would you know which part of pipeline is 'breaking'? If you see
> your pipeline as.
> Source ----->processor ---->sink then you could put it through a unit
> test. Create some source data and pass it through.
> If your processor is fine you could then build a small pipeline which just
> read from source.
> If the issue is writing to be in the sink - most of my problems - you will
> do similar
> Sorry to say something you already know..but I am debugging all my
> pipelines like this..I found this as the quickest way
> Hth
>
> On Tue, Oct 27, 2020, 12:30 PM Thiago Chiarato <ch...@gmail.com> wrote:
>
>> Hi,
>>
>> I have a pipeline running on Dataflow that is reading messages (already
>> published) to a Pub/Sub topic and save to BigQuery. I’m running into an
>> issue that after processing tons of millions of messages the pipeline get
>> stuck and stop reading more messages from Pub/Sub. How can I start
>> investigating to find what’s going on?
>>
>
Re: Dataflow Pipeline Stucked While Processing
Posted by Thiago Chiarato <ch...@gmail.com>.
Thanks, I didn’t try this yet.
It’s just weird to see the throughput decreasing from millions of rows per second to just a few hundred per second.
Em 27 de out. de 2020 10:03 -0300, Sofia’s World <mm...@gmail.com> escreveu:
> Hey
> 2 cents. Would you know which part of pipeline is 'breaking'? If you see your pipeline as.
> Source ----->processor ---->sink then you could put it through a unit test. Create some source data and pass it through.
> If your processor is fine you could then build a small pipeline which just read from source.
> If the issue is writing to be in the sink - most of my problems - you will do similar
> Sorry to say something you already know..but I am debugging all my pipelines like this..I found this as the quickest way
> Hth
>
> > On Tue, Oct 27, 2020, 12:30 PM Thiago Chiarato <ch...@gmail.com> wrote:
> > > Hi,
> > >
> > > I have a pipeline running on Dataflow that is reading messages (already published) to a Pub/Sub topic and save to BigQuery. I’m running into an issue that after processing tons of millions of messages the pipeline get stuck and stop reading more messages from Pub/Sub. How can I start investigating to find what’s going on?
Re: Dataflow Pipeline Stucked While Processing
Posted by Sofia’s World <mm...@gmail.com>.
Hey
2 cents. Would you know which part of pipeline is 'breaking'? If you see
your pipeline as.
Source ----->processor ---->sink then you could put it through a unit test.
Create some source data and pass it through.
If your processor is fine you could then build a small pipeline which just
read from source.
If the issue is writing to be in the sink - most of my problems - you will
do similar
Sorry to say something you already know..but I am debugging all my
pipelines like this..I found this as the quickest way
Hth
On Tue, Oct 27, 2020, 12:30 PM Thiago Chiarato <ch...@gmail.com> wrote:
> Hi,
>
> I have a pipeline running on Dataflow that is reading messages (already
> published) to a Pub/Sub topic and save to BigQuery. I’m running into an
> issue that after processing tons of millions of messages the pipeline get
> stuck and stop reading more messages from Pub/Sub. How can I start
> investigating to find what’s going on?
>