You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Chaim Turkel <ch...@behalf.com> on 2017/09/10 05:45:32 UTC
PipelineResult
Hi,
I am having trouble figuring out what to do with the results. I have
multiple collections running on the pipeline, and since the sink does
not give me the option to get the result, i need to wait for the
pipeline to finish and then poll the results.
From what i can see my only option is to use the metrics, is there
another way to pass information from the collections to the results?
chaim
Re: PipelineResult
Posted by Chaim Turkel <ch...@behalf.com>.
but how can i write when the last point is the bigquery sink -> there
is nothing i can do after this
On Sun, Sep 10, 2017 at 6:16 PM, Eugene Kirpichov
<ki...@google.com.invalid> wrote:
> The common pattern for this sort of thing is to have the pipeline write its
> results to somewhere, eg to files, and have the client read them back. This
> also makes sense for pipelines that never finish.
>
> On Sun, Sep 10, 2017, 3:56 AM Chaim Turkel <ch...@behalf.com> wrote:
>
>> thanks for the reply.
>> I think it would have been nicer if i could get this information as a
>> combine at the end.
>> The current solution means that the client that deploys to job, needs
>> to wait for the results (block) and if the client machine dies, then i
>> loose the information.
>> If I could add it to the collection pipeline as a combine at then end
>> it would be better
>>
>> chaim
>>
>> On Sun, Sep 10, 2017 at 9:03 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>> > Hi Chaim,
>> >
>> > The pipeline result provides the metrics to you. You can periodically
>> poll
>> > (with a thread) the pipeline result object to get the updated data.
>> >
>> > Regards
>> > JB
>> >
>> >
>> > On 09/10/2017 07:45 AM, Chaim Turkel wrote:
>> >>
>> >> Hi,
>> >> I am having trouble figuring out what to do with the results. I have
>> >> multiple collections running on the pipeline, and since the sink does
>> >> not give me the option to get the result, i need to wait for the
>> >> pipeline to finish and then poll the results.
>> >> From what i can see my only option is to use the metrics, is there
>> >> another way to pass information from the collections to the results?
>> >>
>> >> chaim
>> >>
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbonofre@apache.org
>> > http://blog.nanthrax.net
>> > Talend - http://www.talend.com
>>
Re: PipelineResult
Posted by Eugene Kirpichov <ki...@google.com.INVALID>.
The common pattern for this sort of thing is to have the pipeline write its
results to somewhere, eg to files, and have the client read them back. This
also makes sense for pipelines that never finish.
On Sun, Sep 10, 2017, 3:56 AM Chaim Turkel <ch...@behalf.com> wrote:
> thanks for the reply.
> I think it would have been nicer if i could get this information as a
> combine at the end.
> The current solution means that the client that deploys to job, needs
> to wait for the results (block) and if the client machine dies, then i
> loose the information.
> If I could add it to the collection pipeline as a combine at then end
> it would be better
>
> chaim
>
> On Sun, Sep 10, 2017 at 9:03 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> > Hi Chaim,
> >
> > The pipeline result provides the metrics to you. You can periodically
> poll
> > (with a thread) the pipeline result object to get the updated data.
> >
> > Regards
> > JB
> >
> >
> > On 09/10/2017 07:45 AM, Chaim Turkel wrote:
> >>
> >> Hi,
> >> I am having trouble figuring out what to do with the results. I have
> >> multiple collections running on the pipeline, and since the sink does
> >> not give me the option to get the result, i need to wait for the
> >> pipeline to finish and then poll the results.
> >> From what i can see my only option is to use the metrics, is there
> >> another way to pass information from the collections to the results?
> >>
> >> chaim
> >>
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
>
Re: PipelineResult
Posted by Chaim Turkel <ch...@behalf.com>.
thanks for the reply.
I think it would have been nicer if i could get this information as a
combine at the end.
The current solution means that the client that deploys to job, needs
to wait for the results (block) and if the client machine dies, then i
loose the information.
If I could add it to the collection pipeline as a combine at then end
it would be better
chaim
On Sun, Sep 10, 2017 at 9:03 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi Chaim,
>
> The pipeline result provides the metrics to you. You can periodically poll
> (with a thread) the pipeline result object to get the updated data.
>
> Regards
> JB
>
>
> On 09/10/2017 07:45 AM, Chaim Turkel wrote:
>>
>> Hi,
>> I am having trouble figuring out what to do with the results. I have
>> multiple collections running on the pipeline, and since the sink does
>> not give me the option to get the result, i need to wait for the
>> pipeline to finish and then poll the results.
>> From what i can see my only option is to use the metrics, is there
>> another way to pass information from the collections to the results?
>>
>> chaim
>>
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
Re: PipelineResult
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Chaim,
The pipeline result provides the metrics to you. You can periodically poll (with
a thread) the pipeline result object to get the updated data.
Regards
JB
On 09/10/2017 07:45 AM, Chaim Turkel wrote:
> Hi,
> I am having trouble figuring out what to do with the results. I have
> multiple collections running on the pipeline, and since the sink does
> not give me the option to get the result, i need to wait for the
> pipeline to finish and then poll the results.
> From what i can see my only option is to use the metrics, is there
> another way to pass information from the collections to the results?
>
> chaim
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: PipelineResult
Posted by Chaim Turkel <ch...@behalf.com>.
it would be much smarter if the BigQueryIO would return a collection of void
chaim
On Mon, Sep 11, 2017 at 8:47 AM, Reuven Lax <re...@google.com.invalid> wrote:
> On Sun, Sep 10, 2017 at 10:12 PM, Chaim Turkel <ch...@behalf.com> wrote:
>
>> I am migrating multiple tables from mongo to bigquery. So i loop over
>> each table and create a PCollection for each table.
>> I would like to update a status row for each run (time, records...).
>> So I want to write the results at the end.
>> I would like to write a ParDo method after the sink but currently
>> bigqueryIO does not support this.
>>
>
> Makes sense. I wonder if BigQueryIO.Write should return a PCollection of
> metadata objects (table name, rows written, etc.) as part of the
> WriteResult.
>
>
>>
>> Using metrics is an option but then the client needs to block and wait
>> for the pipline to finish - and it might loose the information
>>
>
> If using the Dataflow runner, you can always use the job id to get the
> metrics even if the main program crashes.
>
> Chaim
>>
>> On Sun, Sep 10, 2017 at 8:05 PM, Reuven Lax <re...@google.com.invalid>
>> wrote:
>> > Can you explain what you mean by multiple collections running on the
>> > pipeline? What do you need the results for?
>> >
>> > On Sat, Sep 9, 2017 at 10:45 PM, Chaim Turkel <ch...@behalf.com> wrote:
>> >
>> >> Hi,
>> >> I am having trouble figuring out what to do with the results. I have
>> >> multiple collections running on the pipeline, and since the sink does
>> >> not give me the option to get the result, i need to wait for the
>> >> pipeline to finish and then poll the results.
>> >> From what i can see my only option is to use the metrics, is there
>> >> another way to pass information from the collections to the results?
>> >>
>> >> chaim
>> >>
>>
Re: PipelineResult
Posted by Reuven Lax <re...@google.com.INVALID>.
On Sun, Sep 10, 2017 at 10:12 PM, Chaim Turkel <ch...@behalf.com> wrote:
> I am migrating multiple tables from mongo to bigquery. So i loop over
> each table and create a PCollection for each table.
> I would like to update a status row for each run (time, records...).
> So I want to write the results at the end.
> I would like to write a ParDo method after the sink but currently
> bigqueryIO does not support this.
>
Makes sense. I wonder if BigQueryIO.Write should return a PCollection of
metadata objects (table name, rows written, etc.) as part of the
WriteResult.
>
> Using metrics is an option but then the client needs to block and wait
> for the pipline to finish - and it might loose the information
>
If using the Dataflow runner, you can always use the job id to get the
metrics even if the main program crashes.
Chaim
>
> On Sun, Sep 10, 2017 at 8:05 PM, Reuven Lax <re...@google.com.invalid>
> wrote:
> > Can you explain what you mean by multiple collections running on the
> > pipeline? What do you need the results for?
> >
> > On Sat, Sep 9, 2017 at 10:45 PM, Chaim Turkel <ch...@behalf.com> wrote:
> >
> >> Hi,
> >> I am having trouble figuring out what to do with the results. I have
> >> multiple collections running on the pipeline, and since the sink does
> >> not give me the option to get the result, i need to wait for the
> >> pipeline to finish and then poll the results.
> >> From what i can see my only option is to use the metrics, is there
> >> another way to pass information from the collections to the results?
> >>
> >> chaim
> >>
>
Re: PipelineResult
Posted by Chaim Turkel <ch...@behalf.com>.
I am migrating multiple tables from mongo to bigquery. So i loop over
each table and create a PCollection for each table.
I would like to update a status row for each run (time, records...).
So I want to write the results at the end.
I would like to write a ParDo method after the sink but currently
bigqueryIO does not support this.
Using metrics is an option but then the client needs to block and wait
for the pipline to finish - and it might loose the information
Chaim
On Sun, Sep 10, 2017 at 8:05 PM, Reuven Lax <re...@google.com.invalid> wrote:
> Can you explain what you mean by multiple collections running on the
> pipeline? What do you need the results for?
>
> On Sat, Sep 9, 2017 at 10:45 PM, Chaim Turkel <ch...@behalf.com> wrote:
>
>> Hi,
>> I am having trouble figuring out what to do with the results. I have
>> multiple collections running on the pipeline, and since the sink does
>> not give me the option to get the result, i need to wait for the
>> pipeline to finish and then poll the results.
>> From what i can see my only option is to use the metrics, is there
>> another way to pass information from the collections to the results?
>>
>> chaim
>>
Re: PipelineResult
Posted by Reuven Lax <re...@google.com.INVALID>.
Can you explain what you mean by multiple collections running on the
pipeline? What do you need the results for?
On Sat, Sep 9, 2017 at 10:45 PM, Chaim Turkel <ch...@behalf.com> wrote:
> Hi,
> I am having trouble figuring out what to do with the results. I have
> multiple collections running on the pipeline, and since the sink does
> not give me the option to get the result, i need to wait for the
> pipeline to finish and then poll the results.
> From what i can see my only option is to use the metrics, is there
> another way to pass information from the collections to the results?
>
> chaim
>