You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Chaim Turkel <ch...@behalf.com> on 2019/06/19 12:42:00 UTC
pipeline status tracing
Hi,
I would like to write a status at the end of my pipline. I had
written about this in the past, and wanted to know if there are any
new ideas.
The problem is the end of the pipeline return PDone, and i can't do
anything with this.
So my senario is after i export data from mongo to google storage i
want to write to a db that the job was done with some extra
information.
Chaim Turkel
--
Loans are funded by
FinWise Bank, a Utah-chartered bank located in Sandy,
Utah, member FDIC, Equal
Opportunity Lender. Merchant Cash Advances are
made by Behalf. For more
information on ECOA, click here
<https://www.behalf.com/legal/ecoa/>. For important information about
opening a new
account, review Patriot Act procedures here
<https://www.behalf.com/legal/patriot/>.
Visit Legal
<https://www.behalf.com/legal/> to
review our comprehensive program terms,
conditions, and disclosures.
Re: pipeline status tracing
Posted by Eugene Kirpichov <ki...@google.com>.
I see. This was discussed in this thread [1] - however looks like nobody
has picked it up yet.
[1]
https://lists.apache.org/thread.html/235b1f683ef0eee66934fdf0908b3b85ed4e7d3627cbcfcf12b27de4@%3Cuser.beam.apache.org%3E
On Sun, Jun 23, 2019 at 4:06 AM Chaim Turkel <ch...@behalf.com> wrote:
> i am writing to bigquery
>
> On Fri, Jun 21, 2019 at 7:12 PM Alexey Romanenko
> <ar...@gmail.com> wrote:
> >
> > I see that similar questions happen quite often in the last time, so,
> probably, it would make sense to add this “Wait.on()”-pattern to
> corresponding website documentation page [1].
> >
> > [1]
> https://beam.apache.org/documentation/patterns/file-processing-patterns/
> >
> > On 20 Jun 2019, at 23:00, Eugene Kirpichov <ki...@google.com> wrote:
> >
> > If you're writing to files, you can already do this: FileIO.write()
> returns WriteFilesResult that you can use in combination with Wait.on() and
> JdbcIO.write() to write something to a database afterwards.
> > Something like:
> >
> > PCollection<..> writeResult = data.apply(FileIO.write()...);
> >
> Create.of("dummy").apply(Wait.on(writeResult)).apply(JdbcIO.write(...write
> to database...))
> >
> >
> >
> > On Wed, Jun 19, 2019 at 11:49 PM Chaim Turkel <ch...@behalf.com> wrote:
> >>
> >> is it something you can do, or point me in the right direction?
> >> chaim
> >>
> >> On Wed, Jun 19, 2019 at 8:36 PM Kenneth Knowles <ke...@apache.org>
> wrote:
> >> >
> >> > This sounds like a fairly simply and useful change to the IO to
> output information about where it has written. It seems generally useful to
> always output something like this, instead of PDone.
> >> >
> >> > Kenn
> >> >
> >> > On Wed, Jun 19, 2019 at 5:42 AM Chaim Turkel <ch...@behalf.com>
> wrote:
> >> >>
> >> >> Hi,
> >> >> I would like to write a status at the end of my pipline. I had
> >> >> written about this in the past, and wanted to know if there are any
> >> >> new ideas.
> >> >> The problem is the end of the pipeline return PDone, and i can't do
> >> >> anything with this.
> >> >> So my senario is after i export data from mongo to google storage i
> >> >> want to write to a db that the job was done with some extra
> >> >> information.
> >> >>
> >> >> Chaim Turkel
> >> >>
> >> >> --
> >> >>
> >> >>
> >> >> Loans are funded by
> >> >> FinWise Bank, a Utah-chartered bank located in Sandy,
> >> >> Utah, member FDIC, Equal
> >> >> Opportunity Lender. Merchant Cash Advances are
> >> >> made by Behalf. For more
> >> >> information on ECOA, click here
> >> >> <https://www.behalf.com/legal/ecoa/>. For important information
> about
> >> >> opening a new
> >> >> account, review Patriot Act procedures here
> >> >> <https://www.behalf.com/legal/patriot/>.
> >> >> Visit Legal
> >> >> <https://www.behalf.com/legal/> to
> >> >> review our comprehensive program terms,
> >> >> conditions, and disclosures.
> >>
> >> --
> >>
> >>
> >> Loans are funded by
> >> FinWise Bank, a Utah-chartered bank located in Sandy,
> >> Utah, member FDIC, Equal
> >> Opportunity Lender. Merchant Cash Advances are
> >> made by Behalf. For more
> >> information on ECOA, click here
> >> <https://www.behalf.com/legal/ecoa/>. For important information about
> >> opening a new
> >> account, review Patriot Act procedures here
> >> <https://www.behalf.com/legal/patriot/>.
> >> Visit Legal
> >> <https://www.behalf.com/legal/> to
> >> review our comprehensive program terms,
> >> conditions, and disclosures.
> >
> >
>
> --
>
>
> Loans are funded by
> FinWise Bank, a Utah-chartered bank located in Sandy,
> Utah, member FDIC, Equal
> Opportunity Lender. Merchant Cash Advances are
> made by Behalf. For more
> information on ECOA, click here
> <https://www.behalf.com/legal/ecoa/>. For important information about
> opening a new
> account, review Patriot Act procedures here
> <https://www.behalf.com/legal/patriot/>.
> Visit Legal
> <https://www.behalf.com/legal/> to
> review our comprehensive program terms,
> conditions, and disclosures.
>
Re: pipeline status tracing
Posted by Chaim Turkel <ch...@behalf.com>.
i am writing to bigquery
On Fri, Jun 21, 2019 at 7:12 PM Alexey Romanenko
<ar...@gmail.com> wrote:
>
> I see that similar questions happen quite often in the last time, so, probably, it would make sense to add this “Wait.on()”-pattern to corresponding website documentation page [1].
>
> [1] https://beam.apache.org/documentation/patterns/file-processing-patterns/
>
> On 20 Jun 2019, at 23:00, Eugene Kirpichov <ki...@google.com> wrote:
>
> If you're writing to files, you can already do this: FileIO.write() returns WriteFilesResult that you can use in combination with Wait.on() and JdbcIO.write() to write something to a database afterwards.
> Something like:
>
> PCollection<..> writeResult = data.apply(FileIO.write()...);
> Create.of("dummy").apply(Wait.on(writeResult)).apply(JdbcIO.write(...write to database...))
>
>
>
> On Wed, Jun 19, 2019 at 11:49 PM Chaim Turkel <ch...@behalf.com> wrote:
>>
>> is it something you can do, or point me in the right direction?
>> chaim
>>
>> On Wed, Jun 19, 2019 at 8:36 PM Kenneth Knowles <ke...@apache.org> wrote:
>> >
>> > This sounds like a fairly simply and useful change to the IO to output information about where it has written. It seems generally useful to always output something like this, instead of PDone.
>> >
>> > Kenn
>> >
>> > On Wed, Jun 19, 2019 at 5:42 AM Chaim Turkel <ch...@behalf.com> wrote:
>> >>
>> >> Hi,
>> >> I would like to write a status at the end of my pipline. I had
>> >> written about this in the past, and wanted to know if there are any
>> >> new ideas.
>> >> The problem is the end of the pipeline return PDone, and i can't do
>> >> anything with this.
>> >> So my senario is after i export data from mongo to google storage i
>> >> want to write to a db that the job was done with some extra
>> >> information.
>> >>
>> >> Chaim Turkel
>> >>
>> >> --
>> >>
>> >>
>> >> Loans are funded by
>> >> FinWise Bank, a Utah-chartered bank located in Sandy,
>> >> Utah, member FDIC, Equal
>> >> Opportunity Lender. Merchant Cash Advances are
>> >> made by Behalf. For more
>> >> information on ECOA, click here
>> >> <https://www.behalf.com/legal/ecoa/>. For important information about
>> >> opening a new
>> >> account, review Patriot Act procedures here
>> >> <https://www.behalf.com/legal/patriot/>.
>> >> Visit Legal
>> >> <https://www.behalf.com/legal/> to
>> >> review our comprehensive program terms,
>> >> conditions, and disclosures.
>>
>> --
>>
>>
>> Loans are funded by
>> FinWise Bank, a Utah-chartered bank located in Sandy,
>> Utah, member FDIC, Equal
>> Opportunity Lender. Merchant Cash Advances are
>> made by Behalf. For more
>> information on ECOA, click here
>> <https://www.behalf.com/legal/ecoa/>. For important information about
>> opening a new
>> account, review Patriot Act procedures here
>> <https://www.behalf.com/legal/patriot/>.
>> Visit Legal
>> <https://www.behalf.com/legal/> to
>> review our comprehensive program terms,
>> conditions, and disclosures.
>
>
--
Loans are funded by
FinWise Bank, a Utah-chartered bank located in Sandy,
Utah, member FDIC, Equal
Opportunity Lender. Merchant Cash Advances are
made by Behalf. For more
information on ECOA, click here
<https://www.behalf.com/legal/ecoa/>. For important information about
opening a new
account, review Patriot Act procedures here
<https://www.behalf.com/legal/patriot/>.
Visit Legal
<https://www.behalf.com/legal/> to
review our comprehensive program terms,
conditions, and disclosures.
Re: pipeline status tracing
Posted by Alexey Romanenko <ar...@gmail.com>.
I see that similar questions happen quite often in the last time, so, probably, it would make sense to add this “Wait.on()”-pattern to corresponding website documentation page [1].
[1] https://beam.apache.org/documentation/patterns/file-processing-patterns/ <https://beam.apache.org/documentation/patterns/file-processing-patterns/>
> On 20 Jun 2019, at 23:00, Eugene Kirpichov <ki...@google.com> wrote:
>
> If you're writing to files, you can already do this: FileIO.write() returns WriteFilesResult that you can use in combination with Wait.on() and JdbcIO.write() to write something to a database afterwards.
> Something like:
>
> PCollection<..> writeResult = data.apply(FileIO.write()...);
> Create.of("dummy").apply(Wait.on(writeResult)).apply(JdbcIO.write(...write to database...))
>
>
>
> On Wed, Jun 19, 2019 at 11:49 PM Chaim Turkel <chaim@behalf.com <ma...@behalf.com>> wrote:
> is it something you can do, or point me in the right direction?
> chaim
>
> On Wed, Jun 19, 2019 at 8:36 PM Kenneth Knowles <kenn@apache.org <ma...@apache.org>> wrote:
> >
> > This sounds like a fairly simply and useful change to the IO to output information about where it has written. It seems generally useful to always output something like this, instead of PDone.
> >
> > Kenn
> >
> > On Wed, Jun 19, 2019 at 5:42 AM Chaim Turkel <chaim@behalf.com <ma...@behalf.com>> wrote:
> >>
> >> Hi,
> >> I would like to write a status at the end of my pipline. I had
> >> written about this in the past, and wanted to know if there are any
> >> new ideas.
> >> The problem is the end of the pipeline return PDone, and i can't do
> >> anything with this.
> >> So my senario is after i export data from mongo to google storage i
> >> want to write to a db that the job was done with some extra
> >> information.
> >>
> >> Chaim Turkel
> >>
> >> --
> >>
> >>
> >> Loans are funded by
> >> FinWise Bank, a Utah-chartered bank located in Sandy,
> >> Utah, member FDIC, Equal
> >> Opportunity Lender. Merchant Cash Advances are
> >> made by Behalf. For more
> >> information on ECOA, click here
> >> <https://www.behalf.com/legal/ecoa/ <https://www.behalf.com/legal/ecoa/>>. For important information about
> >> opening a new
> >> account, review Patriot Act procedures here
> >> <https://www.behalf.com/legal/patriot/ <https://www.behalf.com/legal/patriot/>>.
> >> Visit Legal
> >> <https://www.behalf.com/legal/ <https://www.behalf.com/legal/>> to
> >> review our comprehensive program terms,
> >> conditions, and disclosures.
>
> --
>
>
> Loans are funded by
> FinWise Bank, a Utah-chartered bank located in Sandy,
> Utah, member FDIC, Equal
> Opportunity Lender. Merchant Cash Advances are
> made by Behalf. For more
> information on ECOA, click here
> <https://www.behalf.com/legal/ecoa/ <https://www.behalf.com/legal/ecoa/>>. For important information about
> opening a new
> account, review Patriot Act procedures here
> <https://www.behalf.com/legal/patriot/ <https://www.behalf.com/legal/patriot/>>.
> Visit Legal
> <https://www.behalf.com/legal/ <https://www.behalf.com/legal/>> to
> review our comprehensive program terms,
> conditions, and disclosures.
Re: pipeline status tracing
Posted by Eugene Kirpichov <ki...@google.com>.
If you're writing to files, you can already do this: FileIO.write() returns
WriteFilesResult that you can use in combination with Wait.on() and
JdbcIO.write() to write something to a database afterwards.
Something like:
PCollection<..> writeResult = data.apply(FileIO.write()...);
Create.of("dummy").apply(Wait.on(writeResult)).apply(JdbcIO.write(...write
to database...))
On Wed, Jun 19, 2019 at 11:49 PM Chaim Turkel <ch...@behalf.com> wrote:
> is it something you can do, or point me in the right direction?
> chaim
>
> On Wed, Jun 19, 2019 at 8:36 PM Kenneth Knowles <ke...@apache.org> wrote:
> >
> > This sounds like a fairly simply and useful change to the IO to output
> information about where it has written. It seems generally useful to always
> output something like this, instead of PDone.
> >
> > Kenn
> >
> > On Wed, Jun 19, 2019 at 5:42 AM Chaim Turkel <ch...@behalf.com> wrote:
> >>
> >> Hi,
> >> I would like to write a status at the end of my pipline. I had
> >> written about this in the past, and wanted to know if there are any
> >> new ideas.
> >> The problem is the end of the pipeline return PDone, and i can't do
> >> anything with this.
> >> So my senario is after i export data from mongo to google storage i
> >> want to write to a db that the job was done with some extra
> >> information.
> >>
> >> Chaim Turkel
> >>
> >> --
> >>
> >>
> >> Loans are funded by
> >> FinWise Bank, a Utah-chartered bank located in Sandy,
> >> Utah, member FDIC, Equal
> >> Opportunity Lender. Merchant Cash Advances are
> >> made by Behalf. For more
> >> information on ECOA, click here
> >> <https://www.behalf.com/legal/ecoa/>. For important information about
> >> opening a new
> >> account, review Patriot Act procedures here
> >> <https://www.behalf.com/legal/patriot/>.
> >> Visit Legal
> >> <https://www.behalf.com/legal/> to
> >> review our comprehensive program terms,
> >> conditions, and disclosures.
>
> --
>
>
> Loans are funded by
> FinWise Bank, a Utah-chartered bank located in Sandy,
> Utah, member FDIC, Equal
> Opportunity Lender. Merchant Cash Advances are
> made by Behalf. For more
> information on ECOA, click here
> <https://www.behalf.com/legal/ecoa/>. For important information about
> opening a new
> account, review Patriot Act procedures here
> <https://www.behalf.com/legal/patriot/>.
> Visit Legal
> <https://www.behalf.com/legal/> to
> review our comprehensive program terms,
> conditions, and disclosures.
>
Re: pipeline status tracing
Posted by Chaim Turkel <ch...@behalf.com>.
is it something you can do, or point me in the right direction?
chaim
On Wed, Jun 19, 2019 at 8:36 PM Kenneth Knowles <ke...@apache.org> wrote:
>
> This sounds like a fairly simply and useful change to the IO to output information about where it has written. It seems generally useful to always output something like this, instead of PDone.
>
> Kenn
>
> On Wed, Jun 19, 2019 at 5:42 AM Chaim Turkel <ch...@behalf.com> wrote:
>>
>> Hi,
>> I would like to write a status at the end of my pipline. I had
>> written about this in the past, and wanted to know if there are any
>> new ideas.
>> The problem is the end of the pipeline return PDone, and i can't do
>> anything with this.
>> So my senario is after i export data from mongo to google storage i
>> want to write to a db that the job was done with some extra
>> information.
>>
>> Chaim Turkel
>>
>> --
>>
>>
>> Loans are funded by
>> FinWise Bank, a Utah-chartered bank located in Sandy,
>> Utah, member FDIC, Equal
>> Opportunity Lender. Merchant Cash Advances are
>> made by Behalf. For more
>> information on ECOA, click here
>> <https://www.behalf.com/legal/ecoa/>. For important information about
>> opening a new
>> account, review Patriot Act procedures here
>> <https://www.behalf.com/legal/patriot/>.
>> Visit Legal
>> <https://www.behalf.com/legal/> to
>> review our comprehensive program terms,
>> conditions, and disclosures.
--
Loans are funded by
FinWise Bank, a Utah-chartered bank located in Sandy,
Utah, member FDIC, Equal
Opportunity Lender. Merchant Cash Advances are
made by Behalf. For more
information on ECOA, click here
<https://www.behalf.com/legal/ecoa/>. For important information about
opening a new
account, review Patriot Act procedures here
<https://www.behalf.com/legal/patriot/>.
Visit Legal
<https://www.behalf.com/legal/> to
review our comprehensive program terms,
conditions, and disclosures.
Re: pipeline status tracing
Posted by Kenneth Knowles <ke...@apache.org>.
This sounds like a fairly simply and useful change to the IO to output
information about where it has written. It seems generally useful to always
output something like this, instead of PDone.
Kenn
On Wed, Jun 19, 2019 at 5:42 AM Chaim Turkel <ch...@behalf.com> wrote:
> Hi,
> I would like to write a status at the end of my pipline. I had
> written about this in the past, and wanted to know if there are any
> new ideas.
> The problem is the end of the pipeline return PDone, and i can't do
> anything with this.
> So my senario is after i export data from mongo to google storage i
> want to write to a db that the job was done with some extra
> information.
>
> Chaim Turkel
>
> --
>
>
> Loans are funded by
> FinWise Bank, a Utah-chartered bank located in Sandy,
> Utah, member FDIC, Equal
> Opportunity Lender. Merchant Cash Advances are
> made by Behalf. For more
> information on ECOA, click here
> <https://www.behalf.com/legal/ecoa/>. For important information about
> opening a new
> account, review Patriot Act procedures here
> <https://www.behalf.com/legal/patriot/>.
> Visit Legal
> <https://www.behalf.com/legal/> to
> review our comprehensive program terms,
> conditions, and disclosures.
>