You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Chaim Turkel <ch...@behalf.com> on 2019/07/07 07:59:04 UTC
pipeline timeout
Hi,
I have a pipeline that usually takes 15-30 minutes. Sometimes things
get stuck (from 3rd party side). I would like to know if there is a
way to cancel the job if it is running for more than x minutes? I know
there is a cli command but i would like it either on the pipeline
config or in the python sdk.
Any ideas?
Chaim Turkel
--
Loans are funded by
FinWise Bank, a Utah-chartered bank located in Sandy,
Utah, member FDIC, Equal
Opportunity Lender. Merchant Cash Advances are
made by Behalf. For more
information on ECOA, click here
<https://www.behalf.com/legal/ecoa/>. For important information about
opening a new
account, review Patriot Act procedures here
<https://www.behalf.com/legal/patriot/>.
Visit Legal
<https://www.behalf.com/legal/> to
review our comprehensive program terms,
conditions, and disclosures.
Re: pipeline timeout
Posted by Hannah Jiang <ha...@google.com>.
Hi Chaim
If you want to cancel the job from airflow, here is how you can set
timeout. (Using *execution_timeout*)
aggregate_db_message_job = BashOperator(
task_id='aggregate_db_message_job',
execution_timeout=timedelta(hours=3),
pool='ep_data_pipeline_db_msg_agg',
bash_command=aggregate_db_message_job_cmd,
dag=dag)
Hope this helps.
Hannah
On Tue, Jul 9, 2019 at 11:15 AM Rui Wang <ru...@google.com> wrote:
> Just curious: how do you use python to kick off a Java pipeline in
> airflow?
>
> And, could you go to a direction that you have an airflow operator(maybe
> a bash operator to execute a Jar?) to run such Java code:
>
> result = p.run()
> result.wait_until_finish(duration=TIMEOUT_SEC)
> if not PipelineState.is_terminal(result.state):
> result.cancel()
>
> ?
>
> On Tue, Jul 9, 2019 at 1:14 AM Chaim Turkel <ch...@behalf.com> wrote:
>
>> sorry for not being explicit. My pipeline is in java and i am
>> executing it using python in airflow.
>> I would like from airflow to cancel the pipeline if running for more
>> than x minutes.
>> Currently i am doing this using the cli, but it is not optimal
>>
>> chaim
>>
>> On Mon, Jul 8, 2019 at 7:24 PM Mark Liu <ma...@google.com> wrote:
>> >
>> > Hi Chaim,
>> >
>> > You can checkout PipelineResult class and do something like:
>> >
>> > result = p.run()
>> > result.wait_until_finish(duration=TIMEOUT_SEC)
>> > if not PipelineState.is_terminal(result.state):
>> > result.cancel()
>> >
>> > The implementation of PipelineResult depends on what runner you choose.
>> And you may find more useful functions in its subclass.
>> >
>> > Mark
>> >
>> >
>> > On Sun, Jul 7, 2019 at 12:59 AM Chaim Turkel <ch...@behalf.com> wrote:
>> >>
>> >> Hi,
>> >> I have a pipeline that usually takes 15-30 minutes. Sometimes things
>> >> get stuck (from 3rd party side). I would like to know if there is a
>> >> way to cancel the job if it is running for more than x minutes? I know
>> >> there is a cli command but i would like it either on the pipeline
>> >> config or in the python sdk.
>> >> Any ideas?
>> >>
>> >> Chaim Turkel
>> >>
>> >> --
>> >>
>> >>
>> >> Loans are funded by
>> >> FinWise Bank, a Utah-chartered bank located in Sandy,
>> >> Utah, member FDIC, Equal
>> >> Opportunity Lender. Merchant Cash Advances are
>> >> made by Behalf. For more
>> >> information on ECOA, click here
>> >> <https://www.behalf.com/legal/ecoa/>. For important information about
>> >> opening a new
>> >> account, review Patriot Act procedures here
>> >> <https://www.behalf.com/legal/patriot/>.
>> >> Visit Legal
>> >> <https://www.behalf.com/legal/> to
>> >> review our comprehensive program terms,
>> >> conditions, and disclosures.
>>
>> --
>>
>>
>> Loans are funded by
>> FinWise Bank, a Utah-chartered bank located in Sandy,
>> Utah, member FDIC, Equal
>> Opportunity Lender. Merchant Cash Advances are
>> made by Behalf. For more
>> information on ECOA, click here
>> <https://www.behalf.com/legal/ecoa/>. For important information about
>> opening a new
>> account, review Patriot Act procedures here
>> <https://www.behalf.com/legal/patriot/>.
>> Visit Legal
>> <https://www.behalf.com/legal/> to
>> review our comprehensive program terms,
>> conditions, and disclosures.
>>
>
Re: pipeline timeout
Posted by Rui Wang <ru...@google.com>.
Just curious: how do you use python to kick off a Java pipeline in airflow?
And, could you go to a direction that you have an airflow operator(maybe a
bash operator to execute a Jar?) to run such Java code:
result = p.run()
result.wait_until_finish(duration=TIMEOUT_SEC)
if not PipelineState.is_terminal(result.state):
result.cancel()
?
On Tue, Jul 9, 2019 at 1:14 AM Chaim Turkel <ch...@behalf.com> wrote:
> sorry for not being explicit. My pipeline is in java and i am
> executing it using python in airflow.
> I would like from airflow to cancel the pipeline if running for more
> than x minutes.
> Currently i am doing this using the cli, but it is not optimal
>
> chaim
>
> On Mon, Jul 8, 2019 at 7:24 PM Mark Liu <ma...@google.com> wrote:
> >
> > Hi Chaim,
> >
> > You can checkout PipelineResult class and do something like:
> >
> > result = p.run()
> > result.wait_until_finish(duration=TIMEOUT_SEC)
> > if not PipelineState.is_terminal(result.state):
> > result.cancel()
> >
> > The implementation of PipelineResult depends on what runner you choose.
> And you may find more useful functions in its subclass.
> >
> > Mark
> >
> >
> > On Sun, Jul 7, 2019 at 12:59 AM Chaim Turkel <ch...@behalf.com> wrote:
> >>
> >> Hi,
> >> I have a pipeline that usually takes 15-30 minutes. Sometimes things
> >> get stuck (from 3rd party side). I would like to know if there is a
> >> way to cancel the job if it is running for more than x minutes? I know
> >> there is a cli command but i would like it either on the pipeline
> >> config or in the python sdk.
> >> Any ideas?
> >>
> >> Chaim Turkel
> >>
> >> --
> >>
> >>
> >> Loans are funded by
> >> FinWise Bank, a Utah-chartered bank located in Sandy,
> >> Utah, member FDIC, Equal
> >> Opportunity Lender. Merchant Cash Advances are
> >> made by Behalf. For more
> >> information on ECOA, click here
> >> <https://www.behalf.com/legal/ecoa/>. For important information about
> >> opening a new
> >> account, review Patriot Act procedures here
> >> <https://www.behalf.com/legal/patriot/>.
> >> Visit Legal
> >> <https://www.behalf.com/legal/> to
> >> review our comprehensive program terms,
> >> conditions, and disclosures.
>
> --
>
>
> Loans are funded by
> FinWise Bank, a Utah-chartered bank located in Sandy,
> Utah, member FDIC, Equal
> Opportunity Lender. Merchant Cash Advances are
> made by Behalf. For more
> information on ECOA, click here
> <https://www.behalf.com/legal/ecoa/>. For important information about
> opening a new
> account, review Patriot Act procedures here
> <https://www.behalf.com/legal/patriot/>.
> Visit Legal
> <https://www.behalf.com/legal/> to
> review our comprehensive program terms,
> conditions, and disclosures.
>
Re: pipeline timeout
Posted by Chaim Turkel <ch...@behalf.com>.
sorry for not being explicit. My pipeline is in java and i am
executing it using python in airflow.
I would like from airflow to cancel the pipeline if running for more
than x minutes.
Currently i am doing this using the cli, but it is not optimal
chaim
On Mon, Jul 8, 2019 at 7:24 PM Mark Liu <ma...@google.com> wrote:
>
> Hi Chaim,
>
> You can checkout PipelineResult class and do something like:
>
> result = p.run()
> result.wait_until_finish(duration=TIMEOUT_SEC)
> if not PipelineState.is_terminal(result.state):
> result.cancel()
>
> The implementation of PipelineResult depends on what runner you choose. And you may find more useful functions in its subclass.
>
> Mark
>
>
> On Sun, Jul 7, 2019 at 12:59 AM Chaim Turkel <ch...@behalf.com> wrote:
>>
>> Hi,
>> I have a pipeline that usually takes 15-30 minutes. Sometimes things
>> get stuck (from 3rd party side). I would like to know if there is a
>> way to cancel the job if it is running for more than x minutes? I know
>> there is a cli command but i would like it either on the pipeline
>> config or in the python sdk.
>> Any ideas?
>>
>> Chaim Turkel
>>
>> --
>>
>>
>> Loans are funded by
>> FinWise Bank, a Utah-chartered bank located in Sandy,
>> Utah, member FDIC, Equal
>> Opportunity Lender. Merchant Cash Advances are
>> made by Behalf. For more
>> information on ECOA, click here
>> <https://www.behalf.com/legal/ecoa/>. For important information about
>> opening a new
>> account, review Patriot Act procedures here
>> <https://www.behalf.com/legal/patriot/>.
>> Visit Legal
>> <https://www.behalf.com/legal/> to
>> review our comprehensive program terms,
>> conditions, and disclosures.
--
Loans are funded by
FinWise Bank, a Utah-chartered bank located in Sandy,
Utah, member FDIC, Equal
Opportunity Lender. Merchant Cash Advances are
made by Behalf. For more
information on ECOA, click here
<https://www.behalf.com/legal/ecoa/>. For important information about
opening a new
account, review Patriot Act procedures here
<https://www.behalf.com/legal/patriot/>.
Visit Legal
<https://www.behalf.com/legal/> to
review our comprehensive program terms,
conditions, and disclosures.
Re: pipeline timeout
Posted by Mark Liu <ma...@google.com>.
Hi Chaim,
You can checkout PipelineResult
<https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/runner.py#L342>
class and do something like:
result = p.run()
result.wait_until_finish(duration=TIMEOUT_SEC)
if not PipelineState.is_terminal(result.state):
result.cancel()
The implementation of PipelineResult
<https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/runner.py#L342>
depends on what runner you choose. And you may find more useful functions
in its subclass.
Mark
On Sun, Jul 7, 2019 at 12:59 AM Chaim Turkel <ch...@behalf.com> wrote:
> Hi,
> I have a pipeline that usually takes 15-30 minutes. Sometimes things
> get stuck (from 3rd party side). I would like to know if there is a
> way to cancel the job if it is running for more than x minutes? I know
> there is a cli command but i would like it either on the pipeline
> config or in the python sdk.
> Any ideas?
>
> Chaim Turkel
>
> --
>
>
> Loans are funded by
> FinWise Bank, a Utah-chartered bank located in Sandy,
> Utah, member FDIC, Equal
> Opportunity Lender. Merchant Cash Advances are
> made by Behalf. For more
> information on ECOA, click here
> <https://www.behalf.com/legal/ecoa/>. For important information about
> opening a new
> account, review Patriot Act procedures here
> <https://www.behalf.com/legal/patriot/>.
> Visit Legal
> <https://www.behalf.com/legal/> to
> review our comprehensive program terms,
> conditions, and disclosures.
>