You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Johan de Braak <jo...@eleaf.com> on 2018/02/06 09:27:33 UTC
Failing Tasks Indication
Hi,
I am trying to make airflow mark tasks as failed when an exception is raised. I have saved a python script to disk which I would like to run with the BashOperator like this:
import sys
def myfunc(a):
if a == True:
return
else:
raise Exception
if __name__ == '__main__':
try:
myfunc(False)
except:
sys.exit(1)
When running code like this, I would expect that I would see a red circle (indicating failure) in my overview since the except part is excecuted. However, I see dark green indicating that the task was a succes. Could anyone explain how to make the tasks indicate it when they failed?
Thanks in advance,
Johan de Braak
Data Analyst
[Description: Beschrijving: Description: Description: M:\Project_Acquisition\ACQUISITION DOCUMENTS\09. Huisstijl en plaatjes\Huisstijl\LOGO ELEAF High Res.jpg]
johan.de.braak@eleaf.com<ma...@eleaf.com>
eLEAF Competence Center * www.eleaf.com<http://www.eleaf.com/>
This message is intended for the exclusive use of the person(s) mentioned as recipient(s) and may contain personal and/or confidential information. If you have received this message in error, please notify the sender and delete this message from your system immediately. Directly or indirectly copying, disclosing, distributing, printing, publicizing and/or in any way using this message or any part thereof by any means is strictly prohibited if you are not the intended recipient(s). This message and any associated attachments cannot be deemed as legally binding under any circumstances without the express written consent from eLEAF(c). eLEAF(c) is not responsible for any loss and/or damages arising from any errors and/or omissions in its e-mail messages. Reference to the own general terms and conditions by the other party is explicitly rejected by eLEAF(c). Please read our general terms and condition on http://www.eleaf.com/legal.
P You save 200 ml of water, 2g of CO2 and 2g of wood per page: Please print only if it is really necessary.
Re: Failing Tasks Indication
Posted by Taylor Edmiston <te...@gmail.com>.
Like Sean mentioned, bash (in scripts) swallows exit status codes from
commands by default.
https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/
Not sure if this fits your use case but the PythonOperator with your
existing code would work as expected.
By the way, for Boolean comparisons in Python it's preferred to use is over
==.
https://stackoverflow.com/a/27276841/149428
Best,
Taylor
On Thu, Feb 8, 2018 at 2:52 PM Sean Fern <fe...@fixlist.co> wrote:
> Is the bash script wrapping this python command running with "set -e" or
> "set -o errexit"?
>
> On Tue, Feb 6, 2018 at 4:27 AM, Johan de Braak <jo...@eleaf.com>
> wrote:
>
> > Hi,
> >
> >
> >
> > I am trying to make airflow mark tasks as failed when an exception is
> > raised. I have saved a python script to disk which I would like to run
> with
> > the BashOperator like this:
> >
> >
> >
> > import sys
> >
> > def myfunc(a):
> >
> > if a == True:
> >
> > return
> >
> > else:
> >
> > raise Exception
> >
> >
> >
> > if __name__ == '__main__':
> >
> > try:
> >
> > myfunc(False)
> >
> > except:
> >
> > sys.exit(1)
> >
> >
> >
> > When running code like this, I would expect that I would see a red circle
> > (indicating failure) in my overview since the except part is excecuted.
> > However, I see dark green indicating that the task was a succes. Could
> > anyone explain how to make the tasks indicate it when they failed?
> >
> >
> >
> > Thanks in advance,
> >
> >
> >
> > *Johan de Braak*
> >
> > Data Analyst
> >
> >
> >
> > [image: Description: Beschrijving: Description: Description:
> > M:\Project_Acquisition\ACQUISITION DOCUMENTS\09. Huisstijl en
> > plaatjes\Huisstijl\LOGO ELEAF High Res.jpg]
> >
> >
> > *johan.de.braak@eleaf.com <jo...@eleaf.com> *
> > eLEAF Competence Center • *www.eleaf.com <http://www.eleaf.com/>*
> >
> > This message is intended for the exclusive use of the person(s) mentioned
> > as recipient(s) and may contain personal and/or confidential information.
> > If you have received this message in error, please notify the sender and
> > delete this message from your system immediately. Directly or indirectly
> > copying, disclosing, distributing, printing, publicizing and/or in any
> way
> > using this message or any part thereof by any means is strictly
> prohibited
> > if you are not the intended recipient(s). This message and any associated
> > attachments cannot be deemed as legally binding under any circumstances
> > without the express written consent from eLEAF©. eLEAF© is not
> responsible
> > for any loss and/or damages arising from any errors and/or omissions in
> its
> > e-mail messages. Reference to the own general terms and conditions by the
> > other party is explicitly rejected by eLEAF©. Please read our general
> terms
> > and condition on http://www.eleaf.com/legal.
> >
> > P You save 200 ml of water, 2g of CO2 and 2g of wood per page: Please
> > print only if it is really necessary.
> >
> >
> >
>
--
*Taylor Edmiston*
TEdmiston.com <https://www.tedmiston.com/> | Blog
<http://blog.tedmiston.com>
Stack Overflow CV <https://stackoverflow.com/story/taylor> | LinkedIn
<https://www.linkedin.com/in/tedmiston/> | AngelList
<https://angel.co/taylor>
Re: Failing Tasks Indication
Posted by Sean Fern <fe...@fixlist.co>.
Is the bash script wrapping this python command running with "set -e" or
"set -o errexit"?
On Tue, Feb 6, 2018 at 4:27 AM, Johan de Braak <jo...@eleaf.com>
wrote:
> Hi,
>
>
>
> I am trying to make airflow mark tasks as failed when an exception is
> raised. I have saved a python script to disk which I would like to run with
> the BashOperator like this:
>
>
>
> import sys
>
> def myfunc(a):
>
> if a == True:
>
> return
>
> else:
>
> raise Exception
>
>
>
> if __name__ == '__main__':
>
> try:
>
> myfunc(False)
>
> except:
>
> sys.exit(1)
>
>
>
> When running code like this, I would expect that I would see a red circle
> (indicating failure) in my overview since the except part is excecuted.
> However, I see dark green indicating that the task was a succes. Could
> anyone explain how to make the tasks indicate it when they failed?
>
>
>
> Thanks in advance,
>
>
>
> *Johan de Braak*
>
> Data Analyst
>
>
>
> [image: Description: Beschrijving: Description: Description:
> M:\Project_Acquisition\ACQUISITION DOCUMENTS\09. Huisstijl en
> plaatjes\Huisstijl\LOGO ELEAF High Res.jpg]
>
>
> *johan.de.braak@eleaf.com <jo...@eleaf.com> *
> eLEAF Competence Center • *www.eleaf.com <http://www.eleaf.com/>*
>
> This message is intended for the exclusive use of the person(s) mentioned
> as recipient(s) and may contain personal and/or confidential information.
> If you have received this message in error, please notify the sender and
> delete this message from your system immediately. Directly or indirectly
> copying, disclosing, distributing, printing, publicizing and/or in any way
> using this message or any part thereof by any means is strictly prohibited
> if you are not the intended recipient(s). This message and any associated
> attachments cannot be deemed as legally binding under any circumstances
> without the express written consent from eLEAF©. eLEAF© is not responsible
> for any loss and/or damages arising from any errors and/or omissions in its
> e-mail messages. Reference to the own general terms and conditions by the
> other party is explicitly rejected by eLEAF©. Please read our general terms
> and condition on http://www.eleaf.com/legal.
>
> P You save 200 ml of water, 2g of CO2 and 2g of wood per page: Please
> print only if it is really necessary.
>
>
>