You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by heng gu <he...@yahoo.com.INVALID> on 2020/02/24 14:50:27 UTC
task failed without running
I have this dag with a branchpythonoperator task kicking off many of 24 tasks, in this case, 4 tasks. 2 of the tasks were successful, the other two (register_YZ, register_ZY) were failed without running (see the attached UI screen shots). There is no log for tasks register_YZ and register_ZY. I am using Celery Executor and running 12 workers executing register_XX tasks. I am using airflow version 1.10.6. Any idea how to fix it?
RE: task failed without running
Posted by "Shaw, Damian P. " <da...@credit-suisse.com>.
I get the same error in airflow-worker.err, the real underlying cause only revealed itself in airflow-worker.out
Damian
-----Original Message-----
From: heng gu <he...@yahoo.com.INVALID>
Sent: Wednesday, February 26, 2020 12:35
To: dev@airflow.apache.org
Subject: Re: task failed without running
thanks for your help! I have similar DB timeout problem with other tasks, but not this one. The error message I got for this one from airflow-worker.err is
File "python3.6/site-packages/airflow/executors/celery_executor.py", line 67, in execute_command close_fds=True, env=env) File "pypacks/lib/python3.6/subprocess.py", line 311, in check_call raise CalledProcessError(retcode, cmd) Apparently Celery executor ran the worker in subprocess.py and returned non-zero retcode.
Heng
On Wednesday, February 26, 2020, 09:57:29 AM EST, Shaw, Damian P. <da...@credit-suisse.com> wrote:
FYI,
I've had this issue for a few different reasons, the first was that my worker failed to read the file (this for me was a permission issue). Now I'm having it a second time and it appears Airflow timeouts connecting to the DB while setting up the task, for me I can see this in the stdout of the Airflow worker (not in stderr though).
I am looking to increase the connect timeout using this config item: https://airflow.readthedocs.io/en/latest/configurations-ref.html#sql-alchemy-connect-args and this sqlalchemy option: https://stackoverflow.com/a/35640876/2958068
(side note: concerningly this option doesn't seem to be documented in the new documentation https://airflow.apache.org/docs/stable/configurations-ref.html . But I found the exact code in question: https://github.com/apache/airflow/blob/6ec9664a6be151e8fa94ef207cd98b459d392af2/airflow/settings.py#L187 )
This is a snippet of the exception I am getting, it seems to be caused during session.merge(self) in _check_and_change_state_before_execution. Check if you are logging your Airflow worker stdout somewhere and see if you have a similar exception:
Traceback (most recent call last):
File ".../lib/python3.7/site-packages/pymysql/connections.py", line 583, in connect
**kwargs)
File ".../lib/python3.7/socket.py", line 727, in create_connection
raise err
File ".../lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ".../lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect
return fn()
...
File ".../lib/python3.7/site-packages/pymysql/connections.py", line 630, in connect
raise exc
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'my_db.my_domain.net' (timed out)")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File ".../bin/airflow", line 37, in <module>
args.func(args)
...
File ".../lib/python3.7/site-packages/airflow/models/taskinstance.py", line 852, in _check_and_change_state_before_execution
session.merge(self)
...
File ".../lib/python3.7/site-packages/pymysql/connections.py", line 630, in connect
raise exc
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'my_db.my_domain.net' (timed out)")
Regards
Damian
-----Original Message-----
From: Mehmet Ersoy <me...@gmail.com>
Sent: Wednesday, February 26, 2020 03:18
To: dev@airflow.apache.org
Subject: Re: task failed without running
Hi Heng,
I have same problem. Failed jobs do not dump log files. Do you have this problem too?
Have you been using Redis as message queue? Also can you send configurations related parallelism and concurrent in airflow.cfg file?
Best regards,
Mehmet.
heng gu <he...@yahoo.com.invalid>, 24 Şub 2020 Pzt, 17:51 tarihinde şunu
yazdı:
> I have this dag with a branchpythonoperator task kicking off many of
> 24 tasks, in this case, 4 tasks. 2 of the tasks were successful, the
> other two (register_YZ, register_ZY) were failed without running (see
> the attached UI screen shots). There is no log for tasks register_YZ
> and register_ZY. I am using Celery Executor and running 12 workers
> executing register_XX tasks. I am using airflow version 1.10.6. Any idea how to fix it?
>
--
Mehmet ERSOY
===============================================================================
Please access the attached hyperlink for an important electronic communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===============================================================================
===============================================================================
Please access the attached hyperlink for an important electronic communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===============================================================================
Re: task failed without running
Posted by heng gu <he...@yahoo.com.INVALID>.
thanks for your help! I have similar DB timeout problem with other tasks, but not this one. The error message I got for this one from airflow-worker.err is
File "python3.6/site-packages/airflow/executors/celery_executor.py", line 67, in execute_command close_fds=True, env=env) File "pypacks/lib/python3.6/subprocess.py", line 311, in check_call raise CalledProcessError(retcode, cmd)
Apparently Celery executor ran the worker in subprocess.py and returned non-zero retcode.
Heng
On Wednesday, February 26, 2020, 09:57:29 AM EST, Shaw, Damian P. <da...@credit-suisse.com> wrote:
FYI,
I've had this issue for a few different reasons, the first was that my worker failed to read the file (this for me was a permission issue). Now I'm having it a second time and it appears
Airflow timeouts connecting to the DB while setting up the task, for me I can see this in the stdout of the Airflow worker (not in stderr though).
I am looking to increase the connect timeout using this config item: https://airflow.readthedocs.io/en/latest/configurations-ref.html#sql-alchemy-connect-args and this sqlalchemy option: https://stackoverflow.com/a/35640876/2958068
(side note: concerningly this option doesn't seem to be documented in the new documentation https://airflow.apache.org/docs/stable/configurations-ref.html . But I found the exact code in question: https://github.com/apache/airflow/blob/6ec9664a6be151e8fa94ef207cd98b459d392af2/airflow/settings.py#L187 )
This is a snippet of the exception I am getting, it seems to be caused during session.merge(self) in _check_and_change_state_before_execution. Check if you are logging your Airflow worker stdout somewhere and see if you have a similar exception:
Traceback (most recent call last):
File ".../lib/python3.7/site-packages/pymysql/connections.py", line 583, in connect
**kwargs)
File ".../lib/python3.7/socket.py", line 727, in create_connection
raise err
File ".../lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ".../lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect
return fn()
...
File ".../lib/python3.7/site-packages/pymysql/connections.py", line 630, in connect
raise exc
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'my_db.my_domain.net' (timed out)")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File ".../bin/airflow", line 37, in <module>
args.func(args)
...
File ".../lib/python3.7/site-packages/airflow/models/taskinstance.py", line 852, in _check_and_change_state_before_execution
session.merge(self)
...
File ".../lib/python3.7/site-packages/pymysql/connections.py", line 630, in connect
raise exc
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'my_db.my_domain.net' (timed out)")
Regards
Damian
-----Original Message-----
From: Mehmet Ersoy <me...@gmail.com>
Sent: Wednesday, February 26, 2020 03:18
To: dev@airflow.apache.org
Subject: Re: task failed without running
Hi Heng,
I have same problem. Failed jobs do not dump log files. Do you have this problem too?
Have you been using Redis as message queue? Also can you send configurations related parallelism and concurrent in airflow.cfg file?
Best regards,
Mehmet.
heng gu <he...@yahoo.com.invalid>, 24 Şub 2020 Pzt, 17:51 tarihinde şunu
yazdı:
> I have this dag with a branchpythonoperator task kicking off many of
> 24 tasks, in this case, 4 tasks. 2 of the tasks were successful, the
> other two (register_YZ, register_ZY) were failed without running (see
> the attached UI screen shots). There is no log for tasks register_YZ
> and register_ZY. I am using Celery Executor and running 12 workers
> executing register_XX tasks. I am using airflow version 1.10.6. Any idea how to fix it?
>
--
Mehmet ERSOY
===============================================================================
Please access the attached hyperlink for an important electronic communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===============================================================================
RE: task failed without running
Posted by "Shaw, Damian P. " <da...@credit-suisse.com>.
FYI,
I've had this issue for a few different reasons, the first was that my worker failed to read the file (this for me was a permission issue). Now I'm having it a second time and it appears
Airflow timeouts connecting to the DB while setting up the task, for me I can see this in the stdout of the Airflow worker (not in stderr though).
I am looking to increase the connect timeout using this config item: https://airflow.readthedocs.io/en/latest/configurations-ref.html#sql-alchemy-connect-args and this sqlalchemy option: https://stackoverflow.com/a/35640876/2958068
(side note: concerningly this option doesn't seem to be documented in the new documentation https://airflow.apache.org/docs/stable/configurations-ref.html . But I found the exact code in question: https://github.com/apache/airflow/blob/6ec9664a6be151e8fa94ef207cd98b459d392af2/airflow/settings.py#L187 )
This is a snippet of the exception I am getting, it seems to be caused during session.merge(self) in _check_and_change_state_before_execution. Check if you are logging your Airflow worker stdout somewhere and see if you have a similar exception:
Traceback (most recent call last):
File ".../lib/python3.7/site-packages/pymysql/connections.py", line 583, in connect
**kwargs)
File ".../lib/python3.7/socket.py", line 727, in create_connection
raise err
File ".../lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ".../lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect
return fn()
...
File ".../lib/python3.7/site-packages/pymysql/connections.py", line 630, in connect
raise exc
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'my_db.my_domain.net' (timed out)")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File ".../bin/airflow", line 37, in <module>
args.func(args)
...
File ".../lib/python3.7/site-packages/airflow/models/taskinstance.py", line 852, in _check_and_change_state_before_execution
session.merge(self)
...
File ".../lib/python3.7/site-packages/pymysql/connections.py", line 630, in connect
raise exc
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'my_db.my_domain.net' (timed out)")
Regards
Damian
-----Original Message-----
From: Mehmet Ersoy <me...@gmail.com>
Sent: Wednesday, February 26, 2020 03:18
To: dev@airflow.apache.org
Subject: Re: task failed without running
Hi Heng,
I have same problem. Failed jobs do not dump log files. Do you have this problem too?
Have you been using Redis as message queue? Also can you send configurations related parallelism and concurrent in airflow.cfg file?
Best regards,
Mehmet.
heng gu <he...@yahoo.com.invalid>, 24 Şub 2020 Pzt, 17:51 tarihinde şunu
yazdı:
> I have this dag with a branchpythonoperator task kicking off many of
> 24 tasks, in this case, 4 tasks. 2 of the tasks were successful, the
> other two (register_YZ, register_ZY) were failed without running (see
> the attached UI screen shots). There is no log for tasks register_YZ
> and register_ZY. I am using Celery Executor and running 12 workers
> executing register_XX tasks. I am using airflow version 1.10.6. Any idea how to fix it?
>
--
Mehmet ERSOY
===============================================================================
Please access the attached hyperlink for an important electronic communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===============================================================================
RE: task failed without running
Posted by "Shaw, Damian P. " <da...@credit-suisse.com>.
You should look in airflow-worker.out for an exception that's happening at about the same time.
Damian
-----Original Message-----
From: heng gu <he...@yahoo.com.INVALID>
Sent: Wednesday, February 26, 2020 11:55
To: dev@airflow.apache.org
Subject: Re: task failed without running
I got an advice from slack to check airflow-worker.err, and here is what I found
File "python3.6/site-packages/airflow/executors/celery_executor.py", line 67, in execute_command close_fds=True, env=env) File "pypacks/lib/python3.6/subprocess.py", line 311, in check_call raise CalledProcessError(retcode, cmd) I am using rabbitmq. And here is the parallelism and concurrency set in cfg file:
# The amount of parallelism as a setting to the executor. This defines# the max number of task instances that should run simultaneously# on this airflow installationparallelism = 32 # The number of task instances allowed to run concurrently by the schedulerdag_concurrency = 16
for this dag, I set concurrency=16, max_active_runs=8 Thanks, Heng
On Wednesday, February 26, 2020, 03:18:27 AM EST, Mehmet Ersoy <me...@gmail.com> wrote:
Hi Heng,
I have same problem. Failed jobs do not dump log files. Do you have this problem too?
Have you been using Redis as message queue? Also can you send configurations related parallelism and concurrent in airflow.cfg file?
Best regards,
Mehmet.
heng gu <he...@yahoo.com.invalid>, 24 Şub 2020 Pzt, 17:51 tarihinde şunu
yazdı:
> I have this dag with a branchpythonoperator task kicking off many of
> 24 tasks, in this case, 4 tasks. 2 of the tasks were successful, the
> other two (register_YZ, register_ZY) were failed without running (see
> the attached UI screen shots). There is no log for tasks register_YZ
> and register_ZY. I am using Celery Executor and running 12 workers
> executing register_XX tasks. I am using airflow version 1.10.6. Any idea how to fix it?
>
--
Mehmet ERSOY
===============================================================================
Please access the attached hyperlink for an important electronic communications disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===============================================================================
Re: task failed without running
Posted by heng gu <he...@yahoo.com.INVALID>.
I got an advice from slack to check airflow-worker.err, and here is what I found
File "python3.6/site-packages/airflow/executors/celery_executor.py", line 67, in execute_command close_fds=True, env=env) File "pypacks/lib/python3.6/subprocess.py", line 311, in check_call raise CalledProcessError(retcode, cmd)
I am using rabbitmq. And here is the parallelism and concurrency set in cfg file:
# The amount of parallelism as a setting to the executor. This defines# the max number of task instances that should run simultaneously# on this airflow installationparallelism = 32
# The number of task instances allowed to run concurrently by the schedulerdag_concurrency = 16
for this dag, I set concurrency=16, max_active_runs=8
Thanks,
Heng
On Wednesday, February 26, 2020, 03:18:27 AM EST, Mehmet Ersoy <me...@gmail.com> wrote:
Hi Heng,
I have same problem. Failed jobs do not dump log files. Do you have this
problem too?
Have you been using Redis as message queue? Also can you send
configurations related parallelism and concurrent in airflow.cfg file?
Best regards,
Mehmet.
heng gu <he...@yahoo.com.invalid>, 24 Şub 2020 Pzt, 17:51 tarihinde şunu
yazdı:
> I have this dag with a branchpythonoperator task kicking off many of 24
> tasks, in this case, 4 tasks. 2 of the tasks were successful, the other two
> (register_YZ, register_ZY) were failed without running (see the attached UI
> screen shots). There is no log for tasks register_YZ and register_ZY. I am
> using Celery Executor and running 12 workers executing register_XX tasks. I
> am using airflow version 1.10.6. Any idea how to fix it?
>
--
Mehmet ERSOY
Re: task failed without running
Posted by Mehmet Ersoy <me...@gmail.com>.
Hi Heng,
I have same problem. Failed jobs do not dump log files. Do you have this
problem too?
Have you been using Redis as message queue? Also can you send
configurations related parallelism and concurrent in airflow.cfg file?
Best regards,
Mehmet.
heng gu <he...@yahoo.com.invalid>, 24 Şub 2020 Pzt, 17:51 tarihinde şunu
yazdı:
> I have this dag with a branchpythonoperator task kicking off many of 24
> tasks, in this case, 4 tasks. 2 of the tasks were successful, the other two
> (register_YZ, register_ZY) were failed without running (see the attached UI
> screen shots). There is no log for tasks register_YZ and register_ZY. I am
> using Celery Executor and running 12 workers executing register_XX tasks. I
> am using airflow version 1.10.6. Any idea how to fix it?
>
--
Mehmet ERSOY