You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@airflow.apache.org by Daniel Kipnis <dk...@ceceliahealth.com> on 2020/02/05 23:40:37 UTC
DAG task no issues from airflow test, but abends with no messages
Hi,
Created new DAG with 3 tasks, successfully verified all tasks by using “airflow test”, but still failed from scheduler or run without any log details.
1. airflow test five9_load_dag f9_ext_s3 2020-01-01 generated proper result and messages
2. airflow run five9_load_dag f9_ext_s3 2020-01-01 {timeout.py:42} ERROR - Process timed out, PID: 9921
3. From Airflow task Log: No information
[cid:image001.jpg@01D5DC53.C138A950]
1. From Task Instance Details specified log
[2020-02-05 23:31:07,971] {logging_mixin.py:95} INFO - Sending to executor.
[2020-02-05 23:31:07,971] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:07,971[0m] {[34mbase_executor.py:[0m59} INFO[0m - Adding to queue: [1m['airflow', 'run', 'five9_load_dag', 'f9_ext_s3', '2020-01-01T00:00:00+00:00', '--local', '--pool', 'default_pool', '-sd', 'DAGS_FOLDER/five9_load_dag.py'][0m[0m
[2020-02-05 23:31:10,088] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:10,088[0m] {[34mtimeout.py:[0m42} [31mERROR[0m - [31mProcess timed out, PID: [1m10601[0m[0m
[2020-02-05 23:31:16,345] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:16,344[0m] {[34mtimeout.py:[0m42} [31mERROR[0m - [31mProcess timed out, PID: [1m10647[0m[0m
[2020-02-05 23:31:18,460] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:18,460[0m] {[34mtimeout.py:[0m42} [31mERROR[0m - [31mProcess timed out, PID: [1m10658[0m[0m
No other errors or details.
Thanks for support.
Regards, Daniel Kipnis
________________________________
________________________________
The information contained in this transmission may contain privileged and confidential information, including patient information protected by federal and state privacy laws. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original messages.
Re: DAG task no issues from airflow test, but abends with no messages
Posted by Daniel Kipnis <dk...@ceceliahealth.com>.
The issue may related to differences between test and run commands as they produce different result. In case of any long running processes, test command should also fail, but it consistently running with result within seconds.
Regards, Daniel Kipnis.
On Feb 5, 2020, at 6:42 PM, Ash Berlin-Taylor <as...@apache.org> wrote:
The "Process timed out" means that something is taking a long time and getting killed by a timeout before it can do anything.
Is the machine or DB where this is running under a lot of load?
On Feb 5 2020, at 11:40 pm, Daniel Kipnis <dk...@ceceliahealth.com> wrote:
Hi,
Created new DAG with 3 tasks, successfully verified all tasks by using “airflow test”, but still failed from scheduler or run without any log details.
1.
airflow test five9_load_dag f9_ext_s3 2020-01-01 generated proper result and messages
2.
airflow run five9_load_dag f9_ext_s3 2020-01-01 {timeout.py:42} ERROR - Process timed out, PID: 9921
3.
From Airflow task Log: No information
1.
From Task Instance Details specified log
[2020-02-05 23:31:07,971] {logging_mixin.py:95} INFO - Sending to executor.
[2020-02-05 23:31:07,971] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:07,971[0m] {[34mbase_executor.py:[0m59} INFO[0m - Adding to queue: [1m['airflow', 'run', 'five9_load_dag', 'f9_ext_s3', '2020-01-01T00:00:00+00:00', '--local', '--pool', 'default_pool', '-sd', 'DAGS_FOLDER/five9_load_dag.py'][0m[0m
[2020-02-05 23:31:10,088] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:10,088[0m] {[34mtimeout.py:[0m42} [31mERROR[0m - [31mProcess timed out, PID: [1m10601[0m[0m
[2020-02-05 23:31:16,345] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:16,344[0m] {[34mtimeout.py:[0m42} [31mERROR[0m - [31mProcess timed out, PID: [1m10647[0m[0m
[2020-02-05 23:31:18,460] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:18,460[0m] {[34mtimeout.py:[0m42} [31mERROR[0m - [31mProcess timed out, PID: [1m10658[0m[0m
No other errors or details.
Thanks for support.
Regards, Daniel Kipnis
The information contained in this transmission may contain privileged and confidential information, including patient information protected by federal and state privacy laws. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original messages.
Re: DAG task no issues from airflow test, but abends with no messages
Posted by Daniel Kipnis <dk...@ceceliahealth.com>.
The issue may related to differences between test and run commands as they produce different result. In case of any long running processes, test command should also fail, but it consistently running with result within seconds.
Regards, Daniel Kipnis.
On Feb 5, 2020, at 6:42 PM, Ash Berlin-Taylor <as...@apache.org> wrote:
The "Process timed out" means that something is taking a long time and getting killed by a timeout before it can do anything.
Is the machine or DB where this is running under a lot of load?
On Feb 5 2020, at 11:40 pm, Daniel Kipnis <dk...@ceceliahealth.com> wrote:
Hi,
Created new DAG with 3 tasks, successfully verified all tasks by using “airflow test”, but still failed from scheduler or run without any log details.
1.
airflow test five9_load_dag f9_ext_s3 2020-01-01 generated proper result and messages
2.
airflow run five9_load_dag f9_ext_s3 2020-01-01 {timeout.py:42} ERROR - Process timed out, PID: 9921
3.
From Airflow task Log: No information
1.
From Task Instance Details specified log
[2020-02-05 23:31:07,971] {logging_mixin.py:95} INFO - Sending to executor.
[2020-02-05 23:31:07,971] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:07,971[0m] {[34mbase_executor.py:[0m59} INFO[0m - Adding to queue: [1m['airflow', 'run', 'five9_load_dag', 'f9_ext_s3', '2020-01-01T00:00:00+00:00', '--local', '--pool', 'default_pool', '-sd', 'DAGS_FOLDER/five9_load_dag.py'][0m[0m
[2020-02-05 23:31:10,088] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:10,088[0m] {[34mtimeout.py:[0m42} [31mERROR[0m - [31mProcess timed out, PID: [1m10601[0m[0m
[2020-02-05 23:31:16,345] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:16,344[0m] {[34mtimeout.py:[0m42} [31mERROR[0m - [31mProcess timed out, PID: [1m10647[0m[0m
[2020-02-05 23:31:18,460] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:18,460[0m] {[34mtimeout.py:[0m42} [31mERROR[0m - [31mProcess timed out, PID: [1m10658[0m[0m
No other errors or details.
Thanks for support.
Regards, Daniel Kipnis
The information contained in this transmission may contain privileged and confidential information, including patient information protected by federal and state privacy laws. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original messages.
Re: DAG task no issues from airflow test, but abends with no
messages
Posted by Ash Berlin-Taylor <as...@apache.org>.
The "Process timed out" means that something is taking a long time and getting killed by a timeout before it can do anything.
Is the machine or DB where this is running under a lot of load?
On Feb 5 2020, at 11:40 pm, Daniel Kipnis <dk...@ceceliahealth.com> wrote:
>
> Hi,
>
>
> Created new DAG with 3 tasks, successfully verified all tasks by using “airflow test”, but still failed from scheduler or run without any log details.
>
>
>
> airflow test five9_load_dag f9_ext_s3 2020-01-01 generated proper result and messages
> airflow run five9_load_dag f9_ext_s3 2020-01-01 {timeout.py:42} ERROR - Process timed out, PID: 9921
>
> From Airflow task Log: No information
>
>
>
>
> From Task Instance Details specified log
> [2020-02-05 23:31:07,971] {logging_mixin.py:95} INFO - Sending to executor.
>
> [2020-02-05 23:31:07,971] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:07,971[0m] {[34mbase_executor.py:[0m59} INFO[0m - Adding to queue: [1m['airflow', 'run', 'five9_load_dag', 'f9_ext_s3', '2020-01-01T00:00:00+00:00', '--local', '--pool', 'default_pool', '-sd', 'DAGS_FOLDER/five9_load_dag.py'][0m[0m
> [2020-02-05 23:31:10,088] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:10,088[0m] {[34mtimeout.py:[0m42} [31mERROR[0m - [31mProcess timed out, PID: [1m10601[0m[0m
> [2020-02-05 23:31:16,345] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:16,344[0m] {[34mtimeout.py:[0m42} [31mERROR[0m - [31mProcess timed out, PID: [1m10647[0m[0m
> [2020-02-05 23:31:18,460] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:18,460[0m] {[34mtimeout.py:[0m42} [31mERROR[0m - [31mProcess timed out, PID: [1m10658[0m[0m
>
> No other errors or details.
>
> Thanks for support.
> Regards, Daniel Kipnis
> The information contained in this transmission may contain privileged and confidential information, including patient information protected by federal and state privacy laws. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original messages.
>
Re: DAG task no issues from airflow test, but abends with no
messages
Posted by Ash Berlin-Taylor <as...@apache.org>.
The "Process timed out" means that something is taking a long time and getting killed by a timeout before it can do anything.
Is the machine or DB where this is running under a lot of load?
On Feb 5 2020, at 11:40 pm, Daniel Kipnis <dk...@ceceliahealth.com> wrote:
>
> Hi,
>
>
> Created new DAG with 3 tasks, successfully verified all tasks by using “airflow test”, but still failed from scheduler or run without any log details.
>
>
>
> airflow test five9_load_dag f9_ext_s3 2020-01-01 generated proper result and messages
> airflow run five9_load_dag f9_ext_s3 2020-01-01 {timeout.py:42} ERROR - Process timed out, PID: 9921
>
> From Airflow task Log: No information
>
>
>
>
> From Task Instance Details specified log
> [2020-02-05 23:31:07,971] {logging_mixin.py:95} INFO - Sending to executor.
>
> [2020-02-05 23:31:07,971] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:07,971[0m] {[34mbase_executor.py:[0m59} INFO[0m - Adding to queue: [1m['airflow', 'run', 'five9_load_dag', 'f9_ext_s3', '2020-01-01T00:00:00+00:00', '--local', '--pool', 'default_pool', '-sd', 'DAGS_FOLDER/five9_load_dag.py'][0m[0m
> [2020-02-05 23:31:10,088] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:10,088[0m] {[34mtimeout.py:[0m42} [31mERROR[0m - [31mProcess timed out, PID: [1m10601[0m[0m
> [2020-02-05 23:31:16,345] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:16,344[0m] {[34mtimeout.py:[0m42} [31mERROR[0m - [31mProcess timed out, PID: [1m10647[0m[0m
> [2020-02-05 23:31:18,460] {logging_mixin.py:95} INFO - [[34m2020-02-05 23:31:18,460[0m] {[34mtimeout.py:[0m42} [31mERROR[0m - [31mProcess timed out, PID: [1m10658[0m[0m
>
> No other errors or details.
>
> Thanks for support.
> Regards, Daniel Kipnis
> The information contained in this transmission may contain privileged and confidential information, including patient information protected by federal and state privacy laws. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original messages.
>