You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@airflow.apache.org by Reed Villanueva <rv...@ucera.org> on 2020/01/16 01:37:40 UTC
Airflow webserver can't turn on DAG?
Having problem where unable to turn on a DAG in the airflow webserver UI.
One thing to note is that the DAG in question originally was causing
timeout errors when trying to run so I have edited the airflow.cfg file to
have line...
dagbag_import_timeout = 300
Now after making this change, running...
airflow list_dags
can see the dag gets built successfully.
Then going to webserver, refresh dag in UI, switch the DAG status to "On",
click on DAG to attempt to see the graph view.
Either get message about PID timeout or webserver page shows some browser
error like "page sent no data" and after reloading, I see that the DAG has
been switched off (in either case, no indication of problem in the
airflow-webserver.log).
More debugging info if it helps:
[root@airflowetl airflow]# ps -aux | grep webserver
airflow 16740 0.8 0.2 782620 134804 ? S 15:17 0:06
[ready] gunicorn: worker [airflow-webserver]
airflow 29758 2.3 0.2 756164 108644 ? S 15:26 0:03
[ready] gunicorn: worker [airflow-webserver]
airflow 33820 14.8 0.1 724788 78036 ? S 15:29 0:01
gunicorn: worker [airflow-webserver]
airflow 33854 26.7 0.1 724784 78032 ? S 15:29 0:01
gunicorn: worker [airflow-webserver]
airflow 33855 26.5 0.1 724816 78064 ? S 15:29 0:01
gunicorn: worker [airflow-webserver]
root 34072 0.0 0.0 112712 968 pts/0 S+ 15:29 0:00 grep
--color=auto webserver
airflow 91174 1.6 0.1 735708 82468 ? S 14:14 1:14
/usr/bin/python3 /home/airflow/.local/bin/airflow webserver -D
airflow 91211 0.0 0.1 355040 53472 ? S 14:14 0:01
gunicorn: master [airflow-webserver]
Anyone with more airflow experience have any ideas why this could be
happening and how to fix? (Maybe some airflow.cfg timeout config that I
should extend)?
--
This electronic message is intended only for the named
recipient, and may
contain information that is confidential or
privileged. If you are not the
intended recipient, you are
hereby notified that any disclosure, copying,
distribution or
use of the contents of this message is strictly
prohibited. If
you have received this message in error or are not the
named
recipient, please notify us immediately by contacting the
sender at
the electronic mail address noted above, and delete
and destroy all copies
of this message. Thank you.
Re: Airflow webserver can't turn on DAG?
Posted by Kaxil Naik <ka...@gmail.com>.
I will add some docs on using airflow_local_settings.py when I am back from
holidays.
On Sat, Jan 18, 2020, 18:19 Ash Berlin-Taylor <as...@apache.org> wrote:
> Setting that JSON setting is optional (and it defaults to the built in
> JSON lib)
>
> But yeah, that feature isn't well documented. If that file is found in
> your python include path Airflow will load it it can override a few more
> advanced settings not easily controllable from a config file.
>
> The `confog/` folder under airflow home is added to the search path and is
> a good place for this file.
>
> Ash
>
> On 17 January 2020 02:57:21 GMT+03:00, Daniel Standish <
> dpstandish@gmail.com> wrote:
>>
>> I think it's in general not advisable to mess with your site packages
>> directory. It would get overwritten with an upgrade.
>>
>> The configurable stuff usually gets deposited in your AIRFLOW_HOME
>> or at least it ought to i would think?
>> like these:
>> airflow.cfg
>> unittests.cfg
>> webserver_config.py
>>
>> I suspect this is probably supposed to go in AIRFLOW_HOME but it's just
>> not getting autocreated. And maybe there is some documentation missing. I
>> can find a few references to this file in docs but nothing explicitly
>> discussing it like there is with airflow.cfg IIRC
>>
>>
>>
>>
>> On Thu, Jan 16, 2020 at 3:41 PM Reed Villanueva <rv...@ucera.org>
>> wrote:
>>
>>> @daniel
>>>
>>> That file is part of the airflow package (not the AIRFLOW_HOME dir as I
>>> (and I am guessing, you) assumed). Can find on your machine with something
>>> like...
>>>
>>> [airflow@airflowetl]$ find ~/ -name airflow_local_settings.py
>>>
>>> /home/airflow/.local/lib/python3.6/site-packages/airflow/config_templates/airflow_local_settings.py
>>>
>>> I think this is what the docs are referring to.
>>>
>>>
>>> On Thu, Jan 16, 2020 at 1:17 PM Daniel Standish <dp...@gmail.com>
>>> wrote:
>>>
>>>> @ash reading that page i notice a reference to airflow_local_settings.py
>>>>
>>>> you need to define a json variable in local Airflow settings (
>>>>> airflow_local_settings.py)
>>>>
>>>>
>>>> is that a thing? do we have settings configurability with a python
>>>> file? or does this mean to point us to airflow.cfg?
>>>>
>>>> i tried searching docs but found nothing indicating how to use
>>>> airflow_local_settings.py to configure airflow.
>>>>
>>>
>>> This electronic message is intended only for the named
>>> recipient, and may contain information that is confidential or
>>> privileged. If you are not the intended recipient, you are
>>> hereby notified that any disclosure, copying, distribution or
>>> use of the contents of this message is strictly prohibited. If
>>> you have received this message in error or are not the named
>>> recipient, please notify us immediately by contacting the
>>> sender at the electronic mail address noted above, and delete
>>> and destroy all copies of this message. Thank you.
>>>
>>
Re: Airflow webserver can't turn on DAG?
Posted by Ash Berlin-Taylor <as...@apache.org>.
Setting that JSON setting is optional (and it defaults to the built in JSON lib)
But yeah, that feature isn't well documented. If that file is found in your python include path Airflow will load it it can override a few more advanced settings not easily controllable from a config file.
The `confog/` folder under airflow home is added to the search path and is a good place for this file.
Ash
On 17 January 2020 02:57:21 GMT+03:00, Daniel Standish <dp...@gmail.com> wrote:
>I think it's in general not advisable to mess with your site packages
>directory. It would get overwritten with an upgrade.
>
>The configurable stuff usually gets deposited in your AIRFLOW_HOME
>or at least it ought to i would think?
>like these:
>airflow.cfg
>unittests.cfg
>webserver_config.py
>
>I suspect this is probably supposed to go in AIRFLOW_HOME but it's just
>not
>getting autocreated. And maybe there is some documentation missing. I
>can
>find a few references to this file in docs but nothing explicitly
>discussing it like there is with airflow.cfg IIRC
>
>
>
>
>On Thu, Jan 16, 2020 at 3:41 PM Reed Villanueva <rv...@ucera.org>
>wrote:
>
>> @daniel
>>
>> That file is part of the airflow package (not the AIRFLOW_HOME dir as
>I
>> (and I am guessing, you) assumed). Can find on your machine with
>something
>> like...
>>
>> [airflow@airflowetl]$ find ~/ -name airflow_local_settings.py
>>
>>
>/home/airflow/.local/lib/python3.6/site-packages/airflow/config_templates/airflow_local_settings.py
>>
>> I think this is what the docs are referring to.
>>
>>
>> On Thu, Jan 16, 2020 at 1:17 PM Daniel Standish
><dp...@gmail.com>
>> wrote:
>>
>>> @ash reading that page i notice a reference to
>airflow_local_settings.py
>>>
>>> you need to define a json variable in local Airflow settings (
>>>> airflow_local_settings.py)
>>>
>>>
>>> is that a thing? do we have settings configurability with a python
>>> file? or does this mean to point us to airflow.cfg?
>>>
>>> i tried searching docs but found nothing indicating how to use
>>> airflow_local_settings.py to configure airflow.
>>>
>>
>> This electronic message is intended only for the named
>> recipient, and may contain information that is confidential or
>> privileged. If you are not the intended recipient, you are
>> hereby notified that any disclosure, copying, distribution or
>> use of the contents of this message is strictly prohibited. If
>> you have received this message in error or are not the named
>> recipient, please notify us immediately by contacting the
>> sender at the electronic mail address noted above, and delete
>> and destroy all copies of this message. Thank you.
>>
Re: Airflow webserver can't turn on DAG?
Posted by Daniel Standish <dp...@gmail.com>.
I think it's in general not advisable to mess with your site packages
directory. It would get overwritten with an upgrade.
The configurable stuff usually gets deposited in your AIRFLOW_HOME
or at least it ought to i would think?
like these:
airflow.cfg
unittests.cfg
webserver_config.py
I suspect this is probably supposed to go in AIRFLOW_HOME but it's just not
getting autocreated. And maybe there is some documentation missing. I can
find a few references to this file in docs but nothing explicitly
discussing it like there is with airflow.cfg IIRC
On Thu, Jan 16, 2020 at 3:41 PM Reed Villanueva <rv...@ucera.org>
wrote:
> @daniel
>
> That file is part of the airflow package (not the AIRFLOW_HOME dir as I
> (and I am guessing, you) assumed). Can find on your machine with something
> like...
>
> [airflow@airflowetl]$ find ~/ -name airflow_local_settings.py
>
> /home/airflow/.local/lib/python3.6/site-packages/airflow/config_templates/airflow_local_settings.py
>
> I think this is what the docs are referring to.
>
>
> On Thu, Jan 16, 2020 at 1:17 PM Daniel Standish <dp...@gmail.com>
> wrote:
>
>> @ash reading that page i notice a reference to airflow_local_settings.py
>>
>> you need to define a json variable in local Airflow settings (
>>> airflow_local_settings.py)
>>
>>
>> is that a thing? do we have settings configurability with a python
>> file? or does this mean to point us to airflow.cfg?
>>
>> i tried searching docs but found nothing indicating how to use
>> airflow_local_settings.py to configure airflow.
>>
>
> This electronic message is intended only for the named
> recipient, and may contain information that is confidential or
> privileged. If you are not the intended recipient, you are
> hereby notified that any disclosure, copying, distribution or
> use of the contents of this message is strictly prohibited. If
> you have received this message in error or are not the named
> recipient, please notify us immediately by contacting the
> sender at the electronic mail address noted above, and delete
> and destroy all copies of this message. Thank you.
>
Re: Airflow webserver can't turn on DAG?
Posted by Reed Villanueva <rv...@ucera.org>.
@daniel
That file is part of the airflow package (not the AIRFLOW_HOME dir as I
(and I am guessing, you) assumed). Can find on your machine with something
like...
[airflow@airflowetl]$ find ~/ -name airflow_local_settings.py
/home/airflow/.local/lib/python3.6/site-packages/airflow/config_templates/airflow_local_settings.py
I think this is what the docs are referring to.
On Thu, Jan 16, 2020 at 1:17 PM Daniel Standish <dp...@gmail.com>
wrote:
> @ash reading that page i notice a reference to airflow_local_settings.py
>
> you need to define a json variable in local Airflow settings (
>> airflow_local_settings.py)
>
>
> is that a thing? do we have settings configurability with a python file?
> or does this mean to point us to airflow.cfg?
>
> i tried searching docs but found nothing indicating how to use
> airflow_local_settings.py to configure airflow.
>
--
This electronic message is intended only for the named
recipient, and may
contain information that is confidential or
privileged. If you are not the
intended recipient, you are
hereby notified that any disclosure, copying,
distribution or
use of the contents of this message is strictly
prohibited. If
you have received this message in error or are not the
named
recipient, please notify us immediately by contacting the
sender at
the electronic mail address noted above, and delete
and destroy all copies
of this message. Thank you.
Re: Airflow webserver can't turn on DAG?
Posted by Daniel Standish <dp...@gmail.com>.
@ash reading that page i notice a reference to airflow_local_settings.py
you need to define a json variable in local Airflow settings (
> airflow_local_settings.py)
is that a thing? do we have settings configurability with a python file?
or does this mean to point us to airflow.cfg?
i tried searching docs but found nothing indicating how to use
airflow_local_settings.py to configure airflow.
Re: Airflow webserver can't turn on DAG?
Posted by Reed Villanueva <rv...@ucera.org>.
@ash
Interesting, thank for the info. Will try to upgrade and set up.
Could you help me understand something about this statement:
"The reason the webserver takes longer than list dags does is because there
are (by default) 4 webserver worker processes that are all trying to load
all he dags at the same time, so start up of webserver is expensive."
So when I time the list_dags command (when I have the modified dag causing
timeout issues in the webserver) and it says it takes 2m37s, given what you
said above, should I / would there be any benefit to setting the
web_server_worker_timeout value to at least 4 times that observed time?
On Thu, Jan 16, 2020 at 11:53 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> Hi Reed,
>
> You might want to try out DAG serialization
> http://airflow.apache.org/docs/stable/dag-serialization.html - it means
> the webserver doesn't try to load the dags directly, but instead relies on
> the scheduler to parse and write info in the db. (Note there is a bug in
> 1.10.7 with this mode that causes a 500 error when trying to view logs.)
>
> The reason the webserver takes longer than list dags does is because there
> are (by default) 4 webserver worker processes that are all trying to load
> all he dags at the same time, so start up of webserver is expensive.
>
> Ash
>
> On 16 January 2020 21:27:07 GMT, Reed Villanueva <rv...@ucera.org>
> wrote:
>>
>> Timing the airflow list_dags command (just running with time utility)
>> with and without the one change, the time difference is...
>>
>> before change: real 1m31.201s
>> after change: real 2m39.744s
>>
>> Looking at the airflow.cfg file for anything related to the webserver
>> and timeouts (especially timeout values < 2m39s) where the seconds count
>> was less that the new dag build time, found...
>>
>> # Number of seconds the webserver waits before killing gunicorn master that doesn't respond
>> web_server_master_timeout = 120
>>
>> # Number of seconds the gunicorn webserver waits before timing out on a worker
>> web_server_worker_timeout = 120
>> ...
>> # The amount of time (in secs) webserver will wait for initial handshake
>> # while fetching logs from other worker machine
>> log_fetch_timeout_sec = 5
>>
>> Running the webserver (not as a deamon process) and just watching the
>> output, when it attempts to filll the dab bag with the newly modified dag,
>> I see errors like...
>>
>> [2020-01-16 11:12:22 -1000] [137034] [CRITICAL] WORKER TIMEOUT (pid:137039)[2020-01-16 11:12:22 -1000] [137034] [CRITICAL] WORKER TIMEOUT (pid:137040)[2020-01-16 11:12:22 -1000] [137034] [CRITICAL] WORKER TIMEOUT (pid:137041)[2020-01-16 11:12:22 -1000] [137034] [CRITICAL] WORKER TIMEOUT (pid:137042)[2020-01-16 11:12:22 -1000] [137039] [INFO] Worker exiting (pid: 137039)[2020-01-16 11:12:22 -1000] [137041] [INFO] Worker exiting (pid: 137041)[2020-01-16 11:12:22 -1000] [137042] [INFO] Worker exiting (pid: 137042)[2020-01-16 11:12:22 -1000] [137040] [INFO] Worker exiting (pid: 137040)
>>
>> *Changing web_server_worker_timeout from 120 to 300 (5 min.)* and
>> testing accessing the modified problem dag (tree and graph view) in the
>> webserver seems to be much faster (and after seeing a timeout error
>> continue to pop up in the webserver after initially starting) the problem
>> appears to be fixed.
>> ------------------------------
>>
>> *Note that still seeing timeout errors pop up (sometimes when refreshing
>> main page) in the webserver (though can't find in the webserver logs). Not
>> totally sure what underlying mechanics of this problem are and would
>> appreciate any further explanation in the comments, but the problem does
>> appear to be fixed at this point.*
>>
>> On Thu, Jan 16, 2020 at 10:52 AM Reed Villanueva <rv...@ucera.org>
>> wrote:
>>
>>> Update:
>>>
>>> After further debugging, the problem appears to be with a particular
>>> task is configured/created in the dag. The DAG definition itself is not
>>> very straight forward and very application specific, so would need to try
>>> to parse it a bit more into something sensical and readable before posting.
>>> Though this still does not explain why the dag appears to build during
>>> airflow list_dags, but not in the webserver.
>>>
>>> Going with what I can measure, timing the airflow list_dags command
>>> (just running with time utility) with and without the one change, the
>>> time difference is...
>>>
>>> before change: real 2m39.744s
>>> after change: real 1m31.201s
>>>
>>> ------------------------------
>>>
>>> Update:
>>>
>>> After more debugging, I suspect the issue is ultimately with the
>>> webserver. Always able to build the dag when running airflow list_dags, but *when
>>> other dags are running*, not able to click on the dag in the webserver
>>> w/out timeout errors thrown. *When no other dags are running*, able to
>>> view the dag (tree and graph) in the webserver, but upon going back to the
>>> main screen, see the same "Broken DAG; ... Timeout, PID: 1234" error as
>>> before
>>>
>>> On Wed, Jan 15, 2020 at 3:37 PM Reed Villanueva <rv...@ucera.org>
>>> wrote:
>>>
>>>> Having problem where unable to turn on a DAG in the airflow webserver
>>>> UI.
>>>>
>>>> One thing to note is that the DAG in question originally was causing
>>>> timeout errors when trying to run so I have edited the airflow.cfg file to
>>>> have line...
>>>>
>>>> dagbag_import_timeout = 300
>>>>
>>>> Now after making this change, running...
>>>>
>>>> airflow list_dags
>>>>
>>>> can see the dag gets built successfully.
>>>>
>>>> Then going to webserver, refresh dag in UI, switch the DAG status to
>>>> "On", click on DAG to attempt to see the graph view.
>>>>
>>>> Either get message about PID timeout or webserver page shows some
>>>> browser error like "page sent no data" and after reloading, I see that the
>>>> DAG has been switched off (in either case, no indication of problem in the
>>>> airflow-webserver.log).
>>>>
>>>> More debugging info if it helps:
>>>>
>>>> [root@airflowetl airflow]# ps -aux | grep webserver
>>>> airflow 16740 0.8 0.2 782620 134804 ? S 15:17 0:06 [ready] gunicorn: worker [airflow-webserver]
>>>> airflow 29758 2.3 0.2 756164 108644 ? S 15:26 0:03 [ready] gunicorn: worker [airflow-webserver]
>>>> airflow 33820 14.8 0.1 724788 78036 ? S 15:29 0:01 gunicorn: worker [airflow-webserver]
>>>> airflow 33854 26.7 0.1 724784 78032 ? S 15:29 0:01 gunicorn: worker [airflow-webserver]
>>>> airflow 33855 26.5 0.1 724816 78064 ? S 15:29 0:01 gunicorn: worker [airflow-webserver]
>>>> root 34072 0.0 0.0 112712 968 pts/0 S+ 15:29 0:00 grep --color=auto webserver
>>>> airflow 91174 1.6 0.1 735708 82468 ? S 14:14 1:14 /usr/bin/python3 /home/airflow/.local/bin/airflow webserver -D
>>>> airflow 91211 0.0 0.1 355040 53472 ? S 14:14 0:01 gunicorn: master [airflow-webserver]
>>>>
>>>> Anyone with more airflow experience have any ideas why this could be
>>>> happening and how to fix? (Maybe some airflow.cfg timeout config that I
>>>> should extend)?
>>>>
>>>
>> This electronic message is intended only for the named
>> recipient, and may contain information that is confidential or
>> privileged. If you are not the intended recipient, you are
>> hereby notified that any disclosure, copying, distribution or
>> use of the contents of this message is strictly prohibited. If
>> you have received this message in error or are not the named
>> recipient, please notify us immediately by contacting the
>> sender at the electronic mail address noted above, and delete
>> and destroy all copies of this message. Thank you.
>>
>
--
This electronic message is intended only for the named
recipient, and may
contain information that is confidential or
privileged. If you are not the
intended recipient, you are
hereby notified that any disclosure, copying,
distribution or
use of the contents of this message is strictly
prohibited. If
you have received this message in error or are not the
named
recipient, please notify us immediately by contacting the
sender at
the electronic mail address noted above, and delete
and destroy all copies
of this message. Thank you.
Re: Airflow webserver can't turn on DAG?
Posted by Ash Berlin-Taylor <as...@apache.org>.
Hi Reed,
You might want to try out DAG serialization http://airflow.apache.org/docs/stable/dag-serialization.html - it means the webserver doesn't try to load the dags directly, but instead relies on the scheduler to parse and write info in the db. (Note there is a bug in 1.10.7 with this mode that causes a 500 error when trying to view logs.)
The reason the webserver takes longer than list dags does is because there are (by default) 4 webserver worker processes that are all trying to load all he dags at the same time, so start up of webserver is expensive.
Ash
On 16 January 2020 21:27:07 GMT, Reed Villanueva <rv...@ucera.org> wrote:
>Timing the airflow list_dags command (just running with time utility)
>with
>and without the one change, the time difference is...
>
>before change: real 1m31.201s
>after change: real 2m39.744s
>
>Looking at the airflow.cfg file for anything related to the webserver
>and
>timeouts (especially timeout values < 2m39s) where the seconds count
>was
>less that the new dag build time, found...
>
># Number of seconds the webserver waits before killing gunicorn master
>that doesn't respond
>web_server_master_timeout = 120
>
># Number of seconds the gunicorn webserver waits before timing out on a
>worker
>web_server_worker_timeout = 120
>...
># The amount of time (in secs) webserver will wait for initial
>handshake
># while fetching logs from other worker machine
>log_fetch_timeout_sec = 5
>
>Running the webserver (not as a deamon process) and just watching the
>output, when it attempts to filll the dab bag with the newly modified
>dag,
>I see errors like...
>
>[2020-01-16 11:12:22 -1000] [137034] [CRITICAL] WORKER TIMEOUT
>(pid:137039)[2020-01-16 11:12:22 -1000] [137034] [CRITICAL] WORKER
>TIMEOUT (pid:137040)[2020-01-16 11:12:22 -1000] [137034] [CRITICAL]
>WORKER TIMEOUT (pid:137041)[2020-01-16 11:12:22 -1000] [137034]
>[CRITICAL] WORKER TIMEOUT (pid:137042)[2020-01-16 11:12:22 -1000]
>[137039] [INFO] Worker exiting (pid: 137039)[2020-01-16 11:12:22
>-1000] [137041] [INFO] Worker exiting (pid: 137041)[2020-01-16
>11:12:22 -1000] [137042] [INFO] Worker exiting (pid:
>137042)[2020-01-16 11:12:22 -1000] [137040] [INFO] Worker exiting
>(pid: 137040)
>
>*Changing web_server_worker_timeout from 120 to 300 (5 min.)* and
>testing
>accessing the modified problem dag (tree and graph view) in the
>webserver
>seems to be much faster (and after seeing a timeout error continue to
>pop
>up in the webserver after initially starting) the problem appears to be
>fixed.
>------------------------------
>
>*Note that still seeing timeout errors pop up (sometimes when
>refreshing
>main page) in the webserver (though can't find in the webserver logs).
>Not
>totally sure what underlying mechanics of this problem are and would
>appreciate any further explanation in the comments, but the problem
>does
>appear to be fixed at this point.*
>
>On Thu, Jan 16, 2020 at 10:52 AM Reed Villanueva
><rv...@ucera.org>
>wrote:
>
>> Update:
>>
>> After further debugging, the problem appears to be with a particular
>task
>> is configured/created in the dag. The DAG definition itself is not
>very
>> straight forward and very application specific, so would need to try
>to
>> parse it a bit more into something sensical and readable before
>posting.
>> Though this still does not explain why the dag appears to build
>during
>> airflow list_dags, but not in the webserver.
>>
>> Going with what I can measure, timing the airflow list_dags command
>(just
>> running with time utility) with and without the one change, the time
>> difference is...
>>
>> before change: real 2m39.744s
>> after change: real 1m31.201s
>>
>> ------------------------------
>>
>> Update:
>>
>> After more debugging, I suspect the issue is ultimately with the
>> webserver. Always able to build the dag when running airflow
>list_dags, but *when
>> other dags are running*, not able to click on the dag in the
>webserver
>> w/out timeout errors thrown. *When no other dags are running*, able
>to
>> view the dag (tree and graph) in the webserver, but upon going back
>to the
>> main screen, see the same "Broken DAG; ... Timeout, PID: 1234" error
>as
>> before
>>
>> On Wed, Jan 15, 2020 at 3:37 PM Reed Villanueva
><rv...@ucera.org>
>> wrote:
>>
>>> Having problem where unable to turn on a DAG in the airflow
>webserver UI.
>>>
>>> One thing to note is that the DAG in question originally was causing
>>> timeout errors when trying to run so I have edited the airflow.cfg
>file to
>>> have line...
>>>
>>> dagbag_import_timeout = 300
>>>
>>> Now after making this change, running...
>>>
>>> airflow list_dags
>>>
>>> can see the dag gets built successfully.
>>>
>>> Then going to webserver, refresh dag in UI, switch the DAG status to
>>> "On", click on DAG to attempt to see the graph view.
>>>
>>> Either get message about PID timeout or webserver page shows some
>browser
>>> error like "page sent no data" and after reloading, I see that the
>DAG has
>>> been switched off (in either case, no indication of problem in the
>>> airflow-webserver.log).
>>>
>>> More debugging info if it helps:
>>>
>>> [root@airflowetl airflow]# ps -aux | grep webserver
>>> airflow 16740 0.8 0.2 782620 134804 ? S 15:17 0:06
>[ready] gunicorn: worker [airflow-webserver]
>>> airflow 29758 2.3 0.2 756164 108644 ? S 15:26 0:03
>[ready] gunicorn: worker [airflow-webserver]
>>> airflow 33820 14.8 0.1 724788 78036 ? S 15:29 0:01
>gunicorn: worker [airflow-webserver]
>>> airflow 33854 26.7 0.1 724784 78032 ? S 15:29 0:01
>gunicorn: worker [airflow-webserver]
>>> airflow 33855 26.5 0.1 724816 78064 ? S 15:29 0:01
>gunicorn: worker [airflow-webserver]
>>> root 34072 0.0 0.0 112712 968 pts/0 S+ 15:29 0:00
>grep --color=auto webserver
>>> airflow 91174 1.6 0.1 735708 82468 ? S 14:14 1:14
>/usr/bin/python3 /home/airflow/.local/bin/airflow webserver -D
>>> airflow 91211 0.0 0.1 355040 53472 ? S 14:14 0:01
>gunicorn: master [airflow-webserver]
>>>
>>> Anyone with more airflow experience have any ideas why this could be
>>> happening and how to fix? (Maybe some airflow.cfg timeout config
>that I
>>> should extend)?
>>>
>>
>
>--
>This electronic message is intended only for the named
>recipient, and may
>contain information that is confidential or
>privileged. If you are not the
>intended recipient, you are
>hereby notified that any disclosure, copying,
>distribution or
>use of the contents of this message is strictly
>prohibited. If
>you have received this message in error or are not the
>named
>recipient, please notify us immediately by contacting the
>sender at
>the electronic mail address noted above, and delete
>and destroy all copies
>of this message. Thank you.
Re: Airflow webserver can't turn on DAG?
Posted by Reed Villanueva <rv...@ucera.org>.
Timing the airflow list_dags command (just running with time utility) with
and without the one change, the time difference is...
before change: real 1m31.201s
after change: real 2m39.744s
Looking at the airflow.cfg file for anything related to the webserver and
timeouts (especially timeout values < 2m39s) where the seconds count was
less that the new dag build time, found...
# Number of seconds the webserver waits before killing gunicorn master
that doesn't respond
web_server_master_timeout = 120
# Number of seconds the gunicorn webserver waits before timing out on a worker
web_server_worker_timeout = 120
...
# The amount of time (in secs) webserver will wait for initial handshake
# while fetching logs from other worker machine
log_fetch_timeout_sec = 5
Running the webserver (not as a deamon process) and just watching the
output, when it attempts to filll the dab bag with the newly modified dag,
I see errors like...
[2020-01-16 11:12:22 -1000] [137034] [CRITICAL] WORKER TIMEOUT
(pid:137039)[2020-01-16 11:12:22 -1000] [137034] [CRITICAL] WORKER
TIMEOUT (pid:137040)[2020-01-16 11:12:22 -1000] [137034] [CRITICAL]
WORKER TIMEOUT (pid:137041)[2020-01-16 11:12:22 -1000] [137034]
[CRITICAL] WORKER TIMEOUT (pid:137042)[2020-01-16 11:12:22 -1000]
[137039] [INFO] Worker exiting (pid: 137039)[2020-01-16 11:12:22
-1000] [137041] [INFO] Worker exiting (pid: 137041)[2020-01-16
11:12:22 -1000] [137042] [INFO] Worker exiting (pid:
137042)[2020-01-16 11:12:22 -1000] [137040] [INFO] Worker exiting
(pid: 137040)
*Changing web_server_worker_timeout from 120 to 300 (5 min.)* and testing
accessing the modified problem dag (tree and graph view) in the webserver
seems to be much faster (and after seeing a timeout error continue to pop
up in the webserver after initially starting) the problem appears to be
fixed.
------------------------------
*Note that still seeing timeout errors pop up (sometimes when refreshing
main page) in the webserver (though can't find in the webserver logs). Not
totally sure what underlying mechanics of this problem are and would
appreciate any further explanation in the comments, but the problem does
appear to be fixed at this point.*
On Thu, Jan 16, 2020 at 10:52 AM Reed Villanueva <rv...@ucera.org>
wrote:
> Update:
>
> After further debugging, the problem appears to be with a particular task
> is configured/created in the dag. The DAG definition itself is not very
> straight forward and very application specific, so would need to try to
> parse it a bit more into something sensical and readable before posting.
> Though this still does not explain why the dag appears to build during
> airflow list_dags, but not in the webserver.
>
> Going with what I can measure, timing the airflow list_dags command (just
> running with time utility) with and without the one change, the time
> difference is...
>
> before change: real 2m39.744s
> after change: real 1m31.201s
>
> ------------------------------
>
> Update:
>
> After more debugging, I suspect the issue is ultimately with the
> webserver. Always able to build the dag when running airflow list_dags, but *when
> other dags are running*, not able to click on the dag in the webserver
> w/out timeout errors thrown. *When no other dags are running*, able to
> view the dag (tree and graph) in the webserver, but upon going back to the
> main screen, see the same "Broken DAG; ... Timeout, PID: 1234" error as
> before
>
> On Wed, Jan 15, 2020 at 3:37 PM Reed Villanueva <rv...@ucera.org>
> wrote:
>
>> Having problem where unable to turn on a DAG in the airflow webserver UI.
>>
>> One thing to note is that the DAG in question originally was causing
>> timeout errors when trying to run so I have edited the airflow.cfg file to
>> have line...
>>
>> dagbag_import_timeout = 300
>>
>> Now after making this change, running...
>>
>> airflow list_dags
>>
>> can see the dag gets built successfully.
>>
>> Then going to webserver, refresh dag in UI, switch the DAG status to
>> "On", click on DAG to attempt to see the graph view.
>>
>> Either get message about PID timeout or webserver page shows some browser
>> error like "page sent no data" and after reloading, I see that the DAG has
>> been switched off (in either case, no indication of problem in the
>> airflow-webserver.log).
>>
>> More debugging info if it helps:
>>
>> [root@airflowetl airflow]# ps -aux | grep webserver
>> airflow 16740 0.8 0.2 782620 134804 ? S 15:17 0:06 [ready] gunicorn: worker [airflow-webserver]
>> airflow 29758 2.3 0.2 756164 108644 ? S 15:26 0:03 [ready] gunicorn: worker [airflow-webserver]
>> airflow 33820 14.8 0.1 724788 78036 ? S 15:29 0:01 gunicorn: worker [airflow-webserver]
>> airflow 33854 26.7 0.1 724784 78032 ? S 15:29 0:01 gunicorn: worker [airflow-webserver]
>> airflow 33855 26.5 0.1 724816 78064 ? S 15:29 0:01 gunicorn: worker [airflow-webserver]
>> root 34072 0.0 0.0 112712 968 pts/0 S+ 15:29 0:00 grep --color=auto webserver
>> airflow 91174 1.6 0.1 735708 82468 ? S 14:14 1:14 /usr/bin/python3 /home/airflow/.local/bin/airflow webserver -D
>> airflow 91211 0.0 0.1 355040 53472 ? S 14:14 0:01 gunicorn: master [airflow-webserver]
>>
>> Anyone with more airflow experience have any ideas why this could be
>> happening and how to fix? (Maybe some airflow.cfg timeout config that I
>> should extend)?
>>
>
--
This electronic message is intended only for the named
recipient, and may
contain information that is confidential or
privileged. If you are not the
intended recipient, you are
hereby notified that any disclosure, copying,
distribution or
use of the contents of this message is strictly
prohibited. If
you have received this message in error or are not the
named
recipient, please notify us immediately by contacting the
sender at
the electronic mail address noted above, and delete
and destroy all copies
of this message. Thank you.
Re: Airflow webserver can't turn on DAG?
Posted by Reed Villanueva <rv...@ucera.org>.
Update:
After further debugging, the problem appears to be with a particular task
is configured/created in the dag. The DAG definition itself is not very
straight forward and very application specific, so would need to try to
parse it a bit more into something sensical and readable before posting.
Though this still does not explain why the dag appears to build during
airflow list_dags, but not in the webserver.
Going with what I can measure, timing the airflow list_dags command (just
running with time utility) with and without the one change, the time
difference is...
before change: real 2m39.744s
after change: real 1m31.201s
------------------------------
Update:
After more debugging, I suspect the issue is ultimately with the webserver.
Always able to build the dag when running airflow list_dags, but *when
other dags are running*, not able to click on the dag in the webserver
w/out timeout errors thrown. *When no other dags are running*, able to view
the dag (tree and graph) in the webserver, but upon going back to the main
screen, see the same "Broken DAG; ... Timeout, PID: 1234" error as before
On Wed, Jan 15, 2020 at 3:37 PM Reed Villanueva <rv...@ucera.org>
wrote:
> Having problem where unable to turn on a DAG in the airflow webserver UI.
>
> One thing to note is that the DAG in question originally was causing
> timeout errors when trying to run so I have edited the airflow.cfg file to
> have line...
>
> dagbag_import_timeout = 300
>
> Now after making this change, running...
>
> airflow list_dags
>
> can see the dag gets built successfully.
>
> Then going to webserver, refresh dag in UI, switch the DAG status to "On",
> click on DAG to attempt to see the graph view.
>
> Either get message about PID timeout or webserver page shows some browser
> error like "page sent no data" and after reloading, I see that the DAG has
> been switched off (in either case, no indication of problem in the
> airflow-webserver.log).
>
> More debugging info if it helps:
>
> [root@airflowetl airflow]# ps -aux | grep webserver
> airflow 16740 0.8 0.2 782620 134804 ? S 15:17 0:06 [ready] gunicorn: worker [airflow-webserver]
> airflow 29758 2.3 0.2 756164 108644 ? S 15:26 0:03 [ready] gunicorn: worker [airflow-webserver]
> airflow 33820 14.8 0.1 724788 78036 ? S 15:29 0:01 gunicorn: worker [airflow-webserver]
> airflow 33854 26.7 0.1 724784 78032 ? S 15:29 0:01 gunicorn: worker [airflow-webserver]
> airflow 33855 26.5 0.1 724816 78064 ? S 15:29 0:01 gunicorn: worker [airflow-webserver]
> root 34072 0.0 0.0 112712 968 pts/0 S+ 15:29 0:00 grep --color=auto webserver
> airflow 91174 1.6 0.1 735708 82468 ? S 14:14 1:14 /usr/bin/python3 /home/airflow/.local/bin/airflow webserver -D
> airflow 91211 0.0 0.1 355040 53472 ? S 14:14 0:01 gunicorn: master [airflow-webserver]
>
> Anyone with more airflow experience have any ideas why this could be
> happening and how to fix? (Maybe some airflow.cfg timeout config that I
> should extend)?
>
--
This electronic message is intended only for the named
recipient, and may
contain information that is confidential or
privileged. If you are not the
intended recipient, you are
hereby notified that any disclosure, copying,
distribution or
use of the contents of this message is strictly
prohibited. If
you have received this message in error or are not the
named
recipient, please notify us immediately by contacting the
sender at
the electronic mail address noted above, and delete
and destroy all copies
of this message. Thank you.
Re: Airflow webserver can't turn on DAG?
Posted by Anton Erholt <an...@epidemicsound.com>.
This looks to me like it's the gunicorn master process timing out workers
that are unresponsive. Most likely due to blocking while handling a request
for too long.
When we ran into this problem, we used the default sync worker_type. We
solved it by switching to one of the other concurrency models(tornado +
asyncio), which allowed the worker to stay responsive even though request
handling took longer than `--timeout` seconds.
In other words, we switched to asyncronous workers in order to allow the
worker to yield and respond to heartbeats from the master, making sure they
don't get killed.
The switch was pretty non-trivial due to several factors, the biggest one
being that concurrency in Python is pretty hard to reason about. Asyncio
eased our pain.
Note: Raising the timeout is a great way to mask the actual problem, even
though it may very well work as a quickfix.
I guess this chould be discussed further on a dev-list / ticket. I'm new to
Airflow development, but have looked at this particular problem before and
I might be able to help out. Maybe someone can help out here?
concurrent
Æ
regards,
With
best
Den tors 16 jan. 2020 02:41Reed Villanueva <rv...@ucera.org> skrev:
> Having problem where unable to turn on a DAG in the airflow webserver UI.
>
> One thing to note is that the DAG in question originally was causing
> timeout errors when trying to run so I have edited the airflow.cfg file to
> have line...
>
> dagbag_import_timeout = 300
>
> Now after making this change, running...
>
> airflow list_dags
>
> can see the dag gets built successfully.
>
> Then going to webserver, refresh dag in UI, switch the DAG status to "On",
> click on DAG to attempt to see the graph view.
>
> Either get message about PID timeout or webserver page shows some browser
> error like "page sent no data" and after reloading, I see that the DAG has
> been switched off (in either case, no indication of problem in the
> airflow-webserver.log).
>
> More debugging info if it helps:
>
> [root@airflowetl airflow]# ps -aux | grep webserver
> airflow 16740 0.8 0.2 782620 134804 ? S 15:17 0:06 [ready] gunicorn: worker [airflow-webserver]
> airflow 29758 2.3 0.2 756164 108644 ? S 15:26 0:03 [ready] gunicorn: worker [airflow-webserver]
> airflow 33820 14.8 0.1 724788 78036 ? S 15:29 0:01 gunicorn: worker [airflow-webserver]
> airflow 33854 26.7 0.1 724784 78032 ? S 15:29 0:01 gunicorn: worker [airflow-webserver]
> airflow 33855 26.5 0.1 724816 78064 ? S 15:29 0:01 gunicorn: worker [airflow-webserver]
> root 34072 0.0 0.0 112712 968 pts/0 S+ 15:29 0:00 grep --color=auto webserver
> airflow 91174 1.6 0.1 735708 82468 ? S 14:14 1:14 /usr/bin/python3 /home/airflow/.local/bin/airflow webserver -D
> airflow 91211 0.0 0.1 355040 53472 ? S 14:14 0:01 gunicorn: master [airflow-webserver]
>
> Anyone with more airflow experience have any ideas why this could be
> happening and how to fix? (Maybe some airflow.cfg timeout config that I
> should extend)?
>
> This electronic message is intended only for the named
> recipient, and may contain information that is confidential or
> privileged. If you are not the intended recipient, you are
> hereby notified that any disclosure, copying, distribution or
> use of the contents of this message is strictly prohibited. If
> you have received this message in error or are not the named
> recipient, please notify us immediately by contacting the
> sender at the electronic mail address noted above, and delete
> and destroy all copies of this message. Thank you.
>