You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Sunil Varma Chiluvuri <Su...@equifax.com> on 2018/10/15 19:32:01 UTC
RE: [IE] Re: [External] RE: [IE] airflow ui not showing logs
Frank,
The serve_logs process usually starts automatically with the 'airflow worker' command. Try restarting your worker process and see if it also starts the serve_logs process.
I've had a situation where the serve_logs process went down for some reason and restarting the worker fixed it. It doesn't explain the cause but could possibly resolve your issue.
Sunil
-----Original Message-----
From: Frank Maritato [mailto:fmaritato@opentable.com.INVALID]
Sent: Monday, October 15, 2018 2:14 PM
To: dev@airflow.incubator.apache.org
Subject: [IE] Re: [External] RE: [IE] airflow ui not showing logs
Hi Sunil,
I don't see this process running. I have never had to run this command
previously. Should it have started as part of the 'airflow webserver'
command?
I ran it manually from the command line as 'airflow serve_logs &' (does not
have a daemon option I guess) but I am still not seeing logs in the ui. I
did verify with netstat that it is running on port 8793.
On Mon, Oct 15, 2018 at 12:02 PM Sunil Varma Chiluvuri <
SunilVarma.Chiluvuri@equifax.com> wrote:
> Check if the serve_logs process is running alongside your worker
> process(es). This is the process that takes the log files written to disk
> and serves them to the web UI. It should be running on port 8793 by default.
>
> Sunil
>
> -----Original Message-----
> From: Frank Maritato [mailto:fmaritato@opentable.com.INVALID]
> Sent: Monday, October 15, 2018 1:42 PM
> To: dev@airflow.incubator.apache.org
> Subject: [IE] airflow ui not showing logs
>
> Hi All,
>
> I'm running airflow 1.10.0 and the ui isn't showing the task logs anymore.
> This has worked in the past so I'm not sure what changed. I was able to
> verify that the logs are definitely being written to the same local
> directory as what is specified in the airflow.cfg 'base_log_folder'. I
> don't see any errors or debug in the airflow-webserver.{out|err} logs. I
> tried restarting the process and it still doesn't work.
>
> Anyone know how I can track this down?
> --
> Frank Maritato
> This message contains proprietary information from Equifax which may be
> confidential. If you are not an intended recipient, please refrain from any
> disclosure, copying, distribution or use of this information and note that
> such actions are prohibited. If you have received this transmission in
> error, please notify by e-mail postmaster@equifax.com. Equifax® is a
> registered trademark of Equifax Inc. All rights reserved.
>
--
Frank Maritato
This message contains proprietary information from Equifax which may be confidential. If you are not an intended recipient, please refrain from any disclosure, copying, distribution or use of this information and note that such actions are prohibited. If you have received this transmission in error, please notify by e-mail postmaster@equifax.com. Equifax® is a registered trademark of Equifax Inc. All rights reserved.
Re: [IE] Re: [External] RE: [IE] airflow ui not showing logs
Posted by Ash Berlin-Taylor <as...@apache.org>.
I added a check for this that will be in 1.10.1 so Airflow will 1) warn you about this, and 2) try to correct it automatically
https://github.com/apache/incubator-airflow/blob/a1e922fe6d7ac3aa19848a1ad34836b61fccf24d/airflow/logging_config.py#L79-L106 <https://github.com/apache/incubator-airflow/blob/a1e922fe6d7ac3aa19848a1ad34836b61fccf24d/airflow/logging_config.py#L79-L106>
> On 25 Oct 2018, at 22:04, Frank Maritato <fm...@opentable.com.INVALID> wrote:
>
> Ok, I finally had time to try and track down the issue. Looks like the cfg
> option 'task_log_reader' was changed at some point prior to 1.10.0.
> Previously, my airflow.cfg had:
>
> task_log_reader = file.task
>
> But now it needs to be set to
>
> task_log_reader = task
>
>
> On Mon, Oct 15, 2018 at 1:57 PM Frank Maritato <fm...@opentable.com>
> wrote:
>
>> James - No, my configuration doesn't have a setting for rbac, so I'm
>> assuming the default is False.
>>
>> Sunil - We are currently running the LocalExecutor. Is the worker process
>> just for Celery? According to the quickstart, I thought the only processes
>> we need to start are the webserver and scheduler.
>>
>>
>> On Mon, Oct 15, 2018 at 12:32 PM Sunil Varma Chiluvuri <
>> SunilVarma.Chiluvuri@equifax.com> wrote:
>>
>>> Frank,
>>>
>>> The serve_logs process usually starts automatically with the 'airflow
>>> worker' command. Try restarting your worker process and see if it also
>>> starts the serve_logs process.
>>>
>>> I've had a situation where the serve_logs process went down for some
>>> reason and restarting the worker fixed it. It doesn't explain the cause
>>> but could possibly resolve your issue.
>>>
>>> Sunil
>>>
>>> -----Original Message-----
>>> From: Frank Maritato [mailto:fmaritato@opentable.com.INVALID]
>>> Sent: Monday, October 15, 2018 2:14 PM
>>> To: dev@airflow.incubator.apache.org
>>> Subject: [IE] Re: [External] RE: [IE] airflow ui not showing logs
>>>
>>> Hi Sunil,
>>>
>>> I don't see this process running. I have never had to run this command
>>> previously. Should it have started as part of the 'airflow webserver'
>>> command?
>>> I ran it manually from the command line as 'airflow serve_logs &' (does
>>> not
>>> have a daemon option I guess) but I am still not seeing logs in the ui. I
>>> did verify with netstat that it is running on port 8793.
>>>
>>>
>>> On Mon, Oct 15, 2018 at 12:02 PM Sunil Varma Chiluvuri <
>>> SunilVarma.Chiluvuri@equifax.com> wrote:
>>>
>>>> Check if the serve_logs process is running alongside your worker
>>>> process(es). This is the process that takes the log files written to
>>> disk
>>>> and serves them to the web UI. It should be running on port 8793 by
>>> default.
>>>>
>>>> Sunil
>>>>
>>>> -----Original Message-----
>>>> From: Frank Maritato [mailto:fmaritato@opentable.com.INVALID]
>>>> Sent: Monday, October 15, 2018 1:42 PM
>>>> To: dev@airflow.incubator.apache.org
>>>> Subject: [IE] airflow ui not showing logs
>>>>
>>>> Hi All,
>>>>
>>>> I'm running airflow 1.10.0 and the ui isn't showing the task logs
>>> anymore.
>>>> This has worked in the past so I'm not sure what changed. I was able to
>>>> verify that the logs are definitely being written to the same local
>>>> directory as what is specified in the airflow.cfg 'base_log_folder'. I
>>>> don't see any errors or debug in the airflow-webserver.{out|err} logs. I
>>>> tried restarting the process and it still doesn't work.
>>>>
>>>> Anyone know how I can track this down?
>>>> --
>>>> Frank Maritato
>>>> This message contains proprietary information from Equifax which may be
>>>> confidential. If you are not an intended recipient, please refrain from
>>> any
>>>> disclosure, copying, distribution or use of this information and note
>>> that
>>>> such actions are prohibited. If you have received this transmission in
>>>> error, please notify by e-mail postmaster@equifax.com. Equifax® is a
>>>> registered trademark of Equifax Inc. All rights reserved.
>>>>
>>>
>>>
>>> --
>>>
>>> Frank Maritato
>>> This message contains proprietary information from Equifax which may be
>>> confidential. If you are not an intended recipient, please refrain from any
>>> disclosure, copying, distribution or use of this information and note that
>>> such actions are prohibited. If you have received this transmission in
>>> error, please notify by e-mail postmaster@equifax.com. Equifax® is a
>>> registered trademark of Equifax Inc. All rights reserved.
>>>
>>
>>
>> --
>>
>> Frank Maritato
>>
>
>
> --
> Frank Maritato
Re: [IE] Re: [External] RE: [IE] airflow ui not showing logs
Posted by Frank Maritato <fm...@opentable.com.INVALID>.
Ok, I finally had time to try and track down the issue. Looks like the cfg
option 'task_log_reader' was changed at some point prior to 1.10.0.
Previously, my airflow.cfg had:
task_log_reader = file.task
But now it needs to be set to
task_log_reader = task
On Mon, Oct 15, 2018 at 1:57 PM Frank Maritato <fm...@opentable.com>
wrote:
> James - No, my configuration doesn't have a setting for rbac, so I'm
> assuming the default is False.
>
> Sunil - We are currently running the LocalExecutor. Is the worker process
> just for Celery? According to the quickstart, I thought the only processes
> we need to start are the webserver and scheduler.
>
>
> On Mon, Oct 15, 2018 at 12:32 PM Sunil Varma Chiluvuri <
> SunilVarma.Chiluvuri@equifax.com> wrote:
>
>> Frank,
>>
>> The serve_logs process usually starts automatically with the 'airflow
>> worker' command. Try restarting your worker process and see if it also
>> starts the serve_logs process.
>>
>> I've had a situation where the serve_logs process went down for some
>> reason and restarting the worker fixed it. It doesn't explain the cause
>> but could possibly resolve your issue.
>>
>> Sunil
>>
>> -----Original Message-----
>> From: Frank Maritato [mailto:fmaritato@opentable.com.INVALID]
>> Sent: Monday, October 15, 2018 2:14 PM
>> To: dev@airflow.incubator.apache.org
>> Subject: [IE] Re: [External] RE: [IE] airflow ui not showing logs
>>
>> Hi Sunil,
>>
>> I don't see this process running. I have never had to run this command
>> previously. Should it have started as part of the 'airflow webserver'
>> command?
>> I ran it manually from the command line as 'airflow serve_logs &' (does
>> not
>> have a daemon option I guess) but I am still not seeing logs in the ui. I
>> did verify with netstat that it is running on port 8793.
>>
>>
>> On Mon, Oct 15, 2018 at 12:02 PM Sunil Varma Chiluvuri <
>> SunilVarma.Chiluvuri@equifax.com> wrote:
>>
>> > Check if the serve_logs process is running alongside your worker
>> > process(es). This is the process that takes the log files written to
>> disk
>> > and serves them to the web UI. It should be running on port 8793 by
>> default.
>> >
>> > Sunil
>> >
>> > -----Original Message-----
>> > From: Frank Maritato [mailto:fmaritato@opentable.com.INVALID]
>> > Sent: Monday, October 15, 2018 1:42 PM
>> > To: dev@airflow.incubator.apache.org
>> > Subject: [IE] airflow ui not showing logs
>> >
>> > Hi All,
>> >
>> > I'm running airflow 1.10.0 and the ui isn't showing the task logs
>> anymore.
>> > This has worked in the past so I'm not sure what changed. I was able to
>> > verify that the logs are definitely being written to the same local
>> > directory as what is specified in the airflow.cfg 'base_log_folder'. I
>> > don't see any errors or debug in the airflow-webserver.{out|err} logs. I
>> > tried restarting the process and it still doesn't work.
>> >
>> > Anyone know how I can track this down?
>> > --
>> > Frank Maritato
>> > This message contains proprietary information from Equifax which may be
>> > confidential. If you are not an intended recipient, please refrain from
>> any
>> > disclosure, copying, distribution or use of this information and note
>> that
>> > such actions are prohibited. If you have received this transmission in
>> > error, please notify by e-mail postmaster@equifax.com. Equifax® is a
>> > registered trademark of Equifax Inc. All rights reserved.
>> >
>>
>>
>> --
>>
>> Frank Maritato
>> This message contains proprietary information from Equifax which may be
>> confidential. If you are not an intended recipient, please refrain from any
>> disclosure, copying, distribution or use of this information and note that
>> such actions are prohibited. If you have received this transmission in
>> error, please notify by e-mail postmaster@equifax.com. Equifax® is a
>> registered trademark of Equifax Inc. All rights reserved.
>>
>
>
> --
>
> Frank Maritato
>
--
Frank Maritato
Re: [IE] Re: [External] RE: [IE] airflow ui not showing logs
Posted by Frank Maritato <fm...@opentable.com.INVALID>.
James - No, my configuration doesn't have a setting for rbac, so I'm
assuming the default is False.
Sunil - We are currently running the LocalExecutor. Is the worker process
just for Celery? According to the quickstart, I thought the only processes
we need to start are the webserver and scheduler.
On Mon, Oct 15, 2018 at 12:32 PM Sunil Varma Chiluvuri <
SunilVarma.Chiluvuri@equifax.com> wrote:
> Frank,
>
> The serve_logs process usually starts automatically with the 'airflow
> worker' command. Try restarting your worker process and see if it also
> starts the serve_logs process.
>
> I've had a situation where the serve_logs process went down for some
> reason and restarting the worker fixed it. It doesn't explain the cause
> but could possibly resolve your issue.
>
> Sunil
>
> -----Original Message-----
> From: Frank Maritato [mailto:fmaritato@opentable.com.INVALID]
> Sent: Monday, October 15, 2018 2:14 PM
> To: dev@airflow.incubator.apache.org
> Subject: [IE] Re: [External] RE: [IE] airflow ui not showing logs
>
> Hi Sunil,
>
> I don't see this process running. I have never had to run this command
> previously. Should it have started as part of the 'airflow webserver'
> command?
> I ran it manually from the command line as 'airflow serve_logs &' (does not
> have a daemon option I guess) but I am still not seeing logs in the ui. I
> did verify with netstat that it is running on port 8793.
>
>
> On Mon, Oct 15, 2018 at 12:02 PM Sunil Varma Chiluvuri <
> SunilVarma.Chiluvuri@equifax.com> wrote:
>
> > Check if the serve_logs process is running alongside your worker
> > process(es). This is the process that takes the log files written to disk
> > and serves them to the web UI. It should be running on port 8793 by
> default.
> >
> > Sunil
> >
> > -----Original Message-----
> > From: Frank Maritato [mailto:fmaritato@opentable.com.INVALID]
> > Sent: Monday, October 15, 2018 1:42 PM
> > To: dev@airflow.incubator.apache.org
> > Subject: [IE] airflow ui not showing logs
> >
> > Hi All,
> >
> > I'm running airflow 1.10.0 and the ui isn't showing the task logs
> anymore.
> > This has worked in the past so I'm not sure what changed. I was able to
> > verify that the logs are definitely being written to the same local
> > directory as what is specified in the airflow.cfg 'base_log_folder'. I
> > don't see any errors or debug in the airflow-webserver.{out|err} logs. I
> > tried restarting the process and it still doesn't work.
> >
> > Anyone know how I can track this down?
> > --
> > Frank Maritato
> > This message contains proprietary information from Equifax which may be
> > confidential. If you are not an intended recipient, please refrain from
> any
> > disclosure, copying, distribution or use of this information and note
> that
> > such actions are prohibited. If you have received this transmission in
> > error, please notify by e-mail postmaster@equifax.com. Equifax® is a
> > registered trademark of Equifax Inc. All rights reserved.
> >
>
>
> --
>
> Frank Maritato
> This message contains proprietary information from Equifax which may be
> confidential. If you are not an intended recipient, please refrain from any
> disclosure, copying, distribution or use of this information and note that
> such actions are prohibited. If you have received this transmission in
> error, please notify by e-mail postmaster@equifax.com. Equifax® is a
> registered trademark of Equifax Inc. All rights reserved.
>
--
Frank Maritato