You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by manish ranjan <cs...@gmail.com> on 2017/10/25 20:42:43 UTC
Airflow always dumps logs to $AIRFLOW_HOME in 1.8.2
Hi All,
My airflow runs in production on AWS and as it is also the cluster of my
Spark Cluster, I tried to redirect the logs to a separate folder /mnt/xyz
However, I always see logs being generated under
* /home/hadoop/airflow/logs/scheduler/.*
I have already provided this option in airflow.cfg (See screen shot below)
and restarted the webserver and scheduler (I am not sure if it was
required, but I did it anyway).
Am I doing anything wrong here ?
~Manish
Re: Airflow always dumps logs to $AIRFLOW_HOME in 1.8.2
Posted by manish ranjan <cs...@gmail.com>.
Thanks for the response Daniel.
If others could not see my screenshot either, I am pasting the content
*base_log_folder = /mnt1/logs*
*child_process_log_directory = /mnt1/logs/scheduler*
Followed by a restart of airflow-scheduler and webserver (Again I did it as
a cautionary measure, may not have been required). But it continued to log
the files under * /home/hadoop/airflow/logs/scheduler/. *and I could never
get it to log to /mnt location.
Any help is appreciated.
On Wed, Oct 25, 2017 at 1:47 PM Daniel Huang <dx...@gmail.com> wrote:
> I can't view your screenshot, but note that there are two different configs
> for log file path in airflow.cfg. There is core:base_log_folder and
> scheduler:child_process_log_directory, the latter being the location of the
> scheduler logs.
>
> On Wed, Oct 25, 2017 at 1:44 PM, manish ranjan <cs...@gmail.com>
> wrote:
>
> > The airflow version is 1.8.0. Sorry about the typo.
> >
> > Manish
> >
> > ~Manish
> >
> >
> >
> > On Wed, Oct 25, 2017 at 1:42 PM, manish ranjan <cs...@gmail.com>
> > wrote:
> >
> >> Hi All,
> >>
> >> My airflow runs in production on AWS and as it is also the cluster of my
> >> Spark Cluster, I tried to redirect the logs to a separate folder
> /mnt/xyz
> >> However, I always see logs being generated under
> >> * /home/hadoop/airflow/logs/scheduler/.*
> >>
> >> I have already provided this option in airflow.cfg (See screen shot
> >> below) and restarted the webserver and scheduler (I am not sure if it
> was
> >> required, but I did it anyway).
> >>
> >>
> >>
> >> Am I doing anything wrong here ?
> >>
> >>
> >> ~Manish
> >>
> >>
> >>
> >
>
Re: Airflow always dumps logs to $AIRFLOW_HOME in 1.8.2
Posted by Daniel Huang <dx...@gmail.com>.
I can't view your screenshot, but note that there are two different configs
for log file path in airflow.cfg. There is core:base_log_folder and
scheduler:child_process_log_directory, the latter being the location of the
scheduler logs.
On Wed, Oct 25, 2017 at 1:44 PM, manish ranjan <cs...@gmail.com>
wrote:
> The airflow version is 1.8.0. Sorry about the typo.
>
> Manish
>
> ~Manish
>
>
>
> On Wed, Oct 25, 2017 at 1:42 PM, manish ranjan <cs...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> My airflow runs in production on AWS and as it is also the cluster of my
>> Spark Cluster, I tried to redirect the logs to a separate folder /mnt/xyz
>> However, I always see logs being generated under
>> * /home/hadoop/airflow/logs/scheduler/.*
>>
>> I have already provided this option in airflow.cfg (See screen shot
>> below) and restarted the webserver and scheduler (I am not sure if it was
>> required, but I did it anyway).
>>
>>
>>
>> Am I doing anything wrong here ?
>>
>>
>> ~Manish
>>
>>
>>
>
Re: Airflow always dumps logs to $AIRFLOW_HOME in 1.8.2
Posted by manish ranjan <cs...@gmail.com>.
The airflow version is 1.8.0. Sorry about the typo.
Manish
~Manish
On Wed, Oct 25, 2017 at 1:42 PM, manish ranjan <cs...@gmail.com>
wrote:
> Hi All,
>
> My airflow runs in production on AWS and as it is also the cluster of my
> Spark Cluster, I tried to redirect the logs to a separate folder /mnt/xyz
> However, I always see logs being generated under
> * /home/hadoop/airflow/logs/scheduler/.*
>
> I have already provided this option in airflow.cfg (See screen shot below)
> and restarted the webserver and scheduler (I am not sure if it was
> required, but I did it anyway).
>
>
>
> Am I doing anything wrong here ?
>
>
> ~Manish
>
>
>