You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Daniel (Jira)" <ji...@apache.org> on 2019/11/20 13:11:00 UTC

[jira] [Updated] (AIRFLOW-4578) Set Container for Logs to Azure Blob Storage

     [ https://issues.apache.org/jira/browse/AIRFLOW-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel updated AIRFLOW-4578:
----------------------------
    Description: 
Currently it is possible to set the container that the logs to azure are written to when following the instructions.

However, there remain two issues:
 # The instructions do not mention the lines that need to be changed in the code to effect using a different container.
 # The instructions are completely unnecessary to use blob storage. 

Additionally, there is no option to remove files from local log storage. Which is the most obvious use-case.

I suggest an update that, when setting {color:#707070}remote_base_log_folder{color} in airflow.cfg, sets this as the Container in Azure. (The current setup forces the user to setup a container called airflow_logs and creates a folder in airflow_logs with the name set in {color:#707070}remote_base_log_folder{color}. This does not appear to be the likely behavior. I also propose that one can delete local logs via an environment variable.

Secondly, I suggest updating the instructions such that they resemble the logging setup for both s3 and gcs. Setting three items in {color:#707070}airflow.cfg{color} allows logs to be sent to azure, but it currently requires a container labelled {color:#707070}airflow-logs{color} in the storage account. These three items are:

{color:#707070}remote_logging=True{color}

{color:#707070}remote_base_log_folder=wasb-<choose a name here>{color}

{color:#a9b7c6}remote_log_conn_id= <wasb-default>{color}

 

  was:
Currently it is possible to set the container that the logs to azure are written to when following the instructions.

However, there remain two issues:
 # The instructions do not mention the lines that need to be changed in the code to effect using a different container.
 # The instructions are completely unnecessary to use blob storage. 

I suggest an update that, when setting {color:#707070}remote_base_log_folder{color} in airflow.cfg, sets this as the Container in Azure. (The current setup forces the user to setup a container called airflow_logs and creates a folder in airflow_logs with the name set in {color:#707070}remote_base_log_folder{color}. This does not appear to be the likely behavior.


Secondly, I suggest updating the instructions such that they resemble the logging setup for both s3 and gcs. Setting three items in {color:#707070}airflow.cfg{color} allows logs to be sent to azure, but it currently requires a container labelled {color:#707070}airflow-logs{color} in the storage account. These three items are:


{color:#707070}remote_logging=True
{color}

{color:#707070}remote_base_log_folder=wasb-<choose a name here>{color}

{color:#a9b7c6}{color:#707070}remote_log_conn_id= <wasb-default>{color}
{color}

 


> Set Container for Logs to Azure Blob Storage
> --------------------------------------------
>
>                 Key: AIRFLOW-4578
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4578
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: configuration
>    Affects Versions: 1.10.3
>            Reporter: Daniel
>            Assignee: Daniel
>            Priority: Trivial
>   Original Estimate: 5h
>  Remaining Estimate: 5h
>
> Currently it is possible to set the container that the logs to azure are written to when following the instructions.
> However, there remain two issues:
>  # The instructions do not mention the lines that need to be changed in the code to effect using a different container.
>  # The instructions are completely unnecessary to use blob storage. 
> Additionally, there is no option to remove files from local log storage. Which is the most obvious use-case.
> I suggest an update that, when setting {color:#707070}remote_base_log_folder{color} in airflow.cfg, sets this as the Container in Azure. (The current setup forces the user to setup a container called airflow_logs and creates a folder in airflow_logs with the name set in {color:#707070}remote_base_log_folder{color}. This does not appear to be the likely behavior. I also propose that one can delete local logs via an environment variable.
> Secondly, I suggest updating the instructions such that they resemble the logging setup for both s3 and gcs. Setting three items in {color:#707070}airflow.cfg{color} allows logs to be sent to azure, but it currently requires a container labelled {color:#707070}airflow-logs{color} in the storage account. These three items are:
> {color:#707070}remote_logging=True{color}
> {color:#707070}remote_base_log_folder=wasb-<choose a name here>{color}
> {color:#a9b7c6}remote_log_conn_id= <wasb-default>{color}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)