[jira] [Closed] (AIRFLOW-3000) Allow to print into the log from operators (ability to Base Operator)

["Ash Berlin-Taylor (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-3000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ash Berlin-Taylor closed AIRFLOW-3000.
--------------------------------------
    Resolution: Won't Do

The example given in the stack-over flow answer (of sub-classing the Operator) is one way to do to this.

However this is an anti-pattern: this is going to get printed _every time Airflow parses the DAG_ - not just when it runs, but every time Airflow goes around it's parsing loop. This is going to be noisy and create possibly many GB of logs per day.

> Allow to print into the log from operators (ability to Base Operator)
> ---------------------------------------------------------------------
>
>                 Key: AIRFLOW-3000
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3000
>             Project: Apache Airflow
>          Issue Type: Task
>    Affects Versions: 1.10.0
>            Reporter: jack
>            Priority: Minor
>
> As described on stack overflow: [https://stackoverflow.com/questions/52144108/how-to-print-a-unique-message-in-airflow-operator]
>  
> Any print in the code will be shown on the log file. However it a problem when creating operators dinamicly
>  
> assume this code:
>  
> {code:java}
> for i in range(5, 0, -1):
>  print("My name is load_ads_to_BigQuery-{}".format{i))
>  update_bigquery = GoogleCloudStorageToBigQueryOperator   (task_id='load_ads_to_BigQuery-{}'.format(I),…){code}
>  
> This creates 5 operators.
> The print will be executed 5 times per each operator.
> meaning that if you go to the log of 
> {code:java}
> load_ads_to_BigQuery-1 {code}
> you will see:
>  
>  
> {code:java}
> My name is load_ads_to_BigQuery-1
> My name is load_ads_to_BigQuery-2
> My name is load_ads_to_BigQuery-3
> My name is load_ads_to_BigQuery-4
> My name is load_ads_to_BigQuery-5
> {code}
>  
>  
> This is a problem because it logs messages of the other operators.
>  
> Each operator is unique only with-in the operator itself. meaning that the print should be inside the operator as:
> for i in range(5, 0, -1):
>  
>  
> {code:java}
> update_bigquery = GoogleCloudStorageToBigQueryOperator (task_id='load_ads_to_BigQuery-{}'.format(I) , print("My name is load_ads_to_BigQuery-{}".format{i)) , …){code}
>  
> or something like it. However Airflow does not support printing inside operators. It's not one of the allowed arguments.
> Add optional parameter called 
> {code:java}
> msg_log {code}
> that if assigned with value it will print the value to the log when the operator is executed.
>  
> Please add argument on the Base Operator for printing and extend it as optional ability to all operators.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)