[jira] [Created] (AIRFLOW-2620) New property called "RetainSameConnection"

["Flávio Clésio (JIRA)" <ji...@apache.org>]

Flávio Clésio created AIRFLOW-2620:
--------------------------------------

             Summary: New property called "RetainSameConnection"
                 Key: AIRFLOW-2620
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2620
             Project: Apache Airflow
          Issue Type: Wish
          Components: configuration, operators, redshift
    Affects Versions: Airflow 2.0
            Reporter: Flávio Clésio


Some ETL stuff relies on the -high- reusability of the connections in the middle of the data flow, and part of this is achieved using temporary tables.

I performed a test [1] in Airflow and the behaviour is the for each command that we use the "PostgresOperator" hits the database again, and if the DAG had a lot of connections it can be a nightmare. 

As I said in SO, for several cases where you have cases of processing some tables in stages the reuse a temporary table across multiple tasks can be very useful and plus: can remove a lot of pressure in the RDBMS cause you won't need to hit several times.

That's why a purpose a feature of the connection called "RetainSameConnection" that will keep the same connection during the execution of the DAG.
 
Reference: [1] - [My question in Stack Overflow|https://stackoverflow.com/questions/50858770/airflow-retain-the-same-database-connection/] about it



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)