You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Duan Shiqiang (JIRA)" <ji...@apache.org> on 2018/12/10 05:16:00 UTC

[jira] [Created] (AIRFLOW-3487) PostgresToGoogleCloudStorageOperator can't handle big sql result

Duan Shiqiang created AIRFLOW-3487:
--------------------------------------

             Summary: PostgresToGoogleCloudStorageOperator can't handle big sql result
                 Key: AIRFLOW-3487
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3487
             Project: Apache Airflow
          Issue Type: Bug
          Components: contrib, operators
    Affects Versions: 1.10.1
            Reporter: Duan Shiqiang


PostgresToGoogleCloudStorageOperator is using psycopg2's default cursor to query postgres which will execute and fetch all results in memory. For bigger sql result it won't work if the airflow worker doesn't have that much memory.

The operator should consider using server side cursor to handle big volume of data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)