You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jack (JIRA)" <ji...@apache.org> on 2018/06/26 09:27:00 UTC

[jira] [Created] (AIRFLOW-2679) GoogleCloudStorageToBigQueryOperator to support UPSERT

jack created AIRFLOW-2679:
-----------------------------

             Summary: GoogleCloudStorageToBigQueryOperator to support UPSERT
                 Key: AIRFLOW-2679
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2679
             Project: Apache Airflow
          Issue Type: Improvement
            Reporter: jack
             Fix For: 1.10.0, 1.10


Currently the {color:#222222}GoogleCloudStorageToBigQueryOp{color}{color:#222222}erator support incremental load using *{color:#404040}max_id_key{color}*{color:#404040} {color}.{color}

 

{color:#222222}However many systems actually needs "UPSERT" in terms of - if row exists update it, if not insert/copy it.{color}

{color:#222222}Currently the operator assumes that we only need to insert new data, it can't handle update of data. Most of the time data is not static it changes with time. Yesterday order status was NEW today it's Processing, tomorrow it's SENT in a month it will be REFUNDED etc... {color}

 

{color:#222222} {color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)