You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/01/10 09:39:03 UTC

[GitHub] [airflow] jithin97 opened a new pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

jithin97 opened a new pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126
 
 
   ### JIRA Issue
   Link to JIRA issue: https://issues.apache.org/jira/browse/AIRFLOW-6405
   
   ### Description
   
   1. Currently there is no a BigQuery Hook named `run_table_upsert` but there is no BigQuery Table Upsert Operator utilizing this hook. This PR includes a new GCP BigQuery Operator to include this.
   
   2. This new operator allows updating the given table with given parameters or else creates a new table if it is not already present.
   
   3. It also adds a few test cases for the existing hook.
   
   ---
   Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   <sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-573288529
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=h1) Report
   > Merging [#7126](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/26c0c53e2009d1d4aaefbd5666f2aea97d7f360f?src=pr&el=desc) will **decrease** coverage by `1.07%`.
   > The diff coverage is `93.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7126/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7126      +/-   ##
   ==========================================
   - Coverage   85.34%   84.27%   -1.08%     
   ==========================================
     Files         791      791              
     Lines       40128    40153      +25     
   ==========================================
   - Hits        34249    33838     -411     
   - Misses       5879     6315     +436
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.87% <100%> (+0.05%)` | :arrow_up: |
   | [airflow/gcp/example\_dags/example\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYmlncXVlcnkucHk=) | `100% <100%> (ø)` | :arrow_up: |
   | [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `91.46% <91.3%> (-0.13%)` | :arrow_down: |
   | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
   | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0%> (-54.93%)` | :arrow_down: |
   | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50% <0%> (-50%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | ... and [17 more](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=footer). Last update [26c0c53...4b9b0c5](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#discussion_r376326526
 
 

 ##########
 File path: airflow/gcp/hooks/bigquery.py
 ##########
 @@ -1109,6 +1110,16 @@ def run_table_upsert(self, dataset_id: str, table_resource: Dict,
         """
         service = self.get_service()
         # check to see if the table exists
+        if 'tableReference' not in table_resource:
+            raise AirflowBadRequest(
+                '"tableReference" is required within table_resource parameter. '
+                'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+            )
+        if 'tableId' not in table_resource["tableReference"]:
+            raise AirflowBadRequest(
+                '"tableId" is required within table_resource["tableReference"]. '
+                'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+            )
 
 Review comment:
   Users can use this method in their own custom operators that are not a part of Airflow. In this case, we have to be sure that backward compatibility is preserved... especially because it's a BQ which is quite popular in Airflow.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-573288529
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=h1) Report
   > Merging [#7126](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/445ed2fcaedd63ce763858e125d04fe78627b35b?src=pr&el=desc) will **decrease** coverage by `0.27%`.
   > The diff coverage is `92.59%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7126/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7126      +/-   ##
   ==========================================
   - Coverage   85.24%   84.96%   -0.28%     
   ==========================================
     Files         683      683              
     Lines       39152    39182      +30     
   ==========================================
   - Hits        33375    33292      -83     
   - Misses       5777     5890     +113
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.46% <100%> (+0.06%)` | :arrow_up: |
   | [airflow/gcp/example\_dags/example\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYmlncXVlcnkucHk=) | `100% <100%> (ø)` | :arrow_up: |
   | [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `91.97% <90%> (-0.11%)` | :arrow_down: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.47%)` | :arrow_down: |
   | [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `99.28% <0%> (-0.01%)` | :arrow_down: |
   | [airflow/cli/commands/config\_command.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY29uZmlnX2NvbW1hbmQucHk=) | `100% <0%> (ø)` | :arrow_up: |
   | ... and [2 more](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=footer). Last update [445ed2f...69306c3](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-573288529
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=h1) Report
   > Merging [#7126](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/ba85bde8efaa3e2a87607ddc97fa40a0b7e16e1e?src=pr&el=desc) will **increase** coverage by `0.04%`.
   > The diff coverage is `93.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7126/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7126      +/-   ##
   ==========================================
   + Coverage   85.86%   85.91%   +0.04%     
   ==========================================
     Files         863      863              
     Lines       40484    40870     +386     
   ==========================================
   + Hits        34763    35114     +351     
   - Misses       5721     5756      +35
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/providers/google/cloud/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2JpZ3F1ZXJ5LnB5) | `91.87% <100%> (+0.05%)` | :arrow_up: |
   | [...ders/google/cloud/example\_dags/example\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2JpZ3F1ZXJ5LnB5) | `100% <100%> (ø)` | :arrow_up: |
   | [...rflow/providers/google/cloud/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9iaWdxdWVyeS5weQ==) | `91.48% <91.3%> (-0.14%)` | :arrow_down: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: |
   | [airflow/providers/papermill/operators/papermill.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcGFwZXJtaWxsL29wZXJhdG9ycy9wYXBlcm1pbGwucHk=) | `96.42% <0%> (-3.58%)` | :arrow_down: |
   | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.72% <0%> (-1.16%)` | :arrow_down: |
   | ... and [10 more](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=footer). Last update [ba85bde...172322e](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-573025848
 
 
   BigQuery is undergoing massive modernization. On Wednesday, one big patch will appear, which will change a bit how this PR should look. Can we wait for it to be reviewed then?  Is this very important for you?  
   
   CC: @TobKed 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] boring-cyborg[bot] commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-572952087
 
 
   Congratulations on your first Pull Request and welcome to the Apache Airflow community!
   If you have any issues or are unsure about any anything please check our
   Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
   
   Here are some useful points:
   - Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits](
   https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks)
   will help you with that.
   - In case of a new feature add useful documentation (in docstrings or in `docs/` directory).
   Adding a new operator? Consider adding an example DAG that shows how users should use it.
   - Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing
   locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
   - Be patient and persistent. It might take some time to get a review or get the final approval from
   Committers.
   
   Apache Airflow is a community-driven project and together we are making it better 🚀.
   
   In case of doubts contact the developers at:
   Mailing List: dev@airflow.apache.org
   Slack: https://apache-airflow-slack.herokuapp.com/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-574072213
 
 
   Hi @jithin97 , it seems that the mentioned changes are ready could you rebase? @TobKed 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-577704404
 
 
   > Can you rebase, please?
   
   @nuclearpinguin done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] jithin97 commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
jithin97 commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#discussion_r376342272
 
 

 ##########
 File path: airflow/gcp/hooks/bigquery.py
 ##########
 @@ -1109,6 +1110,16 @@ def run_table_upsert(self, dataset_id: str, table_resource: Dict,
         """
         service = self.get_service()
         # check to see if the table exists
+        if 'tableReference' not in table_resource:
+            raise AirflowBadRequest(
+                '"tableReference" is required within table_resource parameter. '
+                'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+            )
+        if 'tableId' not in table_resource["tableReference"]:
+            raise AirflowBadRequest(
+                '"tableId" is required within table_resource["tableReference"]. '
+                'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+            )
 
 Review comment:
   @nuclearpinguin removed https://github.com/apache/airflow/pull/7126/commits/e735ac1438970f333c8c3c681e9f93d5543021f9 and rebased 🙇 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] jithin97 commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
jithin97 commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#discussion_r372935998
 
 

 ##########
 File path: airflow/gcp/hooks/bigquery.py
 ##########
 @@ -1109,6 +1110,16 @@ def run_table_upsert(self, dataset_id: str, table_resource: Dict,
         """
         service = self.get_service()
         # check to see if the table exists
+        if 'tableReference' not in table_resource:
+            raise AirflowBadRequest(
+                '"tableReference" is required within table_resource parameter. '
+                'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+            )
+        if 'tableId' not in table_resource["tableReference"]:
+            raise AirflowBadRequest(
+                '"tableId" is required within table_resource["tableReference"]. '
+                'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+            )
 
 Review comment:
   @nuclearpinguin If your concern is about the operators then I think we are safe because currently none of the operators are using this method. The only calls are from test cases.
   
   Memo:
   ```
   :~/airflow|master 
   ⇒  git grep run_table_upsert
   
   CHANGELOG.txt:- [AIRFLOW-314] Fix BigQuery cursor run_table_upsert method
   CHANGELOG.txt:- Add two methods to bigquery hook's base cursor: run_table_upsert, which adds a table or updates an existing table; and run_grant_dataset_view_access, which grants view access to a given dataset for a given table.
   UPDATING.md:| airflow.gcp.hooks.bigquery.BigQueryBaseCursor.run_table_upsert              | airflow.gcp.hooks.bigquery.BigQueryHook.run_table_upsert              |
   airflow/gcp/hooks/bigquery.py:    def run_table_upsert(self, dataset_id: str, table_resource: Dict,
   airflow/gcp/hooks/bigquery.py:    def run_table_upsert(self, *args, **kwargs) -> Dict:
   airflow/gcp/hooks/bigquery.py:        Please use `airflow.gcp.hooks.bigquery.BigQueryHook.run_table_upsert`
   airflow/gcp/hooks/bigquery.py:            "Please use `airflow.gcp.hooks.bigquery.BigQueryHook.run_table_upsert`",
   airflow/gcp/hooks/bigquery.py:        return self.hook.run_table_upsert(*args, **kwargs)
   tests/gcp/hooks/test_bigquery.py:        bq_hook.run_table_upsert(dataset_id=DATASET_ID, table_resource=table_resource)
   tests/gcp/hooks/test_bigquery.py:        bq_hook.run_table_upsert(dataset_id=DATASET_ID, table_resource=table_resource)
   tests/gcp/hooks/test_bigquery.py:        ("run_table_upsert",),
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-586629212
 
 
   @jithin97 can you rebase please? I've already restarted the build ~3 times :<

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-574201521
 
 
   > bigquery have upsert function? I searched bigquery docs and the word upsert doesn't appear in it even once.
   > 
   > I think they call it merge.
   > Why to use different name from what bigquery decided to call the operation?
   
   @RosterIn 
   
   I am not sure which merge operation you are talking about. But if it is this https://cloud.google.com/bigquery/docs/reference/standard-sql/dml-syntax#merge_statement
   then I think the context of this PR is a bit different.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] jithin97 commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
jithin97 commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#discussion_r372935998
 
 

 ##########
 File path: airflow/gcp/hooks/bigquery.py
 ##########
 @@ -1109,6 +1110,16 @@ def run_table_upsert(self, dataset_id: str, table_resource: Dict,
         """
         service = self.get_service()
         # check to see if the table exists
+        if 'tableReference' not in table_resource:
+            raise AirflowBadRequest(
+                '"tableReference" is required within table_resource parameter. '
+                'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+            )
+        if 'tableId' not in table_resource["tableReference"]:
+            raise AirflowBadRequest(
+                '"tableId" is required within table_resource["tableReference"]. '
+                'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+            )
 
 Review comment:
   @nuclearpinguin If your concern is about the operators then I think we are safe because currently none of the operators are using this method. The only calls are from test cases.
   
   Memo:
   ```
   :~/Desktop/search_work/airflow|master 
   ⇒  git grep run_table_upsert
   
   CHANGELOG.txt:- [AIRFLOW-314] Fix BigQuery cursor run_table_upsert method
   CHANGELOG.txt:- Add two methods to bigquery hook's base cursor: run_table_upsert, which adds a table or updates an existing table; and run_grant_dataset_view_access, which grants view access to a given dataset for a given table.
   UPDATING.md:| airflow.gcp.hooks.bigquery.BigQueryBaseCursor.run_table_upsert              | airflow.gcp.hooks.bigquery.BigQueryHook.run_table_upsert              |
   airflow/gcp/hooks/bigquery.py:    def run_table_upsert(self, dataset_id: str, table_resource: Dict,
   airflow/gcp/hooks/bigquery.py:    def run_table_upsert(self, *args, **kwargs) -> Dict:
   airflow/gcp/hooks/bigquery.py:        Please use `airflow.gcp.hooks.bigquery.BigQueryHook.run_table_upsert`
   airflow/gcp/hooks/bigquery.py:            "Please use `airflow.gcp.hooks.bigquery.BigQueryHook.run_table_upsert`",
   airflow/gcp/hooks/bigquery.py:        return self.hook.run_table_upsert(*args, **kwargs)
   tests/gcp/hooks/test_bigquery.py:        bq_hook.run_table_upsert(dataset_id=DATASET_ID, table_resource=table_resource)
   tests/gcp/hooks/test_bigquery.py:        bq_hook.run_table_upsert(dataset_id=DATASET_ID, table_resource=table_resource)
   tests/gcp/hooks/test_bigquery.py:        ("run_table_upsert",),
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-573033641
 
 
   > BigQuery is undergoing massive modernization. On Wednesday, one big patch will appear, which will change a bit how this PR should look. Can we wait for it to be reviewed then? Is this very important for you?
   > 
   > CC: @TobKed
   
   @mik-laj 
   OK. No hurries. I would wait till then

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] jithin97 commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
jithin97 commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#discussion_r376327268
 
 

 ##########
 File path: airflow/gcp/hooks/bigquery.py
 ##########
 @@ -1109,6 +1110,16 @@ def run_table_upsert(self, dataset_id: str, table_resource: Dict,
         """
         service = self.get_service()
         # check to see if the table exists
+        if 'tableReference' not in table_resource:
+            raise AirflowBadRequest(
+                '"tableReference" is required within table_resource parameter. '
+                'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+            )
+        if 'tableId' not in table_resource["tableReference"]:
+            raise AirflowBadRequest(
+                '"tableId" is required within table_resource["tableReference"]. '
+                'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+            )
 
 Review comment:
   I see... Well even though these are quite trivial checks and google API would any fail for such requests.
   But still, let me remove them for now.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#discussion_r371205169
 
 

 ##########
 File path: airflow/gcp/hooks/bigquery.py
 ##########
 @@ -1109,6 +1110,16 @@ def run_table_upsert(self, dataset_id: str, table_resource: Dict,
         """
         service = self.get_service()
         # check to see if the table exists
+        if 'tableReference' not in table_resource:
+            raise AirflowBadRequest(
+                '"tableReference" is required within table_resource parameter. '
+                'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+            )
+        if 'tableId' not in table_resource["tableReference"]:
+            raise AirflowBadRequest(
+                '"tableId" is required within table_resource["tableReference"]. '
+                'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+            )
 
 Review comment:
   I just wonder, does it break backward compatibility?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-573288529
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=h1) Report
   > Merging [#7126](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/445ed2fcaedd63ce763858e125d04fe78627b35b?src=pr&el=desc) will **decrease** coverage by `0.27%`.
   > The diff coverage is `92.59%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7126/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7126      +/-   ##
   ==========================================
   - Coverage   85.24%   84.96%   -0.28%     
   ==========================================
     Files         683      683              
     Lines       39152    39182      +30     
   ==========================================
   - Hits        33375    33292      -83     
   - Misses       5777     5890     +113
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.46% <100%> (+0.06%)` | :arrow_up: |
   | [airflow/gcp/example\_dags/example\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYmlncXVlcnkucHk=) | `100% <100%> (ø)` | :arrow_up: |
   | [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `91.97% <90%> (-0.11%)` | :arrow_down: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.47%)` | :arrow_down: |
   | [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `99.28% <0%> (-0.01%)` | :arrow_down: |
   | [airflow/cli/commands/config\_command.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY29uZmlnX2NvbW1hbmQucHk=) | `100% <0%> (ø)` | :arrow_up: |
   | ... and [2 more](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=footer). Last update [445ed2f...69306c3](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-573288529
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=h1) Report
   > Merging [#7126](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/445ed2fcaedd63ce763858e125d04fe78627b35b?src=pr&el=desc) will **decrease** coverage by `0.27%`.
   > The diff coverage is `92.59%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7126/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7126      +/-   ##
   ==========================================
   - Coverage   85.24%   84.96%   -0.28%     
   ==========================================
     Files         683      683              
     Lines       39152    39182      +30     
   ==========================================
   - Hits        33375    33292      -83     
   - Misses       5777     5890     +113
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.46% <100%> (+0.06%)` | :arrow_up: |
   | [airflow/gcp/example\_dags/example\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYmlncXVlcnkucHk=) | `100% <100%> (ø)` | :arrow_up: |
   | [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `91.97% <90%> (-0.11%)` | :arrow_down: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.47%)` | :arrow_down: |
   | [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `99.28% <0%> (-0.01%)` | :arrow_down: |
   | [airflow/cli/commands/config\_command.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY29uZmlnX2NvbW1hbmQucHk=) | `100% <0%> (ø)` | :arrow_up: |
   | ... and [2 more](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=footer). Last update [445ed2f...69306c3](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin merged pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
nuclearpinguin merged pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-574116522
 
 
   > Hi @jithin97 , it seems that the mentioned changes are ready could you rebase? @TobKed
   
   @nuclearpinguin I have rebased it to latest master. Please take a look 🙇 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-577694474
 
 
   Can you rebase, please?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] RosterIn commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
RosterIn commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-574166835
 
 
   bigquery have upsert function? I searched bigquery docs and the word upsert doesn't appear in it even once.
   
   I think they call it merge.
   Why to use different name from what bigquery decided to call the operation? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-573288529
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=h1) Report
   > Merging [#7126](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/26c0c53e2009d1d4aaefbd5666f2aea97d7f360f?src=pr&el=desc) will **decrease** coverage by `1.07%`.
   > The diff coverage is `93.33%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7126/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7126      +/-   ##
   ==========================================
   - Coverage   85.34%   84.27%   -1.08%     
   ==========================================
     Files         791      791              
     Lines       40128    40153      +25     
   ==========================================
   - Hits        34249    33838     -411     
   - Misses       5879     6315     +436
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.87% <100%> (+0.05%)` | :arrow_up: |
   | [airflow/gcp/example\_dags/example\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYmlncXVlcnkucHk=) | `100% <100%> (ø)` | :arrow_up: |
   | [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `91.46% <91.3%> (-0.13%)` | :arrow_down: |
   | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
   | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0%> (-54.93%)` | :arrow_down: |
   | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50% <0%> (-50%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | ... and [17 more](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=footer). Last update [26c0c53...4b9b0c5](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] jithin97 commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
jithin97 commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#discussion_r376327268
 
 

 ##########
 File path: airflow/gcp/hooks/bigquery.py
 ##########
 @@ -1109,6 +1110,16 @@ def run_table_upsert(self, dataset_id: str, table_resource: Dict,
         """
         service = self.get_service()
         # check to see if the table exists
+        if 'tableReference' not in table_resource:
+            raise AirflowBadRequest(
+                '"tableReference" is required within table_resource parameter. '
+                'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+            )
+        if 'tableId' not in table_resource["tableReference"]:
+            raise AirflowBadRequest(
+                '"tableId" is required within table_resource["tableReference"]. '
+                'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+            )
 
 Review comment:
   I see... Well even though these are quite trivial tests and google API would any fail for such requests.
   But still, let me remove them for now.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-586716431
 
 
   > @jithin97 can you rebase please? I've already restarted the build ~3 times :<
   
   @nuclearpinguin done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-583712099
 
 
   I restarted the build 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-573288529
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=h1) Report
   > Merging [#7126](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/7c8bf94b3d4f2e848fce6e5fc4a2676ab1650955?src=pr&el=desc) will **decrease** coverage by `0.27%`.
   > The diff coverage is `93.54%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7126/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7126      +/-   ##
   ==========================================
   - Coverage   85.45%   85.18%   -0.28%     
   ==========================================
     Files         710      710              
     Lines       39474    39499      +25     
   ==========================================
   - Hits        33734    33646      -88     
   - Misses       5740     5853     +113
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.87% <100%> (+0.05%)` | :arrow_up: |
   | [airflow/gcp/example\_dags/example\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYmlncXVlcnkucHk=) | `100% <100%> (ø)` | :arrow_up: |
   | [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `91.46% <91.66%> (-0.13%)` | :arrow_down: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
   | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.49%)` | :arrow_down: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=footer). Last update [7c8bf94...f2a401d](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services