You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/01/10 09:39:03 UTC
[GitHub] [airflow] jithin97 opened a new pull request #7126: [AIRFLOW-6405]
Add GCP BigQuery Table Upsert Operator
jithin97 opened a new pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126
### JIRA Issue
Link to JIRA issue: https://issues.apache.org/jira/browse/AIRFLOW-6405
### Description
1. Currently there is no a BigQuery Hook named `run_table_upsert` but there is no BigQuery Table Upsert Operator utilizing this hook. This PR includes a new GCP BigQuery Operator to include this.
2. This new operator allows updating the given table with given parameters or else creates a new table if it is not already present.
3. It also adds a few test cases for the existing hook.
---
Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
<sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7126:
[AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-573288529
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=h1) Report
> Merging [#7126](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/26c0c53e2009d1d4aaefbd5666f2aea97d7f360f?src=pr&el=desc) will **decrease** coverage by `1.07%`.
> The diff coverage is `93.33%`.
[![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7126/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7126 +/- ##
==========================================
- Coverage 85.34% 84.27% -1.08%
==========================================
Files 791 791
Lines 40128 40153 +25
==========================================
- Hits 34249 33838 -411
- Misses 5879 6315 +436
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.87% <100%> (+0.05%)` | :arrow_up: |
| [airflow/gcp/example\_dags/example\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYmlncXVlcnkucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `91.46% <91.3%> (-0.13%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0%> (-54.93%)` | :arrow_down: |
| [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50% <0%> (-50%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| ... and [17 more](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=footer). Last update [26c0c53...4b9b0c5](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#discussion_r376326526
##########
File path: airflow/gcp/hooks/bigquery.py
##########
@@ -1109,6 +1110,16 @@ def run_table_upsert(self, dataset_id: str, table_resource: Dict,
"""
service = self.get_service()
# check to see if the table exists
+ if 'tableReference' not in table_resource:
+ raise AirflowBadRequest(
+ '"tableReference" is required within table_resource parameter. '
+ 'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+ )
+ if 'tableId' not in table_resource["tableReference"]:
+ raise AirflowBadRequest(
+ '"tableId" is required within table_resource["tableReference"]. '
+ 'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+ )
Review comment:
Users can use this method in their own custom operators that are not a part of Airflow. In this case, we have to be sure that backward compatibility is preserved... especially because it's a BQ which is quite popular in Airflow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7126:
[AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-573288529
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=h1) Report
> Merging [#7126](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/445ed2fcaedd63ce763858e125d04fe78627b35b?src=pr&el=desc) will **decrease** coverage by `0.27%`.
> The diff coverage is `92.59%`.
[![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7126/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7126 +/- ##
==========================================
- Coverage 85.24% 84.96% -0.28%
==========================================
Files 683 683
Lines 39152 39182 +30
==========================================
- Hits 33375 33292 -83
- Misses 5777 5890 +113
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.46% <100%> (+0.06%)` | :arrow_up: |
| [airflow/gcp/example\_dags/example\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYmlncXVlcnkucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `91.97% <90%> (-0.11%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.47%)` | :arrow_down: |
| [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `99.28% <0%> (-0.01%)` | :arrow_down: |
| [airflow/cli/commands/config\_command.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY29uZmlnX2NvbW1hbmQucHk=) | `100% <0%> (ø)` | :arrow_up: |
| ... and [2 more](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=footer). Last update [445ed2f...69306c3](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7126:
[AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-573288529
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=h1) Report
> Merging [#7126](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/ba85bde8efaa3e2a87607ddc97fa40a0b7e16e1e?src=pr&el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `93.33%`.
[![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7126/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7126 +/- ##
==========================================
+ Coverage 85.86% 85.91% +0.04%
==========================================
Files 863 863
Lines 40484 40870 +386
==========================================
+ Hits 34763 35114 +351
- Misses 5721 5756 +35
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/providers/google/cloud/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2JpZ3F1ZXJ5LnB5) | `91.87% <100%> (+0.05%)` | :arrow_up: |
| [...ders/google/cloud/example\_dags/example\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2JpZ3F1ZXJ5LnB5) | `100% <100%> (ø)` | :arrow_up: |
| [...rflow/providers/google/cloud/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9iaWdxdWVyeS5weQ==) | `91.48% <91.3%> (-0.14%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: |
| [airflow/providers/papermill/operators/papermill.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcGFwZXJtaWxsL29wZXJhdG9ycy9wYXBlcm1pbGwucHk=) | `96.42% <0%> (-3.58%)` | :arrow_down: |
| [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.72% <0%> (-1.16%)` | :arrow_down: |
| ... and [10 more](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=footer). Last update [ba85bde...172322e](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #7126: [AIRFLOW-6405] Add GCP
BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-573025848
BigQuery is undergoing massive modernization. On Wednesday, one big patch will appear, which will change a bit how this PR should look. Can we wait for it to be reviewed then? Is this very important for you?
CC: @TobKed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] boring-cyborg[bot] commented on issue #7126:
[AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-572952087
Congratulations on your first Pull Request and welcome to the Apache Airflow community!
If you have any issues or are unsure about any anything please check our
Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits](
https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks)
will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory).
Adding a new operator? Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing
locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from
Committers.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on issue #7126: [AIRFLOW-6405]
Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-574072213
Hi @jithin97 , it seems that the mentioned changes are ready could you rebase? @TobKed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP
BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-577704404
> Can you rebase, please?
@nuclearpinguin done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] jithin97 commented on a change in pull request #7126:
[AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
jithin97 commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#discussion_r376342272
##########
File path: airflow/gcp/hooks/bigquery.py
##########
@@ -1109,6 +1110,16 @@ def run_table_upsert(self, dataset_id: str, table_resource: Dict,
"""
service = self.get_service()
# check to see if the table exists
+ if 'tableReference' not in table_resource:
+ raise AirflowBadRequest(
+ '"tableReference" is required within table_resource parameter. '
+ 'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+ )
+ if 'tableId' not in table_resource["tableReference"]:
+ raise AirflowBadRequest(
+ '"tableId" is required within table_resource["tableReference"]. '
+ 'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+ )
Review comment:
@nuclearpinguin removed https://github.com/apache/airflow/pull/7126/commits/e735ac1438970f333c8c3c681e9f93d5543021f9 and rebased 🙇
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] jithin97 commented on a change in pull request #7126:
[AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
jithin97 commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#discussion_r372935998
##########
File path: airflow/gcp/hooks/bigquery.py
##########
@@ -1109,6 +1110,16 @@ def run_table_upsert(self, dataset_id: str, table_resource: Dict,
"""
service = self.get_service()
# check to see if the table exists
+ if 'tableReference' not in table_resource:
+ raise AirflowBadRequest(
+ '"tableReference" is required within table_resource parameter. '
+ 'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+ )
+ if 'tableId' not in table_resource["tableReference"]:
+ raise AirflowBadRequest(
+ '"tableId" is required within table_resource["tableReference"]. '
+ 'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+ )
Review comment:
@nuclearpinguin If your concern is about the operators then I think we are safe because currently none of the operators are using this method. The only calls are from test cases.
Memo:
```
:~/airflow|master
⇒ git grep run_table_upsert
CHANGELOG.txt:- [AIRFLOW-314] Fix BigQuery cursor run_table_upsert method
CHANGELOG.txt:- Add two methods to bigquery hook's base cursor: run_table_upsert, which adds a table or updates an existing table; and run_grant_dataset_view_access, which grants view access to a given dataset for a given table.
UPDATING.md:| airflow.gcp.hooks.bigquery.BigQueryBaseCursor.run_table_upsert | airflow.gcp.hooks.bigquery.BigQueryHook.run_table_upsert |
airflow/gcp/hooks/bigquery.py: def run_table_upsert(self, dataset_id: str, table_resource: Dict,
airflow/gcp/hooks/bigquery.py: def run_table_upsert(self, *args, **kwargs) -> Dict:
airflow/gcp/hooks/bigquery.py: Please use `airflow.gcp.hooks.bigquery.BigQueryHook.run_table_upsert`
airflow/gcp/hooks/bigquery.py: "Please use `airflow.gcp.hooks.bigquery.BigQueryHook.run_table_upsert`",
airflow/gcp/hooks/bigquery.py: return self.hook.run_table_upsert(*args, **kwargs)
tests/gcp/hooks/test_bigquery.py: bq_hook.run_table_upsert(dataset_id=DATASET_ID, table_resource=table_resource)
tests/gcp/hooks/test_bigquery.py: bq_hook.run_table_upsert(dataset_id=DATASET_ID, table_resource=table_resource)
tests/gcp/hooks/test_bigquery.py: ("run_table_upsert",),
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on issue #7126: [AIRFLOW-6405]
Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-586629212
@jithin97 can you rebase please? I've already restarted the build ~3 times :<
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP
BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-574201521
> bigquery have upsert function? I searched bigquery docs and the word upsert doesn't appear in it even once.
>
> I think they call it merge.
> Why to use different name from what bigquery decided to call the operation?
@RosterIn
I am not sure which merge operation you are talking about. But if it is this https://cloud.google.com/bigquery/docs/reference/standard-sql/dml-syntax#merge_statement
then I think the context of this PR is a bit different.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] jithin97 commented on a change in pull request #7126:
[AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
jithin97 commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#discussion_r372935998
##########
File path: airflow/gcp/hooks/bigquery.py
##########
@@ -1109,6 +1110,16 @@ def run_table_upsert(self, dataset_id: str, table_resource: Dict,
"""
service = self.get_service()
# check to see if the table exists
+ if 'tableReference' not in table_resource:
+ raise AirflowBadRequest(
+ '"tableReference" is required within table_resource parameter. '
+ 'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+ )
+ if 'tableId' not in table_resource["tableReference"]:
+ raise AirflowBadRequest(
+ '"tableId" is required within table_resource["tableReference"]. '
+ 'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+ )
Review comment:
@nuclearpinguin If your concern is about the operators then I think we are safe because currently none of the operators are using this method. The only calls are from test cases.
Memo:
```
:~/Desktop/search_work/airflow|master
⇒ git grep run_table_upsert
CHANGELOG.txt:- [AIRFLOW-314] Fix BigQuery cursor run_table_upsert method
CHANGELOG.txt:- Add two methods to bigquery hook's base cursor: run_table_upsert, which adds a table or updates an existing table; and run_grant_dataset_view_access, which grants view access to a given dataset for a given table.
UPDATING.md:| airflow.gcp.hooks.bigquery.BigQueryBaseCursor.run_table_upsert | airflow.gcp.hooks.bigquery.BigQueryHook.run_table_upsert |
airflow/gcp/hooks/bigquery.py: def run_table_upsert(self, dataset_id: str, table_resource: Dict,
airflow/gcp/hooks/bigquery.py: def run_table_upsert(self, *args, **kwargs) -> Dict:
airflow/gcp/hooks/bigquery.py: Please use `airflow.gcp.hooks.bigquery.BigQueryHook.run_table_upsert`
airflow/gcp/hooks/bigquery.py: "Please use `airflow.gcp.hooks.bigquery.BigQueryHook.run_table_upsert`",
airflow/gcp/hooks/bigquery.py: return self.hook.run_table_upsert(*args, **kwargs)
tests/gcp/hooks/test_bigquery.py: bq_hook.run_table_upsert(dataset_id=DATASET_ID, table_resource=table_resource)
tests/gcp/hooks/test_bigquery.py: bq_hook.run_table_upsert(dataset_id=DATASET_ID, table_resource=table_resource)
tests/gcp/hooks/test_bigquery.py: ("run_table_upsert",),
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP
BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-573033641
> BigQuery is undergoing massive modernization. On Wednesday, one big patch will appear, which will change a bit how this PR should look. Can we wait for it to be reviewed then? Is this very important for you?
>
> CC: @TobKed
@mik-laj
OK. No hurries. I would wait till then
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] jithin97 commented on a change in pull request #7126:
[AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
jithin97 commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#discussion_r376327268
##########
File path: airflow/gcp/hooks/bigquery.py
##########
@@ -1109,6 +1110,16 @@ def run_table_upsert(self, dataset_id: str, table_resource: Dict,
"""
service = self.get_service()
# check to see if the table exists
+ if 'tableReference' not in table_resource:
+ raise AirflowBadRequest(
+ '"tableReference" is required within table_resource parameter. '
+ 'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+ )
+ if 'tableId' not in table_resource["tableReference"]:
+ raise AirflowBadRequest(
+ '"tableId" is required within table_resource["tableReference"]. '
+ 'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+ )
Review comment:
I see... Well even though these are quite trivial checks and google API would any fail for such requests.
But still, let me remove them for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#discussion_r371205169
##########
File path: airflow/gcp/hooks/bigquery.py
##########
@@ -1109,6 +1110,16 @@ def run_table_upsert(self, dataset_id: str, table_resource: Dict,
"""
service = self.get_service()
# check to see if the table exists
+ if 'tableReference' not in table_resource:
+ raise AirflowBadRequest(
+ '"tableReference" is required within table_resource parameter. '
+ 'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+ )
+ if 'tableId' not in table_resource["tableReference"]:
+ raise AirflowBadRequest(
+ '"tableId" is required within table_resource["tableReference"]. '
+ 'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+ )
Review comment:
I just wonder, does it break backward compatibility?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7126: [AIRFLOW-6405] Add
GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-573288529
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=h1) Report
> Merging [#7126](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/445ed2fcaedd63ce763858e125d04fe78627b35b?src=pr&el=desc) will **decrease** coverage by `0.27%`.
> The diff coverage is `92.59%`.
[![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7126/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7126 +/- ##
==========================================
- Coverage 85.24% 84.96% -0.28%
==========================================
Files 683 683
Lines 39152 39182 +30
==========================================
- Hits 33375 33292 -83
- Misses 5777 5890 +113
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.46% <100%> (+0.06%)` | :arrow_up: |
| [airflow/gcp/example\_dags/example\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYmlncXVlcnkucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `91.97% <90%> (-0.11%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.47%)` | :arrow_down: |
| [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `99.28% <0%> (-0.01%)` | :arrow_down: |
| [airflow/cli/commands/config\_command.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY29uZmlnX2NvbW1hbmQucHk=) | `100% <0%> (ø)` | :arrow_up: |
| ... and [2 more](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=footer). Last update [445ed2f...69306c3](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7126:
[AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-573288529
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=h1) Report
> Merging [#7126](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/445ed2fcaedd63ce763858e125d04fe78627b35b?src=pr&el=desc) will **decrease** coverage by `0.27%`.
> The diff coverage is `92.59%`.
[![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7126/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7126 +/- ##
==========================================
- Coverage 85.24% 84.96% -0.28%
==========================================
Files 683 683
Lines 39152 39182 +30
==========================================
- Hits 33375 33292 -83
- Misses 5777 5890 +113
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.46% <100%> (+0.06%)` | :arrow_up: |
| [airflow/gcp/example\_dags/example\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYmlncXVlcnkucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `91.97% <90%> (-0.11%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.47%)` | :arrow_down: |
| [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `99.28% <0%> (-0.01%)` | :arrow_down: |
| [airflow/cli/commands/config\_command.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY29uZmlnX2NvbW1hbmQucHk=) | `100% <0%> (ø)` | :arrow_up: |
| ... and [2 more](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=footer). Last update [445ed2f...69306c3](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin merged pull request #7126: [AIRFLOW-6405]
Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin merged pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP
BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-574116522
> Hi @jithin97 , it seems that the mentioned changes are ready could you rebase? @TobKed
@nuclearpinguin I have rebased it to latest master. Please take a look 🙇
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on issue #7126: [AIRFLOW-6405]
Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-577694474
Can you rebase, please?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] RosterIn commented on issue #7126: [AIRFLOW-6405] Add GCP
BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
RosterIn commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-574166835
bigquery have upsert function? I searched bigquery docs and the word upsert doesn't appear in it even once.
I think they call it merge.
Why to use different name from what bigquery decided to call the operation?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7126:
[AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-573288529
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=h1) Report
> Merging [#7126](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/26c0c53e2009d1d4aaefbd5666f2aea97d7f360f?src=pr&el=desc) will **decrease** coverage by `1.07%`.
> The diff coverage is `93.33%`.
[![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7126/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7126 +/- ##
==========================================
- Coverage 85.34% 84.27% -1.08%
==========================================
Files 791 791
Lines 40128 40153 +25
==========================================
- Hits 34249 33838 -411
- Misses 5879 6315 +436
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.87% <100%> (+0.05%)` | :arrow_up: |
| [airflow/gcp/example\_dags/example\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYmlncXVlcnkucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `91.46% <91.3%> (-0.13%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0%> (-54.93%)` | :arrow_down: |
| [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50% <0%> (-50%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| ... and [17 more](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=footer). Last update [26c0c53...4b9b0c5](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] jithin97 commented on a change in pull request #7126:
[AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
jithin97 commented on a change in pull request #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#discussion_r376327268
##########
File path: airflow/gcp/hooks/bigquery.py
##########
@@ -1109,6 +1110,16 @@ def run_table_upsert(self, dataset_id: str, table_resource: Dict,
"""
service = self.get_service()
# check to see if the table exists
+ if 'tableReference' not in table_resource:
+ raise AirflowBadRequest(
+ '"tableReference" is required within table_resource parameter. '
+ 'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+ )
+ if 'tableId' not in table_resource["tableReference"]:
+ raise AirflowBadRequest(
+ '"tableId" is required within table_resource["tableReference"]. '
+ 'See https://cloud.google.com/bigquery/docs/reference/v2/tables#resource'
+ )
Review comment:
I see... Well even though these are quite trivial tests and google API would any fail for such requests.
But still, let me remove them for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP
BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
jithin97 commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-586716431
> @jithin97 can you rebase please? I've already restarted the build ~3 times :<
@nuclearpinguin done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on issue #7126: [AIRFLOW-6405]
Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-583712099
I restarted the build
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7126:
[AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7126: [AIRFLOW-6405] Add GCP BigQuery Table Upsert Operator
URL: https://github.com/apache/airflow/pull/7126#issuecomment-573288529
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=h1) Report
> Merging [#7126](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/7c8bf94b3d4f2e848fce6e5fc4a2676ab1650955?src=pr&el=desc) will **decrease** coverage by `0.27%`.
> The diff coverage is `93.54%`.
[![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7126/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7126 +/- ##
==========================================
- Coverage 85.45% 85.18% -0.28%
==========================================
Files 710 710
Lines 39474 39499 +25
==========================================
- Hits 33734 33646 -88
- Misses 5740 5853 +113
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.87% <100%> (+0.05%)` | :arrow_up: |
| [airflow/gcp/example\_dags/example\_bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYmlncXVlcnkucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `91.46% <91.66%> (-0.13%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7126/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.49%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=footer). Last update [7c8bf94...f2a401d](https://codecov.io/gh/apache/airflow/pull/7126?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services