You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/08 20:00:06 UTC
[GitHub] [beam] ahmedabu98 opened a new pull request, #21757: Handling invalid WriteToBigQuery use case
ahmedabu98 opened a new pull request, #21757:
URL: https://github.com/apache/beam/pull/21757
WriteToBigQuery with copy jobs doesn't support writing to a single table partition.
This PR is to throw an early error before beam starts up a copy job.
Also adding some documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #21757: Handling invalid WriteToBigQuery use case
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21757:
URL: https://github.com/apache/beam/pull/21757#issuecomment-1464904453
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on pull request #21757: Handling invalid WriteToBigQuery use case
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on PR #21757:
URL: https://github.com/apache/beam/pull/21757#issuecomment-1203183008
Run Python PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #21757: Handling invalid WriteToBigQuery use case
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21757:
URL: https://github.com/apache/beam/pull/21757#issuecomment-1682215103
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on pull request #21757: Handling invalid WriteToBigQuery use case
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on PR #21757:
URL: https://github.com/apache/beam/pull/21757#issuecomment-1150399500
R: @chamikaramj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on a diff in pull request #21757: Handling invalid WriteToBigQuery use case
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on code in PR #21757:
URL: https://github.com/apache/beam/pull/21757#discussion_r956205223
##########
sdks/python/apache_beam/io/gcp/bigquery_file_loads.py:
##########
@@ -725,6 +725,13 @@ def process(self, element):
partitions.append(latest_partition.files)
if len(partitions) > 1:
+ # Copy jobs with BigQuery API do not support having
+ # a table partition as the destination
+ if '$' in destination:
Review Comment:
Is there a better way to identify that the table is partitioned beyond looking for a $?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on pull request #21757: Handling invalid WriteToBigQuery use case
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on PR #21757:
URL: https://github.com/apache/beam/pull/21757#issuecomment-1238424283
Run Python 3.8 PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on pull request #21757: Handling invalid WriteToBigQuery use case
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on PR #21757:
URL: https://github.com/apache/beam/pull/21757#issuecomment-1197397205
Run Python PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on a diff in pull request #21757: Handling invalid WriteToBigQuery use case
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on code in PR #21757:
URL: https://github.com/apache/beam/pull/21757#discussion_r964875252
##########
sdks/python/apache_beam/io/gcp/bigquery_file_loads.py:
##########
@@ -725,6 +725,13 @@ def process(self, element):
partitions.append(latest_partition.files)
if len(partitions) > 1:
+ # Copy jobs with BigQuery API do not support having
+ # a table partition as the destination
+ if '$' in destination:
Review Comment:
Hmm. I would reach out to the BQ team on this. looking for $ feels like it could be inconsistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #21757: Handling invalid WriteToBigQuery use case
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21757:
URL: https://github.com/apache/beam/pull/21757#issuecomment-1584513261
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #21757: Handling invalid WriteToBigQuery use case
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #21757:
URL: https://github.com/apache/beam/pull/21757#issuecomment-1150345616
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on a diff in pull request #21757: Handling invalid WriteToBigQuery use case
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on code in PR #21757:
URL: https://github.com/apache/beam/pull/21757#discussion_r961835526
##########
sdks/python/apache_beam/io/gcp/bigquery_file_loads.py:
##########
@@ -725,6 +725,13 @@ def process(self, element):
partitions.append(latest_partition.files)
if len(partitions) > 1:
+ # Copy jobs with BigQuery API do not support having
+ # a table partition as the destination
+ if '$' in destination:
Review Comment:
From what I have found online, you can tell if a given table is partitioned by performing a query: https://stackoverflow.com/questions/44881100/is-there-a-metadata-table-to-check-if-the-table-in-bigquery-is-partitioned
However, I did not find another way to tell if the destination itself is a partition of a table. BigQuery recognizes the `$` as a partition decorator: https://cloud.google.com/bigquery/docs/managing-partitioned-table-data#partition_decorators
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #21757: Handling invalid WriteToBigQuery use case
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #21757:
URL: https://github.com/apache/beam/pull/21757#issuecomment-1324403572
Any updates ? If this change is not needed anymore, we should just close the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #21757: Handling invalid WriteToBigQuery use case
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #21757:
URL: https://github.com/apache/beam/pull/21757#issuecomment-1203203226
# [Codecov](https://codecov.io/gh/apache/beam/pull/21757?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#21757](https://codecov.io/gh/apache/beam/pull/21757?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7825cd4) into [master](https://codecov.io/gh/apache/beam/commit/e62ae391985fc13c7df1ee6e088525835ceaa560?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e62ae39) will **increase** coverage by `0.02%`.
> The diff coverage is `50.00%`.
```diff
@@ Coverage Diff @@
## master #21757 +/- ##
==========================================
+ Coverage 74.03% 74.06% +0.02%
==========================================
Files 698 703 +5
Lines 92131 93032 +901
==========================================
+ Hits 68213 68905 +692
- Misses 22667 22876 +209
Partials 1251 1251
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `83.54% <50.00%> (-0.10%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/21757?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/21757/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `70.37% <ø> (+5.40%)` | :arrow_up: |
| [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/21757/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `87.52% <50.00%> (-0.18%)` | :arrow_down: |
| [...eam/transforms/py\_dataflow\_distribution\_counter.py](https://codecov.io/gh/apache/beam/pull/21757/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9weV9kYXRhZmxvd19kaXN0cmlidXRpb25fY291bnRlci5weQ==) | `91.42% <0.00%> (-4.87%)` | :arrow_down: |
| [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/21757/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.09% <0.00%> (-4.77%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/io.py](https://codecov.io/gh/apache/beam/pull/21757/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2lvLnB5) | `88.78% <0.00%> (-3.26%)` | :arrow_down: |
| [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/21757/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `95.12% <0.00%> (-2.44%)` | :arrow_down: |
| [sdks/python/apache\_beam/utils/counters.py](https://codecov.io/gh/apache/beam/pull/21757/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvY291bnRlcnMucHk=) | `85.39% <0.00%> (-1.36%)` | :arrow_down: |
| [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/21757/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `90.06% <0.00%> (-1.33%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/21757/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `90.97% <0.00%> (-0.76%)` | :arrow_down: |
| [...ache\_beam/runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/21757/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | `90.12% <0.00%> (-0.62%)` | :arrow_down: |
| ... and [97 more](https://codecov.io/gh/apache/beam/pull/21757/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] closed pull request #21757: Handling invalid WriteToBigQuery use case
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #21757: Handling invalid WriteToBigQuery use case
URL: https://github.com/apache/beam/pull/21757
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #21757: Handling invalid WriteToBigQuery use case
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21757:
URL: https://github.com/apache/beam/pull/21757#issuecomment-1671255440
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org