You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/02 15:14:29 UTC
[GitHub] [beam] ahmedabu98 opened a new pull request, #23011: (BQ Python) Pass project field from options or parameter when writing with dynamic destinations
ahmedabu98 opened a new pull request, #23011:
URL: https://github.com/apache/beam/pull/23011
Currently dynamic destinations cannot be passed in as <DATASET>.<TABLE>. For single destinations, the project ID is added from pipeline options or from the WriteToBigQuery parameter earlier in the workflow. This isn't applied to dynamic destinations because the actual destinations are determined later.
These changes will add the project ID at the moment when dynamic destinations are determined in `AppendDestinationsFn`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #23011: (BQ Python) Pass project field from options or parameter when writing with dynamic destinations
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #23011:
URL: https://github.com/apache/beam/pull/23011#issuecomment-1238404579
# [Codecov](https://codecov.io/gh/apache/beam/pull/23011?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#23011](https://codecov.io/gh/apache/beam/pull/23011?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b9d4313) into [master](https://codecov.io/gh/apache/beam/commit/e83192c4fe5e2bc2b8e7f6c3568d00a49d983cdb?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e83192c) will **decrease** coverage by `0.02%`.
> The diff coverage is `100.00%`.
```diff
@@ Coverage Diff @@
## master #23011 +/- ##
==========================================
- Coverage 73.66% 73.63% -0.03%
==========================================
Files 713 716 +3
Lines 94970 95194 +224
==========================================
+ Hits 69960 70098 +138
- Misses 23709 23795 +86
Partials 1301 1301
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `83.43% <100.00%> (-0.08%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/23011?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/23011/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `74.24% <ø> (ø)` | |
| [...s/python/apache\_beam/io/gcp/bigquery\_file\_loads.py](https://codecov.io/gh/apache/beam/pull/23011/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X2ZpbGVfbG9hZHMucHk=) | `87.76% <100.00%> (+0.05%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/23011/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `85.84% <100.00%> (+0.11%)` | :arrow_up: |
| [sdks/python/apache\_beam/typehints/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/23011/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL19faW5pdF9fLnB5) | `77.77% <0.00%> (-22.23%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/statecache.py](https://codecov.io/gh/apache/beam/pull/23011/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc3RhdGVjYWNoZS5weQ==) | `89.69% <0.00%> (-6.47%)` | :arrow_down: |
| [...dks/python/apache\_beam/metrics/monitoring\_infos.py](https://codecov.io/gh/apache/beam/pull/23011/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWV0cmljcy9tb25pdG9yaW5nX2luZm9zLnB5) | `92.50% <0.00%> (-4.50%)` | :arrow_down: |
| [...python/apache\_beam/runners/worker/worker\_status.py](https://codecov.io/gh/apache/beam/pull/23011/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvd29ya2VyX3N0YXR1cy5weQ==) | `76.66% <0.00%> (-3.05%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/schemas.py](https://codecov.io/gh/apache/beam/pull/23011/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3NjaGVtYXMucHk=) | `96.62% <0.00%> (-1.05%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/23011/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.84% <0.00%> (-0.31%)` | :arrow_down: |
| [...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/23011/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXIucHk=) | `82.87% <0.00%> (-0.14%)` | :arrow_down: |
| ... and [23 more](https://codecov.io/gh/apache/beam/pull/23011/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on pull request #23011: (BQ Python) Pass project field from options or parameter when writing with dynamic destinations
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on PR #23011:
URL: https://github.com/apache/beam/pull/23011#issuecomment-1239695274
R: @johnjcasey
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on a diff in pull request #23011: (BQ Python) Pass project field from options or parameter when writing with dynamic destinations
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on code in PR #23011:
URL: https://github.com/apache/beam/pull/23011#discussion_r965130397
##########
sdks/python/apache_beam/io/gcp/bigquery.py:
##########
@@ -2278,8 +2278,8 @@ def expand(self, pcoll):
if (isinstance(self.table_reference, TableReference) and
self.table_reference.projectId is None):
- self.table_reference.projectId = pcoll.pipeline.options.view_as(
- GoogleCloudOptions).project
+ self.table_reference.projectId = self._project or \
Review Comment:
Because this overrides the existing default, we should add a comment to that effect if someone passes the project flag in
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on pull request #23011: (BQ Python) Pass project field from options or parameter when writing with dynamic destinations
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on PR #23011:
URL: https://github.com/apache/beam/pull/23011#issuecomment-1244009880
Run Python 3.8 PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on pull request #23011: (BQ Python) Pass project field from options or parameter when writing with dynamic destinations
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on PR #23011:
URL: https://github.com/apache/beam/pull/23011#issuecomment-1244550640
Run Python 3.8 PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on a diff in pull request #23011: (BQ Python) Pass project field from options or parameter when writing with dynamic destinations
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on code in PR #23011:
URL: https://github.com/apache/beam/pull/23011#discussion_r965150134
##########
sdks/python/apache_beam/io/gcp/bigquery.py:
##########
@@ -2278,8 +2278,8 @@ def expand(self, pcoll):
if (isinstance(self.table_reference, TableReference) and
self.table_reference.projectId is None):
- self.table_reference.projectId = pcoll.pipeline.options.view_as(
- GoogleCloudOptions).project
+ self.table_reference.projectId = self._project or \
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on pull request #23011: (BQ Python) Pass project field from options or parameter when writing with dynamic destinations
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on PR #23011:
URL: https://github.com/apache/beam/pull/23011#issuecomment-1238373701
Run Python 3.8 PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on pull request #23011: (BQ Python) Pass project field from options or parameter when writing with dynamic destinations
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on PR #23011:
URL: https://github.com/apache/beam/pull/23011#issuecomment-1239666480
Run Python 3.8 PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on pull request #23011: (BQ Python) Pass project field from options or parameter when writing with dynamic destinations
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on PR #23011:
URL: https://github.com/apache/beam/pull/23011#issuecomment-1243711717
Run Python 3.8 PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey merged pull request #23011: (BQ Python) Pass project field from options or parameter when writing with dynamic destinations
Posted by GitBox <gi...@apache.org>.
johnjcasey merged PR #23011:
URL: https://github.com/apache/beam/pull/23011
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on pull request #23011: (BQ Python) Pass project field from options or parameter when writing with dynamic destinations
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on PR #23011:
URL: https://github.com/apache/beam/pull/23011#issuecomment-1239659583
Run Python 3.8 PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #23011: (BQ Python) Pass project field from options or parameter when writing with dynamic destinations
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23011:
URL: https://github.com/apache/beam/pull/23011#issuecomment-1239828419
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ahmedabu98 commented on pull request #23011: (BQ Python) Pass project field from options or parameter when writing with dynamic destinations
Posted by GitBox <gi...@apache.org>.
ahmedabu98 commented on PR #23011:
URL: https://github.com/apache/beam/pull/23011#issuecomment-1235621494
Run Python 3.8 PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #23011: (BQ Python) Pass project field from options or parameter when writing with dynamic destinations
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23011:
URL: https://github.com/apache/beam/pull/23011#issuecomment-1235711457
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @yeandy for label python.
R: @Abacn for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org