You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "tarun-google (via GitHub)" <gi...@apache.org> on 2025/09/12 18:54:27 UTC
[PR] Managed Bigquery Java Read&Write Transforms [beam]
tarun-google opened a new pull request, #36139:
URL: https://github.com/apache/beam/pull/36139
Migrating Bigquery java transforms to managedIO. There should not be any change for Read and Streaming write. But for Batch write we are migrating to FILE_LOADS method, which stages files in temp path before writing. This is not a breaking change for the existing users.
Batch write Test:
<img width="397" height="312" alt="Screenshot 2025-09-12 at 11 19 36 AM" src="https://github.com/user-attachments/assets/27a61d5a-16d1-44f7-959d-3345aee0cf3a" />
Read Test:
<img width="397" height="312" alt="Screenshot 2025-09-12 at 11 22 02 AM" src="https://github.com/user-attachments/assets/ee6135f8-d6bf-43bc-acf1-084a5345f1f9" />
Streaming write Test:
<img width="397" height="356" alt="Screenshot 2025-09-12 at 11 53 12 AM" src="https://github.com/user-attachments/assets/3aff7ced-79c7-4e10-937e-ccbd3880b683" />
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3314935465
Reminder, please take a look at this pr: @liferoad
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]
Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3315089662
Blocking this on #33074 issue and fix #36209
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3286929291
Did your end-to-end test runs confirm that the managed transforms upgrade is occurring ?
You should notice "Managed Transform(s)" logs mentioned here: https://cloud.google.com/dataflow/docs/guides/managed-io
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]
Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3325575194
Re-tested with fix and updated main description with results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3324639160
waiting on author
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #36139:
URL: https://github.com/apache/beam/pull/36139#discussion_r2345470949
##########
sdks/python/apache_beam/yaml/standard_io.yaml:
##########
@@ -45,9 +45,15 @@
type: beamJar
transforms:
'ReadFromBigQuery': 'beam:schematransform:org.apache.beam:bigquery_storage_read:v1'
- 'WriteToBigQuery': 'beam:schematransform:org.apache.beam:bigquery_storage_write:v2'
+ 'WriteToBigQuery': 'beam:schematransform:org.apache.beam:bigquery_write:v1'
config:
gradle_target: 'sdks:java:extensions:sql:expansion-service:shadowJar'
+ managed_replacement:
+ # Following transforms may be replaced with equivalent managed transforms,
+ # if the pipelines 'updateCompatibilityBeamVersion' match the provided
+ # version.
+ 'ReadFromBigQuery': '2.65.0'
Review Comment:
This should be the version the change will go to. So Beam 2.69.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]
Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3325813981
Run Python_Examples PreCommit 3.9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3286818175
Assigning reviewers:
R: @liferoad for label python.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj merged PR #36139:
URL: https://github.com/apache/beam/pull/36139
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]
Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3286747945
Run Python_Examples PreCommit 3.11
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3325609545
LGTM. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3324640313
noting that the two blockers are closed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org