You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "tarun-google (via GitHub)" <gi...@apache.org> on 2025/09/12 18:54:27 UTC

[PR] Managed Bigquery Java Read&Write Transforms [beam]

tarun-google opened a new pull request, #36139:
URL: https://github.com/apache/beam/pull/36139

   Migrating Bigquery java transforms to managedIO. There should not be any change for Read and Streaming write. But for Batch write we are migrating to FILE_LOADS method, which stages files in temp path before writing. This is not a breaking change for the existing users. 
   
   Batch write Test:
    
   <img width="397" height="312" alt="Screenshot 2025-09-12 at 11 19 36 AM" src="https://github.com/user-attachments/assets/27a61d5a-16d1-44f7-959d-3345aee0cf3a" />
   
   Read Test:
   
   <img width="397" height="312" alt="Screenshot 2025-09-12 at 11 22 02 AM" src="https://github.com/user-attachments/assets/ee6135f8-d6bf-43bc-acf1-084a5345f1f9" />
   
   Streaming write Test:
   <img width="397" height="356" alt="Screenshot 2025-09-12 at 11 53 12 AM" src="https://github.com/user-attachments/assets/3aff7ced-79c7-4e10-937e-ccbd3880b683" />
   
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/actions/workflows/build_wheels.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/actions/workflows/python_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/actions/workflows/java_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/actions/workflows/go_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3314935465

   Reminder, please take a look at this pr: @liferoad 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]

Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3315089662

   Blocking this on #33074 issue and fix #36209 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]

Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3286929291

   Did your end-to-end test runs confirm that the managed transforms upgrade is occurring ?
   
   You should notice "Managed Transform(s)" logs mentioned here: https://cloud.google.com/dataflow/docs/guides/managed-io


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]

Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3325575194

   Re-tested with fix and updated main description with results 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3324639160

   waiting on author


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]

Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #36139:
URL: https://github.com/apache/beam/pull/36139#discussion_r2345470949


##########
sdks/python/apache_beam/yaml/standard_io.yaml:
##########
@@ -45,9 +45,15 @@
       type: beamJar
       transforms:
         'ReadFromBigQuery': 'beam:schematransform:org.apache.beam:bigquery_storage_read:v1'
-        'WriteToBigQuery': 'beam:schematransform:org.apache.beam:bigquery_storage_write:v2'
+        'WriteToBigQuery': 'beam:schematransform:org.apache.beam:bigquery_write:v1'
       config:
         gradle_target: 'sdks:java:extensions:sql:expansion-service:shadowJar'
+        managed_replacement:
+          # Following transforms may be replaced with equivalent managed transforms,
+          # if the pipelines 'updateCompatibilityBeamVersion' match the provided
+          # version.
+          'ReadFromBigQuery': '2.65.0'

Review Comment:
   This should be the version the change will go to. So Beam 2.69.0



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]

Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3325813981

   Run Python_Examples PreCommit 3.9


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3286818175

   Assigning reviewers:
   
   R: @liferoad for label python.
   
   
   Note: If you would like to opt out of this review, comment `assign to next reviewer`.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]

Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj merged PR #36139:
URL: https://github.com/apache/beam/pull/36139


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]

Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3286747945

   Run Python_Examples PreCommit 3.11


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]

Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3325609545

   LGTM. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Managed Bigquery Java Read&Write Transforms [beam]

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #36139:
URL: https://github.com/apache/beam/pull/36139#issuecomment-3324640313

   noting that the two blockers are closed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org