You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "iht (via GitHub)" <gi...@apache.org> on 2024/05/27 15:10:30 UTC

[I] [Bug]: SpannerIO creates "under the hood" new Pipeline optoins [beam]

iht opened a new issue, #31411:
URL: https://github.com/apache/beam/issues/31411

   ### What happened?
   
   When using `SpannerIO` with `readChangeStream()`, a new pipeline option "pops up" in your jobs.
   
   This option is [actually created by SpannerIO](https://github.com/apache/beam/blob/ca00272434131d607bd710aff79310906e99c524/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java#L1728-L1732). This can cause all types of issues: clash with user defined options, clash if you read from two change streams using different metadata tables, etc.
   
   `SpannerIO` should not alter the pipeline options of the job.
   
   ### Issue Priority
   
   Priority: 2 (default / most bugs should be filed as P2)
   
   ### Issue Components
   
   - [ ] Component: Python SDK
   - [X] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [ ] Component: IO connector
   - [ ] Component: Beam YAML
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Samza Runner
   - [ ] Component: Twister2 Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug]: SpannerIO creates "under the hood" new Pipeline optoins [beam]

Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb closed issue #31411: [Bug]: SpannerIO creates "under the hood" new Pipeline optoins
URL: https://github.com/apache/beam/issues/31411


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug]: SpannerIO creates "under the hood" new Pipeline optoins [beam]

Posted by "iht (via GitHub)" <gi...@apache.org>.
iht commented on issue #31411:
URL: https://github.com/apache/beam/issues/31411#issuecomment-2133795759

   I removed that code, run again a test job, and everything is working normally (the metadata table is created, populated, etc). I will submit a PR removing that code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org