You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "johnjcasey (via GitHub)" <gi...@apache.org> on 2023/05/03 15:28:24 UTC

[GitHub] [beam] johnjcasey opened a new issue, #26520: [Bug]: BQIO fails to keep track of offset correctly, causing write exceptions when using Storage Write API in batch mode

johnjcasey opened a new issue, #26520:
URL: https://github.com/apache/beam/issues/26520

   ### What happened?
   
   BQIO was lazily creating streams, which means that all our appends to a given stream would be appended at index 0.
   This meant that for any second append to a stream, we would get an exception from BQ, and be forced to retry our work. This results in bad performance by itself, but combined with __ resulted in data loss for beam versions 2.44 - 2.47 inclusive.
   
   ### Issue Priority
   
   Priority: 1 (data loss / total loss of function)
   
   ### Issue Components
   
   - [ ] Component: Python SDK
   - [X] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [X] Component: IO connector
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Samza Runner
   - [ ] Component: Twister2 Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] johnjcasey closed issue #26520: [Bug]: BQIO fails to keep track of offset correctly, causing write exceptions when using Storage Write API in batch mode

Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey closed issue #26520: [Bug]: BQIO fails to keep track of offset correctly, causing write exceptions when using Storage Write API in batch mode
URL: https://github.com/apache/beam/issues/26520


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] johnjcasey commented on issue #26520: [Bug]: BQIO fails to keep track of offset correctly, causing write exceptions when using Storage Write API in batch mode

Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on issue #26520:
URL: https://github.com/apache/beam/issues/26520#issuecomment-1533272354

   Fixed by https://github.com/apache/beam/pull/26503


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org