You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "scwhittle (via GitHub)" <gi...@apache.org> on 2023/04/20 12:39:11 UTC

[GitHub] [beam] scwhittle opened a new issue, #26369: [Bug]: BagUserState should chunk appends

scwhittle opened a new issue, #26369:
URL: https://github.com/apache/beam/issues/26369

   ### What happened?
   
   Currently and appends made to a user state bag end up in a single append over the FnApi.
   From BagUserState.java
   
       if (!newValues.isEmpty()) {
         ByteStringOutputStream out = new ByteStringOutputStream();
         for (T newValue : newValues) {
           // TODO: Replace with chunking output stream
           valueCoder.encode(newValue, out);
         }
         beamFnStateClient.handle(
             request
                 .toBuilder()
                 .setAppend(StateAppendRequest.newBuilder().setData(out.toByteString())));
       }
   
   This results in spiky memory and the possibility to exceed byte limits of storage.  Bags are designed to support multiple elements so it seems that this could instead create multiple appends.
   
   ### Issue Priority
   
   Priority: 2 (default / most bugs should be filed as P2)
   
   ### Issue Components
   
   - [ ] Component: Python SDK
   - [X] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [ ] Component: IO connector
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Samza Runner
   - [ ] Component: Twister2 Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] robertwb closed issue #26369: [Bug]: BagUserState should chunk appends

Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb closed issue #26369: [Bug]: BagUserState should chunk appends 
URL: https://github.com/apache/beam/issues/26369


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] scwhittle commented on issue #26369: [Bug]: BagUserState should chunk appends

Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on issue #26369:
URL: https://github.com/apache/beam/issues/26369#issuecomment-1516257650

   .take-issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org