You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 22:50:41 UTC

[GitHub] [beam] damccorm opened a new issue, #21366: Migrate SnsIO.write to async implementation

damccorm opened a new issue, #21366:
URL: https://github.com/apache/beam/issues/21366

   In  BEAM-13203 `SnsIO.writeAsync` was deprecated to address the risk of data loss in that transform. Unfortunately, it has considerable issues in design and implementation with respect to the beam programming model as well as general async programming:
    * Output for a bundle may be emitted even after the bundle is finished (lack of join / waiting for pending futures), additionally output is emitted from multiple threads risking concurrent modification issues on underlying data structures.
    * Exceptions disappear into the void (respective tests are implemented incorrectly as they don't throw asynchronously and could make one think this works).
    * Parallelism is unlimited causing the writer to instantly accumulate everything into memory.
   
   With BEAM-13563 it will be possible to change `SnsIO.write` into a more performant async implementation without any breaking changes to the public user facing API making `SnsIO.writeAsync` obsolete.
   
   Imported from Jira [BEAM-13824](https://issues.apache.org/jira/browse/BEAM-13824). Original Jira may contain additional context.
   Reported by: mosche.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] mosche commented on issue #21366: Migrate SnsIO.write to async implementation

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on issue #21366:
URL: https://github.com/apache/beam/issues/21366#issuecomment-1535857475

   A new async implementation should also use the batch API, see related
   https://github.com/apache/beam/issues/21429


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org