You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "mosche (via GitHub)" <gi...@apache.org> on 2023/04/03 09:02:58 UTC

[GitHub] [beam] mosche commented on issue #25991: [Bug]: BEAM-12329 causes performance issues

mosche commented on issue #25991:
URL: https://github.com/apache/beam/issues/25991#issuecomment-1493945808

   @vatanrathi Thanks for raising this, your pointer to aws/aws-sdk-java-v2/issues/2117 is very helpful 👍  Trying to drain the input stream in all cases is certainly dangerous considering that files might be very large and the byte range requested was from "position" to the very end.
   
   As a quick workaround using `abort` (if `position != contentLength`) seems totally eligible, particularly when dealing with large files. Though, drawback is respective connections cannot be reused.
   
   > If it is not desired to read remaining data from the stream, you can explicitly abort the connection via abort(). Note that this will close the underlying connection and require establishing an HTTP connection which may outweigh the cost of reading the additional data.
   
   I'll have a closer look the next days and will think about alternatives. It will probably make sense to read the data in chunks to minimize the overhead when closing but also allow reusing connections.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org