You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Janek Bevendorff (Jira)" <ji...@apache.org> on 2021/12/31 08:59:00 UTC

[jira] [Created] (BEAM-13585) Python SDK S3 reader vastly inefficient

Janek Bevendorff created BEAM-13585:
---------------------------------------

             Summary: Python SDK S3 reader vastly inefficient
                 Key: BEAM-13585
                 URL: https://issues.apache.org/jira/browse/BEAM-13585
             Project: Beam
          Issue Type: Improvement
          Components: sdk-py-core
    Affects Versions: 2.34.0
            Reporter: Janek Bevendorff


This is an "after-the-fact" Jira issue for my [GitHub PR|https://github.com/apache/beam/pull/15931#issuecomment-999945083] to make S3 streaming in the Python SDK vastly more efficient.

The issue with the old implementation is that a new connection is opened for each range request, which is very inefficient for both the client and the server, adding a lot of unnecessary latency. The new implementation tries to reused an existing connection and continues reading from the same HTTP stream if possible.

Speed gain: 1.7-12x in benchmarks, more like 10x in real-word applications.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)