[jira] [Updated] (BEAM-13585) Python SDK S3 reader 2x - 10x performance gain

["Kenneth Knowles (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-13585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kenneth Knowles updated BEAM-13585:
-----------------------------------
    Summary: Python SDK S3 reader 2x - 10x performance gain  (was: Python SDK S3 reader vastly inefficient)

> Python SDK S3 reader 2x - 10x performance gain
> ----------------------------------------------
>
>                 Key: BEAM-13585
>                 URL: https://issues.apache.org/jira/browse/BEAM-13585
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-py-core
>    Affects Versions: 2.34.0, 2.35.0
>            Reporter: Janek Bevendorff
>            Assignee: Janek Bevendorff
>            Priority: P2
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> This is an "after-the-fact" Jira issue for my [GitHub PR|https://github.com/apache/beam/pull/15931#issuecomment-999945083] to make S3 streaming in the Python SDK vastly more efficient.
> The issue with the old implementation was that a new connection was opened for each range request, which is very inefficient for both the client and the server, adding a lot of unnecessary latency. The new implementation tries to reused an existing connection and continues reading from the same HTTP stream if possible.
> Speed gain: 1.7-12x in benchmarks, more like 10x in real-word applications.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)