You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Kenneth Knowles (Jira)" <ji...@apache.org> on 2022/01/11 13:34:00 UTC
[jira] [Assigned] (BEAM-13585) Python SDK S3 reader vastly inefficient
[ https://issues.apache.org/jira/browse/BEAM-13585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kenneth Knowles reassigned BEAM-13585:
--------------------------------------
Assignee: Janek Bevendorff
> Python SDK S3 reader vastly inefficient
> ---------------------------------------
>
> Key: BEAM-13585
> URL: https://issues.apache.org/jira/browse/BEAM-13585
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
> Affects Versions: 2.34.0
> Reporter: Janek Bevendorff
> Assignee: Janek Bevendorff
> Priority: P2
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is an "after-the-fact" Jira issue for my [GitHub PR|https://github.com/apache/beam/pull/15931#issuecomment-999945083] to make S3 streaming in the Python SDK vastly more efficient.
> The issue with the old implementation was that a new connection was opened for each range request, which is very inefficient for both the client and the server, adding a lot of unnecessary latency. The new implementation tries to reused an existing connection and continues reading from the same HTTP stream if possible.
> Speed gain: 1.7-12x in benchmarks, more like 10x in real-word applications.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)