You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@solr.apache.org by "Houston Putman (Jira)" <ji...@apache.org> on 2025/04/30 23:23:00 UTC

[jira] [Commented] (SOLR-17750) S3 file download does not gracefully handle connection errors

    [ https://issues.apache.org/jira/browse/SOLR-17750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17948604#comment-17948604 ] 

Houston Putman commented on SOLR-17750:
---------------------------------------

I've tried a custom "ResumableInputStream" implementation that submits a new GetObject request with the already-read-bytes offset, and it outperforms the [S3 TransferManager|https://github.com/aws/aws-sdk-java-v2/blob/master/services-custom/s3-transfer-manager/README.md] significantly. The TransferManager is about 5x slower than just doing it ourselves. So that's probably the best way forward, since the TransferManager also requires that we use the AsyncClient for AWS, which we don't currently.

> S3 file download does not gracefully handle connection errors
> -------------------------------------------------------------
>
>                 Key: SOLR-17750
>                 URL: https://issues.apache.org/jira/browse/SOLR-17750
>             Project: Solr
>          Issue Type: Bug
>          Components: contrib - S3 Repository
>            Reporter: Houston Putman
>            Priority: Major
>
> When downloading large object from S3, it's not uncommon to get a ConnectionClosedException or something else similar. We should be able to handle this gracefully and restart the download from the last place we have recorded.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org