You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@arrow.apache.org by Luke <vi...@gmail.com> on 2022/11/09 15:37:01 UTC

pyarrow 10

in the docs for pyarrow.fs.S3FileSystem I see 2 options for a
retry_strategy:  AwsStandardS3RetryStrategy and AwsDefaultS3RetryStrategy.

What is the difference between these two?

thanks,
Luke

Re: pyarrow 10

Posted by Luke <vi...@gmail.com>.

Thanks, that helps a lot.

-Luke

On Wed, Nov 9, 2022 at 11:04 AM Duncan MacQuarrie via user <
user@arrow.apache.org> wrote:

> Hi Luke,
>
> The short answer is this:
>
> "Standard" retry strategy uses exponential backoff and catches a wide
> range of rate limiting errors.
>
> The confusingly named "Default" retry strategy is an older approach which
> catches a smaller set of errors, and I wouldn't generally recommend using
> it, but it's there because people might be reliant upon its legacy
> behaviour for whatever reason.
>
> The "standard" and "default" naming comes from AWS.
>
> Some more info at [1] and if you want to go into down C++ AWS SDK rabbit
> hole on this topic, a good starting point is [2].
>
> Hope this helps.
> Duncan
>
> [1]
> https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html
> [2]
> https://sdk.amazonaws.com/cpp/api/0.14.3/class_aws_1_1_client_1_1_default_retry_strategy.html
>
>
>
> On Wed, Nov 9, 2022 at 3:37 PM Luke <vi...@gmail.com> wrote:
>
>> in the docs for pyarrow.fs.S3FileSystem I see 2 options for a
>> retry_strategy:  AwsStandardS3RetryStrategy and
>> AwsDefaultS3RetryStrategy.
>>
>> What is the difference between these two?
>>
>> thanks,
>> Luke
>>
>>

Re: pyarrow 10

Posted by Duncan MacQuarrie via user <us...@arrow.apache.org>.

Hi Luke,

The short answer is this:

"Standard" retry strategy uses exponential backoff and catches a wide range
of rate limiting errors.

The confusingly named "Default" retry strategy is an older approach which
catches a smaller set of errors, and I wouldn't generally recommend using
it, but it's there because people might be reliant upon its legacy
behaviour for whatever reason.

The "standard" and "default" naming comes from AWS.

Some more info at [1] and if you want to go into down C++ AWS SDK rabbit
hole on this topic, a good starting point is [2].

Hope this helps.
Duncan

[1]
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html
[2]
https://sdk.amazonaws.com/cpp/api/0.14.3/class_aws_1_1_client_1_1_default_retry_strategy.html

On Wed, Nov 9, 2022 at 3:37 PM Luke <vi...@gmail.com> wrote:

> in the docs for pyarrow.fs.S3FileSystem I see 2 options for a
> retry_strategy:  AwsStandardS3RetryStrategy and
> AwsDefaultS3RetryStrategy.
>
> What is the difference between these two?
>
> thanks,
> Luke
>
>