[C++] S3 timeout on linux

[Surya Kiran Gullapalli <su...@gmail.com>]

Hello,
I'm facing this weird timeout error when reading a parquet file
(arrow-10.0.0 and arrow-12.0.0)
The parquet file is not huge (hardly 1MB).

Initialized s3 with arrow::fs::EnsureS3Initialized() and set all the access
keys correctly.

When trying to read the file from s3, I'm getting curlCode 28 error. The
same piece of code is working fine on windows.

Also, this time out is happening on an aws ec2 instance. I did not set any
timeouts on aws api.

We're using s3 for other data as well, so, there's another Aws::Init call
happening in our code (apart from arrow::fs::EnsureS3Initialized). I do not
know if this is causing any problem.

Any help would be greatly appreciated.

Surya

Re: [C++] S3 timeout on linux

[Surya Kiran Gullapalli <su...@gmail.com>]

I've set explicit timeouts for client and it is working fine now.

    auto s3Options = arrow::fs::S3Options::FromAccessKey(accessKey,
secretKey);

    s3Options.connect_timeout = 30.0;
    s3Options.request_timeout = 60.0;
    s3Options.region          = "us-east-1"s;

On Tue, Jun 13, 2023 at 2:53 AM Bryce Mecum <br...@gmail.com> wrote:

> Do you still get the timeout if you build a standalone program that
> just uses Arrow C++? It might give us some more information if you run
> your code with S3's log level turned up. So instead of using
> EnsureS3Initialized, initialize S3 manually like,
>
> S3GlobalOptions options;
> options.log_level = S3LogLevel::Debug;
> InitializeS3(options);
>

Re: [C++] S3 timeout on linux

[Bryce Mecum <br...@gmail.com>]

Do you still get the timeout if you build a standalone program that
just uses Arrow C++? It might give us some more information if you run
your code with S3's log level turned up. So instead of using
EnsureS3Initialized, initialize S3 manually like,

S3GlobalOptions options;
options.log_level = S3LogLevel::Debug;
InitializeS3(options);

Re: [C++] S3 timeout on linux

[Surya Kiran Gullapalli <su...@gmail.com>]

If I fetch the file to disk (using aws cpp sdk) and read the parquet file,
it works fine.

Surya

On Mon, Jun 12, 2023 at 8:27 PM Surya Kiran Gullapalli <
suryakiran.gullapalli@gmail.com> wrote:

> Hello,
> I'm facing this weird timeout error when reading a parquet file
> (arrow-10.0.0 and arrow-12.0.0)
> The parquet file is not huge (hardly 1MB).
>
> Initialized s3 with arrow::fs::EnsureS3Initialized() and set all the
> access keys correctly.
>
> When trying to read the file from s3, I'm getting curlCode 28 error. The
> same piece of code is working fine on windows.
>
> Also, this time out is happening on an aws ec2 instance. I did not set any
> timeouts on aws api.
>
> We're using s3 for other data as well, so, there's another Aws::Init call
> happening in our code (apart from arrow::fs::EnsureS3Initialized). I do not
> know if this is causing any problem.
>
> Any help would be greatly appreciated.
>
> Surya
>