You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by Luke <vi...@gmail.com> on 2021/03/27 14:44:53 UTC
timeouts in S3 reads in pyarrow
I have a local S3 compatible object store (using ceph) and am trying to use
the pyarrow fs interface. This seems to work well except on larger objects
I am getting unhandled exceptions. Is there a way to currently tune the
timeouts or retries? Here is the kind of code and error I am seeing:
from pyarrow import fs
s3 =
fs.S3FileSystem(access_key=my_ak,secret_key=my_sk,endpoint_override=my_endpoint,scheme='http')
raw = s3.open_input_stream('test_bucket/example_key').readall()
File "pyarrow/_fs.pyx", line 621, in
pyarrow._fs.FileSystem.open_input_stream
File "pyarrow/error.pxi", line 122, in
pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
OSError: When reading information for key 'example_key' in bucket
'test_bucket': AWS Error [code 99]: curlCode: 28, Timeout was reached
--
install details:
python: python 3.8.6
OS: linux, redhat 7.7
pyarrow version: 3.0.0
thanks for the help,
Luke
Re: timeouts in S3 reads in pyarrow
Posted by Weston Pace <we...@gmail.com>.
The AWS CPP client has 3 configurable timeouts[1]. The first,
httpRequestTimeoutMs, applies to the entire request but it defaults to
0 (disabled) and since Arrow doesn't set it then it shouldn't be
interfering here. The second, requestTimeoutMs, defaults to 3 seconds
and also applies to the entire request but it is tied to the
lowSpeedLimit. So by default it will timeout if there are less than 3
bytes transmitted in any 3 second window. The third,
connectTimeoutMs, defaults to 1 second, but only applies for the time
it takes to connect to the server.
These are in Aws::Client::ClientConfiguration which we do not expose
today but we probably could.
Also, as Wes said, implementing some kind of retry mechanism would
probably also be a good idea.
[1] https://github.com/aws/aws-sdk-cpp/blob/master/aws-cpp-sdk-core/include/aws/core/client/ClientConfiguration.h
On Mon, Mar 29, 2021 at 5:36 AM Wes McKinney <we...@gmail.com> wrote:
>
> I think boto3 automatically retries, see
>
> https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html
>
> We'll likely need to do something similar and make it configurable
>
> On Mon, Mar 29, 2021 at 10:00 AM Antoine Pitrou <an...@python.org> wrote:
> >
> > On Mon, 29 Mar 2021 10:52:04 -0400
> > Luke <vi...@gmail.com> wrote:
> > > No firewall in this setup. And no errors when the keys are small that I
> > > have seen. Also, using boto3 or requests there are no issues that I can
> > > tell (although it may be suppressing timeouts and retries).
> >
> > How large is the file that gives an error, and how fast is your
> > connection to the S3 server?
> > Also, after how long do you get the timeout error?
> >
> > Regards
> >
> > Antoine.
> >
> >
> > >
> > > So you are saying the 'AWS Error [code 99]: curlCode: 28, Timeout was
> > > reached' is coming from the server? Interesting, we haven't seen that
> > > before but our access before this was with requests or s3fs which uses
> > > boto3 under the hood and must handle this if so.
> > >
> > > thanks, will explore more,
> > > Luke
> > >
> > > On Mon, Mar 29, 2021 at 9:54 AM Antoine Pitrou <an...@python.org> wrote:
> > >
> > > >
> > > > Hi Luke,
> > > >
> > > > Given the error message, my intuition is that the timeout is on the
> > > > server side. Arrow does not try to set any timeouts on S3 connections.
> > > >
> > > > Note that this message ("When reading information") happens *before*
> > > > reading the file data, simply when trying to read the file length. So
> > > > perhaps something is weird in your network configuration (is a firewall
> > > > blocking packets?).
> > > >
> > > > Regards
> > > >
> > > > Antoine.
> > > >
> > > >
> > > >
> > > > On Sat, 27 Mar 2021 10:44:53 -0400
> > > > Luke <vi...@gmail.com> wrote:
> > > > > I have a local S3 compatible object store (using ceph) and am trying to
> > > > use
> > > > > the pyarrow fs interface. This seems to work well except on larger
> > > > objects
> > > > > I am getting unhandled exceptions. Is there a way to currently tune the
> > > > > timeouts or retries? Here is the kind of code and error I am seeing:
> > > > >
> > > > > from pyarrow import fs
> > > > >
> > > > >
> > > > >
> > > > > s3 =
> > > > >
> > > > fs.S3FileSystem(access_key=my_ak,secret_key=my_sk,endpoint_override=my_endpoint,scheme='http')
> > > > >
> > > > > raw = s3.open_input_stream('test_bucket/example_key').readall()
> > > > >
> > > > >
> > > > >
> > > > > File "pyarrow/_fs.pyx", line 621, in
> > > > > pyarrow._fs.FileSystem.open_input_stream
> > > > >
> > > > > File "pyarrow/error.pxi", line 122, in
> > > > > pyarrow.lib.pyarrow_internal_check_status
> > > > >
> > > > > File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
> > > > >
> > > > > OSError: When reading information for key 'example_key' in bucket
> > > > > 'test_bucket': AWS Error [code 99]: curlCode: 28, Timeout was reached
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > install details:
> > > > >
> > > > > python: python 3.8.6
> > > > >
> > > > > OS: linux, redhat 7.7
> > > > >
> > > > > pyarrow version: 3.0.0
> > > > >
> > > > >
> > > > > thanks for the help,
> > > > >
> > > > > Luke
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
Re: timeouts in S3 reads in pyarrow
Posted by Wes McKinney <we...@gmail.com>.
I think boto3 automatically retries, see
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html
We'll likely need to do something similar and make it configurable
On Mon, Mar 29, 2021 at 10:00 AM Antoine Pitrou <an...@python.org> wrote:
>
> On Mon, 29 Mar 2021 10:52:04 -0400
> Luke <vi...@gmail.com> wrote:
> > No firewall in this setup. And no errors when the keys are small that I
> > have seen. Also, using boto3 or requests there are no issues that I can
> > tell (although it may be suppressing timeouts and retries).
>
> How large is the file that gives an error, and how fast is your
> connection to the S3 server?
> Also, after how long do you get the timeout error?
>
> Regards
>
> Antoine.
>
>
> >
> > So you are saying the 'AWS Error [code 99]: curlCode: 28, Timeout was
> > reached' is coming from the server? Interesting, we haven't seen that
> > before but our access before this was with requests or s3fs which uses
> > boto3 under the hood and must handle this if so.
> >
> > thanks, will explore more,
> > Luke
> >
> > On Mon, Mar 29, 2021 at 9:54 AM Antoine Pitrou <an...@python.org> wrote:
> >
> > >
> > > Hi Luke,
> > >
> > > Given the error message, my intuition is that the timeout is on the
> > > server side. Arrow does not try to set any timeouts on S3 connections.
> > >
> > > Note that this message ("When reading information") happens *before*
> > > reading the file data, simply when trying to read the file length. So
> > > perhaps something is weird in your network configuration (is a firewall
> > > blocking packets?).
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > >
> > > On Sat, 27 Mar 2021 10:44:53 -0400
> > > Luke <vi...@gmail.com> wrote:
> > > > I have a local S3 compatible object store (using ceph) and am trying to
> > > use
> > > > the pyarrow fs interface. This seems to work well except on larger
> > > objects
> > > > I am getting unhandled exceptions. Is there a way to currently tune the
> > > > timeouts or retries? Here is the kind of code and error I am seeing:
> > > >
> > > > from pyarrow import fs
> > > >
> > > >
> > > >
> > > > s3 =
> > > >
> > > fs.S3FileSystem(access_key=my_ak,secret_key=my_sk,endpoint_override=my_endpoint,scheme='http')
> > > >
> > > > raw = s3.open_input_stream('test_bucket/example_key').readall()
> > > >
> > > >
> > > >
> > > > File "pyarrow/_fs.pyx", line 621, in
> > > > pyarrow._fs.FileSystem.open_input_stream
> > > >
> > > > File "pyarrow/error.pxi", line 122, in
> > > > pyarrow.lib.pyarrow_internal_check_status
> > > >
> > > > File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
> > > >
> > > > OSError: When reading information for key 'example_key' in bucket
> > > > 'test_bucket': AWS Error [code 99]: curlCode: 28, Timeout was reached
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > install details:
> > > >
> > > > python: python 3.8.6
> > > >
> > > > OS: linux, redhat 7.7
> > > >
> > > > pyarrow version: 3.0.0
> > > >
> > > >
> > > > thanks for the help,
> > > >
> > > > Luke
> > > >
> > >
> > >
> > >
> > >
> >
>
>
>
Re: timeouts in S3 reads in pyarrow
Posted by Luke <vi...@gmail.com>.
I don't have great numbers on when it is failing and when not but I believe
I have only seen this for objects larger than 100MB.
Our local S3 compatible object store (via ceph) is in the same datacenter
where I am running arrow code and the least network connection involved
should be 25Gbps.
The rgw gateways (s3 endpoints) can get pretty loaded and I could see a
second or 2 wait happening from time to time in queue waiting to get
serviced.
Hopefully the api is not doing a list on the bucket in this operation, that
can be really slow.
I think being able to configure timeouts and retries would be a
great addition.
thanks again,
Luke
On Mon, Mar 29, 2021 at 11:00 AM Antoine Pitrou <an...@python.org> wrote:
> On Mon, 29 Mar 2021 10:52:04 -0400
> Luke <vi...@gmail.com> wrote:
> > No firewall in this setup. And no errors when the keys are small that I
> > have seen. Also, using boto3 or requests there are no issues that I can
> > tell (although it may be suppressing timeouts and retries).
>
> How large is the file that gives an error, and how fast is your
> connection to the S3 server?
> Also, after how long do you get the timeout error?
>
> Regards
>
> Antoine.
>
>
> >
> > So you are saying the 'AWS Error [code 99]: curlCode: 28, Timeout was
> > reached' is coming from the server? Interesting, we haven't seen that
> > before but our access before this was with requests or s3fs which uses
> > boto3 under the hood and must handle this if so.
> >
> > thanks, will explore more,
> > Luke
> >
> > On Mon, Mar 29, 2021 at 9:54 AM Antoine Pitrou <an...@python.org>
> wrote:
> >
> > >
> > > Hi Luke,
> > >
> > > Given the error message, my intuition is that the timeout is on the
> > > server side. Arrow does not try to set any timeouts on S3 connections.
> > >
> > > Note that this message ("When reading information") happens *before*
> > > reading the file data, simply when trying to read the file length. So
> > > perhaps something is weird in your network configuration (is a firewall
> > > blocking packets?).
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > >
> > > On Sat, 27 Mar 2021 10:44:53 -0400
> > > Luke <vi...@gmail.com> wrote:
> > > > I have a local S3 compatible object store (using ceph) and am trying
> to
> > > use
> > > > the pyarrow fs interface. This seems to work well except on larger
> > > objects
> > > > I am getting unhandled exceptions. Is there a way to currently tune
> the
> > > > timeouts or retries? Here is the kind of code and error I am seeing:
> > > >
> > > > from pyarrow import fs
> > > >
> > > >
> > > >
> > > > s3 =
> > > >
> > >
> fs.S3FileSystem(access_key=my_ak,secret_key=my_sk,endpoint_override=my_endpoint,scheme='http')
>
> > > >
> > > > raw = s3.open_input_stream('test_bucket/example_key').readall()
> > > >
> > > >
> > > >
> > > > File "pyarrow/_fs.pyx", line 621, in
> > > > pyarrow._fs.FileSystem.open_input_stream
> > > >
> > > > File "pyarrow/error.pxi", line 122, in
> > > > pyarrow.lib.pyarrow_internal_check_status
> > > >
> > > > File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
> > > >
> > > > OSError: When reading information for key 'example_key' in bucket
> > > > 'test_bucket': AWS Error [code 99]: curlCode: 28, Timeout was reached
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > install details:
> > > >
> > > > python: python 3.8.6
> > > >
> > > > OS: linux, redhat 7.7
> > > >
> > > > pyarrow version: 3.0.0
> > > >
> > > >
> > > > thanks for the help,
> > > >
> > > > Luke
> > > >
> > >
> > >
> > >
> > >
> >
>
>
>
>
Re: timeouts in S3 reads in pyarrow
Posted by Antoine Pitrou <an...@python.org>.
On Mon, 29 Mar 2021 10:52:04 -0400
Luke <vi...@gmail.com> wrote:
> No firewall in this setup. And no errors when the keys are small that I
> have seen. Also, using boto3 or requests there are no issues that I can
> tell (although it may be suppressing timeouts and retries).
How large is the file that gives an error, and how fast is your
connection to the S3 server?
Also, after how long do you get the timeout error?
Regards
Antoine.
>
> So you are saying the 'AWS Error [code 99]: curlCode: 28, Timeout was
> reached' is coming from the server? Interesting, we haven't seen that
> before but our access before this was with requests or s3fs which uses
> boto3 under the hood and must handle this if so.
>
> thanks, will explore more,
> Luke
>
> On Mon, Mar 29, 2021 at 9:54 AM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Hi Luke,
> >
> > Given the error message, my intuition is that the timeout is on the
> > server side. Arrow does not try to set any timeouts on S3 connections.
> >
> > Note that this message ("When reading information") happens *before*
> > reading the file data, simply when trying to read the file length. So
> > perhaps something is weird in your network configuration (is a firewall
> > blocking packets?).
> >
> > Regards
> >
> > Antoine.
> >
> >
> >
> > On Sat, 27 Mar 2021 10:44:53 -0400
> > Luke <vi...@gmail.com> wrote:
> > > I have a local S3 compatible object store (using ceph) and am trying to
> > use
> > > the pyarrow fs interface. This seems to work well except on larger
> > objects
> > > I am getting unhandled exceptions. Is there a way to currently tune the
> > > timeouts or retries? Here is the kind of code and error I am seeing:
> > >
> > > from pyarrow import fs
> > >
> > >
> > >
> > > s3 =
> > >
> > fs.S3FileSystem(access_key=my_ak,secret_key=my_sk,endpoint_override=my_endpoint,scheme='http')
> > >
> > > raw = s3.open_input_stream('test_bucket/example_key').readall()
> > >
> > >
> > >
> > > File "pyarrow/_fs.pyx", line 621, in
> > > pyarrow._fs.FileSystem.open_input_stream
> > >
> > > File "pyarrow/error.pxi", line 122, in
> > > pyarrow.lib.pyarrow_internal_check_status
> > >
> > > File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
> > >
> > > OSError: When reading information for key 'example_key' in bucket
> > > 'test_bucket': AWS Error [code 99]: curlCode: 28, Timeout was reached
> > >
> > >
> > >
> > > --
> > >
> > > install details:
> > >
> > > python: python 3.8.6
> > >
> > > OS: linux, redhat 7.7
> > >
> > > pyarrow version: 3.0.0
> > >
> > >
> > > thanks for the help,
> > >
> > > Luke
> > >
> >
> >
> >
> >
>
Re: timeouts in S3 reads in pyarrow
Posted by Luke <vi...@gmail.com>.
No firewall in this setup. And no errors when the keys are small that I
have seen. Also, using boto3 or requests there are no issues that I can
tell (although it may be suppressing timeouts and retries).
So you are saying the 'AWS Error [code 99]: curlCode: 28, Timeout was
reached' is coming from the server? Interesting, we haven't seen that
before but our access before this was with requests or s3fs which uses
boto3 under the hood and must handle this if so.
thanks, will explore more,
Luke
On Mon, Mar 29, 2021 at 9:54 AM Antoine Pitrou <an...@python.org> wrote:
>
> Hi Luke,
>
> Given the error message, my intuition is that the timeout is on the
> server side. Arrow does not try to set any timeouts on S3 connections.
>
> Note that this message ("When reading information") happens *before*
> reading the file data, simply when trying to read the file length. So
> perhaps something is weird in your network configuration (is a firewall
> blocking packets?).
>
> Regards
>
> Antoine.
>
>
>
> On Sat, 27 Mar 2021 10:44:53 -0400
> Luke <vi...@gmail.com> wrote:
> > I have a local S3 compatible object store (using ceph) and am trying to
> use
> > the pyarrow fs interface. This seems to work well except on larger
> objects
> > I am getting unhandled exceptions. Is there a way to currently tune the
> > timeouts or retries? Here is the kind of code and error I am seeing:
> >
> > from pyarrow import fs
> >
> >
> >
> > s3 =
> >
> fs.S3FileSystem(access_key=my_ak,secret_key=my_sk,endpoint_override=my_endpoint,scheme='http')
> >
> > raw = s3.open_input_stream('test_bucket/example_key').readall()
> >
> >
> >
> > File "pyarrow/_fs.pyx", line 621, in
> > pyarrow._fs.FileSystem.open_input_stream
> >
> > File "pyarrow/error.pxi", line 122, in
> > pyarrow.lib.pyarrow_internal_check_status
> >
> > File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
> >
> > OSError: When reading information for key 'example_key' in bucket
> > 'test_bucket': AWS Error [code 99]: curlCode: 28, Timeout was reached
> >
> >
> >
> > --
> >
> > install details:
> >
> > python: python 3.8.6
> >
> > OS: linux, redhat 7.7
> >
> > pyarrow version: 3.0.0
> >
> >
> > thanks for the help,
> >
> > Luke
> >
>
>
>
>
Re: timeouts in S3 reads in pyarrow
Posted by Antoine Pitrou <an...@python.org>.
Hi Luke,
Given the error message, my intuition is that the timeout is on the
server side. Arrow does not try to set any timeouts on S3 connections.
Note that this message ("When reading information") happens *before*
reading the file data, simply when trying to read the file length. So
perhaps something is weird in your network configuration (is a firewall
blocking packets?).
Regards
Antoine.
On Sat, 27 Mar 2021 10:44:53 -0400
Luke <vi...@gmail.com> wrote:
> I have a local S3 compatible object store (using ceph) and am trying to use
> the pyarrow fs interface. This seems to work well except on larger objects
> I am getting unhandled exceptions. Is there a way to currently tune the
> timeouts or retries? Here is the kind of code and error I am seeing:
>
> from pyarrow import fs
>
>
>
> s3 =
> fs.S3FileSystem(access_key=my_ak,secret_key=my_sk,endpoint_override=my_endpoint,scheme='http')
>
> raw = s3.open_input_stream('test_bucket/example_key').readall()
>
>
>
> File "pyarrow/_fs.pyx", line 621, in
> pyarrow._fs.FileSystem.open_input_stream
>
> File "pyarrow/error.pxi", line 122, in
> pyarrow.lib.pyarrow_internal_check_status
>
> File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
>
> OSError: When reading information for key 'example_key' in bucket
> 'test_bucket': AWS Error [code 99]: curlCode: 28, Timeout was reached
>
>
>
> --
>
> install details:
>
> python: python 3.8.6
>
> OS: linux, redhat 7.7
>
> pyarrow version: 3.0.0
>
>
> thanks for the help,
>
> Luke
>