You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "glindstr (via GitHub)" <gi...@apache.org> on 2023/12/11 19:41:50 UTC
[I] Creating S3FileSystem using env AWS_PROFILE without profile config entry extremely slow [arrow]
glindstr opened a new issue, #39184:
URL: https://github.com/apache/arrow/issues/39184
### Describe the bug, including details regarding any error messages, version, and platform.
There appears to be a bug slowing down the creation of a S3FileSystem object when supplying a AWS_PROFILE via environment variable but if that profile doesn't have a corresponding .aws/config entry it hangs. I'm using python but the problematic code may be in the underlining c/c++ library.
.aws/credentials
```
[profile_name_with_config]
aws_access_key_id = ...
aws_secret_access_key = ...
[profile_name_without_config]
aws_access_key_id = ...
aws_secret_access_key = ...
```
.aws/config
```
[profile profile_name_with_config]
region = us-west-2
output = json
```
fresh python 3.12.1 environment
```
numpy 1.26.2
pip 23.2.1
pyarrow 14.0.1
```
test code
```python
from pyarrow.fs import S3FileSystem
import time as t
import os
os.environ["AWS_PROFILE"] = "profile_name_with_config"
tic = t.perf_counter()
fs = S3FileSystem()
toc = t.perf_counter()
print(f"S3FileSystem where config exists {toc - tic:0.4f} seconds")
#S3FileSystem where config exists 0.0071 seconds
os.environ["AWS_PROFILE"] = "profile_name_without_config"
tic = t.perf_counter()
fs = S3FileSystem()
toc = t.perf_counter()
print(f"S3FileSystem where config does not exist {toc - tic:0.4f} seconds")
#S3FileSystem where config does not exist 24.0123 seconds
```
It takes S3FileSystem 24 additional seconds to complete when no config entry is provided. A pretty surprising result.
Thank you for reviewing.
### Component(s)
C++, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Creating S3FileSystem using env AWS_PROFILE without profile config entry extremely slow [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #39184:
URL: https://github.com/apache/arrow/issues/39184#issuecomment-1853980297
Given this is a quite common issue (we also use it in our own tests), we should maybe add a mention of this in the S3FileSystem user guide
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Creating S3FileSystem using env AWS_PROFILE without profile config entry extremely slow [arrow]
Posted by "glindstr (via GitHub)" <gi...@apache.org>.
glindstr commented on issue #39184:
URL: https://github.com/apache/arrow/issues/39184#issuecomment-1902162953
> This has to do with the AWS SDK's default parameters. If you don't use it, I recommend you disable metadata lookup using the [`AWS_EC2_METADATA_DISABLED` environment variable](https://docs.aws.amazon.com/sdkref/latest/guide/feature-imds-credentials.html).
Thank you @pitrou. Setting this environment variable fixed the problem for me.
@jorisvandenbossche Please re-open if you want this issue for the user guide mention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Creating S3FileSystem using env AWS_PROFILE without profile config entry extremely slow [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #39184:
URL: https://github.com/apache/arrow/issues/39184#issuecomment-1853964423
This has to do with the AWS SDK's default parameters. If you don't use it, I recommend you disable metadata lookup using the [`AWS_EC2_METADATA_DISABLED` environment variable](https://docs.aws.amazon.com/sdkref/latest/guide/feature-imds-credentials.html).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Creating S3FileSystem using env AWS_PROFILE without profile config entry extremely slow [arrow]
Posted by "glindstr (via GitHub)" <gi...@apache.org>.
glindstr closed issue #39184: Creating S3FileSystem using env AWS_PROFILE without profile config entry extremely slow
URL: https://github.com/apache/arrow/issues/39184
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org