[GitHub] [arrow] stevenmanton opened a new issue, #36416: [Python] Dataset doesn't seem to load named profiles correctly

["stevenmanton (via GitHub)" <gi...@apache.org>]

stevenmanton opened a new issue, #36416:
URL: https://github.com/apache/arrow/issues/36416

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   I use `pyarrow.dataset.dataset` often to load data from S3. However, it's always been a bit finicky regarding named AWS profiles. For example, let's say I have the following `~/.aws/credentials` file that defines two identical profiles and uses an external process to get credentials (which shouldn't matter, but I'm adding it for completeness):
   
   ```
   [default]
   region = us-east-1
   credential_process = /path/to/get-creds.sh
   
   [dev]
   region = us-east-1
   credential_process = /path/to/get-creds.sh
   ```
   
   These commands both work, which validates that both accounts have access to S3:
   ```
   aws s3 ls s3://bucket/path --profile default
   aws s3 ls s3://bucket/path --profile dev
   ```
   
   Now, as I understand it, the only way to specify a named AWS profile with pyarrow is to use environment variables. To test the access, I use a trivial script:
   
   ```python
   # script.py
   import pyarrow.dataset as ds
   
   dataset = ds.dataset("s3://bucket/path/")
   ```
   
   Now, here's the strangeness:
   ```bash
   # Check that no environment variables are set in this shell:
   env | grep AWS
   
   # This works:
   python ./script.py
   # So does this:
   AWS_PROFILE=default python ./script.py
   # But this fails:
   AWS_PROFILE=dev python ./script.py
   ```
   
   The error is:
   ```
   Traceback (most recent call last):
     File "./script.py", line 3, in <module>
       dataset = ds.dataset("s3://bucket/path/")
     File "/home/antonstv/miniconda3/envs/pdna/lib/python3.8/site-packages/pyarrow/dataset.py", line 763, in dataset
       return _filesystem_dataset(source, **kwargs)
     File "/home/antonstv/miniconda3/envs/pdna/lib/python3.8/site-packages/pyarrow/dataset.py", line 446, in _filesystem_dataset
       fs, paths_or_selector = _ensure_single_source(source, filesystem)
     File "/home/antonstv/miniconda3/envs/pdna/lib/python3.8/site-packages/pyarrow/dataset.py", line 413, in _ensure_single_source
       file_info = filesystem.get_file_info(path)
     File "pyarrow/_fs.pyx", line 571, in pyarrow._fs.FileSystem.get_file_info
     File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
     File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
   OSError: When getting information for key 'path' in bucket 'bucket': AWS Error ACCESS_DENIED during HeadObject operation: No response body.
   ```
   
   I'm really quite at a loss for how this could happen. And since the underlying code isn't python, it seems tricky for me to help debug. It's possible that I'm doing something dumb, but I really can't imagine why two identical profiles would give different results like this.
   
   The pyarrow version I'm using is `12.0.1`.
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] stevenmanton closed issue #36416: [Python] Dataset doesn't seem to load named AWS profiles correctly

["stevenmanton (via GitHub)" <gi...@apache.org>]

stevenmanton closed issue #36416: [Python] Dataset doesn't seem to load named AWS profiles correctly
URL: https://github.com/apache/arrow/issues/36416


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] stevenmanton commented on issue #36416: [Python] Dataset doesn't seem to load named AWS profiles correctly

["stevenmanton (via GitHub)" <gi...@apache.org>]

stevenmanton commented on issue #36416:
URL: https://github.com/apache/arrow/issues/36416#issuecomment-1638968630

   Update: This issue seems to originate with the `Aws::Auth::DefaultAWSCredentialsProviderChain` C++ class. It also seems to be related to the use of `credential_process`. If I run the credential process one time and write the result to a named profile, the issue described above goes away.
   
   To summarize: The use of `credential_process` along with a named AWS profile does not work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org