You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "MisterT314 (via GitHub)" <gi...@apache.org> on 2023/10/23 20:39:49 UTC

[I] pyarrow: using role_arn with S3FileSystem results in Anonymous user error [arrow]

MisterT314 opened a new issue, #38421:
URL: https://github.com/apache/arrow/issues/38421

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   Hi, I'm initializing S3FileSystem with a role_arn, to get refreshable temporary credentials, as described in 
   https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html
   
   The init call is the following:
   
   ```
   self._fs = S3FileSystem(
                   region=_AWS_REGION,
                   role_arn=self._role_arn, 
                   anonymous=False,
               ) 
   
   ```
   
   A bit later I'm initializing a parquetwriter with the file system
   ```
       fs = _get_service().get_file_system()
       output_url = output_url.replace("s3://", "")
       schema = Schema.from_pandas(df=data)
       writer = ParquetWriter(where=output_url, schema=schema, filesystem=fs)
   ```
   
   I then get the error:
   
   OSError: When initiating multiple part upload for key 'resources/01HBTZN7DBR1BE62S5TJRQDZXP/data.parquet' in bucket 'my-bucket-name': AWS Error ACCESS_DENIED during CreateMultipartUpload operation: Anonymous users cannot initiate multipart uploads. Please authenticate.
   
   
   I expect that S3FileSystem would use STS to get temporary credentials like described in the documentation. The role is configured to allow the service (glue) to assume the role.
   
   Pyarrow version 12.0.0 
   S3FS 2023.6.0
   Using Pyarrow in a aws glue v4 environment
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] pyarrow: using role_arn with S3FileSystem results in Anonymous user error [arrow]

Posted by "nph (via GitHub)" <gi...@apache.org>.
nph commented on issue #38421:
URL: https://github.com/apache/arrow/issues/38421#issuecomment-1966939799

   @antonio-yuen-zocdoc A potential work around is to get the temporary AWS credentials using [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts/client/assume_role.html) and then pass these directly to `S3FileSystem`, e.g.:
   ```
   import boto3
   
   session = boto3.session.Session()
   sts = session.client('sts')
   response = sts.assume_role(
       RoleArn=role_arn,
       RoleSessionName='my-role-session',
   )
   
   fs = S3FileSystem(
       access_key=response['Credentials']['AccessKeyId'],
       secret_key=response['Credentials']['SecretAccessKey'],
       session_token=response['Credentials']['SessionToken'],
   ```
   I've used a similar approach to create an `S3FileSystem` using credentials for a specific AWS profile name (which currently isn't possible using `S3FileSystem` directly). 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] pyarrow: using role_arn with S3FileSystem results in Anonymous user error [arrow]

Posted by "kharigardner (via GitHub)" <gi...@apache.org>.
kharigardner commented on issue #38421:
URL: https://github.com/apache/arrow/issues/38421#issuecomment-2010937918

   Seeing this issue in `pandas.to_parquet` method, with SSO credentials, but not env vars. Any workaround for that?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] pyarrow: using role_arn with S3FileSystem results in Anonymous user error [arrow]

Posted by "antonio-yuen-zocdoc (via GitHub)" <gi...@apache.org>.
antonio-yuen-zocdoc commented on issue #38421:
URL: https://github.com/apache/arrow/issues/38421#issuecomment-2057114751

   > Seeing this issue in `pandas.to_parquet` method, with SSO credentials, but not env vars. Any workaround for that?
   
   I got rid of pandas on my end and replaced with polars but I believe I was able to solve this issue using this doc.
   https://pandas.pydata.org/docs/user_guide/io.html#reading-writing-remote-files
   
   I believe I set this option on the to_parquet method: `storage_options={"s3": {"anon": False}}`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] pyarrow: using role_arn with S3FileSystem results in Anonymous user error [arrow]

Posted by "antonio-yuen-zocdoc (via GitHub)" <gi...@apache.org>.
antonio-yuen-zocdoc commented on issue #38421:
URL: https://github.com/apache/arrow/issues/38421#issuecomment-1965900687

   @MisterT314  have you been able to resolve this or work around it by any chance? I'm running into the same issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] pyarrow: using role_arn with S3FileSystem results in Anonymous user error [arrow]

Posted by "lauragalera (via GitHub)" <gi...@apache.org>.
lauragalera commented on issue #38421:
URL: https://github.com/apache/arrow/issues/38421#issuecomment-2057096794

   I am facing not the exact but a similar error using pyarrow version 10.0.0 for a glue job.
   
   After calling `S3FileSystem` with an arn_role and using it to read a table, I get an access error
   
   `df = pq.read_table('key', columns = ["_id","preferences"], schema = custom_schema,filesystem=s3).to_pandas()`
   
   `OSError: When getting information for key 'X' in bucket 'X': AWS Error ACCESS_DENIED during HeadObject operation: No response body.`
   
   I thought it could be due to role policies but the role has enough permission on the S3 bucket and the trust-relationship looks fine.
   
   I have come with a workaround using `boto3.client()`, but I'd like to know what is causing the access denied.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org