You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "lozbrown (via GitHub)" <gi...@apache.org> on 2025/03/07 15:59:39 UTC
[I] pyiceberg with hive and S3 fails even when providing creds [iceberg-python]
lozbrown opened a new issue, #1775:
URL: https://github.com/apache/iceberg-python/issues/1775
I'm trying to use pyiceberg within a pod that has access via a role
I've configured PYICEBERG_CATALOG__DEFAULT__S3__ROLE_ARN and AWS_ROLE_ARN environment variables but that fails with a Headobject issue
```
File "/usr/local/lib64/python3.12/site-packages/pyiceberg/catalog/__init__.py", line 420, in create_table_if_not_exists
return self.create_table(identifier, schema, location, partition_spec, sort_order, properties)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.12/site-packages/pyiceberg/catalog/hive.py", line 404, in create_table
self._write_metadata(staged_table.metadata, staged_table.io, staged_table.metadata_location)
File "/usr/local/lib64/python3.12/site-packages/pyiceberg/catalog/__init__.py", line 939, in _write_metadata
ToOutputFile.table_metadata(metadata, io.new_output(metadata_path))
File "/usr/local/lib64/python3.12/site-packages/pyiceberg/serializers.py", line 130, in table_metadata
with output_file.create(overwrite=overwrite) as output_stream:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.12/site-packages/pyiceberg/io/pyarrow.py", line 338, in create
if not overwrite and self.exists() is True:
^^^^^^^^^^^^^
File "/usr/local/lib64/python3.12/site-packages/pyiceberg/io/pyarrow.py", line 282, in exists
self._file_info() # raises FileNotFoundError if it does not exist
^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.12/site-packages/pyiceberg/io/pyarrow.py", line 264, in _file_info
file_info = self._filesystem.get_file_info(self._path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/_fs.pyx", line 590, in pyarrow._fs.FileSystem.get_file_info
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: When getting information for key 'schemas/meta.db/trino_queries_iceberg/metadata/00000-41568416-bc76-4236-afab-a7bec772eb32.metadata.json' in bucket 'REDACTED-BUCKET': AWS Error ACCESS_DENIED during HeadObject operation: No response body.
```
I believe that's this Pyarrow issue https://github.com/apache/arrow/issues/38421
To get past this I set these keys upstream my python
```
import boto3
session = boto3.session.Session()
os.environ['AWS_ACCESS_KEY_ID']=session.get_credentials().access_key
os.environ['AWS_SECRET_ACCESS_KEY']=session.get_credentials().secret_key
os.environ['AWS_SESSION_TOKEN']=session.get_credentials().token
os.environ['PYICEBERG_CATALOG__DEFAULT__S3__ACCESS_KEY_ID']=session.get_credentials().access_key
os.environ['PYICEBERG_CATALOG__DEFAULT__S3__SECRET_ACCESS_KEY']=session.get_credentials().secret_key
os.environ['PYICEBERG_CATALOG__DEFAULT__S3__SESSION_TOKEN']=session.get_credentials().token
```
That gets me past that issue but then we get to the below java error
```
[trino@testpod /]$ python query_logger_iceberg.py
Traceback (most recent call last):
File "//query_logger_iceberg.py", line 57, in <module>
table = catalog.create_table_if_not_exists( identifier=f'{trino_schema_name}.{tablename}',schema=iceberg_table_schema, partition_spec=partition_spec, location=s3_location,sort_order=sort_order )
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.12/site-packages/pyiceberg/catalog/__init__.py", line 420, in create_table_if_not_exists
return self.create_table(identifier, schema, location, partition_spec, sort_order, properties)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.12/site-packages/pyiceberg/catalog/hive.py", line 408, in create_table
self._create_hive_table(open_client, tbl)
File "/usr/local/lib64/python3.12/site-packages/pyiceberg/catalog/hive.py", line 357, in _create_hive_table
open_client.create_table(hive_table)
File "/usr/local/lib64/python3.12/site-packages/hive_metastore/ThriftHiveMetastore.py", line 3431, in create_table
self.recv_create_table()
File "/usr/local/lib64/python3.12/site-packages/hive_metastore/ThriftHiveMetastore.py", line 3457, in recv_create_table
raise result.o3
hive_metastore.ttypes.MetaException: MetaException(message='java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified by setting the fs.s3.awsAccessKeyId and fs.s3.awsSecretAccessKey properties (respectively).')
```
Given that I've already supplied the keys should they not be passed on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] pyiceberg with hive and S3 fails even when providing creds [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #1775: pyiceberg with hive and S3 fails even when providing creds
URL: https://github.com/apache/iceberg-python/issues/1775
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] pyiceberg with hive and S3 fails even when providing creds [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1775:
URL: https://github.com/apache/iceberg-python/issues/1775#issuecomment-2715173895
Thanks for filing this issue!
It looks like we do pass the `s3.role-arn` to the underlying pyarrow S3FileSystem and the issue is in the S3FileSystem itself, as described in https://github.com/apache/arrow/issues/38421
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] pyiceberg with hive and S3 fails even when providing creds [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1775:
URL: https://github.com/apache/iceberg-python/issues/1775#issuecomment-4920444447
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] pyiceberg with hive and S3 fails even when providing creds [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1775:
URL: https://github.com/apache/iceberg-python/issues/1775#issuecomment-3955564910
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] pyiceberg with hive and S3 fails even when providing creds [iceberg-python]
Posted by "rnadhani-ns (via GitHub)" <gi...@apache.org>.
rnadhani-ns commented on issue #1775:
URL: https://github.com/apache/iceberg-python/issues/1775#issuecomment-3233567050
Was there an update on this? I think I am seeing the same issue.
My .pyiceberg looks like:
```catalog:
local:
uri: sqlite:///iceberg_catalog/catalog.db
warehouse: s3://my_bucket/warehouse```
I was just following: https://estuary.dev/blog/getting-started-pyiceberg/
But if I change the warehouse to file://, it all works.
So I suspect the issue is with the S3 file system module. I tried setting the credentials like https://github.com/apache/iceberg-python/issues/1775#issue-2903374205 but still I see issues.
My credentials are set in ~/.aws/credentials and I do set the AWS_PROFILE and AWS_REGION varialble.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org