You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "fivetran-ashokborra (via GitHub)" <gi...@apache.org> on 2025/04/17 13:03:22 UTC
[PR] fix(Catalog): Add List PolarisStorageAction for all metadata read operations [polaris]
fivetran-ashokborra opened a new pull request, #1391:
URL: https://github.com/apache/polaris/pull/1391
<!--
Possible security vulnerabilities: STOP here and contact security@apache.org instead!
Please update the title of the PR with a meaningful message - do not leave it "empty" or "generated"
Please update this summary field:
The summary should cover these topics, if applicable:
* the motivation for the change
* a description of the status quo, for example the current behavior
* the desired behavior
* etc
PR checklist:
- Do a self-review of your code before opening a pull request
- Make sure that there's good test coverage for the changes included in this PR
- Run tests locally before pushing a PR (./gradlew check)
- Code should have comments where applicable. Particularly hard-to-understand
areas deserve good in-line documentation.
- Include changes and enhancements to the documentation (in site/content/in-dev/unreleased)
- For Work In Progress Pull Requests, please use the Draft PR feature.
Make sure to add the information BELOW this comment.
Everything in this comment will NOT be added to the PR description.
-->
Fixes #1380
As per S3 GetObject [documentation](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html), listBucket permission is needed if the object that is requested doesn't exist
```
If the object that you request doesn’t exist, the error that Amazon S3 returns depends on whether you also have the s3:ListBucket permission.
If you have the s3:ListBucket permission on the bucket, Amazon S3 returns an HTTP status code 404 Not Found error.
If you don’t have the s3:ListBucket permission, Amazon S3 returns an HTTP status code 403 Access Denied error.
```
We are only adding [ListBucket statement](https://github.com/apache/polaris/blob/main/polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java#L137) only when PolarisStorageAction.List is passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(Catalog): Add List PolarisStorageAction for all metadata read operations [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on PR #1391:
URL: https://github.com/apache/polaris/pull/1391#issuecomment-2851555480
ping @flyrain, @eric-maynard
can we please include this in 1.0 mile-stone, this PR is very close
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(Catalog): Add List PolarisStorageAction for all metadata read operations [polaris]
Posted by "fivetran-ashokborra (via GitHub)" <gi...@apache.org>.
fivetran-ashokborra commented on code in PR #1391:
URL: https://github.com/apache/polaris/pull/1391#discussion_r2073792267
##########
regtests/t_pyspark/src/test_spark_sql_s3_with_privileges.py:
##########
@@ -1149,6 +1149,78 @@ def test_spark_ctas(snowflake_catalog, polaris_catalog_url, snowman):
spark.sql(f"drop table {table_name}_t2 PURGE")
+@pytest.mark.skipif(os.environ.get('AWS_TEST_ENABLED', 'False').lower() != 'true',
+ reason='AWS_TEST_ENABLED is not set or is false')
+def test_spark_credentials_s3_exceptions(root_client, snowflake_catalog, polaris_catalog_url,
+ snowman, snowman_catalog_client, test_bucket, aws_bucket_base_location_prefix):
+ """
+ Create a using Spark. Then call the loadTable api directly with snowman token to fetch the vended credentials
+ for the first table.
+ Delete the metadata directory and try to access the table using the vended credentials.
+ It should throw 404 exception
+ :param root_client:
+ :param snowflake_catalog:
+ :param polaris_catalog_url:
+ :param snowman_catalog_client:
+ :param reader_catalog_client:
+ :return:
+ """
+ with IcebergSparkSession(credentials=f'{snowman.principal.client_id}:{snowman.credentials.client_secret}',
+ catalog_name=snowflake_catalog.name,
+ polaris_url=polaris_catalog_url) as spark:
+ spark.sql(f'USE {snowflake_catalog.name}')
+ spark.sql('CREATE NAMESPACE db1')
+ spark.sql('CREATE NAMESPACE db1.schema')
+ spark.sql('USE db1.schema')
+ spark.sql('CREATE TABLE iceberg_table (col1 int, col2 string)')
+
+ response = snowman_catalog_client.load_table(snowflake_catalog.name, unquote('db1%1Fschema'),
+ "iceberg_table",
+ "vended-credentials")
+ assert response.config is not None
+ assert 's3.access-key-id' in response.config
+ assert 's3.secret-access-key' in response.config
+ assert 's3.session-token' in response.config
+
+ s3 = boto3.client('s3',
+ aws_access_key_id=response.config['s3.access-key-id'],
+ aws_secret_access_key=response.config['s3.secret-access-key'],
+ aws_session_token=response.config['s3.session-token'])
+
+ objects = s3.list_objects(Bucket=test_bucket, Delimiter='/',
+ Prefix=f'{aws_bucket_base_location_prefix}/snowflake_catalog/db1/schema/iceberg_table/metadata/')
+ assert objects is not None
+ assert 'Contents' in objects
+ assert len(objects['Contents']) > 0
+
+ metadata_file = next(f for f in objects['Contents'] if f['Key'].endswith('metadata.json'))
+ assert metadata_file is not None
+
+ metadata_contents = s3.get_object(Bucket=test_bucket, Key=metadata_file['Key'])
+ assert metadata_contents is not None
+ assert metadata_contents['ContentLength'] > 0
+
+ s3.delete_objects(Bucket=test_bucket,
+ Delete={'Objects': objects})
Review Comment:
Deletion of metadata should be fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(Catalog): Add List PolarisStorageAction for all metadata read operations [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #1391:
URL: https://github.com/apache/polaris/pull/1391#discussion_r2073766661
##########
regtests/t_pyspark/src/test_spark_sql_s3_with_privileges.py:
##########
@@ -1149,6 +1149,78 @@ def test_spark_ctas(snowflake_catalog, polaris_catalog_url, snowman):
spark.sql(f"drop table {table_name}_t2 PURGE")
+@pytest.mark.skipif(os.environ.get('AWS_TEST_ENABLED', 'False').lower() != 'true',
+ reason='AWS_TEST_ENABLED is not set or is false')
+def test_spark_credentials_s3_exceptions(root_client, snowflake_catalog, polaris_catalog_url,
+ snowman, snowman_catalog_client, test_bucket, aws_bucket_base_location_prefix):
+ """
+ Create a using Spark. Then call the loadTable api directly with snowman token to fetch the vended credentials
+ for the first table.
+ Delete the metadata directory and try to access the table using the vended credentials.
+ It should throw 404 exception
+ :param root_client:
+ :param snowflake_catalog:
+ :param polaris_catalog_url:
+ :param snowman_catalog_client:
+ :param reader_catalog_client:
+ :return:
+ """
+ with IcebergSparkSession(credentials=f'{snowman.principal.client_id}:{snowman.credentials.client_secret}',
+ catalog_name=snowflake_catalog.name,
+ polaris_url=polaris_catalog_url) as spark:
+ spark.sql(f'USE {snowflake_catalog.name}')
+ spark.sql('CREATE NAMESPACE db1')
+ spark.sql('CREATE NAMESPACE db1.schema')
+ spark.sql('USE db1.schema')
+ spark.sql('CREATE TABLE iceberg_table (col1 int, col2 string)')
+
+ response = snowman_catalog_client.load_table(snowflake_catalog.name, unquote('db1%1Fschema'),
+ "iceberg_table",
+ "vended-credentials")
+ assert response.config is not None
+ assert 's3.access-key-id' in response.config
+ assert 's3.secret-access-key' in response.config
+ assert 's3.session-token' in response.config
+
+ s3 = boto3.client('s3',
+ aws_access_key_id=response.config['s3.access-key-id'],
+ aws_secret_access_key=response.config['s3.secret-access-key'],
+ aws_session_token=response.config['s3.session-token'])
+
+ objects = s3.list_objects(Bucket=test_bucket, Delimiter='/',
+ Prefix=f'{aws_bucket_base_location_prefix}/snowflake_catalog/db1/schema/iceberg_table/metadata/')
+ assert objects is not None
+ assert 'Contents' in objects
+ assert len(objects['Contents']) > 0
+
+ metadata_file = next(f for f in objects['Contents'] if f['Key'].endswith('metadata.json'))
+ assert metadata_file is not None
+
+ metadata_contents = s3.get_object(Bucket=test_bucket, Key=metadata_file['Key'])
+ assert metadata_contents is not None
+ assert metadata_contents['ContentLength'] > 0
+
+ s3.delete_objects(Bucket=test_bucket,
+ Delete={'Objects': objects})
Review Comment:
so we only delete metadata obj ? I am assuming it's just the metadata.json if deleted should be fine ? or we neeed to delete manifest and manifest list too, to re-produce.
##########
regtests/t_pyspark/src/test_spark_sql_s3_with_privileges.py:
##########
@@ -1149,6 +1149,78 @@ def test_spark_ctas(snowflake_catalog, polaris_catalog_url, snowman):
spark.sql(f"drop table {table_name}_t2 PURGE")
+@pytest.mark.skipif(os.environ.get('AWS_TEST_ENABLED', 'False').lower() != 'true',
+ reason='AWS_TEST_ENABLED is not set or is false')
+def test_spark_credentials_s3_exceptions(root_client, snowflake_catalog, polaris_catalog_url,
+ snowman, snowman_catalog_client, test_bucket, aws_bucket_base_location_prefix):
+ """
+ Create a using Spark. Then call the loadTable api directly with snowman token to fetch the vended credentials
+ for the first table.
+ Delete the metadata directory and try to access the table using the vended credentials.
+ It should throw 404 exception
+ :param root_client:
+ :param snowflake_catalog:
+ :param polaris_catalog_url:
+ :param snowman_catalog_client:
+ :param reader_catalog_client:
+ :return:
+ """
+ with IcebergSparkSession(credentials=f'{snowman.principal.client_id}:{snowman.credentials.client_secret}',
+ catalog_name=snowflake_catalog.name,
+ polaris_url=polaris_catalog_url) as spark:
+ spark.sql(f'USE {snowflake_catalog.name}')
+ spark.sql('CREATE NAMESPACE db1')
+ spark.sql('CREATE NAMESPACE db1.schema')
+ spark.sql('USE db1.schema')
+ spark.sql('CREATE TABLE iceberg_table (col1 int, col2 string)')
+
+ response = snowman_catalog_client.load_table(snowflake_catalog.name, unquote('db1%1Fschema'),
+ "iceberg_table",
+ "vended-credentials")
+ assert response.config is not None
+ assert 's3.access-key-id' in response.config
+ assert 's3.secret-access-key' in response.config
+ assert 's3.session-token' in response.config
+
+ s3 = boto3.client('s3',
+ aws_access_key_id=response.config['s3.access-key-id'],
+ aws_secret_access_key=response.config['s3.secret-access-key'],
+ aws_session_token=response.config['s3.session-token'])
+
+ objects = s3.list_objects(Bucket=test_bucket, Delimiter='/',
+ Prefix=f'{aws_bucket_base_location_prefix}/snowflake_catalog/db1/schema/iceberg_table/metadata/')
+ assert objects is not None
+ assert 'Contents' in objects
+ assert len(objects['Contents']) > 0
Review Comment:
minor : I would add a small comment like
check if there is metadata present.
##########
regtests/t_pyspark/src/test_spark_sql_s3_with_privileges.py:
##########
@@ -1149,6 +1149,78 @@ def test_spark_ctas(snowflake_catalog, polaris_catalog_url, snowman):
spark.sql(f"drop table {table_name}_t2 PURGE")
+@pytest.mark.skipif(os.environ.get('AWS_TEST_ENABLED', 'False').lower() != 'true',
+ reason='AWS_TEST_ENABLED is not set or is false')
+def test_spark_credentials_s3_exceptions(root_client, snowflake_catalog, polaris_catalog_url,
Review Comment:
can we have more descriptive name ?
suggestion
```suggestion
def test_s3_exception_on_deleted_table_via_spark(root_client, snowflake_catalog, polaris_catalog_url,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(Catalog): Add List PolarisStorageAction for all metadata read operations [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on PR #1391:
URL: https://github.com/apache/polaris/pull/1391#issuecomment-2873135641
Merging this, since we have 2 +1s and been there for a week !
Thanks @fivetran-ashokborra for the work ! Thank @eric-maynard for the review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(Catalog): Add List PolarisStorageAction for all metadata read operations [polaris]
Posted by "fivetran-ashokborra (via GitHub)" <gi...@apache.org>.
fivetran-ashokborra commented on PR #1391:
URL: https://github.com/apache/polaris/pull/1391#issuecomment-2814678118
@singhpk234
> mean while can you please add an UT for this ?
This is already covered [here](https://github.com/apache/polaris/blob/main/polaris-core/src/test/java/org/apache/polaris/service/storage/aws/AwsCredentialsStorageIntegrationTest.java#L258) and [here](https://github.com/apache/polaris/blob/main/polaris-core/src/test/java/org/apache/polaris/service/storage/aws/AwsCredentialsStorageIntegrationTest.java#L359). Do we still need 1 more UT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(Catalog): Add List PolarisStorageAction for all metadata read operations [polaris]
Posted by "fivetran-ashokborra (via GitHub)" <gi...@apache.org>.
fivetran-ashokborra commented on code in PR #1391:
URL: https://github.com/apache/polaris/pull/1391#discussion_r2073803831
##########
regtests/t_pyspark/src/test_spark_sql_s3_with_privileges.py:
##########
@@ -1149,6 +1149,78 @@ def test_spark_ctas(snowflake_catalog, polaris_catalog_url, snowman):
spark.sql(f"drop table {table_name}_t2 PURGE")
+@pytest.mark.skipif(os.environ.get('AWS_TEST_ENABLED', 'False').lower() != 'true',
+ reason='AWS_TEST_ENABLED is not set or is false')
+def test_spark_credentials_s3_exceptions(root_client, snowflake_catalog, polaris_catalog_url,
+ snowman, snowman_catalog_client, test_bucket, aws_bucket_base_location_prefix):
+ """
+ Create a using Spark. Then call the loadTable api directly with snowman token to fetch the vended credentials
+ for the first table.
+ Delete the metadata directory and try to access the table using the vended credentials.
+ It should throw 404 exception
+ :param root_client:
+ :param snowflake_catalog:
+ :param polaris_catalog_url:
+ :param snowman_catalog_client:
+ :param reader_catalog_client:
+ :return:
+ """
+ with IcebergSparkSession(credentials=f'{snowman.principal.client_id}:{snowman.credentials.client_secret}',
+ catalog_name=snowflake_catalog.name,
+ polaris_url=polaris_catalog_url) as spark:
+ spark.sql(f'USE {snowflake_catalog.name}')
+ spark.sql('CREATE NAMESPACE db1')
+ spark.sql('CREATE NAMESPACE db1.schema')
+ spark.sql('USE db1.schema')
+ spark.sql('CREATE TABLE iceberg_table (col1 int, col2 string)')
+
+ response = snowman_catalog_client.load_table(snowflake_catalog.name, unquote('db1%1Fschema'),
+ "iceberg_table",
+ "vended-credentials")
+ assert response.config is not None
+ assert 's3.access-key-id' in response.config
+ assert 's3.secret-access-key' in response.config
+ assert 's3.session-token' in response.config
+
+ s3 = boto3.client('s3',
+ aws_access_key_id=response.config['s3.access-key-id'],
+ aws_secret_access_key=response.config['s3.secret-access-key'],
+ aws_session_token=response.config['s3.session-token'])
+
+ objects = s3.list_objects(Bucket=test_bucket, Delimiter='/',
+ Prefix=f'{aws_bucket_base_location_prefix}/snowflake_catalog/db1/schema/iceberg_table/metadata/')
+ assert objects is not None
+ assert 'Contents' in objects
+ assert len(objects['Contents']) > 0
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(Catalog): Add List PolarisStorageAction for all metadata read operations [polaris]
Posted by "fivetran-ashokborra (via GitHub)" <gi...@apache.org>.
fivetran-ashokborra commented on code in PR #1391:
URL: https://github.com/apache/polaris/pull/1391#discussion_r2073793539
##########
regtests/t_pyspark/src/test_spark_sql_s3_with_privileges.py:
##########
@@ -1149,6 +1149,78 @@ def test_spark_ctas(snowflake_catalog, polaris_catalog_url, snowman):
spark.sql(f"drop table {table_name}_t2 PURGE")
+@pytest.mark.skipif(os.environ.get('AWS_TEST_ENABLED', 'False').lower() != 'true',
+ reason='AWS_TEST_ENABLED is not set or is false')
+def test_spark_credentials_s3_exceptions(root_client, snowflake_catalog, polaris_catalog_url,
Review Comment:
I can rename to `test_spark_credentials_s3_exception_on_metadata_file_deletion`. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(Catalog): Add List PolarisStorageAction for all metadata read operations [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on PR #1391:
URL: https://github.com/apache/polaris/pull/1391#issuecomment-2813599943
cc @flyrain @dennishuo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(Catalog): Add List PolarisStorageAction for all metadata read operations [polaris]
Posted by "collado-mike (via GitHub)" <gi...@apache.org>.
collado-mike commented on PR #1391:
URL: https://github.com/apache/polaris/pull/1391#issuecomment-2814537679
> IMHO, we should in general go with a prinicipal of least priviledges, how hard is this to catch this / parse this and re:throw 404 ?
Generally, I'd agree about the least privilege concern, but I'm opposed to reinterpreting AWS exceptions. I think we ought to get AWS to return the correct error code so we don't have to guess at runtime
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(Catalog): Add List PolarisStorageAction for all metadata read operations [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on PR #1391:
URL: https://github.com/apache/polaris/pull/1391#issuecomment-2819127610
@fivetran-ashokborra i was mostly coming from the fact that this was not catched by a test case, so to be it future proof, if some one removes or adds new permission this would fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(Catalog): Add List PolarisStorageAction for all metadata read operations [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 merged PR #1391:
URL: https://github.com/apache/polaris/pull/1391
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org