You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "travis-bowen (via GitHub)" <gi...@apache.org> on 2025/03/27 00:57:34 UTC

[PR] PySpark Reg Test Updates [polaris]

travis-bowen opened a new pull request, #1262:
URL: https://github.com/apache/polaris/pull/1262

   Minor Updates to PySpark Reg Tests
   
   Each test directly cleans the artifacts it created.
   
   Optional bucket prefix location has a UUID appended to allow for multiple runs against the same instance not conflicting if a previous run didn't get cleaned up.
   
   For testing - RAN 
   `./gradlew clean :polaris-quarkus-server:assemble -Dquarkus.container-image.build=true --no-build-cache                                                            docker compose -f ./regtests/docker-compose.yml up --build --exit-code-from regtest`
   `exported AWS_STORAGE_BUCKET, AWS_BUCKET_BASE_LOCATION_PREFIX, AWS_ROLE_ARN,AWS_TEST_ENABLED`
   
   Saw REG tests complete with exit code 0
   
   <!--
       Possible security vulnerabilities: STOP here and contact security@apache.org instead!
   
       Please update the title of the PR with a meaningful message - do not leave it "empty" or "generated"
       Please update this summary field:
   
       The summary should cover these topics, if applicable:
       * the motivation for the change
       * a description of the status quo, for example the current behavior
       * the desired behavior
       * etc
   
       PR checklist:
       - Do a self-review of your code before opening a pull request
       - Make sure that there's good test coverage for the changes included in this PR
       - Run tests locally before pushing a PR (./gradlew check)
       - Code should have comments where applicable. Particularly hard-to-understand
         areas deserve good in-line documentation.
       - Include changes and enhancements to the documentation (in site/content/in-dev/unreleased)
       - For Work In Progress Pull Requests, please use the Draft PR feature.
   
       Make sure to add the information BELOW this comment.
       Everything in this comment will NOT be added to the PR description.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] PySpark Reg Test Updates [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1262:
URL: https://github.com/apache/polaris/pull/1262#discussion_r2015786603


##########
regtests/t_pyspark/src/test_spark_sql_s3_with_privileges.py:
##########
@@ -359,6 +359,14 @@ def test_spark_creates_table_in_custom_namespace_dir(root_client, snowflake_cata
     entries = spark.sql(
       f"SELECT file FROM db1.schema.table_in_custom_namespace_location.metadata_log_entries").collect()
     assert namespace_location in entries[0][0]
+    try:
+        assert spark.sql("SELECT * FROM table_in_custom_namespace_location").count() == 0
+        # check the metadata and assert the custom namespace location is used
+        entries = spark.sql(
+            f"SELECT file FROM db1.schema.table_in_custom_namespace_location.metadata_log_entries").collect()
+        assert namespace_location in entries[0][0]
+    finally:
+        spark.sql('drop table table_in_custom_namespace_location PURGE')

Review Comment:
   ultra nit: inconsistent casing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] PySpark Reg Test Updates [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard merged PR #1262:
URL: https://github.com/apache/polaris/pull/1262


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] PySpark Reg Test Updates [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1262:
URL: https://github.com/apache/polaris/pull/1262#discussion_r2015788315


##########
regtests/t_pyspark/src/conftest.py:
##########
@@ -84,7 +84,8 @@ def aws_bucket_base_location_prefix():
   """
   default_val = 'polaris_test'
   bucket_prefix = os.getenv('AWS_BUCKET_BASE_LOCATION_PREFIX', default_val)
-  return default_val if bucket_prefix == '' else bucket_prefix
+  # Add random string to prefix to prevent base location overlaps
+  return f"{default_val if bucket_prefix == '' else bucket_prefix}_{str(uuid.uuid4())[-5:]}"

Review Comment:
   ultra nit: Any reason to do `[-5:]` instead of the marginally more idiomatic `[:5]`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] PySpark Reg Test Updates [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #1262:
URL: https://github.com/apache/polaris/pull/1262#discussion_r2017160356


##########
regtests/t_pyspark/src/test_spark_sql_s3_with_privileges.py:
##########
@@ -359,6 +359,14 @@ def test_spark_creates_table_in_custom_namespace_dir(root_client, snowflake_cata
     entries = spark.sql(
       f"SELECT file FROM db1.schema.table_in_custom_namespace_location.metadata_log_entries").collect()
     assert namespace_location in entries[0][0]
+    try:
+        assert spark.sql("SELECT * FROM table_in_custom_namespace_location").count() == 0
+        # check the metadata and assert the custom namespace location is used
+        entries = spark.sql(
+            f"SELECT file FROM db1.schema.table_in_custom_namespace_location.metadata_log_entries").collect()
+        assert namespace_location in entries[0][0]
+    finally:
+        spark.sql('drop table table_in_custom_namespace_location PURGE')

Review Comment:
   This is really not important, but `DROP TABLE` is capitalized elsewhere in this test



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] PySpark Reg Test Updates [polaris]

Posted by "travis-bowen (via GitHub)" <gi...@apache.org>.
travis-bowen commented on code in PR #1262:
URL: https://github.com/apache/polaris/pull/1262#discussion_r2017116066


##########
regtests/t_pyspark/src/test_spark_sql_s3_with_privileges.py:
##########
@@ -359,6 +359,14 @@ def test_spark_creates_table_in_custom_namespace_dir(root_client, snowflake_cata
     entries = spark.sql(
       f"SELECT file FROM db1.schema.table_in_custom_namespace_location.metadata_log_entries").collect()
     assert namespace_location in entries[0][0]
+    try:
+        assert spark.sql("SELECT * FROM table_in_custom_namespace_location").count() == 0
+        # check the metadata and assert the custom namespace location is used
+        entries = spark.sql(
+            f"SELECT file FROM db1.schema.table_in_custom_namespace_location.metadata_log_entries").collect()
+        assert namespace_location in entries[0][0]
+    finally:
+        spark.sql('drop table table_in_custom_namespace_location PURGE')

Review Comment:
   Yeah - each test kind of cases things slightly differently. I went ahead and matched the casing at least within this test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] PySpark Reg Test Updates [polaris]

Posted by "travis-bowen (via GitHub)" <gi...@apache.org>.
travis-bowen commented on code in PR #1262:
URL: https://github.com/apache/polaris/pull/1262#discussion_r2017113832


##########
regtests/t_pyspark/src/conftest.py:
##########
@@ -84,7 +84,8 @@ def aws_bucket_base_location_prefix():
   """
   default_val = 'polaris_test'
   bucket_prefix = os.getenv('AWS_BUCKET_BASE_LOCATION_PREFIX', default_val)
-  return default_val if bucket_prefix == '' else bucket_prefix
+  # Add random string to prefix to prevent base location overlaps
+  return f"{default_val if bucket_prefix == '' else bucket_prefix}_{str(uuid.uuid4())[-5:]}"

Review Comment:
   Not in particular! Changed it to your suggestion!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org