Re: [I] [BUG]Data folder is not getting listed with storage type FILE [polaris]

["MonkeyCanCode (via GitHub)" <gi...@apache.org>]

MonkeyCanCode commented on issue #166:
URL: https://github.com/apache/polaris/issues/166#issuecomment-2311455210

   Hello,
   
   I am able to run this locally and here are what I did:
   ```
   # (on terminal 1) start the application
   ./gradlew runApp
   
   # (on terminal 2) create catalog, catalog role, principal, principal, grant, and namespace
   export CLIENT_ID=xxxxx
   export CLIENT_SECRET=xxxxx
   ./polaris catalogs create --storage-type file --default-base-location file:///tmp/test quickstart_catalog
   ./polaris principals create quickstart_user
   ./polaris principal-roles create quickstart_user_role
   ./polaris catalog-roles create --catalog quickstart_catalog quickstart_catalog_role
   ./polaris principal-roles grant --principal quickstart_user quickstart_user_role
   ./polaris catalog-roles grant --catalog quickstart_catalog --principal-role quickstart_user_role quickstart_catalog_role
   ./polaris privileges catalog grant --catalog quickstart_catalog --catalog-role quickstart_catalog_role CATALOG_MANAGE_CONTENT
   ./polaris namespaces create --catalog quickstart_catalog quickstart_namespace
   
   # (on terminal 2) setup spark and default spark conf (sample spark conf below)
   spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
   spark.jars.packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.13:1.5.0,org.apache.hadoop:hadoop-aws:3.4.0,software.amazon.awssdk:bundle:2.23.19,software.amazon.awssdk:url-connection-client:2.23.19
   spark.sql.iceberg.vectorization.enabled false
   spark.sql.catalog.polaris.type rest
   spark.sql.catalog.polaris org.apache.iceberg.spark.SparkCatalog
   spark.sql.catalog.polaris.uri http://localhost:8181/api/catalog
   spark.sql.catalog.polaris.token-refresh-enabled true
   spark.sql.catalog.polaris.credential xxxxx:xxxxx
   spark.sql.catalog.polaris.warehouse quickstart_catalog
   spark.sql.catalog.polaris.scope PRINCIPAL_ROLE:ALL
   spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation true
   spark.sql.catalog.polaris.io-impl org.apache.iceberg.io.ResolvingFileIO
   
   # (on terminal 2) create table via pyspark
   >>> spark.sql("use polaris")
   DataFrame[]
   >>> spark.sql("use quickstart_namespace")
   DataFrame[]
   >>> spark.sql("create table test_tbl (id int, value int)")
   DataFrame[]
   
   # (on terminal 3) check fs path
   xxxxx@DESKTOP:~/polaris(main)$ ls -l /tmp/test
   total 4
   drwxr-xr-x 3 yong yong 4096 Aug 26 21:30 quickstart_namespace
   xxxxx@DESKTOP:~/polaris(main)$ ls -l /tmp/test/quickstart_namespace/
   total 4
   drwxr-xr-x 3 yong yong 4096 Aug 26 21:30 test_tbl
   xxxxx@DESKTOP:~/polaris(main)$ ls -l /tmp/test/quickstart_namespace/test_tbl/
   total 4
   drwxr-xr-x 2 yong yong 4096 Aug 26 21:30 metadata
   
   # (on terminal 2) insert dummy record
   >>> spark.sql("insert into test_tbl values(1,2)")
   DataFrame[]
   
   # (on terminal 3) check fs path
   xxxxx@DESKTOP:~/polaris(main)$ ls -l /tmp/test/quickstart_namespace/test_tbl/data/
   total 4
   -rw-r--r-- 1 yong yong 608 Aug 26 21:31 00000-0-063831d9-208d-4cc0-9cae-f316edff15c1-0-00001.parquet
   xxxxx@DESKTOP:~/polaris(main)$ ls -l /tmp/test/quickstart_namespace/test_tbl/metadata/
   total 24
   -rw-r--r-- 1 yong yong 1028 Aug 26 21:30 00000-16559417-6d2b-481e-a4df-72cf8a17f9f6.metadata.json
   -rw-r--r-- 1 yong yong 2111 Aug 26 21:31 00001-5124a0c6-6ebd-420f-a6e7-5c3b78b2f3c2.metadata.json
   -rw-r--r-- 1 yong yong 6669 Aug 26 21:31 8a607974-9597-48ea-9471-3299891dd0a0-m0.avro
   -rw-r--r-- 1 yong yong 4234 Aug 26 21:31 snap-7058842161523378569-1-8a607974-9597-48ea-9471-3299891dd0a0.avro
   ```
   
   @shraddhagrawal can u try the step above and see if u still see the issue?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org