You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/06/09 14:42:41 UTC

[GitHub] [iceberg] singhpk234 opened a new pull request, #5006: Spark: Show metadata.json path in `DESC TABLE EXTENDED`

singhpk234 opened a new pull request, #5006:
URL: https://github.com/apache/iceberg/pull/5006

   ### About Change
   The change attempts to show metadata location in `DESC TABLE EXTENDED`, this adds `metadata-location` to the table properties.
   
   ### Why this change is Required
   
   We can use this to figure out  the metadata pointer this table points to and use it to register table via [StoredProcedure](https://github.com/apache/iceberg/pull/4810) recently introduced. This could be convenient for users who *only* want to use SPARK-SQL.
   
   ### Testing
   
   Added a UT.
   Add one more UT for path validation


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] singhpk234 commented on pull request #5006: Spark: Show metadata.json path in `DESC TABLE EXTENDED`

Posted by GitBox <gi...@apache.org>.
singhpk234 commented on PR #5006:
URL: https://github.com/apache/iceberg/pull/5006#issuecomment-1153492893

   @rdblue, Considering now we have register table stored-procedure in, was thinking if there is a pure SQL way to find this property. I had a migration use case (i.e migrate hive tables to glue), had to go to glue UI to find this path and then register it via SQL and also vice-versa went to hive table params to find it and register in glue. 
   ```
   MariaDB [hive]> SELECT * from TBLS where TBL_NAME='store_sales'
   MariaDB [hive]> SELECT * FROM TABLE_PARAMS where TBL_ID = xx
   ```
   Apologies if there exists some other SQL way to find it, I thought that having this exposed in describe table could have helped me. I agree this is not a property of the table but a state of the table. This was my rationale behind putting a pr out for this change. Would love to know your thoughts about the same.
   
   Agree with you, there exists an alternative way for us to find this as well, so it's not like it can't be worked around :
   ```java
   Table table = Spark3Util.loadIcebergTable(spark, tableName);
   String metadataJson = ((HiveTableOperations) (((HasTableOperations) table).operations())).currentMetadataLocation();
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] singhpk234 commented on pull request #5006: Spark: Show metadata.json path in `DESC TABLE EXTENDED`

Posted by GitBox <gi...@apache.org>.
singhpk234 commented on PR #5006:
URL: https://github.com/apache/iceberg/pull/5006#issuecomment-1171944839

   >  I don't think that we want to expose this detail to users, but I could be convinced otherwise
   
   Agree with you, hence this https://github.com/apache/iceberg/pull/5063 could be a better way for achieving the same, @jackye1995  also suggested the same, hence opened https://github.com/apache/iceberg/pull/5063 for the same. Apologies I forgot to close the PR.
   
   > Wouldn't you want to export from the current metastore so you don't have a duplicate table? That use case seems like a dangerous way to migrate
   
   In the case above I wanted to have duplicate tables (1 per each catalog), was bench-marking tpc-ds perf using various catalog (hive / glue). This is a very niche use though and very dangerous as well for prod use cases.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] singhpk234 commented on pull request #5006: Spark: Show metadata.json path in `DESC TABLE EXTENDED`

Posted by GitBox <gi...@apache.org>.
singhpk234 commented on PR #5006:
URL: https://github.com/apache/iceberg/pull/5006#issuecomment-1171945326

   Superceded by https://github.com/apache/iceberg/pull/5063


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] singhpk234 closed pull request #5006: Spark: Show metadata.json path in `DESC TABLE EXTENDED`

Posted by GitBox <gi...@apache.org>.
singhpk234 closed pull request #5006: Spark: Show metadata.json path in `DESC TABLE EXTENDED`
URL: https://github.com/apache/iceberg/pull/5006


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #5006: Spark: Show metadata.json path in `DESC TABLE EXTENDED`

Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #5006:
URL: https://github.com/apache/iceberg/pull/5006#issuecomment-1153320832

   @singhpk234, is this something we want to expose?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org