You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Ayush Saxena (Jira)" <ji...@apache.org> on 2023/11/30 10:16:00 UTC
[jira] [Commented] (HIVE-27903) Iceberg: Implement Expire Snapshot with default table properties
[ https://issues.apache.org/jira/browse/HIVE-27903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17791550#comment-17791550 ]
Ayush Saxena commented on HIVE-27903:
-------------------------------------
Committed to master.
Thanx [~jkcldr] for the report & [~zhangbutao] for the review!!!
> Iceberg: Implement Expire Snapshot with default table properties
> ----------------------------------------------------------------
>
> Key: HIVE-27903
> URL: https://issues.apache.org/jira/browse/HIVE-27903
> Project: Hive
> Issue Type: Improvement
> Components: Hive
> Affects Versions: 4.0.0-alpha-2
> Reporter: JK Pasimuthu
> Assignee: Ayush Saxena
> Priority: Major
> Labels: pull-request-available
>
> [https://github.com/apache/iceberg/issues/9123]
> The 'history.expire.max-snapshot-age-ms' option doesn't have any effect while expiring snapshots.
> #
> CREATE TABLE IF NOT EXISTS test5d78b6 (
> id INT, random1 STRING
> )
> PARTITIONED BY (random2 STRING)
> STORED BY ICEBERG
> TBLPROPERTIES (
> 'write.format.default'='orc',
> 'format-version'='2',
> 'write.orc.compression-codec'='none'
> )
> # INSERT INTO test5d78b6 SELECT if(isnull(MAX(id)) ,0 , MAX(id) ) +1, uuid(), uuid() FROM test5d78b6
> # INSERT INTO test5d78b6 SELECT if(isnull(MAX(id)) ,0 , MAX(id) ) +1, uuid(), uuid() FROM test5d78b6
> # SLEEP for 30 seconds
> # INSERT INTO test5d78b6 SELECT if(isnull(MAX(id)) ,0 , MAX(id) ) +1, uuid(), uuid() FROM test5d78b6
> # INSERT INTO test5d78b6 SELECT if(isnull(MAX(id)) ,0 , MAX(id) ) +1, uuid(), uuid() FROM test5d78b6
> # SELECT (UNIX_TIMESTAMP(CURRENT_TIMESTAMP) - UNIX_TIMESTAMP('2023-10-09 13:23:54.455')) * 1000;
> # ALTER TABLE test5d78b6 SET tblproperties('history.expire.max-snapshot-age-ms'='54000'); - the elapsed time in ms from the second insert and current time
> # ALTER TABLE test5d78b6 EXECUTE expire_snapshots('2200-10-10');
> # SELECT COUNT FROM default.test5d78b6.snapshots;
> output: 1. it should be 2 rows. The default 1 is retained an all snapshots are expired as usual, so setting the property has no effect.
> Additional Info: the default value for 'history.expire.max-snapshot-age-ms' is 5 days per this link: [https://iceberg.apache.org/docs/1.3.1/configuration/]
> Now while writing the tests and running them, the expiring snapshots just worked fine within few seconds of the snapshots being created.
> So, I'm assuming that this option doesn't have any effect right now. Having said that, I'm thinking the implications on end user will have if we fix this.
> The end user may not know about this option at all and will have tough time figuring out why the snapshots are not getting expired. One option could be to set the default to 0ms.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)