You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/03/17 07:27:59 UTC
[GitHub] [iceberg] hereTac opened a new issue #2341: `expire_snapshots` is not worked
hereTac opened a new issue #2341:
URL: https://github.com/apache/iceberg/issues/2341
env:
Spark 3.0.2
Iceberg 0.11.0
SparkThriftserver-SparkSQL-Beeline
Steps to reproduceļ¼
`
0: jdbc:hive2://172.25.52.34:50033> drop table hive.dc.tmp_1;
+---------+
| Result |
+---------+
+---------+
No rows selected (0.188 seconds)
0: jdbc:hive2://172.25.52.34:50033> CREATE TABLE hive.dc.tmp_1( day_id string , visitor_user_id string )
. . . . . . . . . . . . . . . . . > USING iceberg
. . . . . . . . . . . . . . . . . > PARTITIONED BY (day_id)
. . . . . . . . . . . . . . . . . > TBLPROPERTIES (
. . . . . . . . . . . . . . . . . > 'compatibility.snapshot-id-inheritance.enabled'='true',
. . . . . . . . . . . . . . . . . > 'history.expire.max-snapshot-age-ms'='60000',
. . . . . . . . . . . . . . . . . > 'history.expire.min-snapshots-to-keep'='1',
. . . . . . . . . . . . . . . . . > 'write.format.default'='orc',
. . . . . . . . . . . . . . . . . > 'write.metadata.previous-versions-max'='1',
. . . . . . . . . . . . . . . . . > 'write.metadata.delete-after-commit.enabled'='true');
+---------+
| Result |
+---------+
+---------+
No rows selected (0.149 seconds)
0: jdbc:hive2://172.25.52.34:50033> delete from hive.dc.tmp_1 where day_id='2021-03-17';
+---------+
| Result |
+---------+
+---------+
No rows selected (0.331 seconds)
0: jdbc:hive2://172.25.52.34:50033> INSERT INTO hive.dc.tmp_1 SELECT to_date('2021-03-17'),'a' ;
+---------+
| Result |
+---------+
+---------+
No rows selected (0.428 seconds)
0: jdbc:hive2://172.25.52.34:50033> delete from hive.dc.tmp_1 where day_id='2021-03-17';
+---------+
| Result |
+---------+
+---------+
No rows selected (0.408 seconds)
0: jdbc:hive2://172.25.52.34:50033> INSERT INTO hive.dc.tmp_1 SELECT to_date('2021-03-17'),'a' ;
+---------+
| Result |
+---------+
+---------+
No rows selected (0.399 seconds)
0: jdbc:hive2://172.25.52.34:50033> SELECT * FROM hive.dc.tmp_1.history;
+--------------------------+----------------------+----------------------+----------------------+
| made_current_at | snapshot_id | parent_id | is_current_ancestor |
+--------------------------+----------------------+----------------------+----------------------+
| 2021-03-17 15:26:02.61 | 21042742172007942 | NULL | true |
| 2021-03-17 15:26:03.509 | 7031723713111976264 | 21042742172007942 | true |
| 2021-03-17 15:26:04.426 | 5932720827666232030 | 7031723713111976264 | true |
| 2021-03-17 15:26:04.834 | 8441372103215065079 | 5932720827666232030 | true |
+--------------------------+----------------------+----------------------+----------------------+
4 rows selected (0.113 seconds)
0: jdbc:hive2://172.25.52.34:50033> CALL hive.system.expire_snapshots(older_than => now(),table => 'dc.tmp_1', retain_last => 1);
Error: Error running query: org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input '(' expecting STRING(line 1, pos 51)
== SQL ==
CALL hive.system.expire_snapshots(older_than => now(),table => 'dc.tmp_1', retain_last => 1)
---------------------------------------------------^^^ (state=,code=0)
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hereTac commented on issue #2341: spark-procedures Metadata management expire_snapshots is not worked
Posted by GitBox <gi...@apache.org>.
hereTac commented on issue #2341:
URL: https://github.com/apache/iceberg/issues/2341#issuecomment-801578179
If it is as your description shows, then the document should be wrong, and it will make people use it incorrectly.
[https://iceberg.apache.org/spark-procedures/#expire_snapshots](url)
date_sub(current_date(), 10)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #2341: spark-procedures Metadata management expire_snapshots is not worked
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #2341:
URL: https://github.com/apache/iceberg/issues/2341#issuecomment-801578675
ah that is true, I copied that from our internal branch which allows all expressions because it uses the standard parser ... We should probably figure out a way to allow arbitrary expressions here as well
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hereTac closed issue #2341: spark-procedures Metadata management expire_snapshots is not worked
Posted by GitBox <gi...@apache.org>.
hereTac closed issue #2341:
URL: https://github.com/apache/iceberg/issues/2341
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #2341: spark-procedures Metadata management expire_snapshots is not worked
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #2341:
URL: https://github.com/apache/iceberg/issues/2341#issuecomment-801118492
Unfortunately, because we use a parser extension, the procedures do not allow arbitrary expressions to be passed to them. This is why "now()" is throwing an error.
Currently we support only constants and stringMaps,
https://github.com/apache/iceberg/blob/master/spark3-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4#L113-L116
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org