You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/03/23 14:12:59 UTC
[GitHub] [iceberg] ajantha-bhat opened a new pull request #4379: Clarify timestamp in stored procedures
ajantha-bhat opened a new pull request #4379:
URL: https://github.com/apache/iceberg/pull/4379
- Spark supports [microseconds precision]((https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/types/TimestampType.scala#L26)) but in code we cast to milliseconds, so better to warn the users.
- Some code comment was about spark precision.
- Remove generic spark expressions for timestamp in expire_snapshots example (as it is not supported)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4379: Docs: Clarify timestamp in stored procedures
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4379:
URL: https://github.com/apache/iceberg/pull/4379#discussion_r833331333
##########
File path: docs/spark/spark-procedures.md
##########
@@ -215,18 +215,12 @@ If `older_than` and `retain_last` are omitted, the table's [expiration propertie
#### Examples
-Remove snapshots older than one day, but retain the last 100 snapshots:
+Remove snapshots older than one day (assuming today is TIMESTAMP '2021-06-31 00:00:00.000'), but retain the last 100 snapshots:
Review comment:
same as above, I would change this to just
"Remove snapshots older than a specific day, but retain the last X snapshots"
Or we could put the day directly in there
"Remove snapshots older than June 30, 2021, but retain the last 100 snapshots"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on pull request #4379: Clarify timestamp in stored procedures
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on pull request #4379:
URL: https://github.com/apache/iceberg/pull/4379#issuecomment-1076425001
cc: @aokolnychyi , @RussellSpitzer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on a change in pull request #4379: Docs: Clarify timestamp in stored procedures
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #4379:
URL: https://github.com/apache/iceberg/pull/4379#discussion_r833441424
##########
File path: docs/spark/spark-procedures.md
##########
@@ -105,7 +105,7 @@ This procedure invalidates all cached Spark plans that reference the affected ta
#### Example
-Roll back `db.sample` to one day
+Roll back `db.sample` to one day before (assuming today is TIMESTAMP '2021-06-31 00:00:00.000')
Review comment:
yeah, I like this suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx merged pull request #4379: Docs: Clarify timestamp in stored procedures
Posted by GitBox <gi...@apache.org>.
openinx merged pull request #4379:
URL: https://github.com/apache/iceberg/pull/4379
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4379: Docs: Clarify timestamp in stored procedures
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4379:
URL: https://github.com/apache/iceberg/pull/4379#discussion_r833328462
##########
File path: docs/spark/spark-procedures.md
##########
@@ -198,7 +198,7 @@ the `expire_snapshots` procedure will never remove files which are still require
| Argument Name | Required? | Type | Description |
|---------------|-----------|------|-------------|
| `table` | ✔️ | string | Name of the table to update |
-| `older_than` | ️ | timestamp | Timestamp before which snapshots will be removed (Default: 5 days ago) |
+| `older_than` | ️ | timestamp | Timestamp before which snapshots will be removed (Default: 5 days ago). Considered upto millisecond precision. |
Review comment:
"up to" - although in general i'm not sure this clarification is needed, do you think this is a confusion among users?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4379: Docs: Clarify timestamp in stored procedures
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4379:
URL: https://github.com/apache/iceberg/pull/4379#discussion_r833459262
##########
File path: docs/spark/spark-procedures.md
##########
@@ -198,7 +198,7 @@ the `expire_snapshots` procedure will never remove files which are still require
| Argument Name | Required? | Type | Description |
|---------------|-----------|------|-------------|
| `table` | ✔️ | string | Name of the table to update |
-| `older_than` | ️ | timestamp | Timestamp before which snapshots will be removed (Default: 5 days ago) |
+| `older_than` | ️ | timestamp | Timestamp before which snapshots will be removed (Default: 5 days ago). Considered upto millisecond precision. |
Review comment:
I'm just not sure how often folks would expect their microsecond values to matter in snapshot expiration, I generally think about snapshots at best at a ms granularity but I think for most real cases uses are going to be on the second granularity or higher.
I just worry sometimes that we add information that is technically correct but not useful for a user setting the value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on a change in pull request #4379: Clarify timestamp in stored procedures
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #4379:
URL: https://github.com/apache/iceberg/pull/4379#discussion_r833318560
##########
File path: docs/spark/spark-procedures.md
##########
@@ -215,18 +215,12 @@ If `older_than` and `retain_last` are omitted, the table's [expiration propertie
#### Examples
-Remove snapshots older than one day, but retain the last 100 snapshots:
+Remove snapshots older than one day (assuming today is TIMESTAMP '2021-06-31 00:00:00.000'), but retain the last 100 snapshots:
```sql
CALL hive_prod.system.expire_snapshots('db.sample', TIMESTAMP '2021-06-30 00:00:00.000', 100)
```
-Erase all snapshots older than the current timestamp but retain the last 5 snapshots:
-
-```sql
Review comment:
Removed generic spark expressions for timestamp as it is not supported
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #4379: Docs: Clarify timestamp in stored procedures
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #4379:
URL: https://github.com/apache/iceberg/pull/4379#discussion_r833329362
##########
File path: docs/spark/spark-procedures.md
##########
@@ -105,7 +105,7 @@ This procedure invalidates all cached Spark plans that reference the affected ta
#### Example
-Roll back `db.sample` to one day
+Roll back `db.sample` to one day before (assuming today is TIMESTAMP '2021-06-31 00:00:00.000')
Review comment:
I would just change this to "Roll back to a specific day", just to keep it simple.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on a change in pull request #4379: Docs: Clarify timestamp in stored procedures
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #4379:
URL: https://github.com/apache/iceberg/pull/4379#discussion_r833495590
##########
File path: docs/spark/spark-procedures.md
##########
@@ -198,7 +198,7 @@ the `expire_snapshots` procedure will never remove files which are still require
| Argument Name | Required? | Type | Description |
|---------------|-----------|------|-------------|
| `table` | ✔️ | string | Name of the table to update |
-| `older_than` | ️ | timestamp | Timestamp before which snapshots will be removed (Default: 5 days ago) |
+| `older_than` | ️ | timestamp | Timestamp before which snapshots will be removed (Default: 5 days ago). Considered upto millisecond precision. |
Review comment:
ok, as snapshots itself is having creation time in milliseconds, we cannot expect to have microsecond precision.
I will remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on a change in pull request #4379: Docs: Clarify timestamp in stored procedures
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #4379:
URL: https://github.com/apache/iceberg/pull/4379#discussion_r833444028
##########
File path: docs/spark/spark-procedures.md
##########
@@ -198,7 +198,7 @@ the `expire_snapshots` procedure will never remove files which are still require
| Argument Name | Required? | Type | Description |
|---------------|-----------|------|-------------|
| `table` | ✔️ | string | Name of the table to update |
-| `older_than` | ️ | timestamp | Timestamp before which snapshots will be removed (Default: 5 days ago) |
+| `older_than` | ️ | timestamp | Timestamp before which snapshots will be removed (Default: 5 days ago). Considered upto millisecond precision. |
Review comment:
I think people assume it is microsecond precision as spark is microsecond precision, but code we are losing the precision (lossy conversion to milli), so I think better to clarify in the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ajantha-bhat commented on pull request #4379: Docs: Clarify timestamp in stored procedures
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on pull request #4379:
URL: https://github.com/apache/iceberg/pull/4379#issuecomment-1077010756
@RussellSpitzer : can you please take another look and help merging this PR ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org