You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/08/19 22:11:50 UTC
[GitHub] [iceberg] ebyhr opened a new issue, #5591: Incorrect table definition after executing rollback_to_snapshot procedure
ebyhr opened a new issue, #5591:
URL: https://github.com/apache/iceberg/issues/5591
### Apache Iceberg version
0.14.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
Spark returns the latest table definition even after executing `rollback_to_snapshot` procedure.
Steps to reproduce
```
> CREATE TABLE test USING iceberg AS SELECT 1 c1;
> ALTER TABLE test ADD COLUMN c2 int;
> INSERT INTO test VALUES (1, 1);
> SELECT * FROM iceberg_test.default.test.snapshots;
2022-08-19 07:32:29.499 2770581293596517273 ...
2022-08-19 07:32:50.006 6893045681966948046 ...
> DESC iceberg_test.default.test.snapshot_id_2770581293596517273;
c1 int
# Partitioning
Not partitioned
> CALL iceberg_test.system.rollback_to_snapshot('default.test', 2770581293596517273);
> DESC iceberg_test.default.test;
c1 int
c2 int
```
The result is same even after I executed `REFRESH TABLE iceberg_test.default.test` after `rollback_to_snapshot`.
Relates to https://apache-iceberg.slack.com/archives/C025PH0G1D4/p1660895079836159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] findepi commented on issue #5591: Incorrect table definition after executing rollback_to_snapshot procedure
Posted by GitBox <gi...@apache.org>.
findepi commented on issue #5591:
URL: https://github.com/apache/iceberg/issues/5591#issuecomment-1236858872
From end-user perspective, there is a difference between
1. querying table state at given snapshot -- at least in Trino, this uses the "schema current at that time", so includes columns that have been dropped since then
2. query table state after rollback_to_snapshot -- if this uses current schema, this doesn't include columns that have been dropped since the snapshot
Now consider example
```
-- add new column
ALTER TABLE orders ADD COLUMN order_timestamp timestamp(6) with time zone;
-- feel in data for new column
UPDATE orders SET order_timestamp = CAST(json_value(order_data, '$.timestamp') AS timestamp(6) with time zone);
-- drop the now-redundant column
ALTER TABLE orders DROP COLUMN order_data;
-- imagine now that comparing this uncovered that `order_data` was encoded in a bad way, so we need to roll this all back
CALL rollback_to_snapshot(.....)
```
As a user, i would expect to see `order_data` column back in my table.
Per this issue, i understand this wouldn't be the case. As a user I would call it a data loss (and so a bug).
cc @alexjo2144 @electrum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #5591: Incorrect table definition after executing rollback_to_snapshot procedure
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #5591:
URL: https://github.com/apache/iceberg/issues/5591#issuecomment-1222816779
Schema evolution rules allow always being able to write with an older schema, so that there is no need to check whether the schema a writer is using is allowed. That cuts down on commit conflicts, but it also means that schemas should always be rolled forward and never backward. I don't think that we want to start adding extra validation to support a case where we roll back a schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #5591: Incorrect table definition after executing rollback_to_snapshot procedure
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #5591:
URL: https://github.com/apache/iceberg/issues/5591#issuecomment-1222524262
This isn't a bug. Metadata and data updates are intended to be separate, although I can see why there are cases where you'd assume that they are not.
If you update the schema and commit data in a single job, then it isn't unreasonable to assume the schema change would be rolled back. But if I concurrently add a column while someone else commits, then a rollback should be independent. Expectations can go both ways.
While expectations differ, Iceberg never rolls back to a previous schema because that operation is unsafe. For example, if someone deletes a required column and then tries to roll that back, there may have been data written without that column. You can recover the column, but you need to make it optional (or in the future, set a read default).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #5591: Incorrect table definition after executing rollback_to_snapshot procedure
Posted by GitBox <gi...@apache.org>.
Fokko commented on issue #5591:
URL: https://github.com/apache/iceberg/issues/5591#issuecomment-1222602329
@rdblue thanks for the background information. I see the point.
One question, are there situations where it is possible to do a rollback, but still have additional data being written? I would expect that the rollback would cause a conflict on a concurrent write. And then it would fail because of a schema mismatch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #5591: Incorrect table definition after executing rollback_to_snapshot procedure
Posted by GitBox <gi...@apache.org>.
Fokko commented on issue #5591:
URL: https://github.com/apache/iceberg/issues/5591#issuecomment-1222384602
@ebyhr Thanks for reporting this, this looks like a bug 🐛 I've created a PR to fix this, but I would love to get some input from other maintainers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] closed issue #5591: Incorrect table definition after executing rollback_to_snapshot procedure
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #5591: Incorrect table definition after executing rollback_to_snapshot procedure
URL: https://github.com/apache/iceberg/issues/5591
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #5591: Incorrect table definition after executing rollback_to_snapshot procedure
Posted by GitBox <gi...@apache.org>.
Fokko commented on issue #5591:
URL: https://github.com/apache/iceberg/issues/5591#issuecomment-1222001655
I'm able to reproduce it and it looks like the data is rolebacked, but the schema isn't.
```sql
%%sql
CREATE TABLE test.test USING iceberg AS SELECT 1 c1;
```
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
<table>
<thead>
<tr>
</tr>
</thead>
<tbody>
</tbody>
</table>
```sql
%%sql
ALTER TABLE test.test ADD COLUMN c2 int;
```
<table>
<thead>
<tr>
</tr>
</thead>
<tbody>
</tbody>
</table>
```sql
%%sql
INSERT INTO test.test VALUES (1, 1);
```
<table>
<thead>
<tr>
</tr>
</thead>
<tbody>
</tbody>
</table>
```sql
%%sql
SELECT * FROM test.test;
```
<table>
<thead>
<tr>
<th>c1</th>
<th>c2</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>None</td>
</tr>
<tr>
<td>1</td>
<td>1</td>
</tr>
</tbody>
</table>
```sql
%%sql
SELECT * FROM test.test.snapshots;
```
<table>
<thead>
<tr>
<th>committed_at</th>
<th>snapshot_id</th>
<th>parent_id</th>
<th>operation</th>
<th>manifest_list</th>
<th>summary</th>
</tr>
</thead>
<tbody>
<tr>
<td>2022-08-22 07:47:45.143000</td>
<td>5161623784319151399</td>
<td>None</td>
<td>append</td>
<td>s3://tabular-wh-us-west-2-dev/8bcb0838-50fc-472d-9ddb-8feb89ef5f1e/03b32560-7561-4f96-992b-01ef730946f5/metadata/snap-5161623784319151399-1-cde4c05b-670c-47dd-8860-bcd6f07756f9.avro</td>
<td>{'added-data-files': '1', 'total-equality-deletes': '0', 'added-records': '1', 'partition-summaries-included': 'true', 'partitions.': 'added-data-files=1,added-records=1,added-files-size=389', 'total-records': '1', 'spark.app.id': 'local-1661154453400', 'changed-partition-count': '1', 'total-position-deletes': '0', 'added-files-size': '389', 'total-delete-files': '0', 'total-files-size': '389', 'total-data-files': '1'}</td>
</tr>
<tr>
<td>2022-08-22 07:47:52.492000</td>
<td>2279922865032537001</td>
<td>5161623784319151399</td>
<td>append</td>
<td>s3://tabular-wh-us-west-2-dev/8bcb0838-50fc-472d-9ddb-8feb89ef5f1e/03b32560-7561-4f96-992b-01ef730946f5/metadata/snap-2279922865032537001-1-58b2d439-1420-49c2-bf06-228c77f1ebd0.avro</td>
<td>{'added-data-files': '1', 'total-equality-deletes': '0', 'added-records': '1', 'partition-summaries-included': 'true', 'partitions.': 'added-data-files=1,added-records=1,added-files-size=599', 'total-records': '2', 'spark.app.id': 'local-1661154453400', 'changed-partition-count': '1', 'total-position-deletes': '0', 'added-files-size': '599', 'total-delete-files': '0', 'total-files-size': '988', 'total-data-files': '2'}</td>
</tr>
</tbody>
</table>
```sql
%%sql
CALL system.rollback_to_snapshot('test.test', 5161623784319151399)
```
<table>
<thead>
<tr>
<th>previous_snapshot_id</th>
<th>current_snapshot_id</th>
</tr>
</thead>
<tbody>
<tr>
<td>2279922865032537001</td>
<td>5161623784319151399</td>
</tr>
</tbody>
</table>
```sql
%%sql
SELECT * FROM test.test;
```
<table>
<thead>
<tr>
<th>c1</th>
<th>c2</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>None</td>
</tr>
</tbody>
</table>
```sql
%%sql
DESCRIBE TABLE test.test;
```
<table>
<thead>
<tr>
<th>col_name</th>
<th>data_type</th>
<th>comment</th>
</tr>
</thead>
<tbody>
<tr>
<td>c1</td>
<td>int</td>
<td></td>
</tr>
<tr>
<td>c2</td>
<td>int</td>
<td></td>
</tr>
<tr>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td># Partitioning</td>
<td></td>
<td></td>
</tr>
<tr>
<td>Not partitioned</td>
<td></td>
<td></td>
</tr>
</tbody>
</table>
```sql
%%sql
SELECT * FROM test.test.snapshots;
```
<table>
<thead>
<tr>
<th>committed_at</th>
<th>snapshot_id</th>
<th>parent_id</th>
<th>operation</th>
<th>manifest_list</th>
<th>summary</th>
</tr>
</thead>
<tbody>
<tr>
<td>2022-08-22 07:47:45.143000</td>
<td>5161623784319151399</td>
<td>None</td>
<td>append</td>
<td>s3://tabular-wh-us-west-2-dev/8bcb0838-50fc-472d-9ddb-8feb89ef5f1e/03b32560-7561-4f96-992b-01ef730946f5/metadata/snap-5161623784319151399-1-cde4c05b-670c-47dd-8860-bcd6f07756f9.avro</td>
<td>{'added-data-files': '1', 'total-equality-deletes': '0', 'added-records': '1', 'partition-summaries-included': 'true', 'partitions.': 'added-data-files=1,added-records=1,added-files-size=389', 'total-records': '1', 'spark.app.id': 'local-1661154453400', 'changed-partition-count': '1', 'total-position-deletes': '0', 'added-files-size': '389', 'total-delete-files': '0', 'total-files-size': '389', 'total-data-files': '1'}</td>
</tr>
<tr>
<td>2022-08-22 07:47:52.492000</td>
<td>2279922865032537001</td>
<td>5161623784319151399</td>
<td>append</td>
<td>s3://tabular-wh-us-west-2-dev/8bcb0838-50fc-472d-9ddb-8feb89ef5f1e/03b32560-7561-4f96-992b-01ef730946f5/metadata/snap-2279922865032537001-1-58b2d439-1420-49c2-bf06-228c77f1ebd0.avro</td>
<td>{'added-data-files': '1', 'total-equality-deletes': '0', 'added-records': '1', 'partition-summaries-included': 'true', 'partitions.': 'added-data-files=1,added-records=1,added-files-size=599', 'total-records': '2', 'spark.app.id': 'local-1661154453400', 'changed-partition-count': '1', 'total-position-deletes': '0', 'added-files-size': '599', 'total-delete-files': '0', 'total-files-size': '988', 'total-data-files': '2'}</td>
</tr>
</tbody>
</table>
```sql
%%sql
REFRESH TABLE test.test
```
<table>
<thead>
<tr>
</tr>
</thead>
<tbody>
</tbody>
</table>
```sql
%%sql
SELECT * FROM test.test;
```
<table>
<thead>
<tr>
<th>c1</th>
<th>c2</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>None</td>
</tr>
</tbody>
</table>
# Waiting one minute
```sql
%%sql
SELECT * FROM test.test;
```
<table>
<thead>
<tr>
<th>c1</th>
<th>c2</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>None</td>
</tr>
</tbody>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ebyhr closed issue #5591: Incorrect table definition after executing rollback_to_snapshot procedure
Posted by GitBox <gi...@apache.org>.
ebyhr closed issue #5591: Incorrect table definition after executing rollback_to_snapshot procedure
URL: https://github.com/apache/iceberg/issues/5591
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #5591: Incorrect table definition after executing rollback_to_snapshot procedure
Posted by GitBox <gi...@apache.org>.
Fokko commented on issue #5591:
URL: https://github.com/apache/iceberg/issues/5591#issuecomment-1222309854
It looks like the `current_schema_id` is not being updated when rolling back to an earlier snapshot:
```
➜ iceberg git:(master) ✗ pyiceberg describe test.test
Table format version 1
Metadata location s3://tabular-wh-us-west-2-dev/8bcb0838-50fc-472d-9ddb-8feb89ef5f1e/03b32560-7561-4f96-992b-01ef730946f5/metadata/00003-8b6eeb0f-76fc-435f-a25f-12db7baf62ff.gz.metadata.json
Table UUID 43fe073e-9360-4f4e-8ee7-d22d4969f9d1
Last Updated 1661154486212
Partition spec []
Sort order []
Current schema Schema, id=1
├── 1: c1: optional int
└── 2: c2: optional int
Current snapshot id=5161623784319151399, schema_id=0
Snapshots Snapshots
├── Snapshot 5161623784319151399, schema 0:
│ s3://tabular-wh-us-west-2-dev/8bcb0838-50fc-472d-9ddb-8feb89ef5f1e/03b32560-7561-4f96-992b-01ef730946f5/metadata/snap-5161623784319151399-1-cde4c05b-670c-47dd-8860-bcd6f07756f9.avro
└── Snapshot 2279922865032537001, schema 1:
s3://tabular-wh-us-west-2-dev/8bcb0838-50fc-472d-9ddb-8feb89ef5f1e/03b32560-7561-4f96-992b-01ef730946f5/metadata/snap-2279922865032537001-1-58b2d439-1420-49c2-bf06-228c77f1ebd0.avro
Properties write.delete.parquet.compression-codec zstd
write.metadata.compression-codec gzip
write.summary.partition-limit 100
write.parquet.compression-codec zstd
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #5591: Incorrect table definition after executing rollback_to_snapshot procedure
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5591:
URL: https://github.com/apache/iceberg/issues/5591#issuecomment-1480411433
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ebyhr commented on issue #5591: Incorrect table definition after executing rollback_to_snapshot procedure
Posted by GitBox <gi...@apache.org>.
ebyhr commented on issue #5591:
URL: https://github.com/apache/iceberg/issues/5591#issuecomment-1235020181
> This isn't a bug.
Could you please document the behavior? I believe the current behavior isn't intuitive and I saw some other people said "it's a bug".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #5591: Incorrect table definition after executing rollback_to_snapshot procedure
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5591:
URL: https://github.com/apache/iceberg/issues/5591#issuecomment-1459066523
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org