You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "oneonestar (via GitHub)" <gi...@apache.org> on 2024/02/13 23:26:19 UTC
[I] Null values in metadata_log_entries [iceberg]
oneonestar opened a new issue, #9723:
URL: https://github.com/apache/iceberg/issues/9723
### Query engine
Spark
### Question
In `metadata_log_entries` table, `latest_snapshot_id, latest_schema_id, latest_sequence_number` return null in some cases. Also, those values became null after `CREATE OR REPLACE`.
The current implementation rely on `snapshot-log` field in metadata file, and `snapshot-log` got reset after after the CREATE OR REPLACE statement.
Is this an intended behavior?
Could someone provide a precise definition for `latest_snapshot_id, latest_schema_id, latest_sequence_number`?
```
spark-sql> create table test.t1 (c1 integer);
spark-sql> alter table test.t1 add columns (c2 string);
spark-sql> SELECT * FROM test.t1.metadata_log_entries;
2024-01-22 18:51:41.331 hdfs://hadoop/metadata/00000-d1ad9769-3bf3-497d-a86e-20f71d2700f7.metadata.json NULL NULL NULL
2024-01-22 18:51:41.593 hdfs://hadoop/metadata/00001-9eb3275c-f53d-4e24-bc23-fdcadae28d92.metadata.json NULL NULL NULL
spark-sql> insert into test.t1 values (1, 'a');
spark-sql> SELECT * FROM test.t1.metadata_log_entries;
2024-01-22 18:51:41.331 hdfs://hadoop/metadata/00000-d1ad9769-3bf3-497d-a86e-20f71d2700f7.metadata.json NULL NULL NULL
2024-01-22 18:51:41.593 hdfs://hadoop/metadata/00001-9eb3275c-f53d-4e24-bc23-fdcadae28d92.metadata.json NULL NULL NULL
2024-01-22 18:51:47.538 hdfs://hadoop/metadata/00002-ec3f0aa6-7dae-4c45-90ad-13839eefbd54.metadata.json 1836642618692808023 1 0
spark-sql> delete from test.t1 where c1 = 1;
spark-sql> SELECT * FROM test.t1.metadata_log_entries;
2024-01-22 18:51:41.331 hdfs://hadoop/metadata/00000-d1ad9769-3bf3-497d-a86e-20f71d2700f7.metadata.json NULL NULL NULL
2024-01-22 18:51:41.593 hdfs://hadoop/metadata/00001-9eb3275c-f53d-4e24-bc23-fdcadae28d92.metadata.json NULL NULL NULL
2024-01-22 18:51:47.538 hdfs://hadoop/metadata/00002-ec3f0aa6-7dae-4c45-90ad-13839eefbd54.metadata.json 1836642618692808023 1 0
2024-01-22 18:51:52.6 hdfs://hadoop/metadata/00003-806771f6-e7f1-44f5-ac80-350f2c084505.metadata.json 8876099574020403871 1 0
spark-sql> CALL local.system.rewrite_data_files('test.t1');
spark-sql> SELECT * FROM test.t1.metadata_log_entries;
2024-01-22 18:51:41.331 hdfs://hadoop/metadata/00000-d1ad9769-3bf3-497d-a86e-20f71d2700f7.metadata.json NULL NULL NULL
2024-01-22 18:51:41.593 hdfs://hadoop/metadata/00001-9eb3275c-f53d-4e24-bc23-fdcadae28d92.metadata.json NULL NULL NULL
2024-01-22 18:51:47.538 hdfs://hadoop/metadata/00002-ec3f0aa6-7dae-4c45-90ad-13839eefbd54.metadata.json 1836642618692808023 1 0
2024-01-22 18:51:52.6 hdfs://hadoop/metadata/00003-806771f6-e7f1-44f5-ac80-350f2c084505.metadata.json 8876099574020403871 1 0
spark-sql> create or replace table test.t1 (c3 integer);
spark-sql> SELECT * FROM test.t1.metadata_log_entries;
2024-01-22 18:51:41.331 hdfs://hadoop/metadata/00000-d1ad9769-3bf3-497d-a86e-20f71d2700f7.metadata.json NULL NULL NULL
2024-01-22 18:51:41.593 hdfs://hadoop/metadata/00001-9eb3275c-f53d-4e24-bc23-fdcadae28d92.metadata.json NULL NULL NULL
2024-01-22 18:51:47.538 hdfs://hadoop/metadata/00002-ec3f0aa6-7dae-4c45-90ad-13839eefbd54.metadata.json NULL NULL NULL
2024-01-22 18:51:52.6 hdfs://hadoop/metadata/00003-806771f6-e7f1-44f5-ac80-350f2c084505.metadata.json NULL NULL NULL
2024-01-22 18:52:01.749 hdfs://hadoop/metadata/00004-d9c74f49-37c8-4463-8565-7b31660723e2.metadata.json NULL NULL NULL
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Null values in metadata_log_entries after replacing Table [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on issue #9723:
URL: https://github.com/apache/iceberg/issues/9723#issuecomment-1952572152
@oneonestar i'm guessing that your Trino developments from https://github.com/trinodb/trino/pull/20410 can continue now because it is clear that the inconsistency found for `CREATE OR REPLACE` was linked to iceberg library internals - see #9732
Thank you for following up on this one.
Thank you @nastra for the fix. 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Null values in metadata_log_entries [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on issue #9723:
URL: https://github.com/apache/iceberg/issues/9723#issuecomment-1943095150
The short answer to your question is yes, I think this is expected behavior for the scenario you described because a replace will reset the history entries (the snapshot-log).
As for definitions:
1.) Metadata log entry -> These log entries that are stored as part of table metadata indicate what are the historical metadata files, and at which point in time the table's metadata file was that particular metadata file
2.) latest_snapshot_id -> This is the snapshot ID that was the state of the table at the time the metadata file in the log entry was the metadata file for the table. Time travel for snapshot IDs is done on the basis of the history, you can check out more details on this [here](https://github.com/apache/iceberg/pull/5364#issuecomment-1227902420) . Since the replace eliminates the previous history it's not possible to determine this snapshot so null is returned.
3.) latest_schema_id -> This is the schema associated with the snpashot in 2). If the snapshot is null, this will also be null.
4.) latest_sequence_number -> This is the sequence number associated with the snapshot in 2. If the snapshot is null, this will also be null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Null values in metadata_log_entries [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #9723:
URL: https://github.com/apache/iceberg/issues/9723#issuecomment-1946724217
I'll take a closer look at this tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Null values in metadata_log_entries after replacing Table [iceberg]
Posted by "oneonestar (via GitHub)" <gi...@apache.org>.
oneonestar commented on issue #9723:
URL: https://github.com/apache/iceberg/issues/9723#issuecomment-1951908193
@nastra @amogh-jahagirdar
Any comments on the [above issue](https://github.com/apache/iceberg/issues/9723#issuecomment-1950232583)?
Just want to confirm that is the correct behavior, so I can continue the implementation for Trino.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Null values in metadata_log_entries after replacing Table [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #9723:
URL: https://github.com/apache/iceberg/issues/9723#issuecomment-1952909031
> @amogh-jahagirdar
>
> After taken a closer look, `latest_schema_id` behaves strangely for ALTER TABLE. ALTER TABLE updates the metadata file and its `current-schema-id`, but it will not update the snapshot. If `latest_schema_id` is associated with the metadata file, the value should be 1. Now the `latest_schema_id` is associated with the snapshot in the metadata file, so the value is 0.
>
> Isn't this confusing consider that this table is `metadata_log_entries` and metadata file should be the main entity to be associated with? A user won't be able to know without documentation or a better name for the column.
>
> IMO,
>
> * `schema_id` in the metadata table should be associated with the metadata file.
> * `schema_id` associated with the snapshot should be added to the snapshot table.
>
> Consider the following example. If `latest_schema_id` in second row is 1, it will clearly show that the schema has been updated in that commit.
>
> ```
> > create table iceberg.default.t1 (c1 integer);
> > alter table iceberg.default.t1 add column c2 varchar;
> timestamp | file | latest_snapshot_id | latest_schema_id | latest_sequence_number
> ------------------------+-----------------------------------------------------------+---------------------+------------------+------------------------
> 2024-01-22 08:11:35.128 | 00000-9e475095-c0f2-4249-92b5-d744f208a624.metadata.json | 3603548602758670925 | 0 | 1
> 2024-01-22 08:11:49.279 | 00001-c357b5f7-1d55-45d8-be6d-f1bbdd0de184.metadata.json | 3603548602758670925 | 0 | 1
> (2 rows)
> ```
@oneonestar the issue was auto-closed when the PR was merged. Can you please open a separate issue for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Null values in metadata_log_entries [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #9723:
URL: https://github.com/apache/iceberg/issues/9723#issuecomment-1947988416
It appears that when we build a replacement for `TableMetadata` [here](https://github.com/apache/iceberg/blob/6e21bbf4c4cd2c8351a64636f91d05d00492dff2/core/src/main/java/org/apache/iceberg/TableMetadata.java#L703-L711) we also remove the `main` branch via `.removeRef(SnapshotRef.MAIN_BRANCH)`. This then cleans the `snapshotLog` [here](https://github.com/apache/iceberg/blob/6e21bbf4c4cd2c8351a64636f91d05d00492dff2/core/src/main/java/org/apache/iceberg/TableMetadata.java#L1249), which was introduced by #4071.
@amogh-jahagirdar do you remember why this is being done? As a comparison, when we do a `CREATE OR REPLACE VIEW` we don't clear the history of a view.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Null values in metadata_log_entries [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on issue #9723:
URL: https://github.com/apache/iceberg/issues/9723#issuecomment-1948850563
I did a little investigation as well, I'm not sure the original context on why the snapshot log was being cleared. But here are my thoughts:
It does indeed some like we should be [preserving the history during RTAS](https://iceberg.apache.org/docs/latest/spark-ddl/#replace-table-as-select). I was thinking the intention of a replace is effectively an entirely new table state with its own history tracking but probably makes more sense to preserve history in case the RTAS is unintentional and someone wants to roll it back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Null values in metadata_log_entries after replacing Table [iceberg]
Posted by "oneonestar (via GitHub)" <gi...@apache.org>.
oneonestar commented on issue #9723:
URL: https://github.com/apache/iceberg/issues/9723#issuecomment-1953473457
Ok. https://github.com/apache/iceberg/issues/9758
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Null values in metadata_log_entries [iceberg]
Posted by "findinpath (via GitHub)" <gi...@apache.org>.
findinpath commented on issue #9723:
URL: https://github.com/apache/iceberg/issues/9723#issuecomment-1943680429
> I think this is expected behavior for the scenario you described because a replace will reset the history entries (the snapshot-log)
Why would replace affect the history entries ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Null values in metadata_log_entries [iceberg]
Posted by "oneonestar (via GitHub)" <gi...@apache.org>.
oneonestar commented on issue #9723:
URL: https://github.com/apache/iceberg/issues/9723#issuecomment-1944060001
> latest_snapshot_id -> This is the snapshot ID that was the state of the table at the time the metadata file in the log entry was the metadata file for the table.
By this definition, `latest_snapshot_id` should be equal to the `current-snapshot-id` in that metadata file.
This should held true even after replace, but now the result is null.
> Since the replace eliminates the previous history it's not possible to determine this snapshot so null is returned.
We can always get the `current-snapshot-id` from that metadata file. I understand that going through all the metadata files is costly. Using timestamp to do a lookup in `metadata-log` and `snapshot-log` is a reasonable implementation decision. However, it doesn't look correct to return NULL when there is actually a `snapshot_id` exist for that metadata file.
Another way is to define `metadata_log_entries` table ONLY return the `metadata-log` lookup with `snapshot-log` in the current metadata file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org