You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "tomtongue (via GitHub)" <gi...@apache.org> on 2026/06/18 08:01:45 UTC

[PR] Core, Spark, Flink: Add `table_properties_log` metadata table [iceberg]

tomtongue opened a new pull request, #16859:
URL: https://github.com/apache/iceberg/pull/16859

   ## Overview
   
   This PR adds a read-only `table_properties_log` metadata table that exposes the history of a table's effective properties from Iceberg metadata files.
   
   For each retained metadata version, the table returns the metadata file timestamp, the metadata file location, the snapshot that was current in that metadata version, and the full `properties` map recorded in that `metadata.json`.
   
   This makes table-property history queryable from the table's own metadata. It allows users to answer questions such as which properties were in effect when a snapshot became current, when a property change appeared in table metadata, and whether a later incident or behavior change lines up with a table-property update.
   
   The change is read-only and engine-level. It does not change the table spec, does not require a format-version bump, and does not change the write path. The new table is registered through the existing metadata-table mechanism, so it is available to Spark and Flink using their existing metadata-table syntax.
   
   ## Motivation
   
   Iceberg versions table state through immutable metadata files. Schemas, partition specs, sort orders, snapshots, `snapshot-log` entries, and `metadata-log` entries are retained as part of table metadata. However, `TableMetadata.properties` is represented as the current effective property map in each metadata file. When a property is changed, the current metadata points at the new map, and there is no direct query surface for comparing that map across previous metadata versions.
   
   * The historical values are still present as long as the corresponding metadata files are retained. Each historical `metadata.json` contains the full property map for that metadata version, and `TableMetadata.previousFiles()` records the retained previous metadata files. Using that information requires locating and parsing those files manually today.
   
   * That gap matters for operational debugging and audit-style questions. For example, an operator may need to determine whether `gc.enabled` was enabled before an expiration run, whether a performance regression started after `write.target-file-size-bytes` or `write.distribution-mode` changed, or whether a read behavior change lines up with `write.delete.mode`, `write.update.mode`, or `write.merge.mode` and more.
   
   `table_properties_log` exposes the history Iceberg already retains, without adding new persisted metadata and without requiring out-of-band catalog logs.
   
   ## Use-cases
   
   The primary use-cases are point-in-time inspection of table configuration from Iceberg metadata.
   
   * **Audit and forensic checks.** Query whether a property had a specific value in a retained metadata version, including values that were later changed back.
   * **Incident RCA.** Correlate a behavior change with the metadata version where a property first changed.
   * **Performance tuning.** Verify that tuning properties took effect and compare the values used before and after a change.
   * **Engine-agnostic access.** Use the same core metadata table from Spark and Flink through the existing metadata-table resolution path.
   
   ## Changes
   
   This PR makes the following changes:
   
   * Adds `TablePropertiesLogTable` in core.
     * The table is modeled on `MetadataLogEntriesTable` and the existing static metadata-table pattern.
     * It reads `TableMetadata.previousFiles()` and appends the current metadata file as the latest row.
     * Each row is built by reading the corresponding `metadata.json` with `TableMetadataParser.read`.
   
   * Adds `MetadataTableType.TABLE_PROPERTIES_LOG`.
     * The metadata table can be resolved by name through `MetadataTableType.from`.
   
   * Registers `TABLE_PROPERTIES_LOG` in `MetadataTableUtils`.
     * Spark can query it as `<table>.table_properties_log`.
     * Flink can query it as `<table>$table_properties_log`.
     * No Spark-specific or Flink-specific implementation is needed.
   
   * Exposes the following schema:
   
     | Column | Type | Description |
     | --- | --- | --- |
     | `timestamp` | `timestamp with zone` | Timestamp of the metadata log entry |
     | `file` | `string` | Location of the `metadata.json` file |
     | `latest_snapshot_id` | `long` | Snapshot that was current in that metadata version; `null` before the first snapshot |
     | `properties` | `map<string, string>` | Effective table properties stored in that metadata version |
   
   * Add the tests verifying the metadata table query results to Spark 4.1 and Flink 2.1
   
   Note that:
   
   * The visible history is bounded by retained metadata files. In practice this is controlled by metadata retention settings such as `write.metadata.previous-versions-max` and whether old metadata files are deleted after commit.
   * The table reads one retained `metadata.json` per emitted metadata version. This is bounded by the metadata log exposed by the current metadata, not by every metadata file ever written for the table.
   * Predicate filtering is handled by the metadata-table scan path and engines; this PR does not add predicate pushdown into historical metadata-file reads.
   * `latest_snapshot_id` is the snapshot current in that metadata version. A metadata-only property update can therefore produce a row with the same `latest_snapshot_id` as the previous row.
   
   ### Example: Query on TablePropertiesLogTable with Spark
   
   ```sql
   CREATE TABLE props_log (id int, name string) USING iceberg TBLPROPERTIES ('key1'='value1')";
   
   SELECT * FROM props_log.table_properties_log;
   /*
   +-----------------------+---------------------------------------------------------------------------------------------+------------------+-------------------------------------------------------------------------+
   |timestamp              |file                                                                                         |latest_snapshot_id|properties                                                               |
   +-----------------------+---------------------------------------------------------------------------------------------+------------------+-------------------------------------------------------------------------+
   |2026-06-17 15:24:33.798|s3://warehouse/db/props_log/metadata/00000-059abc2d-e741-4465-ae11-121b8023d380.metadata.json|NULL              |{key1 -> value1, owner -> spark, write.parquet.compression-codec -> zstd}|
   +-----------------------+---------------------------------------------------------------------------------------------+------------------+-------------------------------------------------------------------------+
   */
   
   ALTER TABLE db.props_log SET TBLPROPERTIES('key2'='value2');
   
   INSERT INTO db.props_log VALUES (1, 'a');
   
   /*
   +-----------------------+---------------------------------------------------------------------------------------------+-------------------+-----------------------------------------------------------------------------------------+
   |timestamp              |file                                                                                         |latest_snapshot_id |properties                                                                               |
   +-----------------------+---------------------------------------------------------------------------------------------+-------------------+-----------------------------------------------------------------------------------------+
   |2026-06-17 15:24:33.798|s3://warehouse/db/props_log/metadata/00000-059abc2d-e741-4465-ae11-121b8023d380.metadata.json|NULL               |{key1 -> value1, owner -> spark, write.parquet.compression-codec -> zstd}                |
   |2026-06-17 15:25:17.637|s3://warehouse/db/props_log/metadata/00001-28f3023b-423f-4b4c-bc18-ef93596baffe.metadata.json|NULL               |{key1 -> value1, owner -> spark, key2 -> value2, write.parquet.compression-codec -> zstd}|
   |2026-06-17 15:26:09.574|s3://warehouse/db/props_log/metadata/00002-6b48f0eb-a243-4146-9890-0400deba828b.metadata.json|1312307084484665830|{key1 -> value1, owner -> spark, key2 -> value2, write.parquet.compression-codec -> zstd}|
   +-----------------------+---------------------------------------------------------------------------------------------+-------------------+-----------------------------------------------------------------------------------------+
   
   */
   ```
   
   ## Alternatives Considered
   
   * **Record table properties into each snapshot summary at commit time.**: This was the original design direction, and the full design is captured in [Preserving Table Property History in Snapshot Summary](https://docs.google.com/document/d/1w3dQ_-m6rqSpsaLGuXTSUYVBNQn3UiYBCWEgt9ztTJo/edit?tab=t.0). That approach would make property history available from `<table>.snapshots` and tie values directly to snapshot retention. However, it requires a write-path change and a spec addition for reserved snapshot-summary keys. It also needs pointer compression or another size-control mechanism to avoid repeating the full property map on every snapshot. The current PR avoids those concerns by exposing property history already present in retained metadata files.
   
   * **Add a new top-level property-history field to `TableMetadata`.**: Another option is to add a dedicated history structure to table metadata, such as a list of property-map versions or property deltas. That would make the history independent of metadata-log retention and could support direct point lookups without reading historical metadata files. However, it would introduce new persisted table metadata that every reader must preserve correctly, and it would require defining retention, compatibility, and upgrade behavior for that new structure. It is also heavier than necessary for this PR because Iceberg already stores the complete property map in each retained `metadata.json`. A read-only metadata table exposes that existing information without changing the table metadata format.
   
   ## Testing
   
   Run the following specific tests:
   
   - `./gradlew :iceberg-spark:iceberg-spark-extensions-4.1_2.13:test --tests "org.apache.iceberg.spark.extensions.TestMetadataTables.testTablePropertiesLog"`
   - `./gradlew :iceberg-flink:iceberg-flink-2.1:test --tests "org.apache.iceberg.flink.source.TestFlinkMetaDataTable.testTablePropertiesLog"`
   
   ## AI Assistance
   
   I used Claude Code and Codex while preparing this PR. For the code changes, I used them to explore the existing codebase and to review my implementation, including checking whether it followed existing coding patterns, whether the implementation and wording were consistent with nearby code, and whether there were simpler alternatives. For the PR description, I used them as reviewers to refine the text and point out missing context. I reviewed the final code and PR text myself and am responsible for the submitted changes.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r3853884110


##########
spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMetadataTables.java:
##########
@@ -644,19 +644,22 @@ public void testMetadataLogEntries() throws Exception {
                 metadataLogEntries.get(0).file(),
                 null,
                 null,
-                null),
+                null,
+                tableMetadata.properties()),

Review Comment:
   Added Spark and Flink tests for each version.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#issuecomment-5709817879

   @szehon-ho (cc: @laskoviymishka @singhpk234 ) Thanks so much for the detailed review. I addressed your comments by using `PROPERTIES` and `shouldLoadProperties `, and adding parallel processing implementation. 
   
   When you have a chance, could you review again?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add `table_properties_log` metadata table [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#issuecomment-4752482642

   @szehon-ho @singhpk234, could I ask for your review if you have time?
   
   (@szehon-ho , tagging you because you have reviewed or authored several metadata-table changes. and @singhpk234, tagging you because this PR is closely modeled on `MetadataLogEntriesTable`, which you originally added.)
   
   The main thing I’d like feedback on is whether the user-facing table name/schema and the approach of reading retained metadata files are reasonable for a new read-only metadata table.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r3851239686


##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -119,6 +140,32 @@ private static StaticDataTask.Row metadataLogEntryToRow(
         // latest snapshot in this file corresponding to the log entry
         latestSnapshotId,
         latestSnapshot != null ? latestSnapshot.schemaId() : null,
-        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null);
+        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null,
+        properties);
+  }
+
+  private static Map<String, String> tablePropertiesResolver(

Review Comment:
   Thanks for pointing this out! Totally makes sense. I'll use `loadTableProperties` instead of `tablePropertiesResolver`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add `table_properties_log` metadata table [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#issuecomment-5124831755

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r3843389313


##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -119,6 +140,32 @@ private static StaticDataTask.Row metadataLogEntryToRow(
         // latest snapshot in this file corresponding to the log entry
         latestSnapshotId,
         latestSnapshot != null ? latestSnapshot.schemaId() : null,
-        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null);
+        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null,
+        properties);
+  }
+
+  private static Map<String, String> tablePropertiesResolver(
+      TableMetadata.MetadataLogEntry metadataLogEntry,
+      FileIO io,
+      TableMetadata current,
+      boolean skipPropertiesLoad) {
+
+    // Avoid loading metadata file when properties are not projected.
+    if (skipPropertiesLoad) {
+      return null;
+    }
+
+    // Reuse the already loaded current metadata.
+    if (metadataLogEntry.file().equals(current.metadataFileLocation())) {
+      return current.properties();
+    }

Review Comment:
   Nice — these two cover the cheap cases well. One more scenario worth a look, not blocking: on a `SELECT *`, or a filter like `WHERE properties['x'] = 'y'`, neither optimization kicks in, so a table with deep metadata history ends up reading and parsing every historical `metadata.json`, one at a time. That's roughly the shape a monitoring job polling this table would hit. Might be worth a javadoc note about the cost at least, or resolving the metadata once per scan — wdyt?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r3853757331


##########
spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMetadataTables.java:
##########
@@ -644,19 +644,22 @@ public void testMetadataLogEntries() throws Exception {
                 metadataLogEntries.get(0).file(),
                 null,
                 null,
-                null),
+                null,
+                tableMetadata.properties()),

Review Comment:
   Thanks. initially I thought testing the behavior in the core package should be enough, but maybe tests in Spark and Flink are should be added. Let me add those tests.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r3854624011


##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -119,6 +140,32 @@ private static StaticDataTask.Row metadataLogEntryToRow(
         // latest snapshot in this file corresponding to the log entry
         latestSnapshotId,
         latestSnapshot != null ? latestSnapshot.schemaId() : null,
-        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null);
+        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null,
+        properties);
+  }
+
+  private static Map<String, String> tablePropertiesResolver(
+      TableMetadata.MetadataLogEntry metadataLogEntry,
+      FileIO io,
+      TableMetadata current,
+      boolean skipPropertiesLoad) {
+
+    // Avoid loading metadata file when properties are not projected.
+    if (skipPropertiesLoad) {
+      return null;
+    }
+
+    // Reuse the already loaded current metadata.
+    if (metadataLogEntry.file().equals(current.metadataFileLocation())) {
+      return current.properties();
+    }
+
+    try {
+      return TableMetadataParser.read(io, metadataLogEntry.file()).properties();
+    } catch (NotFoundException e) {

Review Comment:
   Thanks for calling this out. I chose the current behavior and I think it looks appropriate. Missing historical files can be expected due to metadata cleanup or else, so returning NULL with a warning is reasonable. A `RuntimeIOException` indicates corruption or an access issue, so it should surface instead of being treated as a missing file. I’ll keep the current behavior as is, if there's no other issues.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add `table_properties_log` metadata table [iceberg]

Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r3814630901


##########
core/src/main/java/org/apache/iceberg/MetadataTableUtils.java:
##########
@@ -70,6 +70,8 @@ private static Table createMetadataTableInstance(
         return new SnapshotsTable(baseTable, metadataTableName);
       case METADATA_LOG_ENTRIES:
         return new MetadataLogEntriesTable(baseTable, metadataTableName);
+      case TABLE_PROPERTIES_LOG:
+        return new TablePropertiesLogTable(baseTable, metadataTableName);

Review Comment:
   I don't hold a very strong opinion here, I am fine either ways :), but sharing my thoughts here I feel as far as adding new columns to existing tables is concerned we have done that couple of times !
   partitions
   
   - spec_id — https://github.com/apache/iceberg/pull/4516
   - last_updated_at, last_updated_snapshot_id — https://github.com/apache/iceberg/pull/7581
   - total_data_file_size_in_bytes — https://github.com/apache/iceberg/pull/7920
   - rename of last-updated timestamp column — https://github.com/apache/iceberg/pull/8003
   
   manifests / all_manifests
   - content + added/existing/deleted_delete_files_count — https://github.com/apache/iceberg/pull/4764
   - reference_snapshot_id (all_manifests) — https://github.com/apache/iceberg/pull/4847
   
   we can also just say its null ... if the file is missing, also our default max version is 100, i don't expect that much of an overhead !
   
   
   > Yea my concern is mainly if there's something sensitive in table properties that we'd have to guard more now
   
   +1, infact in EMR we had to gaurd this all the way up to these metadata tables : https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-lf-limitations-cont.html , even paths are sensitive as sometime we have partition_col in the file name ... which can reveal which partition are there, but i feel this is more of the platform responsibility, from the glance of the table props defined nothing sensistive comes up to me, but folks can always add **anything** in the table prop, platforms may feel maintain an allowlist the props they wanna show though 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add `table_properties_log` metadata table [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#issuecomment-5336006275

   @szehon-ho this is the dev mailing list thread: https://lists.apache.org/thread/73tvsbmfnfhsow0nyq2lrblwbmbw00rb
   
   Regarding masking the table properties, I was also thinking about it, and as you mentioned, currently anyone can access the table property with `DESCRIBE EXTENDED`, so I considered it out of scope for this PR. But, if anyone considers this a risk, please feel free to discuss it here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "szehon-ho (via GitHub) via issues" <is...@iceberg.apache.org>.
szehon-ho commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r4461632417


##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -57,20 +84,44 @@ public Schema schema() {
     return METADATA_LOG_ENTRIES_SCHEMA;
   }
 
-  private DataTask task(TableScan scan) {
+  private DataTask task(StaticTableScan scan) {
     TableMetadata current = table().operations().current();
     List<TableMetadata.MetadataLogEntry> metadataLogEntries =
         Lists.newArrayList(current.previousFiles().listIterator());
     metadataLogEntries.add(
         new TableMetadata.MetadataLogEntry(
             current.lastUpdatedMillis(), current.metadataFileLocation()));
+    FileIO io = table().io();
+    Schema projectedSchema = scan.schema();
+    boolean shouldLoadProperties = projectedSchema.findField(PROPERTIES.fieldId()) != null;
+    ConcurrentMap<String, Map<String, String>> propertiesByMetadataFile = Maps.newConcurrentMap();
+
+    if (shouldLoadProperties) {
+      propertiesByMetadataFile.put(current.metadataFileLocation(), current.properties());
+
+      Tasks.Builder<TableMetadata.MetadataLogEntry> propertyLoadTasks =
+          Tasks.foreach(current.previousFiles()).stopOnFailure();

Review Comment:
   Could we apply the `file`/`timestamp` predicates before scheduling these reads? A query selecting properties for just the current metadata file still reads every historical file. `ExpressionUtil.extractByIdInclusive` and `Evaluator` could filter the entries before both prefetching and row assembly.



##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -119,6 +172,19 @@ private static StaticDataTask.Row metadataLogEntryToRow(
         // latest snapshot in this file corresponding to the log entry
         latestSnapshotId,
         latestSnapshot != null ? latestSnapshot.schemaId() : null,
-        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null);
+        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null,
+        properties);
+  }
+
+  private static Map<String, String> loadTableProperties(
+      TableMetadata.MetadataLogEntry metadataLogEntry, FileIO io) {
+    try {
+      return TableMetadataParser.read(io, metadataLogEntry.file()).properties();
+    } catch (NotFoundException e) {

Review Comment:
   Could we ensure missing historical files also return null with `OSSFileIO`? `OSSInputStream.openStream()` lets the SDK’s `OSSException` with `NoSuchKey` escape, so this catch won’t handle a deleted object and the whole query fails. Normalizing that error to `NotFoundException` in the OSS reader and adding backend coverage would preserve the intended fallback without hiding other failures.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r4004263617


##########
spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMetadataTables.java:
##########
@@ -644,19 +644,22 @@ public void testMetadataLogEntries() throws Exception {
                 metadataLogEntries.get(0).file(),
                 null,
                 null,
-                null),
+                null,
+                tableMetadata.properties()),

Review Comment:
   Thanks for calling this out. Based on your comment, I considered using concrete expected maps and adding a comment as below:
   * Concrete maps would not add meaningful history coverage because all entries in this test intentionally have identical properties, and they would couple the test to unrelated default-property changes. 
   * A comment would repeat what the immediately following `testMetadataLogEntriesPropertyHistory` already makes explicit. 
   
   For `testMetadataLogEntries`, I would prefer to keep the existing assertions as-is. This test checks that Spark returns the `properties` column correctly for both `SELECT *` and filtered queries. The separate `testMetadataLogEntriesPropertyHistory` checks that each metadata version returns the properties stored in that version, so the two tests cover different behavior.
   
   I believe this separation provides the intended coverage. If you still see a gap, please let me know and I’ll update the tests.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r3853504245


##########
core/src/test/java/org/apache/iceberg/TestMetadataLogEntriesTableProperties.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+class TestMetadataLogEntriesTableProperties {
+  private static final String PROPERTY = "key";
+  private static final String INITIAL_VALUE = "old";
+  private static final String UPDATED_VALUE = "new";
+  private static final TableIdentifier TABLE_IDENTIFIER = TableIdentifier.of("ns", "table");
+  private static final Schema SCHEMA =
+      new Schema(Types.NestedField.required(1, "id", Types.IntegerType.get()));
+
+  private InMemoryCatalog catalog;
+  private BaseTable table;
+  private Map<String, String> initialProperties;
+  private Map<String, String> updatedProperties;
+
+  @BeforeEach
+  public void setupTableWithProperties() {
+    this.catalog = new InMemoryCatalog();
+    catalog.initialize("test", ImmutableMap.of());
+    catalog.createNamespace(TABLE_IDENTIFIER.namespace());
+    this.table =
+        (BaseTable)
+            catalog.createTable(
+                TABLE_IDENTIFIER,
+                SCHEMA,
+                PartitionSpec.unpartitioned(),
+                ImmutableMap.of(PROPERTY, INITIAL_VALUE));
+    this.initialProperties = ImmutableMap.copyOf(table.properties());
+    table.updateProperties().set(PROPERTY, UPDATED_VALUE).commit();
+    this.updatedProperties = ImmutableMap.copyOf(table.properties());
+  }
+
+  @AfterEach
+  public void after() throws IOException {
+    catalog.dropTable(TABLE_IDENTIFIER);
+    catalog.close();
+  }
+
+  @Test
+  public void loadsHistoricalPropertiesAndReusesCurrentMetadata() throws IOException {
+    TableMetadata current = table.operations().current();
+    TableMetadata.MetadataLogEntry previous = Iterables.getOnlyElement(current.previousFiles());
+    FileIO io = spy(table.io());
+
+    DataTask task = planTask(metadataLogEntriesTable(current, io).newScan().select("properties"));
+
+    assertThat(firstColumnValues(task)).containsExactly(initialProperties, updatedProperties);
+    verify(io).newInputFile(previous.file());
+    verify(io).newInputFile(current.metadataFileLocation());

Review Comment:
   Thanks for reviewing this part as well. I add `times(1)` to validate that just calling it once, and when it's called twice, the test fails. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add `table_properties_log` metadata table [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#issuecomment-5352872091

   @szehon-ho @singhpk234 I remove the `table_properties_log` and instead, add the `properties` to `metadata_log_entries` based on our discussion. Could you review the changes again when you have a chance? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add `table_properties_log` metadata table [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r3819413240


##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -119,6 +140,32 @@ private static StaticDataTask.Row metadataLogEntryToRow(
         // latest snapshot in this file corresponding to the log entry
         latestSnapshotId,
         latestSnapshot != null ? latestSnapshot.schemaId() : null,
-        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null);
+        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null,
+        properties);
+  }
+
+  private static Map<String, String> tablePropertiesResolver(
+      TableMetadata.MetadataLogEntry metadataLogEntry,
+      FileIO io,
+      TableMetadata current,
+      boolean skipPropertiesLoad) {
+
+    // Avoid loading metadata file when properties are not projected.
+    if (skipPropertiesLoad) {
+      return null;
+    }
+
+    // Reuse the already loaded current metadata.
+    if (metadataLogEntry.file().equals(current.metadataFileLocation())) {
+      return current.properties();
+    }

Review Comment:
   This part adds two optimizations:
   
   * First, if the metadata table query doesn't include the `properties` column (e.g., `SELECT file FROM catalog.db.tbl.metadata_log_entries`), `skipPropertiesLoad` is set to `true`, and no metadata files are loaded to retrieve properties.
   * Second, if the metadata log entry points to the current metadata file, the properties from the already loaded current metadata are returned without loading the metadata file again.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r4006282857


##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -119,6 +150,32 @@ private static StaticDataTask.Row metadataLogEntryToRow(
         // latest snapshot in this file corresponding to the log entry
         latestSnapshotId,
         latestSnapshot != null ? latestSnapshot.schemaId() : null,
-        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null);
+        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null,
+        properties);
+  }
+
+  private static Map<String, String> loadTableProperties(
+      TableMetadata.MetadataLogEntry metadataLogEntry,
+      FileIO io,
+      TableMetadata current,
+      boolean skipPropertiesLoad) {
+
+    // Avoid loading metadata file when properties are not projected.
+    if (skipPropertiesLoad) {
+      return null;
+    }
+
+    // Reuse the already loaded current metadata.
+    if (metadataLogEntry.file().equals(current.metadataFileLocation())) {
+      return current.properties();
+    }
+
+    try {
+      return TableMetadataParser.read(io, metadataLogEntry.file()).properties();
+    } catch (NotFoundException e) {

Review Comment:
   Thanks for raising this. I carefully considered your comment but I would prefer to keep the fallback limited to missing historical metadata files. For other read failures, returning `null` would let the query succeed with incomplete property history. This potentially hides a storage or metadata problem. Propagating the error makes the failure visible to the caller.
   
   Catching `RuntimeIOException` also would not cover all unreadable-file cases. For example, some storage errors propagate as SDK exceptions, and invalid metadata can raise validation exceptions. Catching all RuntimeExceptions would additionally risk suppressing programming errors.
   
   So I decided to keep `NotFoundException` as the only exception that produces `null` and a warning, and added a comment explaining that fallback. I also updated the test to verify that `RuntimeIOException` propagates, while retaining the missing-file test. I believe this keeps the behavior aligned with the PR’s documented handling of missing historical metadata files.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r4021505326


##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -19,20 +19,43 @@
 package org.apache.iceberg;
 
 import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.exceptions.NotFoundException;
 import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileIO;
 import org.apache.iceberg.relocated.com.google.common.collect.Lists;
 import org.apache.iceberg.types.Types;
 import org.apache.iceberg.util.SnapshotUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
+/**
+ * A {@link Table} implementation that exposes a table's metadata log as rows.
+ *
+ * <p>Each row represents a historical or current metadata file and includes the snapshot details
+ * and table properties recorded in that file. The current metadata is included as the latest row.
+ *
+ * <p>Queries that reference {@code properties} read each retained historical metadata file. These
+ * additional reads are skipped when {@code properties} is not referenced, and the already loaded
+ * current metadata is reused.
+ */
 public class MetadataLogEntriesTable extends BaseMetadataTable {
 
+  private static final Logger LOG = LoggerFactory.getLogger(MetadataLogEntriesTable.class);
+
+  private static final int PROPERTIES_FIELD_ID = 6;

Review Comment:
   Could we define `PROPERTIES` as a `Types.NestedField` constant and use `PROPERTIES.fieldId()` here? That keeps the field definition and projection check together, while making a separate static assertion unnecessary. A positive `shouldLoadProperties` boolean may also read more clearly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add `table_properties_log` metadata table [iceberg]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#issuecomment-5321336025

   i think it sounds useful, is there a devlist discussion that you end up making?.  any thoughts @singhpk234 ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add `table_properties_log` metadata table [iceberg]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#issuecomment-5334634107

   Sorry, do you have a link to the devlist?  i seem not to find it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r4004531745


##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -19,20 +19,43 @@
 package org.apache.iceberg;
 
 import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.exceptions.NotFoundException;
 import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileIO;
 import org.apache.iceberg.relocated.com.google.common.collect.Lists;
 import org.apache.iceberg.types.Types;
 import org.apache.iceberg.util.SnapshotUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
+/**
+ * A {@link Table} implementation that exposes a table's metadata log as rows.
+ *
+ * <p>Each row represents a historical or current metadata file and includes the snapshot details
+ * and table properties recorded in that file. The current metadata is included as the latest row.
+ *
+ * <p>Queries that reference {@code properties} read each retained historical metadata file. These
+ * additional reads are skipped when {@code properties} is not referenced, and the already loaded
+ * current metadata is reused.
+ */
 public class MetadataLogEntriesTable extends BaseMetadataTable {
 
+  private static final int PROPERTIES_FIELD_ID = 6;

Review Comment:
   Thanks, I totally agree with this validation. I will add this to the next commit.
   And, thanks for pointing out the `LOG` location. Will correct the location.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r4006146932


##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -19,20 +19,43 @@
 package org.apache.iceberg;
 
 import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.exceptions.NotFoundException;
 import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileIO;
 import org.apache.iceberg.relocated.com.google.common.collect.Lists;
 import org.apache.iceberg.types.Types;
 import org.apache.iceberg.util.SnapshotUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
+/**
+ * A {@link Table} implementation that exposes a table's metadata log as rows.
+ *
+ * <p>Each row represents a historical or current metadata file and includes the snapshot details
+ * and table properties recorded in that file. The current metadata is included as the latest row.
+ *
+ * <p>Queries that reference {@code properties} read each retained historical metadata file. These
+ * additional reads are skipped when {@code properties} is not referenced, and the already loaded
+ * current metadata is reused.
+ */
 public class MetadataLogEntriesTable extends BaseMetadataTable {
 
+  private static final int PROPERTIES_FIELD_ID = 6;

Review Comment:
   @laskoviymishka  I revisited adding this check and I would like to revise my previous response.
   
   The `properties` field is declared with `PROPERTIES_FIELD_ID`. And ,the projection check determining whether to load `properties` uses that same constant:
   
   ```java
   // L93 in the current commit
       boolean skipPropertiesLoad = projectedSchema.findField(PROPERTIES_FIELD_ID) == null;
   ```
   
   Field IDs are explicitly assigned, and `findField(int)` looks up a field by ID. Inserting or reordering fields therefore does not change which field the constant identifies.
   
   The assertion could catch a future edit that changes the declaration independently. But I don't think that additional runtime check is needed with the shared constant here. I plan to remove the static block and its unused `Preconditions` import, and keep the `LOG` placement change.
   
   If you see any issues with this approach, please let me know, and I’ll update the implementation accordingly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r3961172200


##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -19,20 +19,43 @@
 package org.apache.iceberg;
 
 import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.exceptions.NotFoundException;
 import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileIO;
 import org.apache.iceberg.relocated.com.google.common.collect.Lists;
 import org.apache.iceberg.types.Types;
 import org.apache.iceberg.util.SnapshotUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
+/**
+ * A {@link Table} implementation that exposes a table's metadata log as rows.
+ *
+ * <p>Each row represents a historical or current metadata file and includes the snapshot details
+ * and table properties recorded in that file. The current metadata is included as the latest row.
+ *
+ * <p>Queries that reference {@code properties} read each retained historical metadata file. These
+ * additional reads are skipped when {@code properties} is not referenced, and the already loaded
+ * current metadata is reused.
+ */
 public class MetadataLogEntriesTable extends BaseMetadataTable {
 
+  private static final int PROPERTIES_FIELD_ID = 6;

Review Comment:
   `PROPERTIES_FIELD_ID = 6` drives both the schema declaration and the `findField(PROPERTIES_FIELD_ID) == null` skip, but nothing checks the two agree — if a field ever gets inserted before `properties`, the constant silently points elsewhere and the skip mis-fires. A small static assertion that `METADATA_LOG_ENTRIES_SCHEMA.findField(PROPERTIES_FIELD_ID)` is non-null and named `properties` would keep them honest.
   
   Tiny thing while we're here: `LOG` usually sits at the top of the constant block by convention.



##########
spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMetadataTables.java:
##########
@@ -644,19 +644,22 @@ public void testMetadataLogEntries() throws Exception {
                 metadataLogEntries.get(0).file(),
                 null,
                 null,
-                null),
+                null,
+                tableMetadata.properties()),

Review Comment:
   these new `properties` assertions all compare against `tableMetadata.properties()`, which is the current (final) map — and since `testMetadataLogEntries` never changes a property, every historical row happens to equal current. So this stays green even if `loadTableProperties` just returned `current.properties()` for every entry.
   
   The real history coverage now lives in `testMetadataLogEntriesPropertyHistory`, so I wouldn't block on this. I'd either assert the concrete expected map per row here, or drop a one-line comment noting the properties are identical across entries in this test so it's clear the divergence check lives elsewhere. Same in the Flink `testMetadataLogEntries` copies. wdyt?



##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -119,6 +150,32 @@ private static StaticDataTask.Row metadataLogEntryToRow(
         // latest snapshot in this file corresponding to the log entry
         latestSnapshotId,
         latestSnapshot != null ? latestSnapshot.schemaId() : null,
-        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null);
+        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null,
+        properties);
+  }
+
+  private static Map<String, String> loadTableProperties(
+      TableMetadata.MetadataLogEntry metadataLogEntry,
+      FileIO io,
+      TableMetadata current,
+      boolean skipPropertiesLoad) {
+
+    // Avoid loading metadata file when properties are not projected.
+    if (skipPropertiesLoad) {
+      return null;
+    }
+
+    // Reuse the already loaded current metadata.
+    if (metadataLogEntry.file().equals(current.metadataFileLocation())) {
+      return current.properties();
+    }
+
+    try {
+      return TableMetadataParser.read(io, metadataLogEntry.file()).properties();
+    } catch (NotFoundException e) {

Review Comment:
   we only catch `NotFoundException` here, but `TableMetadataParser.read` rethrows other IO problems as `RuntimeIOException` — a present-but-unreadable file (permissions, a truncated/corrupt JSON, a transient read error) escapes and fails the whole query instead of degrading to null+warn like the missing-file case.
   
   Since this is already best-effort, I'd broaden the catch (`RuntimeException`, or at least `RuntimeIOException`) and log the class+message. If we'd rather keep it narrow, worth a comment saying only missing files degrade gracefully. wdyt?



##########
flink/v2.3/flink/src/test/java/org/apache/iceberg/flink/source/TestFlinkMetaDataTable.java:
##########
@@ -667,6 +667,23 @@ public void testMetadataLogEntries() {
     }
   }
 
+  @TestTemplate

Review Comment:
   v2.3 only picked up the new `testMetadataLogEntriesPropertyHistory` — the four `getField("properties")` assertions that landed in v1.20/2.1/2.2's `testMetadataLogEntries` didn't make it into this module. Same base commit, so it looks like a copy that got missed rather than a real version difference. I'd add the same four here so the base-case coverage matches across Flink versions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add `table_properties_log` metadata table [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#issuecomment-4786386981

   @nastra (sorry for mentioning) I thought you might be a good reviewer because you have reviewed related metadata-table scan support and `TableMetadata` / `metadata-log` timestamp behavior. If you have time, could I ask you to review this idea and PR? If I should share this with the community first, please let me know. 
   
   @szehon-ho @singhpk234, I'd also appreciate your review on this if you have time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r3843662425


##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -119,6 +140,32 @@ private static StaticDataTask.Row metadataLogEntryToRow(
         // latest snapshot in this file corresponding to the log entry
         latestSnapshotId,
         latestSnapshot != null ? latestSnapshot.schemaId() : null,
-        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null);
+        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null,
+        properties);
+  }
+
+  private static Map<String, String> tablePropertiesResolver(

Review Comment:
   Small thing: `tablePropertiesResolver` reads as a noun, but it's a static helper doing an action, and every other method in this file is a verb phrase (`metadataLogEntryToRow`, `task`, `planFiles`). I'd rename to `resolveTableProperties` (or `loadTableProperties`) — the call site then reads as `loadTableProperties(metadataLogEntry, io, current, skipPropertiesLoad)`.



##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -119,6 +140,32 @@ private static StaticDataTask.Row metadataLogEntryToRow(
         // latest snapshot in this file corresponding to the log entry
         latestSnapshotId,
         latestSnapshot != null ? latestSnapshot.schemaId() : null,
-        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null);
+        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null,
+        properties);
+  }
+
+  private static Map<String, String> tablePropertiesResolver(
+      TableMetadata.MetadataLogEntry metadataLogEntry,
+      FileIO io,
+      TableMetadata current,
+      boolean skipPropertiesLoad) {
+
+    // Avoid loading metadata file when properties are not projected.
+    if (skipPropertiesLoad) {
+      return null;
+    }
+
+    // Reuse the already loaded current metadata.
+    if (metadataLogEntry.file().equals(current.metadataFileLocation())) {
+      return current.properties();
+    }
+
+    try {
+      return TableMetadataParser.read(io, metadataLogEntry.file()).properties();
+    } catch (NotFoundException e) {

Review Comment:
   The null-on-missing-file behavior looks intentional and matches what you noted in the design thread — no issue there. What I'm less sure about is the other failure modes: `TableMetadataParser.read` can also throw `RuntimeIOException` (corrupt or partially-written metadata, permission denied), and those propagate and fail the whole scan. Since the use case is post-incident RCA, that's exactly when a historical file is most likely inconsistent. Do you want those to surface, or fold into the same null+warn path? Either's fine — just worth being deliberate about which.



##########
spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMetadataTables.java:
##########
@@ -644,19 +644,22 @@ public void testMetadataLogEntries() throws Exception {
                 metadataLogEntries.get(0).file(),
                 null,
                 null,
-                null),
+                null,
+                tableMetadata.properties()),

Review Comment:
   These assertions don't quite exercise the feature. The expected value is `tableMetadata.properties()` — the current map — for every row, including the historical ones, and nothing here changes a property between metadata versions, so every metadata file carries the same set. An impl that just returned `current.properties()` for every row regardless of history would pass all of these.
   
   The core unit test does cover the old→new case, so this isn't a correctness gap — but the engine-level tests give no regression protection for the one thing the column is for. In at least one of them I'd `ALTER TABLE ... SET TBLPROPERTIES` between two appends and assert the earlier row shows the old value and the later row the new one.
   
   (Same applies to the Spark v4.0/v4.1 copies and the Flink `TestFlinkMetaDataTable` versions — they share this pattern.)



##########
core/src/test/java/org/apache/iceberg/TestMetadataLogEntriesTableProperties.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+class TestMetadataLogEntriesTableProperties {
+  private static final String PROPERTY = "key";
+  private static final String INITIAL_VALUE = "old";
+  private static final String UPDATED_VALUE = "new";
+  private static final TableIdentifier TABLE_IDENTIFIER = TableIdentifier.of("ns", "table");
+  private static final Schema SCHEMA =
+      new Schema(Types.NestedField.required(1, "id", Types.IntegerType.get()));
+
+  private InMemoryCatalog catalog;
+  private BaseTable table;
+  private Map<String, String> initialProperties;
+  private Map<String, String> updatedProperties;
+
+  @BeforeEach
+  public void setupTableWithProperties() {
+    this.catalog = new InMemoryCatalog();
+    catalog.initialize("test", ImmutableMap.of());
+    catalog.createNamespace(TABLE_IDENTIFIER.namespace());
+    this.table =
+        (BaseTable)
+            catalog.createTable(
+                TABLE_IDENTIFIER,
+                SCHEMA,
+                PartitionSpec.unpartitioned(),
+                ImmutableMap.of(PROPERTY, INITIAL_VALUE));
+    this.initialProperties = ImmutableMap.copyOf(table.properties());
+    table.updateProperties().set(PROPERTY, UPDATED_VALUE).commit();
+    this.updatedProperties = ImmutableMap.copyOf(table.properties());
+  }
+
+  @AfterEach
+  public void after() throws IOException {
+    catalog.dropTable(TABLE_IDENTIFIER);
+    catalog.close();
+  }
+
+  @Test
+  public void loadsHistoricalPropertiesAndReusesCurrentMetadata() throws IOException {
+    TableMetadata current = table.operations().current();
+    TableMetadata.MetadataLogEntry previous = Iterables.getOnlyElement(current.previousFiles());
+    FileIO io = spy(table.io());
+
+    DataTask task = planTask(metadataLogEntriesTable(current, io).newScan().select("properties"));
+
+    assertThat(firstColumnValues(task)).containsExactly(initialProperties, updatedProperties);
+    verify(io).newInputFile(previous.file());
+    verify(io).newInputFile(current.metadataFileLocation());

Review Comment:
   `verify(io).newInputFile(current.metadataFileLocation())` passes because `StaticDataTask.of(...)` calls `newInputFile` once at construction — not because the reuse fast path did anything. So this reads like it's checking the current-metadata reuse, but it's really asserting a side effect of an unrelated code path; if `StaticDataTask` ever stopped calling `newInputFile` on construction, it'd still pass with the optimization broken.
   
   `doesNotLoadPropertiesWhenNotProjected` uses `verify(io, never())` cleanly — could we assert the reuse the same way (no second read of the current file), or at least add a comment on the invariant? The missing-metadata test is well structured.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add `table_properties_log` metadata table [iceberg]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r3817891209


##########
core/src/main/java/org/apache/iceberg/MetadataTableUtils.java:
##########
@@ -70,6 +70,8 @@ private static Table createMetadataTableInstance(
         return new SnapshotsTable(baseTable, metadataTableName);
       case METADATA_LOG_ENTRIES:
         return new MetadataLogEntriesTable(baseTable, metadataTableName);
+      case TABLE_PROPERTIES_LOG:
+        return new TablePropertiesLogTable(baseTable, metadataTableName);

Review Comment:
   Thank you @singhpk234 for sharing the experience (especially wrt security) , im ok then to see the POC.  I suppose by default the spark/flink engines wont mask anything



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r4033624170


##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -119,6 +150,33 @@ private static StaticDataTask.Row metadataLogEntryToRow(
         // latest snapshot in this file corresponding to the log entry
         latestSnapshotId,
         latestSnapshot != null ? latestSnapshot.schemaId() : null,
-        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null);
+        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null,
+        properties);
+  }
+
+  private static Map<String, String> loadTableProperties(
+      TableMetadata.MetadataLogEntry metadataLogEntry,
+      FileIO io,
+      TableMetadata current,
+      boolean skipPropertiesLoad) {
+
+    // Avoid loading metadata file when properties are not projected.
+    if (skipPropertiesLoad) {
+      return null;
+    }
+
+    // Reuse the already loaded current metadata.
+    if (metadataLogEntry.file().equals(current.metadataFileLocation())) {
+      return current.properties();
+    }
+
+    try {
+      return TableMetadataParser.read(io, metadataLogEntry.file()).properties();

Review Comment:
   Thanks for this suggestion as well. I update this part with supporting parallel reading metadata files. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add `table_properties_log` metadata table [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#issuecomment-5352574076

   In addition to those tests, I also tested querying `metadata_log_entries` table from Spark on local. Here's the results and they worked fine.
   
   ```sql
   > CREATE TABLE db.t (id bigint, data string) USING iceberg
   > INSERT INTO db.t VALUES (1, 'a')
   
   > SELECT * FROM db.t.metadata_log_entries
   /*
   +-----------------------+-------------------------------------------------------------------------------------+------------------+----------------+----------------------+---------------------------------------------------------+
   |timestamp              |file                                                                                 |latest_snapshot_id|latest_schema_id|latest_sequence_number|properties                                               |
   +-----------------------+-------------------------------------------------------------------------------------+------------------+----------------+----------------------+---------------------------------------------------------+
   |2026-08-20 06:47:00.286|s3://warehouse/db/t/metadata/00000-4c91df99-f3eb-489c-950e-55415224b460.metadata.json|NULL              |NULL            |NULL                  |{owner -> spark, write.parquet.compression-codec -> zstd}|
   |2026-08-20 06:47:40.675|s3://warehouse/db/t/metadata/00001-8298c092-34a4-4bd1-bf38-5e5c905f04b0.metadata.json|82363968092352335 |0               |1                     |{owner -> spark, write.parquet.compression-codec -> zstd}|
   +-----------------------+-------------------------------------------------------------------------------------+------------------+----------------+----------------------+---------------------------------------------------------+
   
   spark.sql("ALTER TABLE db.t SET TBLPROPERTIES ('key1'='value')")
   spark.sql("SELECT * FROM db.t.metadata_log_entries").show(false)
   +-----------------------+-------------------------------------------------------------------------------------+------------------+----------------+----------------------+------------------------------------------------------------------------+
   |timestamp              |file                                                                                 |latest_snapshot_id|latest_schema_id|latest_sequence_number|properties                                                              |
   +-----------------------+-------------------------------------------------------------------------------------+------------------+----------------+----------------------+------------------------------------------------------------------------+
   |2026-08-20 06:47:00.286|s3://warehouse/db/t/metadata/00000-4c91df99-f3eb-489c-950e-55415224b460.metadata.json|NULL              |NULL            |NULL                  |{owner -> spark, write.parquet.compression-codec -> zstd}               |
   |2026-08-20 06:47:40.675|s3://warehouse/db/t/metadata/00001-8298c092-34a4-4bd1-bf38-5e5c905f04b0.metadata.json|82363968092352335 |0               |1                     |{owner -> spark, write.parquet.compression-codec -> zstd}               |
   |2026-08-20 06:49:29.391|s3://warehouse/db/t/metadata/00002-41e0419e-beec-4ff1-944f-06c41e961f95.metadata.json|82363968092352335 |0               |1                     |{owner -> spark, key1 -> value, write.parquet.compression-codec -> zstd}|
   +-----------------------+-------------------------------------------------------------------------------------+------------------+----------------+----------------------+------------------------------------------------------------------------+
   */
   
   > ALTER TABLE db.t SET TBLPROPERTIES ('key2'='value2')
   > SELECT * FROM db.t.metadata_log_entries
   
   /*
   +-----------------------+-------------------------------------------------------------------------------------+------------------+----------------+----------------------+----------------------------------------------------------------------------------------+
   |timestamp              |file                                                                                 |latest_snapshot_id|latest_schema_id|latest_sequence_number|properties                                                                              |
   +-----------------------+-------------------------------------------------------------------------------------+------------------+----------------+----------------------+----------------------------------------------------------------------------------------+
   |2026-08-20 06:47:00.286|s3://warehouse/db/t/metadata/00000-4c91df99-f3eb-489c-950e-55415224b460.metadata.json|NULL              |NULL            |NULL                  |{owner -> spark, write.parquet.compression-codec -> zstd}                               |
   |2026-08-20 06:47:40.675|s3://warehouse/db/t/metadata/00001-8298c092-34a4-4bd1-bf38-5e5c905f04b0.metadata.json|82363968092352335 |0               |1                     |{owner -> spark, write.parquet.compression-codec -> zstd}                               |
   |2026-08-20 06:49:29.391|s3://warehouse/db/t/metadata/00002-41e0419e-beec-4ff1-944f-06c41e961f95.metadata.json|82363968092352335 |0               |1                     |{owner -> spark, key1 -> value, write.parquet.compression-codec -> zstd}                |
   |2026-08-20 06:49:59.146|s3://warehouse/db/t/metadata/00003-ad8a3f69-a622-4c10-aabe-c083e724b15f.metadata.json|82363968092352335 |0               |1                     |{owner -> spark, key1 -> value, key2 -> value2, write.parquet.compression-codec -> zstd}|
   +-----------------------+-------------------------------------------------------------------------------------+------------------+----------------+----------------------+----------------------------------------------------------------------------------------+
   */
   
   > ALTER TABLE db.t UNSET TBLPROPERTIES ('key1')
   > SELECT * FROM db.t.metadata_log_entries
   
   /*
   +-----------------------+-------------------------------------------------------------------------------------+------------------+----------------+----------------------+----------------------------------------------------------------------------------------+
   |timestamp              |file                                                                                 |latest_snapshot_id|latest_schema_id|latest_sequence_number|properties                                                                              |
   +-----------------------+-------------------------------------------------------------------------------------+------------------+----------------+----------------------+----------------------------------------------------------------------------------------+
   |2026-08-20 06:47:00.286|s3://warehouse/db/t/metadata/00000-4c91df99-f3eb-489c-950e-55415224b460.metadata.json|NULL              |NULL            |NULL                  |{owner -> spark, write.parquet.compression-codec -> zstd}                               |
   |2026-08-20 06:47:40.675|s3://warehouse/db/t/metadata/00001-8298c092-34a4-4bd1-bf38-5e5c905f04b0.metadata.json|82363968092352335 |0               |1                     |{owner -> spark, write.parquet.compression-codec -> zstd}                               |
   |2026-08-20 06:49:29.391|s3://warehouse/db/t/metadata/00002-41e0419e-beec-4ff1-944f-06c41e961f95.metadata.json|82363968092352335 |0               |1                     |{owner -> spark, key1 -> value, write.parquet.compression-codec -> zstd}                |
   |2026-08-20 06:49:59.146|s3://warehouse/db/t/metadata/00003-ad8a3f69-a622-4c10-aabe-c083e724b15f.metadata.json|82363968092352335 |0               |1                     |{owner -> spark, key1 -> value, key2 -> value2, write.parquet.compression-codec -> zstd}|
   |2026-08-20 06:50:40.097|s3://warehouse/db/t/metadata/00004-7bd63877-5476-464b-afa6-2c2fdff2113b.metadata.json|82363968092352335 |0               |1                     |{owner -> spark, key2 -> value2, write.parquet.compression-codec -> zstd}               |
   +-----------------------+-------------------------------------------------------------------------------------+------------------+----------------+----------------------+----------------------------------------------------------------------------------------+
   */
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add `table_properties_log` metadata table [iceberg]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r3809361046


##########
core/src/main/java/org/apache/iceberg/MetadataTableUtils.java:
##########
@@ -70,6 +70,8 @@ private static Table createMetadataTableInstance(
         return new SnapshotsTable(baseTable, metadataTableName);
       case METADATA_LOG_ENTRIES:
         return new MetadataLogEntriesTable(baseTable, metadataTableName);
+      case TABLE_PROPERTIES_LOG:
+        return new TablePropertiesLogTable(baseTable, metadataTableName);

Review Comment:
   Yes I think its cleaner.  Yea my concern is mainly if there's something sensitive in table properties that we'd have to guard more now.  Not sure if @singhpk234 has any thought on that



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#issuecomment-6484265605

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r3851798578


##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -119,6 +140,32 @@ private static StaticDataTask.Row metadataLogEntryToRow(
         // latest snapshot in this file corresponding to the log entry
         latestSnapshotId,
         latestSnapshot != null ? latestSnapshot.schemaId() : null,
-        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null);
+        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null,
+        properties);
+  }
+
+  private static Map<String, String> tablePropertiesResolver(
+      TableMetadata.MetadataLogEntry metadataLogEntry,
+      FileIO io,
+      TableMetadata current,
+      boolean skipPropertiesLoad) {
+
+    // Avoid loading metadata file when properties are not projected.
+    if (skipPropertiesLoad) {
+      return null;
+    }
+
+    // Reuse the already loaded current metadata.
+    if (metadataLogEntry.file().equals(current.metadataFileLocation())) {
+      return current.properties();
+    }

Review Comment:
   Thanks for catching this up. I agreed your feedback. I'll try to add this specification to both Javadoc and spark/flink documents first. If either of them or none of them should be added, we can discuss it. Additionally, the `MetadataLogEntriesTable` class doesn't have the jacadoc, so maybe it's better to add the description about this class. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add `table_properties_log` metadata table [iceberg]

Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r3800187024


##########
core/src/main/java/org/apache/iceberg/MetadataTableUtils.java:
##########
@@ -70,6 +70,8 @@ private static Table createMetadataTableInstance(
         return new SnapshotsTable(baseTable, metadataTableName);
       case METADATA_LOG_ENTRIES:
         return new MetadataLogEntriesTable(baseTable, metadataTableName);
+      case TABLE_PROPERTIES_LOG:
+        return new TablePropertiesLogTable(baseTable, metadataTableName);

Review Comment:
   i wonder, what if we include`properties` to the metadata_log table ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add `table_properties_log` metadata table [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#issuecomment-5322182368

   @szehon-ho Thanks for taking a look at this PR and for your feedback. I’m currently working through the helpful comments from the dev-list discussion (many thanks to @tanmayrauth and @laskoviymishka as well). Apologies for my delayed response. I’ll address the feedback in the PR and reply to the dev-list thread by tomorrow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r3853757331


##########
spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMetadataTables.java:
##########
@@ -644,19 +644,22 @@ public void testMetadataLogEntries() throws Exception {
                 metadataLogEntries.get(0).file(),
                 null,
                 null,
-                null),
+                null,
+                tableMetadata.properties()),

Review Comment:
   Thanks. initially I thought testing the behavior in the core package should be enough, but maybe tests in Spark and Flink are should be added. Let me add thoese tests.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r4021505326


##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -19,20 +19,43 @@
 package org.apache.iceberg;
 
 import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.exceptions.NotFoundException;
 import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileIO;
 import org.apache.iceberg.relocated.com.google.common.collect.Lists;
 import org.apache.iceberg.types.Types;
 import org.apache.iceberg.util.SnapshotUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
+/**
+ * A {@link Table} implementation that exposes a table's metadata log as rows.
+ *
+ * <p>Each row represents a historical or current metadata file and includes the snapshot details
+ * and table properties recorded in that file. The current metadata is included as the latest row.
+ *
+ * <p>Queries that reference {@code properties} read each retained historical metadata file. These
+ * additional reads are skipped when {@code properties} is not referenced, and the already loaded
+ * current metadata is reused.
+ */
 public class MetadataLogEntriesTable extends BaseMetadataTable {
 
+  private static final Logger LOG = LoggerFactory.getLogger(MetadataLogEntriesTable.class);
+
+  private static final int PROPERTIES_FIELD_ID = 6;

Review Comment:
   Could we define `properties` as a `Types.NestedField` constant and use `PROPERTIES.fieldId()` here? That keeps the field definition and projection check together, while making a separate static assertion unnecessary. A positive `shouldLoadProperties` boolean may also read more clearly.



##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -119,6 +150,33 @@ private static StaticDataTask.Row metadataLogEntryToRow(
         // latest snapshot in this file corresponding to the log entry
         latestSnapshotId,
         latestSnapshot != null ? latestSnapshot.schemaId() : null,
-        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null);
+        latestSnapshot != null ? latestSnapshot.sequenceNumber() : null,
+        properties);
+  }
+
+  private static Map<String, String> loadTableProperties(
+      TableMetadata.MetadataLogEntry metadataLogEntry,
+      FileIO io,
+      TableMetadata current,
+      boolean skipPropertiesLoad) {
+
+    // Avoid loading metadata file when properties are not projected.
+    if (skipPropertiesLoad) {
+      return null;
+    }
+
+    // Reuse the already loaded current metadata.
+    if (metadataLogEntry.file().equals(current.metadataFileLocation())) {
+      return current.properties();
+    }
+
+    try {
+      return TableMetadataParser.read(io, metadataLogEntry.file()).properties();

Review Comment:
   Could we prefetch historical properties with `Tasks.foreach`, using the scan’s planning executor when parallel planning is enabled? Results could be stored by metadata-file location and rows assembled afterward in their original order. This would parallelize remote reads while retaining Iceberg’s existing executor controls and error handling.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r4023645021


##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -19,20 +19,43 @@
 package org.apache.iceberg;
 
 import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.exceptions.NotFoundException;
 import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileIO;
 import org.apache.iceberg.relocated.com.google.common.collect.Lists;
 import org.apache.iceberg.types.Types;
 import org.apache.iceberg.util.SnapshotUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
+/**
+ * A {@link Table} implementation that exposes a table's metadata log as rows.
+ *
+ * <p>Each row represents a historical or current metadata file and includes the snapshot details
+ * and table properties recorded in that file. The current metadata is included as the latest row.
+ *
+ * <p>Queries that reference {@code properties} read each retained historical metadata file. These
+ * additional reads are skipped when {@code properties} is not referenced, and the already loaded
+ * current metadata is reused.
+ */
 public class MetadataLogEntriesTable extends BaseMetadataTable {
 
+  private static final Logger LOG = LoggerFactory.getLogger(MetadataLogEntriesTable.class);
+
+  private static final int PROPERTIES_FIELD_ID = 6;

Review Comment:
   thanks for the suggestion, I agree with this. Let me update this part by adding `PROPERTIES` and change the parameter name to `shouldLoadProperties`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#issuecomment-5665969902

   @laskoviymishka Thanks so much for the detailed review. I addressed your comments and responded in regards to Exception handling and field ID handling.
   
   For the follow-ups, 
   
   > missing historical metadata files are handled gracefully, but other IO errors still fail the query
   
   After reconsidering the error handling, I decided to keep this behavior like missing historical metadata files produce null and a warning, while other read failures propagate to the caller as I commended above.
   
   > since this exposes full historical properties, it would be good to track the same surface in iceberg-go/PyIceberg and add a short docs note about sensitive or credential-like properties
   
   I thought that once this PR is merged, I start implementing other modules. Thanks for the suggestions. For the secret table properties, I'm going to make and share a doc based on the current implementation.
   
   ---
   @singhpk234 @szehon-ho If you have a chance, could you review the new changes I applied, and share your feedback?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add `table_properties_log` metadata table [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#issuecomment-5199157653

   Still discussing in dev email thread. Please keep this pr opened.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r4003819853


##########
flink/v2.3/flink/src/test/java/org/apache/iceberg/flink/source/TestFlinkMetaDataTable.java:
##########
@@ -667,6 +667,23 @@ public void testMetadataLogEntries() {
     }
   }
 
+  @TestTemplate

Review Comment:
   Oh sorry I missed the tests, thanks so much for pointing this out. I'll add those tests to v2.3.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r4006146932


##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -19,20 +19,43 @@
 package org.apache.iceberg;
 
 import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.exceptions.NotFoundException;
 import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileIO;
 import org.apache.iceberg.relocated.com.google.common.collect.Lists;
 import org.apache.iceberg.types.Types;
 import org.apache.iceberg.util.SnapshotUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
+/**
+ * A {@link Table} implementation that exposes a table's metadata log as rows.
+ *
+ * <p>Each row represents a historical or current metadata file and includes the snapshot details
+ * and table properties recorded in that file. The current metadata is included as the latest row.
+ *
+ * <p>Queries that reference {@code properties} read each retained historical metadata file. These
+ * additional reads are skipped when {@code properties} is not referenced, and the already loaded
+ * current metadata is reused.
+ */
 public class MetadataLogEntriesTable extends BaseMetadataTable {
 
+  private static final int PROPERTIES_FIELD_ID = 6;

Review Comment:
   I revisited adding this check and I would like to revise my previous response.
   
   The `properties` field is declared with `PROPERTIES_FIELD_ID`. And ,the projection check determining whether to load `properties` uses that same constant:
   
   ```java
   // L93 in the current commit
       boolean skipPropertiesLoad = projectedSchema.findField(PROPERTIES_FIELD_ID) == null;
   ```
   
   Field IDs are explicitly assigned, and `findField(int)` looks up a field by ID. Inserting or reordering fields therefore does not change which field the constant identifies.
   
   The assertion could catch a future edit that changes the declaration independently. But I don't think that additional runtime check is needed with the shared constant here. I plan to remove the static block and its unused Preconditions import, and keep the LOG placement change.
   
   If you see any issues with this approach, please let me know, and I’ll update the implementation accordingly.



##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -19,20 +19,43 @@
 package org.apache.iceberg;
 
 import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.exceptions.NotFoundException;
 import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileIO;
 import org.apache.iceberg.relocated.com.google.common.collect.Lists;
 import org.apache.iceberg.types.Types;
 import org.apache.iceberg.util.SnapshotUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
+/**
+ * A {@link Table} implementation that exposes a table's metadata log as rows.
+ *
+ * <p>Each row represents a historical or current metadata file and includes the snapshot details
+ * and table properties recorded in that file. The current metadata is included as the latest row.
+ *
+ * <p>Queries that reference {@code properties} read each retained historical metadata file. These
+ * additional reads are skipped when {@code properties} is not referenced, and the already loaded
+ * current metadata is reused.
+ */
 public class MetadataLogEntriesTable extends BaseMetadataTable {
 
+  private static final int PROPERTIES_FIELD_ID = 6;

Review Comment:
   @laskoviymishka  I revisited adding this check and I would like to revise my previous response.
   
   The `properties` field is declared with `PROPERTIES_FIELD_ID`. And ,the projection check determining whether to load `properties` uses that same constant:
   
   ```java
   // L93 in the current commit
       boolean skipPropertiesLoad = projectedSchema.findField(PROPERTIES_FIELD_ID) == null;
   ```
   
   Field IDs are explicitly assigned, and `findField(int)` looks up a field by ID. Inserting or reordering fields therefore does not change which field the constant identifies.
   
   The assertion could catch a future edit that changes the declaration independently. But I don't think that additional runtime check is needed with the shared constant here. I plan to remove the static block and its unused Preconditions import, and keep the LOG placement change.
   
   If you see any issues with this approach, please let me know, and I’ll update the implementation accordingly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add `table_properties_log` metadata table [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r3809119990


##########
core/src/main/java/org/apache/iceberg/MetadataTableUtils.java:
##########
@@ -70,6 +70,8 @@ private static Table createMetadataTableInstance(
         return new SnapshotsTable(baseTable, metadataTableName);
       case METADATA_LOG_ENTRIES:
         return new MetadataLogEntriesTable(baseTable, metadataTableName);
+      case TABLE_PROPERTIES_LOG:
+        return new TablePropertiesLogTable(baseTable, metadataTableName);

Review Comment:
   @singhpk234  Thanks so much for the review and suggestion. I think adding `properties` to `metadata_log_entries` is feasible, and it has the benefit of not introducing another metadata table.
   
   Regarding adding the `properties` column to the table, I believe there are a few concerns:
   
   * Even if `properties` is added as an optional field, it changes the schema of an existing metadata table. This may affect existing users (I don't think many use this `metadata_log_entries` table in their operations), especially queries such as `SELECT * ... UNION ...` that rely on the current number or order of columns.
   * To return historical properties, we still need to read the metadata file referenced by each entry. We need to decide how to handle missing or unreadable files, for example whether to fail the whole scan or keep the row with `properties = null`. This is the same read-cost and failure-handling concern raised on the dev list.
   * This also introduces additional I/O to obtain table properties by accessing each metadata file. I plan to make the loading projection-aware so that historical metadata files are not read when `properties` is not required, and to avoid unnecessary reads where possible. However, an unfiltered `SELECT *` will still need to read all retained metadata files.
   
   I would keep the current semantics of the existing columns and only add the new `properties` field.
   
   I’ll try implementing this approach to compare between two patterns of implementations. Does this direction sound reasonable? Please let me know if you see any other concerns (in addition to the concern @szehon-ho mentioned below).
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#issuecomment-5412879278

   @laskoviymishka Thanks so much for the detailed review. I addressed your comments in the latest changes.
   @szehon-ho @singhpk234 If you have a chance, I’d also appreciate any feedback on the current implementation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add `table_properties_log` metadata table [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r3817800257


##########
core/src/main/java/org/apache/iceberg/MetadataTableUtils.java:
##########
@@ -70,6 +70,8 @@ private static Table createMetadataTableInstance(
         return new SnapshotsTable(baseTable, metadataTableName);
       case METADATA_LOG_ENTRIES:
         return new MetadataLogEntriesTable(baseTable, metadataTableName);
+      case TABLE_PROPERTIES_LOG:
+        return new TablePropertiesLogTable(baseTable, metadataTableName);

Review Comment:
   @singhpk234 Thanks for checking each concern and sharing the related PRs. @szehon-ho also, thanks for raising the masking concern.
   
   From those PRs, adding `properties` as the optional column sounds reasonable and should have limited impact.
   
   Regarding masking or allowlisting all versions of table props, I also think it's better handled at the platform or catalog layer as the table access management. 
   
   Based on those feedback, I'll proceed with adding an optional `properties` column to the existing `metadata_log_entries` table first. Also it will return `null` when a referenced metadata file is missing. If there's any issues while implementing it, I'll share with you!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark, Flink: Add table properties to metadata log entries [iceberg]

Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#issuecomment-6488223143

   This is still awaiting for the review. @szehon-ho @singhpk234 Could you review this PR?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org