You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "amogh-jahagirdar (via GitHub)" <gi...@apache.org> on 2025/03/21 05:45:05 UTC

[PR] Core, Spark: Add row lineage metadata columns, and surface them in SparkTable metadata columns [iceberg]

amogh-jahagirdar opened a new pull request, #12596:
URL: https://github.com/apache/iceberg/pull/12596

   This change adds
   
   1. Row id and last updated sequence number metadata columns with field IDs as defined per the spec https://iceberg.apache.org/spec/#reserved-field-ids
   
   2. Exposes these metadata columns in SparkTable for V3 tables. Added a test which ensures that the columns are resolved for V3+ tables, and fail for tables prior to V3.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Add row lineage metadata columns, and surface them in SparkTable metadata columns [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12596:
URL: https://github.com/apache/iceberg/pull/12596#discussion_r2008112104


##########
core/src/main/java/org/apache/iceberg/MetadataColumns.java:
##########
@@ -93,21 +93,34 @@ private MetadataColumns() {}
           "_commit_snapshot_id",
           Types.LongType.get(),
           "Commit snapshot ID");
+  public static final NestedField ROW_ID =
+      NestedField.optional(
+          Integer.MAX_VALUE - 107, "_row_id", Types.LongType.get(), "Row ID for row lineage");

Review Comment:
   Should we state that this is an implicit row ID that is automatically assigned, rather than just "Row ID"? I want to make it clear that people shouldn't expect to be able to set this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Add row lineage metadata columns, and surface them in SparkTable metadata columns [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #12596:
URL: https://github.com/apache/iceberg/pull/12596#issuecomment-2755153435

   Rerunning Spark CI, it failed for an unrelated reason:
   
   ```
   TestCompressionSettings > initializationError FAILED
       java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:41801
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Add row lineage metadata columns, and surface them in SparkTable metadata columns [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #12596:
URL: https://github.com/apache/iceberg/pull/12596#issuecomment-2778926621

   Went ahead and merged since the vote for setting row lineage to always be on in V3 passed, and the spec change was merged. Thank you @rdblue @nastra for reviewing! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Add row lineage metadata columns, and surface them in SparkTable metadata columns [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #12596:
URL: https://github.com/apache/iceberg/pull/12596#discussion_r2006888010


##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkMetadataColumns.java:
##########
@@ -311,6 +317,22 @@ public void testConflictingColumns() {
         sql("SELECT _spec_id, _partition, _renamed_spec_id FROM %s", TABLE_NAME));
   }
 
+  @TestTemplate
+  public void testRowLineageColumnsResolvedInV3OrHigher() {
+    if (formatVersion >= 3) {
+      assertEquals(
+          "Rows must match",
+          ImmutableList.of(),
+          sql("SELECT _row_id, _last_updated_sequence_number, id FROM %s", TABLE_NAME));

Review Comment:
   I intentionally chose an empty table, since the goal was to ensure that the columns could be resolved in V3 or higher. If we insert data , the assertions would later have to change when we add inheritance , which seemed like an unnecessary burden considering the scope of this test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Add row lineage metadata columns, and surface them in SparkTable metadata columns [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #12596:
URL: https://github.com/apache/iceberg/pull/12596#discussion_r2014658672


##########
core/src/main/java/org/apache/iceberg/MetadataColumns.java:
##########
@@ -93,21 +93,34 @@ private MetadataColumns() {}
           "_commit_snapshot_id",
           Types.LongType.get(),
           "Commit snapshot ID");
+  public static final NestedField ROW_ID =
+      NestedField.optional(
+          Integer.MAX_VALUE - 107, "_row_id", Types.LongType.get(), "Row ID for row lineage");
+  public static final NestedField LAST_UPDATED_SEQUENCE_NUMBER =
+      NestedField.optional(
+          Integer.MAX_VALUE - 108,
+          "_last_updated_sequence_number",
+          Types.LongType.get(),
+          "Last updated sequence number for row lineage");

Review Comment:
   Fixed this!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Add row lineage metadata columns, and surface them in SparkTable metadata columns [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12596:
URL: https://github.com/apache/iceberg/pull/12596#discussion_r2008126466


##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkMetadataColumns.java:
##########
@@ -311,6 +317,22 @@ public void testConflictingColumns() {
         sql("SELECT _spec_id, _partition, _renamed_spec_id FROM %s", TABLE_NAME));
   }
 
+  @TestTemplate
+  public void testRowLineageColumnsResolvedInV3OrHigher() {
+    if (formatVersion >= 3) {
+      assertEquals(
+          "Rows must match",
+          ImmutableList.of(),
+          sql("SELECT _row_id, _last_updated_sequence_number, id FROM %s", TABLE_NAME));

Review Comment:
   Seems worth a comment to explain?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Add row lineage metadata columns, and surface them in SparkTable metadata columns [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12596:
URL: https://github.com/apache/iceberg/pull/12596#discussion_r2008122597


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java:
##########
@@ -256,13 +258,23 @@ public Set<TableCapability> capabilities() {
   @Override
   public MetadataColumn[] metadataColumns() {
     DataType sparkPartitionType = SparkSchemaUtil.convert(Partitioning.partitionType(table()));
-    return new MetadataColumn[] {
-      new SparkMetadataColumn(MetadataColumns.SPEC_ID.name(), DataTypes.IntegerType, false),
-      new SparkMetadataColumn(MetadataColumns.PARTITION_COLUMN_NAME, sparkPartitionType, true),
-      new SparkMetadataColumn(MetadataColumns.FILE_PATH.name(), DataTypes.StringType, false),
-      new SparkMetadataColumn(MetadataColumns.ROW_POSITION.name(), DataTypes.LongType, false),
-      new SparkMetadataColumn(MetadataColumns.IS_DELETED.name(), DataTypes.BooleanType, false)
-    };
+    ImmutableList.Builder<SparkMetadataColumn> metadataColumns = ImmutableList.builder();
+    metadataColumns.add(
+        new SparkMetadataColumn(MetadataColumns.SPEC_ID.name(), DataTypes.IntegerType, false),
+        new SparkMetadataColumn(MetadataColumns.PARTITION_COLUMN_NAME, sparkPartitionType, true),
+        new SparkMetadataColumn(MetadataColumns.FILE_PATH.name(), DataTypes.StringType, false),
+        new SparkMetadataColumn(MetadataColumns.ROW_POSITION.name(), DataTypes.LongType, false),
+        new SparkMetadataColumn(MetadataColumns.IS_DELETED.name(), DataTypes.BooleanType, false));
+
+    if (TableUtil.formatVersion(table()) >= 3) {
+      metadataColumns.add(
+          new SparkMetadataColumn(MetadataColumns.ROW_ID.name(), DataTypes.LongType, true));
+      metadataColumns.add(
+          new SparkMetadataColumn(
+              MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.name(), DataTypes.LongType, true));
+    }
+
+    return metadataColumns.build().toArray(new SparkMetadataColumn[0]);

Review Comment:
   Now you can use `toArray(SparkMetadataColumn[]::new)` to avoid the use of a 0-length array.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Add row lineage metadata columns, and surface them in SparkTable metadata columns [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar merged PR #12596:
URL: https://github.com/apache/iceberg/pull/12596


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Add row lineage metadata columns, and surface them in SparkTable metadata columns [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #12596:
URL: https://github.com/apache/iceberg/pull/12596#discussion_r2007878806


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java:
##########
@@ -256,13 +258,23 @@ public Set<TableCapability> capabilities() {
   @Override
   public MetadataColumn[] metadataColumns() {
     DataType sparkPartitionType = SparkSchemaUtil.convert(Partitioning.partitionType(table()));
-    return new MetadataColumn[] {
-      new SparkMetadataColumn(MetadataColumns.SPEC_ID.name(), DataTypes.IntegerType, false),
-      new SparkMetadataColumn(MetadataColumns.PARTITION_COLUMN_NAME, sparkPartitionType, true),
-      new SparkMetadataColumn(MetadataColumns.FILE_PATH.name(), DataTypes.StringType, false),
-      new SparkMetadataColumn(MetadataColumns.ROW_POSITION.name(), DataTypes.LongType, false),
-      new SparkMetadataColumn(MetadataColumns.IS_DELETED.name(), DataTypes.BooleanType, false)
-    };
+    ImmutableList.Builder<SparkMetadataColumn> metadataColumns = ImmutableList.builder();
+    metadataColumns.add(
+        new SparkMetadataColumn(MetadataColumns.SPEC_ID.name(), DataTypes.IntegerType, false),
+        new SparkMetadataColumn(MetadataColumns.PARTITION_COLUMN_NAME, sparkPartitionType, true),
+        new SparkMetadataColumn(MetadataColumns.FILE_PATH.name(), DataTypes.StringType, false),
+        new SparkMetadataColumn(MetadataColumns.ROW_POSITION.name(), DataTypes.LongType, false),
+        new SparkMetadataColumn(MetadataColumns.IS_DELETED.name(), DataTypes.BooleanType, false));
+
+    if (TableUtil.formatVersion(table()) >= 3) {

Review Comment:
   Pending discussion of row lineage being enabled by default in V3



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Add row lineage metadata columns, and surface them in SparkTable metadata columns [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #12596:
URL: https://github.com/apache/iceberg/pull/12596#discussion_r2008116032


##########
core/src/main/java/org/apache/iceberg/MetadataColumns.java:
##########
@@ -93,21 +93,34 @@ private MetadataColumns() {}
           "_commit_snapshot_id",
           Types.LongType.get(),
           "Commit snapshot ID");
+  public static final NestedField ROW_ID =
+      NestedField.optional(
+          Integer.MAX_VALUE - 107, "_row_id", Types.LongType.get(), "Row ID for row lineage");
+  public static final NestedField LAST_UPDATED_SEQUENCE_NUMBER =
+      NestedField.optional(
+          Integer.MAX_VALUE - 108,
+          "_last_updated_sequence_number",
+          Types.LongType.get(),
+          "Last updated sequence number for row lineage");

Review Comment:
   "Sequence number when the row was last updated"? I don't know that we want to call out "row lineage" because people that see this may not know what that is.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Add row lineage metadata columns, and surface them in SparkTable metadata columns [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #12596:
URL: https://github.com/apache/iceberg/pull/12596#discussion_r2010345722


##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkMetadataColumns.java:
##########
@@ -311,6 +317,22 @@ public void testConflictingColumns() {
         sql("SELECT _spec_id, _partition, _renamed_spec_id FROM %s", TABLE_NAME));
   }
 
+  @TestTemplate
+  public void testRowLineageColumnsResolvedInV3OrHigher() {
+    if (formatVersion >= 3) {
+      assertEquals(
+          "Rows must match",
+          ImmutableList.of(),
+          sql("SELECT _row_id, _last_updated_sequence_number, id FROM %s", TABLE_NAME));
+    } else {
+      // Should fail to resolve row lineage metadata columns in V1/V2 tables
+      assertThatThrownBy(() -> sql("SELECT _row_id FROM %s", TABLE_NAME))
+          .isInstanceOf(AnalysisException.class);

Review Comment:
   minor: this and the check below should have a `hasMessage()` check



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Add row lineage metadata columns, and surface them in SparkTable metadata columns [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #12596:
URL: https://github.com/apache/iceberg/pull/12596#discussion_r2007878806


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java:
##########
@@ -256,13 +258,23 @@ public Set<TableCapability> capabilities() {
   @Override
   public MetadataColumn[] metadataColumns() {
     DataType sparkPartitionType = SparkSchemaUtil.convert(Partitioning.partitionType(table()));
-    return new MetadataColumn[] {
-      new SparkMetadataColumn(MetadataColumns.SPEC_ID.name(), DataTypes.IntegerType, false),
-      new SparkMetadataColumn(MetadataColumns.PARTITION_COLUMN_NAME, sparkPartitionType, true),
-      new SparkMetadataColumn(MetadataColumns.FILE_PATH.name(), DataTypes.StringType, false),
-      new SparkMetadataColumn(MetadataColumns.ROW_POSITION.name(), DataTypes.LongType, false),
-      new SparkMetadataColumn(MetadataColumns.IS_DELETED.name(), DataTypes.BooleanType, false)
-    };
+    ImmutableList.Builder<SparkMetadataColumn> metadataColumns = ImmutableList.builder();
+    metadataColumns.add(
+        new SparkMetadataColumn(MetadataColumns.SPEC_ID.name(), DataTypes.IntegerType, false),
+        new SparkMetadataColumn(MetadataColumns.PARTITION_COLUMN_NAME, sparkPartitionType, true),
+        new SparkMetadataColumn(MetadataColumns.FILE_PATH.name(), DataTypes.StringType, false),
+        new SparkMetadataColumn(MetadataColumns.ROW_POSITION.name(), DataTypes.LongType, false),
+        new SparkMetadataColumn(MetadataColumns.IS_DELETED.name(), DataTypes.BooleanType, false));
+
+    if (TableUtil.formatVersion(table()) >= 3) {

Review Comment:
   Pending discussion of row lineage being enabled always in V3



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Add row lineage metadata columns, and surface them in SparkTable metadata columns [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #12596:
URL: https://github.com/apache/iceberg/pull/12596#discussion_r2007885914


##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkMetadataColumns.java:
##########
@@ -94,12 +95,17 @@ public static Object[][] parameters() {
       {FileFormat.PARQUET, true, 1},
       {FileFormat.PARQUET, false, 2},
       {FileFormat.PARQUET, true, 2},
+      {FileFormat.PARQUET, false, 3},
+      {FileFormat.PARQUET, true, 3},

Review Comment:
   Adding these different combinations led to tests increasing in time from 20 to 25 seconds locally for me, so I figured it's worth having the coverage in V3 for each case since it doesn't come at too high a cost.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org