You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "gengliangwang (via GitHub)" <gi...@apache.org> on 2026/04/21 20:37:58 UTC

[PR] [SPARK-55948][SQL][FOLLOWUP] Clarify rowVersion contract in DSv2 Changelog interface [spark]

gengliangwang opened a new pull request, #55460:
URL: https://github.com/apache/spark/pull/55460

   ### What changes were proposed in this pull request?
   
   Refine the Javadoc for two methods on the DSv2 `Changelog` interface introduced in [SPARK-55948](https://issues.apache.org/jira/browse/SPARK-55948):
   
   - `rowVersion()` — redefined as the commit at which the row's *content* was last modified: assigned on initial insert, bumped on update, and **preserved when the row is rewritten by a copy-on-write operation without a content change**. The previous Javadoc described it as an "ordering column", which invited implementers to set it equal to `_commit_version`.
   - `rowId()` — extended the "must override" condition to include `containsCarryoverRows()`, matching the fact that the column is used for carry-over removal in addition to update detection and net change computation.
   
   Documentation only; no code behavior changes.
   
   ### Why are the changes needed?
   
   When `rowVersion` is conflated with `_commit_version`, both halves of a delete+insert pair always carry the same value, so row-version equality cannot discriminate a copy-on-write carry-over from a real update. Post-processing is then forced into full-row equality checks — slower, and incorrect in the edge case where a real update happens to rewrite a row to identical values (it would be silently dropped as a carry-over).
   
   With the clarified contract, carry-over detection becomes a scalar comparison: within a `(rowId, _commit_version)` group, equal `rowVersion` ⇒ copy, different `rowVersion` ⇒ real update.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No. Javadoc only.
   
   ### How was this patch tested?
   
   N/A — documentation only.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Opus 4.7
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55948][SQL][FOLLOWUP] Clarify rowVersion contract in DSv2 Changelog interface [spark]

Posted by "SanJSp (via GitHub)" <gi...@apache.org>.
SanJSp commented on code in PR #55460:
URL: https://github.com/apache/spark/pull/55460#discussion_r3122871340


##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java:
##########
@@ -95,22 +95,39 @@ public interface Changelog {
   ScanBuilder newScanBuilder(CaseInsensitiveStringMap options);
 
   /**
-   * Returns the columns that uniquely identify a row, used for update detection and
-   * net change computation.
+   * Returns the columns that uniquely identify a row, used for carry-over removal, update
+   * detection, and net change computation.
    * <p>
-   * The default implementation throws {@link UnsupportedOperationException}. Connectors
-   * that support update detection or net change computation must override this method.
+   * The default implementation throws {@link UnsupportedOperationException}. Connectors must
+   * override this method when any of {@link #containsCarryoverRows()},
+   * {@link #representsUpdateAsDeleteAndInsert()}, or {@link #containsIntermediateChanges()}
+   * returns {@code true}.
    */
   default NamedReference[] rowId() {
     throw new UnsupportedOperationException("rowId is not supported.");
   }
 
   /**
-   * Returns the column used for ordering changes within the same row identity, used for
-   * update detection.
+   * Returns the column that holds the row version — the commit version at which the row's
+   * content was last modified. The row version has these properties:
+   * <ul>
+   *   <li>Assigned the current commit version when the row is initially inserted.</li>
+   *   <li>Bumped to the current commit version when the row's content is updated.</li>
+   *   <li><b>Preserved</b> when the row is rewritten by a copy-on-write operation without a
+   *       content change — it is NOT bumped to the current commit version.</li>
+   * </ul>
+   * The row version is distinct from {@code _commit_version}. {@code _commit_version}
+   * identifies the commit that emitted this change row; the row version identifies the commit
+   * that last wrote the row's content. For a delete+insert pair produced within a single
+   * commit, both halves share the same row version if the pair is a copy, and carry different

Review Comment:
   Nit to connect to copy-on-write operation above: 
   ```suggestion
      * commit, both halves share the same row version if the pair is a copy-on-write carry-over, and have different
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55948][SQL][FOLLOWUP] Clarify rowVersion contract in DSv2 Changelog interface [spark]

Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang closed pull request #55460: [SPARK-55948][SQL][FOLLOWUP] Clarify rowVersion contract in DSv2 Changelog interface
URL: https://github.com/apache/spark/pull/55460


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55948][SQL][FOLLOWUP] Clarify rowVersion contract in DSv2 Changelog interface [spark]

Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #55460:
URL: https://github.com/apache/spark/pull/55460#issuecomment-4298814255

   @SanJSp Thanks, I have applied all your suggestions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55948][SQL][FOLLOWUP] Clarify rowVersion contract in DSv2 Changelog interface [spark]

Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #55460:
URL: https://github.com/apache/spark/pull/55460#issuecomment-4291622530

   cc @johanl-db @SanJSp as well


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55948][SQL][FOLLOWUP] Clarify rowVersion contract in DSv2 Changelog interface [spark]

Posted by "SanJSp (via GitHub)" <gi...@apache.org>.
SanJSp commented on code in PR #55460:
URL: https://github.com/apache/spark/pull/55460#discussion_r3123056975


##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java:
##########
@@ -95,22 +95,39 @@ public interface Changelog {
   ScanBuilder newScanBuilder(CaseInsensitiveStringMap options);
 
   /**
-   * Returns the columns that uniquely identify a row, used for update detection and
-   * net change computation.
+   * Returns the columns that uniquely identify a row, used for carry-over removal, update
+   * detection, and net change computation.
    * <p>
-   * The default implementation throws {@link UnsupportedOperationException}. Connectors
-   * that support update detection or net change computation must override this method.
+   * The default implementation throws {@link UnsupportedOperationException}. Connectors must
+   * override this method when any of {@link #containsCarryoverRows()},
+   * {@link #representsUpdateAsDeleteAndInsert()}, or {@link #containsIntermediateChanges()}
+   * returns {@code true}.
    */
   default NamedReference[] rowId() {
     throw new UnsupportedOperationException("rowId is not supported.");
   }
 
   /**
-   * Returns the column used for ordering changes within the same row identity, used for
-   * update detection.
+   * Returns the column that holds the row version — the commit version at which the row's
+   * content was last modified. The row version has these properties:
+   * <ul>
+   *   <li>Assigned the current commit version when the row is initially inserted.</li>
+   *   <li>Bumped to the current commit version when the row's content is updated.</li>
+   *   <li><b>Preserved</b> when the row is rewritten by a copy-on-write operation without a
+   *       content change — it is NOT bumped to the current commit version.</li>
+   * </ul>
+   * The row version is distinct from {@code _commit_version}. {@code _commit_version}
+   * identifies the commit that emitted this change row; the row version identifies the commit
+   * that last wrote the row's content. For a delete+insert pair produced within a single
+   * commit, both halves share the same row version if the pair is a copy, and carry different
+   * row versions (old on the delete, new on the insert) if the pair is a true update.
    * <p>
-   * The default implementation throws {@link UnsupportedOperationException}. Connectors
-   * that support update detection must override this method.
+   * Spark uses the row version to distinguish copy from update without scanning data columns,
+   * for both carry-over removal and update detection.
+   * <p>
+   * The default implementation throws {@link UnsupportedOperationException}. Connectors must
+   * override this method when {@link #containsCarryoverRows()} or
+   * {@link #representsUpdateAsDeleteAndInsert()} returns {@code true}.

Review Comment:
   ```suggestion
      * {@link #representsUpdateAsDeleteAndInsert()} returns {@code true}. The referenced
      * column must be a top-level column of {@link #columns()} and must be non-nullable.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55948][SQL][FOLLOWUP] Clarify rowVersion contract in DSv2 Changelog interface [spark]

Posted by "SanJSp (via GitHub)" <gi...@apache.org>.
SanJSp commented on code in PR #55460:
URL: https://github.com/apache/spark/pull/55460#discussion_r3122874613


##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java:
##########
@@ -95,22 +95,39 @@ public interface Changelog {
   ScanBuilder newScanBuilder(CaseInsensitiveStringMap options);
 
   /**
-   * Returns the columns that uniquely identify a row, used for update detection and
-   * net change computation.
+   * Returns the columns that uniquely identify a row, used for carry-over removal, update
+   * detection, and net change computation.
    * <p>
-   * The default implementation throws {@link UnsupportedOperationException}. Connectors
-   * that support update detection or net change computation must override this method.
+   * The default implementation throws {@link UnsupportedOperationException}. Connectors must
+   * override this method when any of {@link #containsCarryoverRows()},
+   * {@link #representsUpdateAsDeleteAndInsert()}, or {@link #containsIntermediateChanges()}
+   * returns {@code true}.
    */
   default NamedReference[] rowId() {
     throw new UnsupportedOperationException("rowId is not supported.");
   }
 
   /**
-   * Returns the column used for ordering changes within the same row identity, used for
-   * update detection.
+   * Returns the column that holds the row version — the commit version at which the row's
+   * content was last modified. The row version has these properties:
+   * <ul>
+   *   <li>Assigned the current commit version when the row is initially inserted.</li>
+   *   <li>Bumped to the current commit version when the row's content is updated.</li>
+   *   <li><b>Preserved</b> when the row is rewritten by a copy-on-write operation without a
+   *       content change — it is NOT bumped to the current commit version.</li>
+   * </ul>
+   * The row version is distinct from {@code _commit_version}. {@code _commit_version}
+   * identifies the commit that emitted this change row; the row version identifies the commit
+   * that last wrote the row's content. For a delete+insert pair produced within a single
+   * commit, both halves share the same row version if the pair is a copy, and carry different
+   * row versions (old on the delete, new on the insert) if the pair is a true update.
    * <p>
-   * The default implementation throws {@link UnsupportedOperationException}. Connectors
-   * that support update detection must override this method.
+   * Spark uses the row version to distinguish copy from update without scanning data columns,

Review Comment:
   Nit: continue using copy-on-write carry-over as term
   ```suggestion
      * Spark uses the row version to distinguish copy-on-write carry-over from update without scanning data columns,
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55948][SQL][FOLLOWUP] Clarify rowVersion contract in DSv2 Changelog interface [spark]

Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #55460:
URL: https://github.com/apache/spark/pull/55460#issuecomment-4299800320

   @SanJSp @viirya thanks for the review. Merging to master


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org