You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "prashantwason (via GitHub)" <gi...@apache.org> on 2025/12/03 01:24:26 UTC

[PR] perf: Faster checking for deleted records by avoiding calling getInsertValue() [hudi]

prashantwason opened a new pull request, #17465:
URL: https://github.com/apache/hudi/pull/17465

   ### Describe the issue this Pull Request addresses
   
   During insert operations, HoodieCreateHandle checks if a record is deleted by calling `payload.getInsertValue().isPresent()`. This approach leads to unnecessary generation of the insert value, which can be computationally expensive for certain payload implementations.
   
   This PR introduces a performance optimization to avoid this overhead.
   
   ### Summary and Changelog
   
   **Summary:** 
   This PR introduces a new method `hasInsertValue()` to the `HoodieRecordPayload` interface that allows payload implementations to efficiently check if they have a valid insert value without actually generating it.
   
   **Changelog:**
   - Added new `hasInsertValue(Schema schema, Properties properties)` method to `HoodieRecordPayload` interface with a default implementation that maintains backward compatibility
   - Updated `HoodieAvroRecord.checkIsDelete()` to use the new `hasInsertValue()` method instead of `getInsertValue().isPresent()`
   - Implemented optimized `hasInsertValue()` in `HoodieMetadataPayload` - checks internal flags without deserializing
   - Implemented optimized `hasInsertValue()` in `RawTripTestPayload` - checks `isDeleted` flag directly
   - Added helper methods `validatePayload()` and `getNestedFieldValue()` in `HoodieMetadataPayload`
   - Added utility methods in test payload classes for improved testing
   
   ### Impact
   
   **Performance Impact:** This change provides a significant performance improvement for write operations, particularly when dealing with payloads where generating the insert value is expensive. Payload implementations can now bypass the costly insert value generation during the deleted record check by implementing custom logic in `hasInsertValue()`.
   
   **Public API Change:** A new method `hasInsertValue()` is added to the `HoodieRecordPayload` interface. The default implementation ensures backward compatibility by delegating to the existing `getInsertValue()` method.
   
   ### Risk Level
   
   **Low**
   
   The change is backward compatible as the new method has a default implementation that preserves existing behavior. Existing payload implementations will continue to work without modification. The risk is mitigated by:
   - Default implementation maintains current behavior
   - Extensive testing with various payload types
   - No changes to external APIs or configuration
   - Internal optimization only, no storage format changes
   
   ### Documentation Update
   
   None required. This is an internal performance optimization that does not introduce new configs, user-facing features, or changes to existing behavior from a user perspective.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Faster checking for deleted records by avoiding calling getInsertValue() [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17465:
URL: https://github.com/apache/hudi/pull/17465#issuecomment-3604649230

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8a4f06f202b5ebb2c6990ece2b57b53803f97664",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9850",
       "triggerID" : "8a4f06f202b5ebb2c6990ece2b57b53803f97664",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8a4f06f202b5ebb2c6990ece2b57b53803f97664 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9850) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Faster checking for deleted records by avoiding calling getInsertValue() [hudi]

Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on PR #17465:
URL: https://github.com/apache/hudi/pull/17465#issuecomment-3708380110

   We will move to the new feature.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Faster checking for deleted records by avoiding calling getInsertValue() [hudi]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17465:
URL: https://github.com/apache/hudi/pull/17465#discussion_r2586226022


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java:
##########
@@ -162,6 +162,23 @@ default Comparable<?> getOrderingValue() {
     return 0;
   }
 
+  /**
+   * Returns true is this payload represents an actual record to be written.
+   *
+   * Default implementation checks if the insert value is present. For better write performance, this should be
+   * overridden by implementations for which generating the insert value is expensive.
+   *
+   * @param schema Schema used for record
+   * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+   * @return true if this payload has a record to be written.
+   *         false if this payload is empty and should be skipped from writing.
+   *         false if this payload represents a record to be deleted.
+   */
+  @PublicAPIMethod(maturity = ApiMaturityLevel.EVOLVING)
+  default boolean hasInsertValue(Schema schema, Properties properties) throws IOException {

Review Comment:
   Payloads have been deprecated in favor of merge modes which have similar optimizations made. Is it possible for your workflow to adopt a custom merger or one of the prebuilt merge modes?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Faster checking for deleted records by avoiding calling getInsertValue() [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17465:
URL: https://github.com/apache/hudi/pull/17465#issuecomment-3604644401

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8a4f06f202b5ebb2c6990ece2b57b53803f97664",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8a4f06f202b5ebb2c6990ece2b57b53803f97664",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8a4f06f202b5ebb2c6990ece2b57b53803f97664 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Faster checking for deleted records by avoiding calling getInsertValue() [hudi]

Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason closed pull request #17465: [MINOR] Faster checking for deleted records by avoiding calling getInsertValue()
URL: https://github.com/apache/hudi/pull/17465


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Faster checking for deleted records by avoiding calling getInsertValue() [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #17465:
URL: https://github.com/apache/hudi/pull/17465#discussion_r2587443840


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java:
##########
@@ -162,6 +162,23 @@ default Comparable<?> getOrderingValue() {
     return 0;
   }
 
+  /**
+   * Returns true is this payload represents an actual record to be written.
+   *
+   * Default implementation checks if the insert value is present. For better write performance, this should be
+   * overridden by implementations for which generating the insert value is expensive.
+   *
+   * @param schema Schema used for record
+   * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+   * @return true if this payload has a record to be written.
+   *         false if this payload is empty and should be skipped from writing.
+   *         false if this payload represents a record to be deleted.
+   */
+  @PublicAPIMethod(maturity = ApiMaturityLevel.EVOLVING)
+  default boolean hasInsertValue(Schema schema, Properties properties) throws IOException {

Review Comment:
   yeah, probably put the patch to just 0.x-branch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Faster checking for deleted records by avoiding calling getInsertValue() [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #17465:
URL: https://github.com/apache/hudi/pull/17465#discussion_r2651942555


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java:
##########
@@ -162,6 +162,23 @@ default Comparable<?> getOrderingValue() {
     return 0;
   }
 
+  /**
+   * Returns true is this payload represents an actual record to be written.
+   *
+   * Default implementation checks if the insert value is present. For better write performance, this should be
+   * overridden by implementations for which generating the insert value is expensive.
+   *
+   * @param schema Schema used for record
+   * @param properties Payload related properties. For example pass the ordering field(s) name to extract from value in storage.
+   * @return true if this payload has a record to be written.
+   *         false if this payload is empty and should be skipped from writing.
+   *         false if this payload represents a record to be deleted.
+   */
+  @PublicAPIMethod(maturity = ApiMaturityLevel.EVOLVING)
+  default boolean hasInsertValue(Schema schema, Properties properties) throws IOException {

Review Comment:
   +1 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [MINOR] Faster checking for deleted records by avoiding calling getInsertValue() [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17465:
URL: https://github.com/apache/hudi/pull/17465#issuecomment-3604737194

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8a4f06f202b5ebb2c6990ece2b57b53803f97664",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9850",
       "triggerID" : "8a4f06f202b5ebb2c6990ece2b57b53803f97664",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8a4f06f202b5ebb2c6990ece2b57b53803f97664 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9850) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org