You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Davis-Zhang-Onehouse (via GitHub)" <gi...@apache.org> on 2025/08/07 21:34:49 UTC
[PR] [ENG-30770] code refactor for better hoodie data clean up protocol [hudi]
Davis-Zhang-Onehouse opened a new pull request, #13695:
URL: https://github.com/apache/hudi/pull/13695
### Change Logs
### Impact
Cleaner and easier hoodie data/hoodie pair data clean up.
### Risk level (write none, low medium or high below)
none
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "Davis-Zhang-Onehouse (via GitHub)" <gi...@apache.org>.
Davis-Zhang-Onehouse commented on code in PR #13695:
URL: https://github.com/apache/hudi/pull/13695#discussion_r2285674558
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieDataUtils.java:
##########
@@ -94,4 +96,76 @@ public static <K, V> Map<K, Set<V>> collectPairDataAsMap(HoodiePairData<K, V> pa
},
HashMap::putAll);
}
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup after use
Review Comment:
HoodieData is an abstraction and its clean method also does not come with any implementation. If we need to explain `cleanup` it means involve child class specific implementation details.
The point I'm trying to make here is like "hoodie data is an auto closable, make sure you call close once you are done with it". I doubt we need to explain what `close` would do as auto closable is merely an interface that requires some standard way of using it.
I can add hoodie RDD data clean up implementation details here - the clean up method would call rdd.unpersist for the direct/indirect rdd dependency of the current hoodie java rdd. lmk your thoughts. thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "Davis-Zhang-Onehouse (via GitHub)" <gi...@apache.org>.
Davis-Zhang-Onehouse commented on code in PR #13695:
URL: https://github.com/apache/hudi/pull/13695#discussion_r2288691721
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieDataUtils.java:
##########
@@ -94,4 +96,76 @@ public static <K, V> Map<K, Set<V>> collectPairDataAsMap(HoodiePairData<K, V> pa
},
HashMap::putAll);
}
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup after use
Review Comment:
added now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13695:
URL: https://github.com/apache/hudi/pull/13695#issuecomment-3211444563
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7620",
"triggerID" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"triggerType" : "PUSH"
}, {
"hash" : "d37ad20317a1dd0101b0f5ac59f6039423aaf5b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7624",
"triggerID" : "d37ad20317a1dd0101b0f5ac59f6039423aaf5b0",
"triggerType" : "PUSH"
}, {
"hash" : "0849d8fe2db94e0182bc6af30fa82804c38adf02",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7982",
"triggerID" : "0849d8fe2db94e0182bc6af30fa82804c38adf02",
"triggerType" : "PUSH"
}, {
"hash" : "0109793c3a98c4298adc9eb1f6d49488c51d314a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8014",
"triggerID" : "0109793c3a98c4298adc9eb1f6d49488c51d314a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0849d8fe2db94e0182bc6af30fa82804c38adf02 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7982)
* 0109793c3a98c4298adc9eb1f6d49488c51d314a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8014)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13695:
URL: https://github.com/apache/hudi/pull/13695#issuecomment-3211706621
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7620",
"triggerID" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"triggerType" : "PUSH"
}, {
"hash" : "d37ad20317a1dd0101b0f5ac59f6039423aaf5b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7624",
"triggerID" : "d37ad20317a1dd0101b0f5ac59f6039423aaf5b0",
"triggerType" : "PUSH"
}, {
"hash" : "0849d8fe2db94e0182bc6af30fa82804c38adf02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7982",
"triggerID" : "0849d8fe2db94e0182bc6af30fa82804c38adf02",
"triggerType" : "PUSH"
}, {
"hash" : "0109793c3a98c4298adc9eb1f6d49488c51d314a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8014",
"triggerID" : "0109793c3a98c4298adc9eb1f6d49488c51d314a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0109793c3a98c4298adc9eb1f6d49488c51d314a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8014)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13695:
URL: https://github.com/apache/hudi/pull/13695#issuecomment-3166013458
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7620",
"triggerID" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77426a9c12bcd027714ba8a3fe575d6e2f81443e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7620)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13695:
URL: https://github.com/apache/hudi/pull/13695#discussion_r2289611537
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieDataUtils.java:
##########
@@ -94,4 +96,76 @@ public static <K, V> Map<K, Set<V>> collectPairDataAsMap(HoodiePairData<K, V> pa
},
HashMap::putAll);
}
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup after use
Review Comment:
This comment is still not updated, is there a local commit you meant to push?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13695:
URL: https://github.com/apache/hudi/pull/13695#issuecomment-3165823307
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77426a9c12bcd027714ba8a3fe575d6e2f81443e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13695:
URL: https://github.com/apache/hudi/pull/13695#discussion_r2283640183
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieDataUtils.java:
##########
@@ -94,4 +96,76 @@ public static <K, V> Map<K, Set<V>> collectPairDataAsMap(HoodiePairData<K, V> pa
},
HashMap::putAll);
}
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup after use
Review Comment:
Let's add context of what `cleanup` means here. For example, will this close any open resources?
Also include when you are required to use this.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieDataUtils.java:
##########
@@ -94,4 +96,76 @@ public static <K, V> Map<K, Set<V>> collectPairDataAsMap(HoodiePairData<K, V> pa
},
HashMap::putAll);
}
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup after use
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUp(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
+ try {
+ return f.apply(hoodieData);
+ } finally {
+ hoodieData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodiePairData and ensures cleanup after use
+ *
+ * @param hoodiePairData The HoodiePairData to use
+ * @param f Function that processes the HoodiePairData
+ * @param <K> Key type parameter of HoodiePairData
+ * @param <V> Value type parameter of HoodiePairData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <K, V, R> R withHoodieDataCleanUp(HoodiePairData<K, V> hoodiePairData, Function<HoodiePairData<K, V>, R> f) {
+ try {
+ return f.apply(hoodiePairData);
+ } finally {
+ hoodiePairData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup only on exception
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUpOnException(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
Review Comment:
Why do we need this when we already have the cleanup in a `finally` block above?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13695:
URL: https://github.com/apache/hudi/pull/13695#discussion_r2286358521
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieDataUtils.java:
##########
@@ -94,4 +96,76 @@ public static <K, V> Map<K, Set<V>> collectPairDataAsMap(HoodiePairData<K, V> pa
},
HashMap::putAll);
}
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup after use
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUp(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
+ try {
+ return f.apply(hoodieData);
+ } finally {
+ hoodieData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodiePairData and ensures cleanup after use
+ *
+ * @param hoodiePairData The HoodiePairData to use
+ * @param f Function that processes the HoodiePairData
+ * @param <K> Key type parameter of HoodiePairData
+ * @param <V> Value type parameter of HoodiePairData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <K, V, R> R withHoodieDataCleanUp(HoodiePairData<K, V> hoodiePairData, Function<HoodiePairData<K, V>, R> f) {
+ try {
+ return f.apply(hoodiePairData);
+ } finally {
+ hoodiePairData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup only on exception
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUpOnException(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
Review Comment:
Maybe this is a larger confusion with these APIs. How do we handle unchecked exceptions? Does every usage now require this helper?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13695:
URL: https://github.com/apache/hudi/pull/13695#issuecomment-3207479740
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7620",
"triggerID" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"triggerType" : "PUSH"
}, {
"hash" : "d37ad20317a1dd0101b0f5ac59f6039423aaf5b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7624",
"triggerID" : "d37ad20317a1dd0101b0f5ac59f6039423aaf5b0",
"triggerType" : "PUSH"
}, {
"hash" : "0849d8fe2db94e0182bc6af30fa82804c38adf02",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7982",
"triggerID" : "0849d8fe2db94e0182bc6af30fa82804c38adf02",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0849d8fe2db94e0182bc6af30fa82804c38adf02 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7982)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13695:
URL: https://github.com/apache/hudi/pull/13695#issuecomment-3166158397
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7620",
"triggerID" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"triggerType" : "PUSH"
}, {
"hash" : "d37ad20317a1dd0101b0f5ac59f6039423aaf5b0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d37ad20317a1dd0101b0f5ac59f6039423aaf5b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77426a9c12bcd027714ba8a3fe575d6e2f81443e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7620)
* d37ad20317a1dd0101b0f5ac59f6039423aaf5b0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13695:
URL: https://github.com/apache/hudi/pull/13695#discussion_r2289612159
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieDataUtils.java:
##########
@@ -94,4 +96,79 @@ public static <K, V> Map<K, Set<V>> collectPairDataAsMap(HoodiePairData<K, V> pa
},
HashMap::putAll);
}
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup after use
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUp(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
+ try {
+ return f.apply(hoodieData);
+ } finally {
+ hoodieData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodiePairData and ensures cleanup after use
+ *
+ * @param hoodiePairData The HoodiePairData to use
+ * @param f Function that processes the HoodiePairData
+ * @param <K> Key type parameter of HoodiePairData
+ * @param <V> Value type parameter of HoodiePairData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <K, V, R> R withHoodieDataCleanUp(HoodiePairData<K, V> hoodiePairData, Function<HoodiePairData<K, V>, R> f) {
+ try {
+ return f.apply(hoodiePairData);
+ } finally {
+ hoodiePairData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup only on exception.
+ *
+ * It calls unpersistWithDependencies which internally close all resources it depends on.
+ * For example, for spark rdd-based API implementation, it cleans up all the upstream cached rdd.
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUpOnException(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
+ try {
+ return f.apply(hoodieData);
+ } catch (Exception e) {
+ hoodieData.unpersistWithDependencies();
+ throw e;
+ }
+ }
+
+ /**
+ * Executes a function with HoodiePairData and ensures cleanup only on exception
Review Comment:
Let's update this javadoc to match the above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "Davis-Zhang-Onehouse (via GitHub)" <gi...@apache.org>.
Davis-Zhang-Onehouse commented on code in PR #13695:
URL: https://github.com/apache/hudi/pull/13695#discussion_r2288656363
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieDataUtils.java:
##########
@@ -94,4 +96,76 @@ public static <K, V> Map<K, Set<V>> collectPairDataAsMap(HoodiePairData<K, V> pa
},
HashMap::putAll);
}
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup after use
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUp(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
+ try {
+ return f.apply(hoodieData);
+ } finally {
+ hoodieData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodiePairData and ensures cleanup after use
+ *
+ * @param hoodiePairData The HoodiePairData to use
+ * @param f Function that processes the HoodiePairData
+ * @param <K> Key type parameter of HoodiePairData
+ * @param <V> Value type parameter of HoodiePairData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <K, V, R> R withHoodieDataCleanUp(HoodiePairData<K, V> hoodiePairData, Function<HoodiePairData<K, V>, R> f) {
+ try {
+ return f.apply(hoodiePairData);
+ } finally {
+ hoodiePairData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup only on exception
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUpOnException(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
Review Comment:
```
myAppFunc() {
rdd4 = myFunc0() {
rdd1 = myFunc1() {
create rdd11
rdd11.persist
rdd11.collect()
return myFunc2(rdd11)
}
rdd3 = rdd1 do more DAG and some other code
return rdd3
}
rdd4.collect() <--- done using the DAG
}
```
here we have 3 types of funcs:
- myFunc1 - deeply nested and call persist and collect and return another rdd
- myFunc0 - some intermediate func that extend the DAG and does not finish the code. I call this "pipeFunc"
- myAppFunc - application level code that defines "when the DAG is done being used" and we can remove cache as code completion clean up.
when the "pipeFunc" does not wrap with cleanUpOnException and throws, what happens here is:
myAppFunc at best catch the exception, yet in order to clean up the cached rdd upstream, it needs the reference to either that rdd itself or any rdd that use the cached rdd as the depdnency, which myAppFunc does not have either of them.
But if the pipeFunc are all wrapped with clean up on exception, then pipefunc will do the clean up if it throws ("the unhappy path" - who becomes unhappy is responsible to clean up)
myAppFunc who defines when we are done using the DAG, must wrap with cleanupfinally as only this func knows when we can "clean up everything" on "the happy path"
So the protocol is summarized as:
- the pipeFunc must guard itself with clean up on exception to handle the unhappy path
- myAppfunc must guard with clean up finally.
Over the entire code base that we work with hoodieData and hoodiePair data we should follow this protocol unless the same requirement is satisfied otherwise.
If this is still not sufficient, feel free to chat offline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "Davis-Zhang-Onehouse (via GitHub)" <gi...@apache.org>.
Davis-Zhang-Onehouse commented on code in PR #13695:
URL: https://github.com/apache/hudi/pull/13695#discussion_r2288656363
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieDataUtils.java:
##########
@@ -94,4 +96,76 @@ public static <K, V> Map<K, Set<V>> collectPairDataAsMap(HoodiePairData<K, V> pa
},
HashMap::putAll);
}
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup after use
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUp(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
+ try {
+ return f.apply(hoodieData);
+ } finally {
+ hoodieData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodiePairData and ensures cleanup after use
+ *
+ * @param hoodiePairData The HoodiePairData to use
+ * @param f Function that processes the HoodiePairData
+ * @param <K> Key type parameter of HoodiePairData
+ * @param <V> Value type parameter of HoodiePairData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <K, V, R> R withHoodieDataCleanUp(HoodiePairData<K, V> hoodiePairData, Function<HoodiePairData<K, V>, R> f) {
+ try {
+ return f.apply(hoodiePairData);
+ } finally {
+ hoodiePairData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup only on exception
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUpOnException(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
Review Comment:
myAppFunc() {
rdd4 = myFunc0() {
rdd1 = myFunc1() {
create rdd11
rdd11.persist
rdd11.collect()
return myFunc2(rdd11)
}
rdd3 = rdd1 do more DAG and some other code
return rdd3
}
rdd4.collect() <--- done using the DAG
}
here we have 3 types of funcs:
- myFunc1 - deeply nested and call persist and collect and return another rdd
- myFunc0 - some intermediate func that extend the DAG and does not finish the code. I call this "pipeFunc"
- myAppFunc - application level code that defines "when the DAG is done being used" and we can remove cache as code completion clean up.
when the "pipeFunc" does not wrap with cleanUpOnException and throws, what happens here is:
myAppFunc at best catch the exception, yet in order to clean up the cached rdd upstream, it needs the reference to either that rdd itself or any rdd that use the cached rdd as the depdnency, which myAppFunc does not have either of them.
But if the pipeFunc are all wrapped with clean up on exception, then pipefunc will do the clean up if it throws ("the unhappy path" - who becomes unhappy is responsible to clean up)
myAppFunc who defines when we are done using the DAG, must wrap with cleanupfinally as only this func knows when we can "clean up everything" on "the happy path"
So the protocol is summarized as:
- the pipeFunc must guard itself with clean up on exception to handle the unhappy path
- myAppfunc must guard with clean up finally.
Over the entire code base that we work with hoodieData and hoodiePair data we should follow this protocol unless the same requirement is satisfied otherwise.
If this is still not sufficient, feel free to chat offline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13695:
URL: https://github.com/apache/hudi/pull/13695#issuecomment-3207077200
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7620",
"triggerID" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"triggerType" : "PUSH"
}, {
"hash" : "d37ad20317a1dd0101b0f5ac59f6039423aaf5b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7624",
"triggerID" : "d37ad20317a1dd0101b0f5ac59f6039423aaf5b0",
"triggerType" : "PUSH"
}, {
"hash" : "0849d8fe2db94e0182bc6af30fa82804c38adf02",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0849d8fe2db94e0182bc6af30fa82804c38adf02",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d37ad20317a1dd0101b0f5ac59f6039423aaf5b0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7624)
* 0849d8fe2db94e0182bc6af30fa82804c38adf02 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "Davis-Zhang-Onehouse (via GitHub)" <gi...@apache.org>.
Davis-Zhang-Onehouse commented on code in PR #13695:
URL: https://github.com/apache/hudi/pull/13695#discussion_r2285656672
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieDataUtils.java:
##########
@@ -94,4 +96,76 @@ public static <K, V> Map<K, Set<V>> collectPairDataAsMap(HoodiePairData<K, V> pa
},
HashMap::putAll);
}
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup after use
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUp(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
+ try {
+ return f.apply(hoodieData);
+ } finally {
+ hoodieData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodiePairData and ensures cleanup after use
+ *
+ * @param hoodiePairData The HoodiePairData to use
+ * @param f Function that processes the HoodiePairData
+ * @param <K> Key type parameter of HoodiePairData
+ * @param <V> Value type parameter of HoodiePairData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <K, V, R> R withHoodieDataCleanUp(HoodiePairData<K, V> hoodiePairData, Function<HoodiePairData<K, V>, R> f) {
+ try {
+ return f.apply(hoodiePairData);
+ } finally {
+ hoodiePairData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup only on exception
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUpOnException(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
Review Comment:
please check the description of PR https://github.com/apache/hudi/pull/13629
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "Davis-Zhang-Onehouse (via GitHub)" <gi...@apache.org>.
Davis-Zhang-Onehouse commented on code in PR #13695:
URL: https://github.com/apache/hudi/pull/13695#discussion_r2291659987
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieDataUtils.java:
##########
@@ -94,4 +96,76 @@ public static <K, V> Map<K, Set<V>> collectPairDataAsMap(HoodiePairData<K, V> pa
},
HashMap::putAll);
}
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup after use
Review Comment:
I should have added them to all 4 APIs, they are all updated now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #13695:
URL: https://github.com/apache/hudi/pull/13695#issuecomment-6700697564
Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #13695: [MINOR] code refactor for better hoodie data clean up protocol
URL: https://github.com/apache/hudi/pull/13695
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13695:
URL: https://github.com/apache/hudi/pull/13695#issuecomment-3165834498
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7620",
"triggerID" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77426a9c12bcd027714ba8a3fe575d6e2f81443e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7620)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13695:
URL: https://github.com/apache/hudi/pull/13695#issuecomment-3166285226
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7620",
"triggerID" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"triggerType" : "PUSH"
}, {
"hash" : "d37ad20317a1dd0101b0f5ac59f6039423aaf5b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7624",
"triggerID" : "d37ad20317a1dd0101b0f5ac59f6039423aaf5b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d37ad20317a1dd0101b0f5ac59f6039423aaf5b0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7624)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13695:
URL: https://github.com/apache/hudi/pull/13695#discussion_r2286358521
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieDataUtils.java:
##########
@@ -94,4 +96,76 @@ public static <K, V> Map<K, Set<V>> collectPairDataAsMap(HoodiePairData<K, V> pa
},
HashMap::putAll);
}
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup after use
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUp(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
+ try {
+ return f.apply(hoodieData);
+ } finally {
+ hoodieData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodiePairData and ensures cleanup after use
+ *
+ * @param hoodiePairData The HoodiePairData to use
+ * @param f Function that processes the HoodiePairData
+ * @param <K> Key type parameter of HoodiePairData
+ * @param <V> Value type parameter of HoodiePairData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <K, V, R> R withHoodieDataCleanUp(HoodiePairData<K, V> hoodiePairData, Function<HoodiePairData<K, V>, R> f) {
+ try {
+ return f.apply(hoodiePairData);
+ } finally {
+ hoodiePairData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup only on exception
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUpOnException(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
Review Comment:
Maybe this is a larger confusion with these APIs. How do we handle unchecked exceptions? Does every block now require this helper?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13695:
URL: https://github.com/apache/hudi/pull/13695#issuecomment-3211436294
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7620",
"triggerID" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"triggerType" : "PUSH"
}, {
"hash" : "d37ad20317a1dd0101b0f5ac59f6039423aaf5b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7624",
"triggerID" : "d37ad20317a1dd0101b0f5ac59f6039423aaf5b0",
"triggerType" : "PUSH"
}, {
"hash" : "0849d8fe2db94e0182bc6af30fa82804c38adf02",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7982",
"triggerID" : "0849d8fe2db94e0182bc6af30fa82804c38adf02",
"triggerType" : "PUSH"
}, {
"hash" : "0109793c3a98c4298adc9eb1f6d49488c51d314a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0109793c3a98c4298adc9eb1f6d49488c51d314a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0849d8fe2db94e0182bc6af30fa82804c38adf02 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7982)
* 0109793c3a98c4298adc9eb1f6d49488c51d314a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "Davis-Zhang-Onehouse (via GitHub)" <gi...@apache.org>.
Davis-Zhang-Onehouse commented on code in PR #13695:
URL: https://github.com/apache/hudi/pull/13695#discussion_r2291660834
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieDataUtils.java:
##########
@@ -94,4 +96,79 @@ public static <K, V> Map<K, Set<V>> collectPairDataAsMap(HoodiePairData<K, V> pa
},
HashMap::putAll);
}
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup after use
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUp(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
+ try {
+ return f.apply(hoodieData);
+ } finally {
+ hoodieData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodiePairData and ensures cleanup after use
+ *
+ * @param hoodiePairData The HoodiePairData to use
+ * @param f Function that processes the HoodiePairData
+ * @param <K> Key type parameter of HoodiePairData
+ * @param <V> Value type parameter of HoodiePairData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <K, V, R> R withHoodieDataCleanUp(HoodiePairData<K, V> hoodiePairData, Function<HoodiePairData<K, V>, R> f) {
+ try {
+ return f.apply(hoodiePairData);
+ } finally {
+ hoodiePairData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup only on exception.
+ *
+ * It calls unpersistWithDependencies which internally close all resources it depends on.
+ * For example, for spark rdd-based API implementation, it cleans up all the upstream cached rdd.
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUpOnException(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
+ try {
+ return f.apply(hoodieData);
+ } catch (Exception e) {
+ hoodieData.unpersistWithDependencies();
+ throw e;
+ }
+ }
+
+ /**
+ * Executes a function with HoodiePairData and ensures cleanup only on exception
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13695:
URL: https://github.com/apache/hudi/pull/13695#discussion_r2286340133
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieDataUtils.java:
##########
@@ -94,4 +96,76 @@ public static <K, V> Map<K, Set<V>> collectPairDataAsMap(HoodiePairData<K, V> pa
},
HashMap::putAll);
}
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup after use
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUp(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
+ try {
+ return f.apply(hoodieData);
+ } finally {
+ hoodieData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodiePairData and ensures cleanup after use
+ *
+ * @param hoodiePairData The HoodiePairData to use
+ * @param f Function that processes the HoodiePairData
+ * @param <K> Key type parameter of HoodiePairData
+ * @param <V> Value type parameter of HoodiePairData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <K, V, R> R withHoodieDataCleanUp(HoodiePairData<K, V> hoodiePairData, Function<HoodiePairData<K, V>, R> f) {
+ try {
+ return f.apply(hoodiePairData);
+ } finally {
+ hoodiePairData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup only on exception
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUpOnException(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
Review Comment:
I read it, still the question remains.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13695:
URL: https://github.com/apache/hudi/pull/13695#discussion_r2286344042
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieDataUtils.java:
##########
@@ -94,4 +96,76 @@ public static <K, V> Map<K, Set<V>> collectPairDataAsMap(HoodiePairData<K, V> pa
},
HashMap::putAll);
}
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup after use
Review Comment:
The javadoc on AutoCloseable gives examples of resources an object may hold onto. I think we can do something similar here so the developer understands what to look out for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13695:
URL: https://github.com/apache/hudi/pull/13695#issuecomment-3166162277
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7620",
"triggerID" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"triggerType" : "PUSH"
}, {
"hash" : "d37ad20317a1dd0101b0f5ac59f6039423aaf5b0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7624",
"triggerID" : "d37ad20317a1dd0101b0f5ac59f6039423aaf5b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77426a9c12bcd027714ba8a3fe575d6e2f81443e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7620)
* d37ad20317a1dd0101b0f5ac59f6039423aaf5b0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7624)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "Davis-Zhang-Onehouse (via GitHub)" <gi...@apache.org>.
Davis-Zhang-Onehouse commented on code in PR #13695:
URL: https://github.com/apache/hudi/pull/13695#discussion_r2288684699
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieDataUtils.java:
##########
@@ -94,4 +96,76 @@ public static <K, V> Map<K, Set<V>> collectPairDataAsMap(HoodiePairData<K, V> pa
},
HashMap::putAll);
}
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup after use
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUp(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
+ try {
+ return f.apply(hoodieData);
+ } finally {
+ hoodieData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodiePairData and ensures cleanup after use
+ *
+ * @param hoodiePairData The HoodiePairData to use
+ * @param f Function that processes the HoodiePairData
+ * @param <K> Key type parameter of HoodiePairData
+ * @param <V> Value type parameter of HoodiePairData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <K, V, R> R withHoodieDataCleanUp(HoodiePairData<K, V> hoodiePairData, Function<HoodiePairData<K, V>, R> f) {
+ try {
+ return f.apply(hoodiePairData);
+ } finally {
+ hoodiePairData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup only on exception
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUpOnException(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
Review Comment:
> How do we handle unchecked exceptions?
as you can see it catches "Exception" class, do clean up and throw again as the code is not responsible for handling the exception that it has no idea about.
> Does every usage now require this helper?
yes, at the time I introduced the code, I make sure all usage that have rdd cache follows this pattern. Verified with functional tests.
In the PR I linked for description, I could not enforce test failure if we found cached rdd in the end, it is because the storage level member is not volatile so multi-threading scenario it has visibility issue from other threads. Also spark only provides listener for rdd unpersist, but not for persist. without changing the spark code we cannot reliably enforce the check in hudi functional test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13695:
URL: https://github.com/apache/hudi/pull/13695#issuecomment-3207084048
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7620",
"triggerID" : "77426a9c12bcd027714ba8a3fe575d6e2f81443e",
"triggerType" : "PUSH"
}, {
"hash" : "d37ad20317a1dd0101b0f5ac59f6039423aaf5b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7624",
"triggerID" : "d37ad20317a1dd0101b0f5ac59f6039423aaf5b0",
"triggerType" : "PUSH"
}, {
"hash" : "0849d8fe2db94e0182bc6af30fa82804c38adf02",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7982",
"triggerID" : "0849d8fe2db94e0182bc6af30fa82804c38adf02",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d37ad20317a1dd0101b0f5ac59f6039423aaf5b0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7624)
* 0849d8fe2db94e0182bc6af30fa82804c38adf02 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7982)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] code refactor for better hoodie data clean up protocol [hudi]
Posted by "Davis-Zhang-Onehouse (via GitHub)" <gi...@apache.org>.
Davis-Zhang-Onehouse commented on code in PR #13695:
URL: https://github.com/apache/hudi/pull/13695#discussion_r2288684699
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieDataUtils.java:
##########
@@ -94,4 +96,76 @@ public static <K, V> Map<K, Set<V>> collectPairDataAsMap(HoodiePairData<K, V> pa
},
HashMap::putAll);
}
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup after use
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUp(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
+ try {
+ return f.apply(hoodieData);
+ } finally {
+ hoodieData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodiePairData and ensures cleanup after use
+ *
+ * @param hoodiePairData The HoodiePairData to use
+ * @param f Function that processes the HoodiePairData
+ * @param <K> Key type parameter of HoodiePairData
+ * @param <V> Value type parameter of HoodiePairData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <K, V, R> R withHoodieDataCleanUp(HoodiePairData<K, V> hoodiePairData, Function<HoodiePairData<K, V>, R> f) {
+ try {
+ return f.apply(hoodiePairData);
+ } finally {
+ hoodiePairData.unpersistWithDependencies();
+ }
+ }
+
+ /**
+ * Executes a function with HoodieData and ensures cleanup only on exception
+ *
+ * @param hoodieData The HoodieData to use
+ * @param f Function that processes the HoodieData
+ * @param <T> Type parameter of HoodieData
+ * @param <R> Return type of the function
+ * @return Result of the function
+ */
+ public static <T, R> R withHoodieDataCleanUpOnException(HoodieData<T> hoodieData, Function<HoodieData<T>, R> f) {
Review Comment:
> How do we handle unchecked exceptions?
as you can see it catches "Exception" class, do clean up and throw again as the code is not responsible for handling the exception that it has no idea about.
> Does every usage now require this helper?
yes, at the time I introduced the code, I make sure all usage that have rdd cache follows this pattern. Verified with functional tests as long as it is index related (col stats, files, RLI, SI, expression idx, MDT validation test, etc)
In the PR I linked for you to check the description, I could not enforce test failure if we found cached rdd in the end, it is because the storage level member is not volatile so multi-threading scenario it has visibility issue from other threads. Also spark only provides listener for rdd unpersist, but not for persist. without changing the spark code we cannot reliably enforce the check in hudi functional test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org