You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "Caideyipi (via GitHub)" <gi...@apache.org> on 2026/07/28 07:17:45 UTC
[PR] Pipe: Handle hybrid meta progress indexes [iotdb]
Caideyipi opened a new pull request, #18331:
URL: https://github.com/apache/iotdb/pull/18331
## Description
### Problem
IoTDBNonDataRegionSource assumed every non-minimum task progress was a MetaProgressIndex. Pipe task progress can also be merged into a HybridProgressIndex, and may be wrapped by a StateProgressIndex. This caused a ClassCastException while restoring a non-data-region source or calculating its remaining events during DataNode shutdown.
### Changes
- Extract the meta progress component from direct, hybrid, and state-wrapped progress indexes.
- Use that component for source recovery and remaining-event accounting.
- Conservatively fall back to snapshot recovery or full queue size when no meta component exists.
- Add unit coverage for hybrid, state-wrapped hybrid, and hybrid-without-meta cases.
### Verification
- mvn spotless:apply -pl iotdb-core/node-commons
- mvn test -pl iotdb-core/node-commons -Dtest=IoTDBNonDataRegionSourceTest
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added unit tests to cover new code paths.
<hr>
##### Key changed/added classes
- IoTDBNonDataRegionSource
- IoTDBNonDataRegionSourceTest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pipe: Handle hybrid meta progress indexes [iotdb]
Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 commented on code in PR #18331:
URL: https://github.com/apache/iotdb/pull/18331#discussion_r3671478928
##########
iotdb-client/client-go:
##########
Review Comment:
？
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/consensus/index/ProgressIndex.java:
##########
@@ -158,6 +159,45 @@ public abstract ProgressIndex updateToMinimumEqualOrIsAfterProgressIndex(
*/
public abstract ProgressIndexType getType();
+ /**
+ * Extracts a progress index of the given type from this progress index.
+ *
+ * <p>{@link StateProgressIndex} and {@link HybridProgressIndex} are recursively unwrapped because
+ * they may contain progress indexes from other causal chains.
+ */
+ public final <T extends ProgressIndex> Optional<T> getProgressIndexByType(
+ final Class<T> progressIndexClass) {
+ if (progressIndexClass.isInstance(this)) {
+ return Optional.of(progressIndexClass.cast(this));
+ }
+
+ if (this instanceof StateProgressIndex) {
+ return ((StateProgressIndex) this)
+ .getInnerProgressIndex()
+ .getProgressIndexByType(progressIndexClass);
+ }
+
+ if (this instanceof HybridProgressIndex) {
+ final Map<Short, ProgressIndex> type2Index = ((HybridProgressIndex) this).getType2Index();
+
+ // Prefer a direct component over one nested in another composite progress index.
+ for (final ProgressIndex progressIndex : type2Index.values()) {
+ if (progressIndexClass.isInstance(progressIndex)) {
+ return Optional.of(progressIndexClass.cast(progressIndex));
+ }
+ }
+ for (final ProgressIndex progressIndex : type2Index.values()) {
+ final Optional<T> extractedProgressIndex =
+ progressIndex.getProgressIndexByType(progressIndexClass);
+ if (extractedProgressIndex.isPresent()) {
+ return extractedProgressIndex;
+ }
+ }
+ }
+
+ return Optional.empty();
+ }
Review Comment:
Let the sub-classes implement it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pipe: Handle hybrid meta progress indexes [iotdb]
Posted by "Caideyipi (via GitHub)" <gi...@apache.org>.
Caideyipi commented on code in PR #18331:
URL: https://github.com/apache/iotdb/pull/18331#discussion_r3679628939
##########
iotdb-client/client-go:
##########
Review Comment:
Removed the unintended client-go submodule pointer change in 1defa441f18. Thanks for catching it.
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/consensus/index/ProgressIndex.java:
##########
@@ -158,6 +159,45 @@ public abstract ProgressIndex updateToMinimumEqualOrIsAfterProgressIndex(
*/
public abstract ProgressIndexType getType();
+ /**
+ * Extracts a progress index of the given type from this progress index.
+ *
+ * <p>{@link StateProgressIndex} and {@link HybridProgressIndex} are recursively unwrapped because
+ * they may contain progress indexes from other causal chains.
+ */
+ public final <T extends ProgressIndex> Optional<T> getProgressIndexByType(
+ final Class<T> progressIndexClass) {
+ if (progressIndexClass.isInstance(this)) {
+ return Optional.of(progressIndexClass.cast(this));
+ }
+
+ if (this instanceof StateProgressIndex) {
+ return ((StateProgressIndex) this)
+ .getInnerProgressIndex()
+ .getProgressIndexByType(progressIndexClass);
+ }
+
+ if (this instanceof HybridProgressIndex) {
+ final Map<Short, ProgressIndex> type2Index = ((HybridProgressIndex) this).getType2Index();
+
+ // Prefer a direct component over one nested in another composite progress index.
+ for (final ProgressIndex progressIndex : type2Index.values()) {
+ if (progressIndexClass.isInstance(progressIndex)) {
+ return Optional.of(progressIndexClass.cast(progressIndex));
+ }
+ }
+ for (final ProgressIndex progressIndex : type2Index.values()) {
+ final Optional<T> extractedProgressIndex =
+ progressIndex.getProgressIndexByType(progressIndexClass);
+ if (extractedProgressIndex.isPresent()) {
+ return extractedProgressIndex;
+ }
+ }
+ }
+
+ return Optional.empty();
+ }
Review Comment:
Done in 1defa441f18. getProgressIndexByType is now abstract on ProgressIndex and implemented by each subclass; StateProgressIndex and HybridProgressIndex retain the recursive unwrapping behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pipe: Handle hybrid meta progress indexes [iotdb]
Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 merged PR #18331:
URL: https://github.com/apache/iotdb/pull/18331
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org