You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "Caideyipi (via GitHub)" <gi...@apache.org> on 2026/08/19 11:44:03 UTC
[PR] [Pipe] Allow startup when a plugin JAR is unavailable [iotdb]
Caideyipi opened a new pull request, #18496:
URL: https://github.com/apache/iotdb/pull/18496
## Description
When ConfigNode cannot provide a registered Pipe plugin JAR, DataNode startup currently fails. This change records the affected plugin load failures, logs them at ERROR, skips unavailable plugins, and allows DataNode startup to continue.
The fetch path now also handles unsuccessful or malformed ConfigNode responses, validates the returned JAR count, and isolates individual JAR save failures. The recorded failure remains visible through SHOW PIPEPLUGINS.
## Validation
- mvn -pl iotdb-core/datanode spotless:check -DskipTests
- git diff --check
This PR has:
- [x] been self-reviewed.
##### Key changed class
- iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/agent/runtime/PipeAgentLauncher.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Pipe] Allow startup when a plugin JAR is unavailable [iotdb]
Posted by "luoluoyuyu (via GitHub)" <gi...@apache.org>.
luoluoyuyu commented on PR #18496:
URL: https://github.com/apache/iotdb/pull/18496#issuecomment-5350157791
Thanks for addressing the DataNode startup failure when a Pipe plugin JAR is unavailable. Allowing the node to start in a degraded state is reasonable, but I see two issues that should be addressed before merging.
1. A single missing JAR can disable the entire batch
getPipePluginJar fails the whole request when any requested JAR is unavailable. In that case, this implementation marks every plugin in pipePluginMetaList as unavailable:
markPipePluginLoadFailures(pipePluginMetaList, exception);
unavailablePipePluginNameSet.addAll(pluginNameList);
Since one RPC can contain up to 10 JARs, one missing JAR may incorrectly prevent up to nine healthy plugins from being registered.
Could we retry the failed batch individually, or recursively split it, so that only the plugins whose JARs are actually unavailable are marked as failed?
2. The new user-facing messages need to use the i18n message classes
The new String.format and logger templates are user/operator-visible English literals, for example:
"%s Plugins: %s, jars: %s, status: %s"
"{} Failed to save jar {} for pipe plugin {}."
According to the repository’s i18n convention, these should be complete constants in both the English and Chinese DataNodePipeMessages classes, with matching placeholders.
It would also be useful to add focused tests covering:
one unavailable JAR alongside healthy JARs in the same batch;
an unsuccessful or malformed ConfigNode response;
a JAR-count mismatch;
failure to save one JAR without affecting subsequent JARs;
the plugin-level failure information exposed through SHOW PIPEPLUGINS.
One additional operational consideration: after startup, an unavailable plugin does not appear to be retried automatically when its JAR becomes available again. If recovery requires restarting the DataNode, that behavior should at least be documented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Pipe] Allow startup when a plugin JAR is unavailable [iotdb]
Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 commented on code in PR #18496:
URL: https://github.com/apache/iotdb/pull/18496#discussion_r3893233025
##########
iotdb-core/datanode/src/main/i18n/en/org/apache/iotdb/db/i18n/DataNodePipeMessages.java:
##########
@@ -759,7 +778,7 @@ public final class DataNodePipeMessages {
public static final String FAILED_TO_CONVERT_STATEMENT_TO_TABLET_FOR =
"Failed to convert statement to tablet for serialization";
public static final String FAILED_TO_CREATE_GROUP_ERROR_CODE_0X =
- "Failed to create group，error code: 0x";
+ "Failed to create group?error code: 0x";
Review Comment:
？
The same below.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/agent/runtime/PipeAgentLauncher.java:
##########
@@ -137,28 +141,152 @@ private static List<PipePluginMeta> getUninstalledOrConflictedPipePluginMetaList
return pipePluginMetaList;
}
- private static void fetchAndSavePipePluginJars(List<PipePluginMeta> pipePluginMetaList)
- throws StartupException {
+ static Set<String> fetchAndSavePipePluginJars(List<PipePluginMeta> pipePluginMetaList) {
+ if (pipePluginMetaList.isEmpty()) {
+ return new HashSet<>();
+ }
Review Comment:
Will the set be written? If not, use emptySet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Pipe] Allow startup when a plugin JAR is unavailable [iotdb]
Posted by "Caideyipi (via GitHub)" <gi...@apache.org>.
Caideyipi commented on PR #18496:
URL: https://github.com/apache/iotdb/pull/18496#issuecomment-5351127372
Thanks for the review. I applied the requested changes in follow-up commit [`8cf4615`](https://github.com/Caideyipi/iotdb/commit/8cf4615423a0753d2ad2b6e85625eabeb320149c).
- Batch fetch failures, unsuccessful responses, and JAR-count mismatches now retry each plugin individually. Only plugins that still fail are marked unavailable; healthy plugins continue through save and registration.
- Individual save failures are isolated so a later plugin is still processed.
- All new operator-facing fetch/save messages are complete EN/ZH `DataNodePipeMessages` constants, and these failure paths log at `ERROR`.
- Added focused tests for batch fallback, count mismatch, isolated save failure, and recorded plugin failure metadata.
Recovery is not polled after startup. If the JAR is restored on ConfigNode, the affected DataNode must be restarted to retry startup loading; recreating the plugin follows the normal ConfigNode-to-DataNode registration flow.
Spotless, Checkstyle, and `git diff --check` pass. The full DataNode `test-compile` is still blocked by pre-existing generated-source/dependency errors in unrelated query-engine fill and aggregation classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Pipe] Allow startup when a plugin JAR is unavailable [iotdb]
Posted by "Caideyipi (via GitHub)" <gi...@apache.org>.
Caideyipi commented on code in PR #18496:
URL: https://github.com/apache/iotdb/pull/18496#discussion_r3946048348
##########
iotdb-core/datanode/src/main/i18n/en/org/apache/iotdb/db/i18n/DataNodePipeMessages.java:
##########
@@ -759,7 +778,7 @@ public final class DataNodePipeMessages {
public static final String FAILED_TO_CONVERT_STATEMENT_TO_TABLET_FOR =
"Failed to convert statement to tablet for serialization";
public static final String FAILED_TO_CREATE_GROUP_ERROR_CODE_0X =
- "Failed to create group，error code: 0x";
+ "Failed to create group?error code: 0x";
Review Comment:
Fixed in https://github.com/Caideyipi/iotdb/commit/443b8257bbd0fe3d2b3fabf8417546a04da37d7b. Restored the affected punctuation and Unicode text in both i18n message classes while retaining the six new EN/ZH message constants.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/agent/runtime/PipeAgentLauncher.java:
##########
@@ -137,28 +141,152 @@ private static List<PipePluginMeta> getUninstalledOrConflictedPipePluginMetaList
return pipePluginMetaList;
}
- private static void fetchAndSavePipePluginJars(List<PipePluginMeta> pipePluginMetaList)
- throws StartupException {
+ static Set<String> fetchAndSavePipePluginJars(List<PipePluginMeta> pipePluginMetaList) {
+ if (pipePluginMetaList.isEmpty()) {
+ return new HashSet<>();
+ }
Review Comment:
Fixed in https://github.com/Caideyipi/iotdb/commit/443b8257bbd0fe3d2b3fabf8417546a04da37d7b. The returned set is only consumed by addAll and is never mutated, so the empty-input path now returns Collections.emptySet().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Pipe] Allow startup when a plugin JAR is unavailable [iotdb]
Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 merged PR #18496:
URL: https://github.com/apache/iotdb/pull/18496
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org