You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "DanielWang2035 (via GitHub)" <gi...@apache.org> on 2023/10/30 13:12:12 UTC
[PR] Pipe: multiple bug fixes [iotdb]
DanielWang2035 opened a new pull request, #11425:
URL: https://github.com/apache/iotdb/pull/11425
This PR includes:
- Disable batch mode for retry connector in case some events are never retried
- Pipe won't collect TsFile without tail magic
- Make async connector stop properly when retry count exceeds limit (exception will be properly reported by HeartbeatEvents)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pipe: Disable batch mode for retry connector in case some events are never retried & Pipe won't collect TsFile without tail magic & Make async connector stop properly when retry count exceeds limit [iotdb]
Posted by "SteveYurongSu (via GitHub)" <gi...@apache.org>.
SteveYurongSu merged PR #11425:
URL: https://github.com/apache/iotdb/pull/11425
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pipe: multiple bug fixes [iotdb]
Posted by "SteveYurongSu (via GitHub)" <gi...@apache.org>.
SteveYurongSu commented on code in PR #11425:
URL: https://github.com/apache/iotdb/pull/11425#discussion_r1376497511
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/protocol/thrift/sync/IoTDBThriftSyncConnector.java:
##########
@@ -231,6 +231,11 @@ public void transfer(TsFileInsertionEvent tsFileInsertionEvent) throws Exception
return;
}
+ if (!((PipeTsFileInsertionEvent) tsFileInsertionEvent).waitForTsFileClose()) {
+ LOGGER.warn("Pipe found a temporary TsFile which shouldn't be transferred. Skipping.");
Review Comment:
print file name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pipe: multiple bug fixes [iotdb]
Posted by "SteveYurongSu (via GitHub)" <gi...@apache.org>.
SteveYurongSu commented on code in PR #11425:
URL: https://github.com/apache/iotdb/pull/11425#discussion_r1376495603
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/protocol/airgap/IoTDBAirGapConnector.java:
##########
@@ -240,6 +240,11 @@ public void transfer(TsFileInsertionEvent tsFileInsertionEvent) throws Exception
return;
}
+ if (!((PipeTsFileInsertionEvent) tsFileInsertionEvent).waitForTsFileClose()) {
+ LOGGER.warn("Pipe found a temporary TsFile which shouldn't be transferred. Skipping.");
Review Comment:
print file name.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/protocol/legacy/IoTDBLegacyPipeConnector.java:
##########
@@ -206,6 +206,11 @@ public void transfer(TsFileInsertionEvent tsFileInsertionEvent) throws Exception
"IoTDBLegacyPipeConnector only support PipeTsFileInsertionEvent.");
}
+ if (!((PipeTsFileInsertionEvent) tsFileInsertionEvent).waitForTsFileClose()) {
+ LOGGER.warn("Pipe found a temporary TsFile which shouldn't be transferred. Skipping.");
Review Comment:
print file name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pipe: multiple bug fixes [iotdb]
Posted by "SteveYurongSu (via GitHub)" <gi...@apache.org>.
SteveYurongSu commented on code in PR #11425:
URL: https://github.com/apache/iotdb/pull/11425#discussion_r1376495956
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/protocol/thrift/async/IoTDBThriftAsyncConnector.java:
##########
@@ -291,26 +297,30 @@ public void transfer(TsFileInsertionEvent tsFileInsertionEvent) throws Exception
return;
}
- if (((EnrichedEvent) tsFileInsertionEvent).shouldParsePatternOrTime()) {
+ final PipeTsFileInsertionEvent pipeTsFileInsertionEvent =
+ (PipeTsFileInsertionEvent) tsFileInsertionEvent;
+ if (!pipeTsFileInsertionEvent.waitForTsFileClose()) {
+ LOGGER.warn("Pipe found a temporary TsFile which shouldn't be transferred. Skipping.");
Review Comment:
print file name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pipe: multiple bug fixes [iotdb]
Posted by "SteveYurongSu (via GitHub)" <gi...@apache.org>.
SteveYurongSu commented on code in PR #11425:
URL: https://github.com/apache/iotdb/pull/11425#discussion_r1376499388
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/heartbeat/PipeHeartbeatEvent.java:
##########
@@ -73,8 +73,12 @@ public PipeHeartbeatEvent(String dataRegionId, boolean shouldPrintMessage) {
this.shouldPrintMessage = shouldPrintMessage;
}
- public PipeHeartbeatEvent(String dataRegionId, long timePublished, boolean shouldPrintMessage) {
- super(null, null);
+ public PipeHeartbeatEvent(
+ String dataRegionId,
+ long timePublished,
+ boolean shouldPrintMessage,
+ PipeTaskMeta pipeTaskMeta) {
Review Comment:
```suggestion
public PipeHeartbeatEvent(
PipeTaskMeta pipeTaskMeta,
String dataRegionId,
long timePublished,
boolean shouldPrintMessage) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pipe: multiple bug fixes [iotdb]
Posted by "SteveYurongSu (via GitHub)" <gi...@apache.org>.
SteveYurongSu commented on code in PR #11425:
URL: https://github.com/apache/iotdb/pull/11425#discussion_r1376495194
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/protocol/thrift/async/IoTDBThriftAsyncConnector.java:
##########
@@ -342,7 +352,13 @@ private void transfer(
@Override
public void transfer(Event event) throws Exception {
- transferQueuedEventsIfNecessary();
+ try {
+ transferQueuedEventsIfNecessary();
+ } catch (Exception e) {
+ LOGGER.warn("caught ex in retry, rethrown.");
+ throw e;
+ }
+ LOGGER.warn("no ex in retry");
Review Comment:
```suggestion
transferQueuedEventsIfNecessary();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pipe: multiple bug fixes [iotdb]
Posted by "SteveYurongSu (via GitHub)" <gi...@apache.org>.
SteveYurongSu commented on code in PR #11425:
URL: https://github.com/apache/iotdb/pull/11425#discussion_r1376498187
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/EnrichedEvent.java:
##########
@@ -181,6 +181,8 @@ public abstract EnrichedEvent shallowCopySelfAndBindPipeTaskMetaForProgressRepor
public void reportException(PipeRuntimeException pipeRuntimeException) {
if (pipeTaskMeta != null) {
PipeAgent.runtime().report(pipeTaskMeta, pipeRuntimeException);
+ } else {
+ LOGGER.warn("Attempt to report pipe exception to a null PipeTaskMeta.");
Review Comment:
```suggestion
LOGGER.warn("Attempt to report pipe exception to a null PipeTaskMeta.", pipeRuntimeException);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org