You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/12/16 08:49:13 UTC
[GitHub] [incubator-doris] morningman opened a new pull request #5093: [Bug] Fix bug that routine load may lost some data
morningman opened a new pull request #5093:
URL: https://github.com/apache/incubator-doris/pull/5093
## Proposed changes
In the previous implementation, whether a subtask is in commit or abort state,
we will try to update the job progress, such as the consumed offset of kafka.
Under normal circumstances, the aborted transaction does not consume any data,
and all progress is 0, so even we update the progress, the progress will remain
unchanged.
However, in the case of high cluster load, the subtask may fail half of the execution on the BE side.
At this time, although the task is aborted, part of the progress is updated.
Cause the next subtask to skip these data for consumption, resulting in data loss.
Also modify some log level and config
## Types of changes
- [x] Bugfix (non-breaking change which fixes an issue)
## Checklist
- [] I have create an issue on (Fix #ISSUE), and have described the bug/feature there in detail
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #5093: [Bug] Fix bug that routine load may lost some data
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #5093:
URL: https://github.com/apache/incubator-doris/pull/5093#discussion_r544216730
##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaProgress.java
##########
@@ -38,6 +42,8 @@
*/
// {"partitionIdToOffset": {}}
public class KafkaProgress extends RoutineLoadProgress {
+ private static final Logger LOG = LogManager.getLogger(KafkaProgress.class);
Review comment:
```suggestion
private static final Logger LOG = LogManager.getLogger(RoutineLoadProgress.class);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #5093: [Bug] Fix bug that routine load may lost some data
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 commented on a change in pull request #5093:
URL: https://github.com/apache/incubator-doris/pull/5093#discussion_r544866386
##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
##########
@@ -1049,8 +1053,9 @@ private void executeTaskOnTxnStatusChanged(RoutineLoadTaskInfo routineLoadTaskIn
+ " maybe task was aborted by master when timeout")
.build());
}
- } else if (checkCommitInfo(rlTaskTxnCommitAttachment, txnState.getTransactionStatus())) {
+ } else if (checkCommitInfo(rlTaskTxnCommitAttachment, txnState)) {
// step2: update job progress
+ // only committed task need to update progress
Review comment:
error commit...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #5093: [Bug] Fix bug that routine load may lost some data
Posted by GitBox <gi...@apache.org>.
morningman merged pull request #5093:
URL: https://github.com/apache/incubator-doris/pull/5093
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #5093: [Bug] Fix bug that routine load may lost some data
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #5093:
URL: https://github.com/apache/incubator-doris/pull/5093#discussion_r544216730
##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaProgress.java
##########
@@ -38,6 +42,8 @@
*/
// {"partitionIdToOffset": {}}
public class KafkaProgress extends RoutineLoadProgress {
+ private static final Logger LOG = LogManager.getLogger(KafkaProgress.class);
Review comment:
```suggestion
private static final Logger LOG = LogManager.getLogger(RoutineLoadProgress.class);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org