You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zddr (via GitHub)" <gi...@apache.org> on 2023/06/05 08:12:05 UTC
[GitHub] [doris] zddr opened a new pull request, #20441: [feature-wip](MTMV)task only log finish status
zddr opened a new pull request, #20441:
URL: https://github.com/apache/doris/pull/20441
## Proposed changes
Issue Number: close #xxx
mtmv task keep only finish status,reduce the loss caused by logging
After changes, unfinished tasks will be directly lost when fe master restart
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] SaintBacchus commented on a diff in pull request #20441: [feature-wip](MTMV)task only log finish status
Posted by "SaintBacchus (via GitHub)" <gi...@apache.org>.
SaintBacchus commented on code in PR #20441:
URL: https://github.com/apache/doris/pull/20441#discussion_r1220944326
##########
fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVTaskManager.java:
##########
@@ -328,86 +318,10 @@ public MTMVTask getTask(String taskId) throws AnalysisException {
}
public void replayCreateJobTask(MTMVTask task) {
- if (task.getState() == TaskState.SUCCESS || task.getState() == TaskState.FAILURE) {
- if (MTMVUtils.getNowTimeStamp() > task.getExpireTime()) {
- return;
- }
- }
-
- switch (task.getState()) {
- case PENDING:
- String jobName = task.getJobName();
- MTMVJob job = mtmvJobManager.getJob(jobName);
- if (job == null) {
- LOG.warn("fail to obtain task name {} because task is null", jobName);
- return;
- }
- MTMVTaskExecutor taskExecutor = MTMVUtils.buildTask(job);
- taskExecutor.setTask(task);
- arrangeToPendingTask(taskExecutor);
- break;
- case RUNNING:
- task.setState(TaskState.FAILURE);
- addHistory(task);
- break;
- case FAILURE:
- case SUCCESS:
- addHistory(task);
- break;
- default:
- break;
- }
- }
-
- public void replayUpdateTask(ChangeMTMVTask changeTask) {
- TaskState fromStatus = changeTask.getFromStatus();
- TaskState toStatus = changeTask.getToStatus();
- Long jobId = changeTask.getJobId();
- if (fromStatus == TaskState.PENDING) {
- Queue<MTMVTaskExecutor> taskQueue = getPendingTaskMap().get(jobId);
- if (taskQueue == null) {
- return;
- }
- if (taskQueue.size() == 0) {
- getPendingTaskMap().remove(jobId);
- return;
- }
-
- MTMVTaskExecutor pendingTask = taskQueue.poll();
- MTMVTask status = pendingTask.getTask();
-
- if (toStatus == TaskState.RUNNING) {
- if (status.getTaskId().equals(changeTask.getTaskId())) {
- status.setState(TaskState.RUNNING);
- getRunningTaskMap().put(jobId, pendingTask);
- }
- } else if (toStatus == TaskState.FAILURE) {
- status.setMessage(changeTask.getErrorMessage());
- status.setErrorCode(changeTask.getErrorCode());
- status.setState(TaskState.FAILURE);
- addHistory(status);
- }
- if (taskQueue.size() == 0) {
- getPendingTaskMap().remove(jobId);
- }
- } else if (fromStatus == TaskState.RUNNING && (toStatus == TaskState.SUCCESS
- || toStatus == TaskState.FAILURE)) {
- MTMVTaskExecutor runningTask = getRunningTaskMap().remove(jobId);
- if (runningTask == null) {
- return;
- }
- MTMVTask status = runningTask.getTask();
- if (status.getTaskId().equals(changeTask.getTaskId())) {
- status.setMessage(changeTask.getErrorMessage());
- status.setErrorCode(changeTask.getErrorCode());
- status.setState(toStatus);
- status.setFinishTime(changeTask.getFinishTime());
- addHistory(status);
- }
- } else {
- LOG.warn("Illegal Task taskId:{} status transform from {} to {}", changeTask.getTaskId(), fromStatus,
- toStatus);
+ if (MTMVUtils.getNowTimeStamp() > task.getExpireTime()) {
Review Comment:
clear expired task in reply. it will cause inconsinst in different fe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zddr commented on pull request #20441: [feature-wip](MTMV)task only log finish status
Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #20441:
URL: https://github.com/apache/doris/pull/20441#issuecomment-1577978790
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20441: [feature-wip](MTMV)task only log finish status
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20441:
URL: https://github.com/apache/doris/pull/20441#issuecomment-1579987613
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20441: [feature-wip](MTMV)task only log finish status
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20441:
URL: https://github.com/apache/doris/pull/20441#issuecomment-1579987671
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] adonis0147 merged pull request #20441: [feature-wip](MTMV) Sync finish status only for tasks
Posted by "adonis0147 (via GitHub)" <gi...@apache.org>.
adonis0147 merged PR #20441:
URL: https://github.com/apache/doris/pull/20441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zddr commented on pull request #20441: [feature-wip](MTMV)task only log finish status
Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #20441:
URL: https://github.com/apache/doris/pull/20441#issuecomment-1578070797
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zddr commented on pull request #20441: [feature-wip](MTMV)task only log finish status
Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #20441:
URL: https://github.com/apache/doris/pull/20441#issuecomment-1579731768
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20441: [feature-wip](MTMV)task only log finish status
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20441:
URL: https://github.com/apache/doris/pull/20441#issuecomment-1580152121
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] adonis0147 commented on pull request #20441: [feature-wip](MTMV)task only log finish status
Posted by "adonis0147 (via GitHub)" <gi...@apache.org>.
adonis0147 commented on PR #20441:
URL: https://github.com/apache/doris/pull/20441#issuecomment-1580150255
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zddr commented on pull request #20441: [feature-wip](MTMV)task only log finish status
Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #20441:
URL: https://github.com/apache/doris/pull/20441#issuecomment-1576556724
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zddr commented on pull request #20441: [feature-wip](MTMV)task only log finish status
Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #20441:
URL: https://github.com/apache/doris/pull/20441#issuecomment-1576560594
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org