You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Kikyou1997 (via GitHub)" <gi...@apache.org> on 2023/04/21 08:45:53 UTC
[GitHub] [doris] Kikyou1997 opened a new pull request, #18901: [feat](stats) Support to kill analyze
Kikyou1997 opened a new pull request, #18901:
URL: https://github.com/apache/doris/pull/18901
# Proposed changes
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #18901: [feat](stats) Support to kill analyze
Posted by "Kikyou1997 (via GitHub)" <gi...@apache.org>.
Kikyou1997 commented on PR #18901:
URL: https://github.com/apache/doris/pull/18901#issuecomment-1521307006
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #18901: [feat](stats) Support to kill analyze
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #18901:
URL: https://github.com/apache/doris/pull/18901#discussion_r1177359381
##########
fe/fe-core/src/main/cup/sql_parser.cup:
##########
@@ -4753,6 +4755,13 @@ kill_stmt ::=
:}
;
+kill_analysis_job_stmt ::=
+ KW_KILL KW_ANALYZE INTEGER_LITERAL:value
Review Comment:
after executing `analyze table XXX`, how can I get the INTEGER_LETEAL(job_id)?
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java:
##########
@@ -329,4 +331,91 @@ public void dropStats(DropStatsStmt dropStatsStmt) throws DdlException {
}
}
+ public void handleKillAnalyzeStmt(KillAnalysisJobStmt killAnalysisJobStmt) throws DdlException {
+ Map<Long, BaseAnalysisTask> analysisTaskInfoMap = analysisJobIdToTaskMap.remove(killAnalysisJobStmt.jobId);
+ if (analysisTaskInfoMap == null) {
+ throw new DdlException("Job not exists or already finished");
+ }
+ BaseAnalysisTask anyTask = analysisTaskInfoMap.values().stream().findFirst().orElse(null);
+ if (anyTask == null) {
+ return;
+ }
+ checkPriv(anyTask);
+ for (BaseAnalysisTask taskInfo : analysisTaskInfoMap.values()) {
+ taskInfo.markAsKilled();
+ }
+ Map<String, String> params = new HashMap<>();
+ params.put("jobState", AnalysisState.FAILED.toString());
+ params.put("message", ", message = 'Killed by user : " + ConnectContext.get().getQualifiedUser() + "'");
+ params.put("updateExecTime", ", last_exec_time_in_ms=" + String.valueOf(System.currentTimeMillis()));
+ params.put("jobId", String.valueOf(killAnalysisJobStmt.jobId));
+ params.put("taskId", "'-1'");
+ params.put("isAllTask", "true");
+ try {
+ StatisticsUtil.execUpdate(new StringSubstitutor(params).replace(UPDATE_JOB_STATE_SQL_TEMPLATE));
+ } catch (Exception e) {
+ LOG.warn("Failed to update status", e);
+ }
+ }
+
+ private void checkPriv(BaseAnalysisTask analysisTask) {
+ String dbName = analysisTask.db.getFullName();
+ String tblName = analysisTask.tbl.getName();
+ if (!Env.getCurrentEnv().getAccessManager()
+ .checkTblPriv(ConnectContext.get(), dbName, tblName, PrivPredicate.SELECT)) {
+ throw new RuntimeException("You need at least SELECT PRIV to corresponding table to kill this analyze"
Review Comment:
this message is not clear enough.
In my understanding, it means user does not have privilege to run this command.
1. SELECT PRIV means read privilege, but here we need write privilege. why SELECT privilege is enough?
2. `SELECT PRIV` is more like a internal term, but not user friendly. how about 'select privilege'
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java:
##########
@@ -76,12 +79,14 @@ public class AnalysisManager {
// The time field that needs to be displayed
private static final String LAST_EXEC_TIME_IN_MS = "last_exec_time_in_ms";
- private final ConcurrentMap<Long, Map<Long, AnalysisTaskInfo>> analysisJobIdToTaskMap;
+ private final ConcurrentMap<Long, Map<Long, BaseAnalysisTask>> analysisJobIdToTaskMap;
Review Comment:
what is the meaning of the inner map's key? (the outer map key means job id)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #18901: [feat](stats) Support to kill analyze
Posted by "Kikyou1997 (via GitHub)" <gi...@apache.org>.
Kikyou1997 commented on PR #18901:
URL: https://github.com/apache/doris/pull/18901#issuecomment-1522129900
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly merged pull request #18901: [feat](stats) Support to kill analyze
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly merged PR #18901:
URL: https://github.com/apache/doris/pull/18901
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #18901: [feat](stats) Support to kill analyze
Posted by "Kikyou1997 (via GitHub)" <gi...@apache.org>.
Kikyou1997 commented on code in PR #18901:
URL: https://github.com/apache/doris/pull/18901#discussion_r1177391193
##########
fe/fe-core/src/main/cup/sql_parser.cup:
##########
@@ -4753,6 +4755,13 @@ kill_stmt ::=
:}
;
+kill_analysis_job_stmt ::=
+ KW_KILL KW_ANALYZE INTEGER_LITERAL:value
Review Comment:
If you submit async analyze, FE would return a job id
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #18901: [feat](stats) Support to kill analyze
Posted by "Kikyou1997 (via GitHub)" <gi...@apache.org>.
Kikyou1997 commented on PR #18901:
URL: https://github.com/apache/doris/pull/18901#issuecomment-1522851027
> I have following concerns
>
> 1. to cancel a analyze job, we mark the job as killed, but do not clear its partial results. is that right?
> 2. could you add a regression test to demonstrate KILL ANALYZE works?
1. Column level stats if is final wroten, so it's ok to leave some partition level stats in table, and it won't cause loading incomplete stats
2. Yes, we should, however, the pipeline is parallel running and it's hard to have a determined job's state for it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #18901: [feat](stats) Support to kill analyze
Posted by "Kikyou1997 (via GitHub)" <gi...@apache.org>.
Kikyou1997 commented on code in PR #18901:
URL: https://github.com/apache/doris/pull/18901#discussion_r1177391592
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java:
##########
@@ -76,12 +79,14 @@ public class AnalysisManager {
// The time field that needs to be displayed
private static final String LAST_EXEC_TIME_IN_MS = "last_exec_time_in_ms";
- private final ConcurrentMap<Long, Map<Long, AnalysisTaskInfo>> analysisJobIdToTaskMap;
+ private final ConcurrentMap<Long, Map<Long, BaseAnalysisTask>> analysisJobIdToTaskMap;
Review Comment:
It means task id
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18901: [feat](stats) Support to kill analyze
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18901:
URL: https://github.com/apache/doris/pull/18901#issuecomment-1522847906
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18901: [feat](stats) Support to kill analyze
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18901:
URL: https://github.com/apache/doris/pull/18901#issuecomment-1522847871
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #18901: [feat](stats) Support to kill analyze
Posted by "Kikyou1997 (via GitHub)" <gi...@apache.org>.
Kikyou1997 commented on PR #18901:
URL: https://github.com/apache/doris/pull/18901#issuecomment-1522135905
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #18901: [feat](stats) Support to kill analyze
Posted by "Kikyou1997 (via GitHub)" <gi...@apache.org>.
Kikyou1997 commented on PR #18901:
URL: https://github.com/apache/doris/pull/18901#issuecomment-1521513054
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on pull request #18901: [feat](stats) Support to kill analyze
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #18901:
URL: https://github.com/apache/doris/pull/18901#issuecomment-1522837808
I have following concerns
1. to cancel a analyze job, we mark the job as killed, but do not clear its partial results. is that right?
2. could you add a regression test to demonstrate KILL ANALYZE works?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #18901: [feat](stats) Support to kill analyze
Posted by "Kikyou1997 (via GitHub)" <gi...@apache.org>.
Kikyou1997 commented on PR #18901:
URL: https://github.com/apache/doris/pull/18901#issuecomment-1522130432
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #18901: [feat](stats) Support to kill analyze
Posted by "Kikyou1997 (via GitHub)" <gi...@apache.org>.
Kikyou1997 commented on PR #18901:
URL: https://github.com/apache/doris/pull/18901#issuecomment-1521614858
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #18901: [feat](stats) Support to kill analyze
Posted by "Kikyou1997 (via GitHub)" <gi...@apache.org>.
Kikyou1997 commented on PR #18901:
URL: https://github.com/apache/doris/pull/18901#issuecomment-1517492198
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #18901: [feat](stats) Support to kill analyze
Posted by "Kikyou1997 (via GitHub)" <gi...@apache.org>.
Kikyou1997 commented on PR #18901:
URL: https://github.com/apache/doris/pull/18901#issuecomment-1517493671
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #18901: [feat](stats) Support to kill analyze
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #18901:
URL: https://github.com/apache/doris/pull/18901#issuecomment-1517573527
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.28 seconds
stream load tsv: 424 seconds loaded 74807831229 Bytes, about 168 MB/s
stream load json: 24 seconds loaded 2358488459 Bytes, about 93 MB/s
stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230421095045_clickbench_pr_132429.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org