You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/04/30 08:05:30 UTC
[GitHub] [incubator-kyuubi] wForget opened a new pull request, #2529: [WIP][KYUUBI #2493] Implement the progress of statement
wForget opened a new pull request, #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
close #2493
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [X] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] turboFei commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement for spark sql engine
Posted by GitBox <gi...@apache.org>.
turboFei commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1120414621
thanks, merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r867334993
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/operation/ExecuteStatement.scala:
##########
@@ -81,6 +81,9 @@ class ExecuteStatement(
while (!isComplete) {
fetchQueryLog()
verifyTStatus(statusResp.getStatus)
+ if (statusResp.getProgressUpdateResponse != null) {
+ setOperationJobProgress(statusResp.getProgressUpdateResponse)
Review Comment:
https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1120181210 here might be the best place we stringify the ProgressUpdateResponse and do the log
And maybe refine L88 together
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r867335430
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/operation/ExecuteStatement.scala:
##########
@@ -81,6 +81,9 @@ class ExecuteStatement(
while (!isComplete) {
fetchQueryLog()
verifyTStatus(statusResp.getStatus)
+ if (statusResp.getProgressUpdateResponse != null) {
+ setOperationJobProgress(statusResp.getProgressUpdateResponse)
Review Comment:
SGTM, I will send a follow-up pr to implement it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1120167308
> since this progress needs to be rendered by itself on the client side.
How hard is it that we do this on the server side?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1120181210
try to make myself clear:
- both 2 features are for job progress and the functionality is duplicated and might be heavy. it's good to merge them.
- IIUC, the existing one is logging progress info to log files and feeds clients who support operation log, the one you add provides progress info through thrift API and feeds clients who implement this feature too.
- So, we just add `ONE MORE log.info` at the server side and the existing one can be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on pull request #2529: [WIP][KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1113946301
`SparkStatusTracker` doesn't seem to distinguish skipped stages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #2529: [WIP][KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1113951686
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/2529?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#2529](https://codecov.io/gh/apache/incubator-kyuubi/pull/2529?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f7f7d42) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/cdfae8d879194a355fdaa03dc33f23578d897402?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cdfae8d) will **increase** coverage by `0.17%`.
> The diff coverage is `87.01%`.
```diff
@@ Coverage Diff @@
## master #2529 +/- ##
============================================
+ Coverage 63.63% 63.80% +0.17%
Complexity 41 41
============================================
Files 373 377 +4
Lines 17829 17981 +152
Branches 2378 2392 +14
============================================
+ Hits 11345 11473 +128
- Misses 5430 5441 +11
- Partials 1054 1067 +13
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/2529?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../spark/operation/progress/SparkStageProgress.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/2529/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZXh0ZXJuYWxzL2t5dXViaS1zcGFyay1zcWwtZW5naW5lL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL2VuZ2luZS9zcGFyay9vcGVyYXRpb24vcHJvZ3Jlc3MvU3BhcmtTdGFnZVByb2dyZXNzLnNjYWxh) | `55.55% <55.55%> (ø)` | |
| [...rg/apache/kyuubi/operation/JobProgressUpdate.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/2529/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vSm9iUHJvZ3Jlc3NVcGRhdGUuc2NhbGE=) | `63.15% <63.15%> (ø)` | |
| [...park/operation/progress/SparkProgressMonitor.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/2529/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZXh0ZXJuYWxzL2t5dXViaS1zcGFyay1zcWwtZW5naW5lL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL2VuZ2luZS9zcGFyay9vcGVyYXRpb24vcHJvZ3Jlc3MvU3BhcmtQcm9ncmVzc01vbml0b3Iuc2NhbGE=) | `88.31% <88.31%> (ø)` | |
| [...uubi/engine/spark/operation/ExecuteStatement.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/2529/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZXh0ZXJuYWxzL2t5dXViaS1zcGFyay1zcWwtZW5naW5lL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL2VuZ2luZS9zcGFyay9vcGVyYXRpb24vRXhlY3V0ZVN0YXRlbWVudC5zY2FsYQ==) | `83.13% <100.00%> (+0.63%)` | :arrow_up: |
| [...park/operation/progress/SparkProgressFetcher.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/2529/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZXh0ZXJuYWxzL2t5dXViaS1zcGFyay1zcWwtZW5naW5lL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL2VuZ2luZS9zcGFyay9vcGVyYXRpb24vcHJvZ3Jlc3MvU3BhcmtQcm9ncmVzc0ZldGNoZXIuc2NhbGE=) | `100.00% <100.00%> (ø)` | |
| [...rg/apache/kyuubi/operation/AbstractOperation.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/2529/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vQWJzdHJhY3RPcGVyYXRpb24uc2NhbGE=) | `84.50% <100.00%> (+0.68%)` | :arrow_up: |
| [.../org/apache/kyuubi/operation/OperationStatus.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/2529/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vT3BlcmF0aW9uU3RhdHVzLnNjYWxh) | `88.88% <100.00%> (+1.38%)` | :arrow_up: |
| [...a/org/apache/kyuubi/service/TFrontendService.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/2529/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2aWNlL1RGcm9udGVuZFNlcnZpY2Uuc2NhbGE=) | `95.04% <100.00%> (+0.12%)` | :arrow_up: |
| [...org/apache/kyuubi/operation/ExecuteStatement.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/2529/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vRXhlY3V0ZVN0YXRlbWVudC5zY2FsYQ==) | `84.33% <100.00%> (+1.67%)` | :arrow_up: |
| [...rg/apache/kyuubi/engine/trino/TrinoSqlEngine.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/2529/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZXh0ZXJuYWxzL2t5dXViaS10cmluby1lbmdpbmUvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9reXV1YmkvZW5naW5lL3RyaW5vL1RyaW5vU3FsRW5naW5lLnNjYWxh) | `52.63% <0.00%> (-7.02%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/2529?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/2529?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [cdfae8d...f7f7d42](https://codecov.io/gh/apache/incubator-kyuubi/pull/2529?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] turboFei commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
turboFei commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r866754306
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecuteStatement.scala:
##########
@@ -157,6 +160,11 @@ class ExecuteStatement(
SparkOperationEvent(this, operationListener.getExecutionId))
}
+ override def getStatus: OperationStatus = {
+ setOperationJobProgress(progressFetcher.getJobProgressUpdate(startTime))
Review Comment:
I see, there is a parameter to enable it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r867311670
##########
docs/deployment/settings.md:
##########
@@ -381,6 +381,7 @@ Key | Default | Meaning | Type | Since
<code>kyuubi.session.engine.trino.main.resource</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'><undefined></div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The package used to create Trino engine remote job. If it is undefined, Kyuubi will use the default</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.5.0</div>
<code>kyuubi.session.idle.timeout</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>PT6H</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>session idle timeout, it will be closed when it's not accessed for this duration</div>|<div style='width: 30pt'>duration</div>|<div style='width: 20pt'>1.2.0</div>
<code>kyuubi.session.name</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'><undefined></div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>A human readable name of session and we use empty string by default. This name will be recorded in event. Note that, we only apply this value from session conf.</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.4.0</div>
+<code>kyuubi.session.progress.enable</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>false</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Whether to enable the operation progress. When true, the operation progress will be returned in `GetOperationStatus`.</div>|<div style='width: 30pt'>boolean</div>|<div style='width: 20pt'>1.6.0</div>
Review Comment:
I don't think we need to define different configurations for different engines because the GetOperationStatus method is common and we can implement it later for other engines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r867334006
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SparkProgressFetcher.scala:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.kyuubi
+
+import org.apache.hive.service.rpc.thrift.TProgressUpdateResp
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.status.api.v1.StageStatus
+
+import org.apache.kyuubi.engine.spark.operation.progress.{SparkProgressMonitor, SparkStage, SparkStageProgress}
+
+class SparkProgressFetcher(spark: SparkSession, jobGroup: String) {
+
+ val statusStore = spark.sparkContext.statusStore
+
+ def getJobProgressUpdate(startTime: Long): TProgressUpdateResp = {
+ val progressMap = getProgressMap
+ val progressMonitor = new SparkProgressMonitor(progressMap)
+ new TProgressUpdateResp(
+ progressMonitor.headers,
+ progressMonitor.rows,
+ progressMonitor.progressedPercentage,
+ progressMonitor.executionStatus,
+ progressMonitor.footerSummary,
+ startTime)
+ }
+
+ private def getProgressMap(): Map[SparkStage, SparkStageProgress] = {
Review Comment:
> How about removing SparkProgressFetcher and merging it into SparkProgressMonitor?
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1120180143
> it works fine for beeline, but other clients need to implement it further.
So, now `kyuubi.session.engine.spark.showProgress` works for both beeline and other clients ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r867320409
##########
docs/deployment/settings.md:
##########
@@ -381,6 +381,7 @@ Key | Default | Meaning | Type | Since
<code>kyuubi.session.engine.trino.main.resource</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'><undefined></div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The package used to create Trino engine remote job. If it is undefined, Kyuubi will use the default</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.5.0</div>
<code>kyuubi.session.idle.timeout</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>PT6H</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>session idle timeout, it will be closed when it's not accessed for this duration</div>|<div style='width: 30pt'>duration</div>|<div style='width: 20pt'>1.2.0</div>
<code>kyuubi.session.name</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'><undefined></div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>A human readable name of session and we use empty string by default. This name will be recorded in event. Note that, we only apply this value from session conf.</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.4.0</div>
+<code>kyuubi.session.progress.enable</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>false</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Whether to enable the operation progress. When true, the operation progress will be returned in `GetOperationStatus`.</div>|<div style='width: 30pt'>boolean</div>|<div style='width: 20pt'>1.6.0</div>
Review Comment:
kyuubi.`operation`.progress.enable`d`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r866488006
##########
externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/operation/SparkOperationProgressSuite.scala:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.engine.spark.operation
+
+import scala.collection.JavaConverters._
+
+import org.apache.hive.service.rpc.thrift.{TExecuteStatementReq, TGetOperationStatusReq, TJobExecutionStatus}
+import org.scalatest.concurrent.PatienceConfiguration.Timeout
+import org.scalatest.exceptions.TestFailedException
+import org.scalatest.time.SpanSugar.convertIntToGrainOfTime
+
+import org.apache.kyuubi.engine.spark.WithSparkSQLEngine
+import org.apache.kyuubi.operation.HiveJDBCTestHelper
+
+class SparkOperationProgressSuite extends WithSparkSQLEngine with HiveJDBCTestHelper {
+
+ override protected def jdbcUrl: String = getJdbcUrl
+ override def withKyuubiConf: Map[String, String] = Map(
+ "spark.master" -> "local[1]",
+ "kyuubi.operation.status.polling.timeout" -> "1000")
+
+ test("test operation progress") {
+ withSessionHandle { (client, handle) =>
+ val preReq = new TExecuteStatementReq()
+ preReq.setStatement("show databases")
+ preReq.setSessionHandle(handle)
+ preReq.setRunAsync(false)
+ client.ExecuteStatement(preReq)
+
+ val sql = "SELECT java_method('java.lang.Thread', 'sleep', 10000l) FROM range(1, 3, 1, 2)"
+ val req = new TExecuteStatementReq()
+ req.setStatement(sql)
+ req.setRunAsync(true)
+ req.setSessionHandle(handle)
+ val resp = client.ExecuteStatement(req)
+ var checkFlag1 = false
+ var checkFlag2 = false
+ var checkFlag3 = false
+ val initStageId = initJobId
+ eventually(Timeout(25.seconds)) {
+ val statusReq = new TGetOperationStatusReq(resp.getOperationHandle)
+ val statusResp = client.GetOperationStatus(statusReq)
+ val headers = statusResp.getProgressUpdateResponse.getHeaderNames
+ val progress = statusResp.getProgressUpdateResponse.getProgressedPercentage
+ val rows = statusResp.getProgressUpdateResponse.getRows
+ val footerSummary = statusResp.getProgressUpdateResponse.getFooterSummary
+ val status = statusResp.getProgressUpdateResponse.getStatus
+ assertResult(Seq(
+ "STAGES",
+ "ATTEMPT",
+ "STATUS",
+ "TOTAL",
+ "COMPLETED",
+ "RUNNING",
+ "PENDING",
+ "FAILED"))(headers.asScala)
+ assert(rows.size() == 1)
+ try {
+ progress match {
+ case 0.0 =>
+ assert(Seq(s"Stage-$initStageId ", "0", "PENDING", "2", "0", "0", "2", "0")
+ == rows.get(0).asScala ||
+ Seq(s"Stage-$initStageId ", "0", "RUNNING", "2", "0", "1", "1", "0")
+ == rows.get(0).asScala ||
+ Seq(s"Stage-$initStageId ", "0", "RUNNING", "2", "0", "2", "0", "0")
+ == rows.get(0).asScala)
+ assert("STAGES: 00/01" === footerSummary)
+ assert(TJobExecutionStatus.IN_PROGRESS === status)
+ checkFlag1 = true
+ case 0.5 =>
+ assertResult(Seq(
+ s"Stage-$initStageId ....",
+ "0",
+ "RUNNING",
+ "2",
+ "1",
+ "1",
+ "0",
+ "0"))(
+ rows.get(0).asScala)
+ assert("STAGES: 00/01" === footerSummary)
+ assert(TJobExecutionStatus.IN_PROGRESS === status)
+ checkFlag2 = true
+ case 1.0 =>
+ assertResult(Seq(
+ s"Stage-$initStageId ........",
+ "0",
+ "FINISHED",
+ "2",
+ "2",
+ "0",
+ "0",
+ "0"))(
+ rows.get(0).asScala)
+ assert("STAGES: 01/01" === footerSummary)
+ checkFlag3 = true
+ }
+ } catch {
+ case tfe: TestFailedException =>
+ // scalastyle:off println
+ println(s"progress: $progress")
+ println(s"rows: $rows")
+ println(s"footerSummary: $footerSummary")
+ println(s"status: $status")
+ // scalastyle:on println
Review Comment:
remove this, if needed we can use `info`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1119547236
shall we remove this feature？
kyuubi.session.engine.spark.showProgress
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1120175457
> How hard is it that we do this on the server side?
It doesn't seem hard, but I prefer to implement rendering on the client side, so that we can interface with other systems well, not just log progress info.
> Logging Progress info might work?
it works fine for beeline, but other clients need to implement it further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r865821832
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/JobProgressUpdate.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.operation
+
+import java.util
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.hive.service.rpc.thrift.TJobExecutionStatus
+
+case class JobProgressUpdate(
+ headers: util.List[String] = null,
+ rows: util.List[util.List[String]] = null,
+ footerSummary: String = null,
+ progressedPercentage: Double = 0.0,
+ startTimeMillis: Long = 0L,
+ status: String = null)
+
+object KyuubiProgressMonitorStatusMapper {
+
+ def forStatus(status: String): TJobExecutionStatus = {
+ if (StringUtils.isEmpty(status)) {
+ return TJobExecutionStatus.NOT_AVAILABLE
+ }
+ KyuubiStatus.withName(status) match {
+ case KyuubiStatus.NOT_AVAILABLE =>
+ TJobExecutionStatus.NOT_AVAILABLE
+ case KyuubiStatus.IN_PROGRESS | KyuubiStatus.PENDING | KyuubiStatus.RUNNING =>
+ TJobExecutionStatus.IN_PROGRESS
+ case KyuubiStatus.COMPLETE => TJobExecutionStatus.COMPLETE
+ case _ => TJobExecutionStatus.COMPLETE
+ }
+ }
+}
+
+object KyuubiStatus extends Enumeration {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1119494260
> In addition we may need a configuration to control whether to fetch operation progress.
@yaooqinn @ulysses-you how about named `kyuubi.session.progress.enable` and default to false?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r866523081
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/progress/SparkProgressMonitor.scala:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kyuubi.engine.spark.operation.progress
+
+import java.util
+
+import scala.collection.JavaConverters._
+
+import org.apache.kyuubi.engine.spark.operation.progress.SparkProgressMonitor._
+import org.apache.kyuubi.operation.KyuubiStatus
+
+class SparkProgressMonitor(progressMap: Map[SparkStage, SparkStageProgress], _startTime: Long = 0) {
+
+ def headers: util.List[String] = HEADERS
+
+ def rows: util.List[util.List[String]] = {
+ val progressRows = progressMap.toSeq.sortBy(_._1).map {
+ case (stage, progress) =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ val running = progress.runningTaskCount
+ val failed = progress.failedTaskCount
+ var state =
+ if (total > 0) {
+ KyuubiStatus.PENDING
+ } else {
+ KyuubiStatus.FINISHED
+ }
+ if (complete > 0 || running > 0 || failed > 0) {
+ state = if (complete < total) KyuubiStatus.RUNNING else KyuubiStatus.FINISHED
+ }
+ val attempt = String.valueOf(stage.attemptId)
+ val stageName = "Stage-" + String.valueOf(stage.stageId)
+ val nameWithProgress = getNameWithProgress(stageName, complete, total)
+ val pending = total - complete - running
+ util.Arrays.asList(
+ nameWithProgress,
+ attempt,
+ state.toString,
+ String.valueOf(total),
+ String.valueOf(complete),
+ String.valueOf(running),
+ String.valueOf(pending),
+ String.valueOf(failed),
+ "")
+ }.toList.asJavaCollection
+ new util.ArrayList[util.List[String]](progressRows)
+ }
+
+ def footerSummary: String = {
+ "STAGES: %02d/%02d".format(getCompletedStages, progressMap.keySet.size)
+ }
+
+ def progressedPercentage: Double = {
+ var sumTotal = 0
+ var sumComplete = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ sumTotal += total
+ sumComplete += complete
+ }
+ if (sumTotal == 0) {
+ 1.0f
+ } else {
+ sumComplete.toFloat / sumTotal.toFloat
+ }
+ }
+
+ def startTime: Long = _startTime
+
+ def executionStatus: String =
+ if (getCompletedStages == progressMap.keySet.size) {
Review Comment:
> does it affect the statement status if here return `TJobExecutionStatus.COMPLETE` ? Seems the progress status is isolation with statement status ?
It doesn't seem to affect, it's hardly used by client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1120159706
> kyuubi.session.engine.spark.showProgress
cc @wForget again `consoleProgressBar` is now duplicated with this feature, shall we remove it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] turboFei commented on pull request #2529: [WIP][KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
turboFei commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1113997306
> SparkStatusTracker doesn't seem to distinguish skipped stages. how about we use AppStatusStore to get stages?
Can we distinguish skipped stages with AppStatusStore?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] turboFei commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
turboFei commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r866753554
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecuteStatement.scala:
##########
@@ -157,6 +160,11 @@ class ExecuteStatement(
SparkOperationEvent(this, operationListener.getExecutionId))
}
+ override def getStatus: OperationStatus = {
+ setOperationJobProgress(progressFetcher.getJobProgressUpdate(startTime))
Review Comment:
nice feature.
I have a question. is the getJobProgressUpdate operation is heavy?
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecuteStatement.scala:
##########
@@ -157,6 +160,11 @@ class ExecuteStatement(
SparkOperationEvent(this, operationListener.getExecutionId))
}
+ override def getStatus: OperationStatus = {
+ setOperationJobProgress(progressFetcher.getJobProgressUpdate(startTime))
Review Comment:
nice feature.
I have a question. is the getJobProgressUpdate operation heavy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] turboFei commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement for spark sql engine
Posted by GitBox <gi...@apache.org>.
turboFei commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1120442900
```
0: jdbc:hive2://kyuubidev-lb.hadoopmaster-dev> set kyuubi.operation.progress.enabled=false;
08:53:01.960 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_kyuubi_dev's query[8ec9d7d6-d6a4-4976-8c13-7a28a3ea0268]: INITIALIZED_STATE -> PENDING_STATE, statement: set kyuubi.operation.progress.enabled=false
08:53:02.073 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_kyuubi_dev's query[8ec9d7d6-d6a4-4976-8c13-7a28a3ea0268]: PENDING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.progress.enabled=false
22/05/08 08:53:01 INFO ExecuteStatement: Processing b_kyuubi_dev's query[86d3a895-db3d-4751-bfa9-05eeb92afa16]: INITIALIZED_STATE -> PENDING_STATE, statement: set kyuubi.operation.progress.enabled=false
22/05/08 08:53:01 INFO ExecuteStatement: Processing b_kyuubi_dev's query[86d3a895-db3d-4751-bfa9-05eeb92afa16]: PENDING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.progress.enabled=false
22/05/08 08:53:01 INFO ExecuteStatement:
Spark application name: kyuubi_zeuslvs_USER_SPARK_SQL_b_kyuubi_dev_default_636f21eb-349b-47f0-bf3a-beef3b7146da
application ID: application_1644980694928_2371390
application web UI: https://zeus-lvs-rm-1.vip.hadoop.ebay.com:50030/proxy/application_1644980694928_2371390,https://zeus-lvs-rm-2.vip.hadoop.ebay.com:50030/proxy/application_1644980694928_2371390
master: yarn
deploy mode: cluster
version: 3.1.1.0.6.0
Start time: 2022-05-08T08:32:47.574
User: b_kyuubi_dev
22/05/08 08:53:01 INFO ExecuteStatement: Execute in full collect mode
22/05/08 08:53:01 INFO ExecuteStatement: Processing b_kyuubi_dev's query[86d3a895-db3d-4751-bfa9-05eeb92afa16]: RUNNING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.progress.enabled=false
22/05/08 08:53:01 INFO ExecuteStatement: Processing b_kyuubi_dev's query[86d3a895-db3d-4751-bfa9-05eeb92afa16]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.progress.enabled=false, time taken: 0.015 seconds
08:53:02.078 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[8ec9d7d6-d6a4-4976-8c13-7a28a3ea0268] in FINISHED_STATE
08:53:02.078 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_kyuubi_dev's query[8ec9d7d6-d6a4-4976-8c13-7a28a3ea0268]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.progress.enabled=false, time taken: 0.005 seconds
+------------------------------------+--------+
| key | value |
+------------------------------------+--------+
| kyuubi.operation.progress.enabled | false |
+------------------------------------+--------+
1 row selected (1.605 seconds)
0: jdbc:hive2://kyuubidev-lb.hadoopmaster-dev> SELECT java_method('java.lang.Thread', 'sleep', 10000l) as res FROM range(1, 3, 1, 2);
08:53:05.361 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_kyuubi_dev's query[700f7531-8d9b-473d-8ab7-e95221d58e6d]: INITIALIZED_STATE -> PENDING_STATE, statement: SELECT java_method('java.lang.Thread', 'sleep', 10000l) as res FROM range(1, 3, 1, 2)
08:53:05.365 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_kyuubi_dev's query[700f7531-8d9b-473d-8ab7-e95221d58e6d]: PENDING_STATE -> RUNNING_STATE, statement: SELECT java_method('java.lang.Thread', 'sleep', 10000l) as res FROM range(1, 3, 1, 2)
22/05/08 08:53:05 INFO ExecuteStatement: Processing b_kyuubi_dev's query[59ab4911-570f-4ed6-abeb-e0ed0b9045f9]: INITIALIZED_STATE -> PENDING_STATE, statement: SELECT java_method('java.lang.Thread', 'sleep', 10000l) as res FROM range(1, 3, 1, 2)
22/05/08 08:53:05 INFO ExecuteStatement: Processing b_kyuubi_dev's query[59ab4911-570f-4ed6-abeb-e0ed0b9045f9]: PENDING_STATE -> RUNNING_STATE, statement: SELECT java_method('java.lang.Thread', 'sleep', 10000l) as res FROM range(1, 3, 1, 2)
22/05/08 08:53:05 INFO ExecuteStatement:
Spark application name: kyuubi_zeuslvs_USER_SPARK_SQL_b_kyuubi_dev_default_636f21eb-349b-47f0-bf3a-beef3b7146da
application ID: application_1644980694928_2371390
application web UI: https://zeus-lvs-rm-1.vip.hadoop.ebay.com:50030/proxy/application_1644980694928_2371390,https://zeus-lvs-rm-2.vip.hadoop.ebay.com:50030/proxy/application_1644980694928_2371390
master: yarn
deploy mode: cluster
version: 3.1.1.0.6.0
Start time: 2022-05-08T08:32:47.574
User: b_kyuubi_dev
22/05/08 08:53:05 INFO ExecuteStatement: Execute in full collect mode
22/05/08 08:53:05 INFO SparkContext: Starting job: collect at ExecuteStatement.scala:135
22/05/08 08:53:05 INFO SQLOperationListener: Query [59ab4911-570f-4ed6-abeb-e0ed0b9045f9]: Job 35 started with 1 stages, 1 active jobs running
22/05/08 08:53:05 INFO SQLOperationListener: Query [59ab4911-570f-4ed6-abeb-e0ed0b9045f9]: Stage 46 started with 2 tasks, 1 active stages running
08:53:10.369 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[700f7531-8d9b-473d-8ab7-e95221d58e6d] in RUNNING_STATE
08:53:15.371 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[700f7531-8d9b-473d-8ab7-e95221d58e6d] in RUNNING_STATE
22/05/08 08:53:15 INFO SQLOperationListener: Finished stage: Stage(46, 0); Name: 'collect at ExecuteStatement.scala:135'; Status: succeeded; numTasks: 2; Took: 10037 msec
22/05/08 08:53:15 INFO DAGScheduler: Job 35 finished: collect at ExecuteStatement.scala:135, took 10.040103 s
22/05/08 08:53:15 INFO StatsReportListener: task runtime:(count: 2, mean: 10032.000000, stdev: 0.000000, max: 10032.000000, min: 10032.000000)
22/05/08 08:53:15 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
22/05/08 08:53:15 INFO StatsReportListener: 10.0 s 10.0 s 10.0 s 10.0 s 10.0 s 10.0 s 10.0 s 10.0 s 10.0 s
22/05/08 08:53:15 INFO StatsReportListener: shuffle bytes written:(count: 2, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
22/05/08 08:53:15 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
22/05/08 08:53:15 INFO StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B
22/05/08 08:53:15 INFO StatsReportListener: fetch wait time:(count: 2, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
22/05/08 08:53:15 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
22/05/08 08:53:15 INFO StatsReportListener: 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms
22/05/08 08:53:15 INFO StatsReportListener: remote bytes read:(count: 2, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
22/05/08 08:53:15 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
22/05/08 08:53:15 INFO StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B
22/05/08 08:53:15 INFO StatsReportListener: task result size:(count: 2, mean: 1257.000000, stdev: 0.000000, max: 1257.000000, min: 1257.000000)
22/05/08 08:53:15 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
22/05/08 08:53:15 INFO StatsReportListener: 1257.0 B 1257.0 B 1257.0 B 1257.0 B 1257.0 B 1257.0 B 1257.0 B 1257.0 B 1257.0 B
22/05/08 08:53:15 INFO StatsReportListener: executor (non-fetch) time pct: (count: 2, mean: 99.750797, stdev: 0.000000, max: 99.750797, min: 99.750797)
22/05/08 08:53:15 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
22/05/08 08:53:15 INFO StatsReportListener: 100 % 100 % 100 % 100 % 100 % 100 % 100 % 100 % 100 %
22/05/08 08:53:15 INFO StatsReportListener: fetch wait time pct: (count: 2, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
22/05/08 08:53:15 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
22/05/08 08:53:15 INFO StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 %
22/05/08 08:53:15 INFO StatsReportListener: other time pct: (count: 2, mean: 0.249203, stdev: 0.000000, max: 0.249203, min: 0.249203)
22/05/08 08:53:15 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
22/05/08 08:53:15 INFO StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 %
22/05/08 08:53:15 INFO SQLOperationListener: Query [59ab4911-570f-4ed6-abeb-e0ed0b9045f9]: Job 35 succeeded, 0 active jobs running
22/05/08 08:53:15 INFO ExecuteStatement: Processing b_kyuubi_dev's query[59ab4911-570f-4ed6-abeb-e0ed0b9045f9]: RUNNING_STATE -> FINISHED_STATE, statement: SELECT java_method('java.lang.Thread', 'sleep', 10000l) as res FROM range(1, 3, 1, 2), time taken: 10.071 seconds
08:53:15.437 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[700f7531-8d9b-473d-8ab7-e95221d58e6d] in FINISHED_STATE
08:53:15.437 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_kyuubi_dev's query[700f7531-8d9b-473d-8ab7-e95221d58e6d]: RUNNING_STATE -> FINISHED_STATE, statement: SELECT java_method('java.lang.Thread', 'sleep', 10000l) as res FROM range(1, 3, 1, 2), time taken: 10.072 seconds
+-------+
| res |
+-------+
| null |
| null |
+-------+
2 rows selected (11.388 seconds)
0: jdbc:hive2://kyuubidev-lb.hadoopmaster-dev> set kyuubi.operation.progress.enabled=true;
08:53:39.589 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_kyuubi_dev's query[31480d05-87a3-4e09-991c-3216b37985ea]: INITIALIZED_STATE -> PENDING_STATE, statement: set kyuubi.operation.progress.enabled=true
08:53:39.593 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_kyuubi_dev's query[31480d05-87a3-4e09-991c-3216b37985ea]: PENDING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.progress.enabled=true
22/05/08 08:53:39 INFO ExecuteStatement: Processing b_kyuubi_dev's query[881f0e75-e0b9-4be0-9945-b7b7f0ad6455]: INITIALIZED_STATE -> PENDING_STATE, statement: set kyuubi.operation.progress.enabled=true
22/05/08 08:53:39 INFO ExecuteStatement: Processing b_kyuubi_dev's query[881f0e75-e0b9-4be0-9945-b7b7f0ad6455]: PENDING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.progress.enabled=true
22/05/08 08:53:39 INFO ExecuteStatement:
Spark application name: kyuubi_zeuslvs_USER_SPARK_SQL_b_kyuubi_dev_default_636f21eb-349b-47f0-bf3a-beef3b7146da
application ID: application_1644980694928_2371390
application web UI: https://zeus-lvs-rm-1.vip.hadoop.ebay.com:50030/proxy/application_1644980694928_2371390,https://zeus-lvs-rm-2.vip.hadoop.ebay.com:50030/proxy/application_1644980694928_2371390
master: yarn
deploy mode: cluster
version: 3.1.1.0.6.0
Start time: 2022-05-08T08:32:47.574
User: b_kyuubi_dev
22/05/08 08:53:39 INFO ExecuteStatement: Execute in full collect mode
22/05/08 08:53:39 INFO ExecuteStatement: Processing b_kyuubi_dev's query[881f0e75-e0b9-4be0-9945-b7b7f0ad6455]: RUNNING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.progress.enabled=true
22/05/08 08:53:39 INFO ExecuteStatement: Processing b_kyuubi_dev's query[881f0e75-e0b9-4be0-9945-b7b7f0ad6455]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.progress.enabled=true, time taken: 0.014 seconds
08:53:39.608 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[31480d05-87a3-4e09-991c-3216b37985ea] in FINISHED_STATE
08:53:39.608 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_kyuubi_dev's query[31480d05-87a3-4e09-991c-3216b37985ea]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.progress.enabled=true, time taken: 0.015 seconds
+------------------------------------+--------+
| key | value |
+------------------------------------+--------+
| kyuubi.operation.progress.enabled | true |
+------------------------------------+--------+
1 row selected (1.519 seconds)
0: jdbc:hive2://kyuubidev-lb.hadoopmaster-dev> SELECT java_method('java.lang.Thread', 'sleep', 10000l) as res FROM range(1, 3, 1, 2);
08:53:43.145 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_kyuubi_dev's query[cd3aec1e-4fd1-44cc-a849-d6fa1e829904]: INITIALIZED_STATE -> PENDING_STATE, statement: SELECT java_method('java.lang.Thread', 'sleep', 10000l) as res FROM range(1, 3, 1, 2)
08:53:43.150 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_kyuubi_dev's query[cd3aec1e-4fd1-44cc-a849-d6fa1e829904]: PENDING_STATE -> RUNNING_STATE, statement: SELECT java_method('java.lang.Thread', 'sleep', 10000l) as res FROM range(1, 3, 1, 2)
22/05/08 08:53:43 INFO ExecuteStatement: Processing b_kyuubi_dev's query[ae64608d-9ed6-42ab-bd62-32886ca24482]: INITIALIZED_STATE -> PENDING_STATE, statement: SELECT java_method('java.lang.Thread', 'sleep', 10000l) as res FROM range(1, 3, 1, 2)
22/05/08 08:53:43 INFO ExecuteStatement: Processing b_kyuubi_dev's query[ae64608d-9ed6-42ab-bd62-32886ca24482]: PENDING_STATE -> RUNNING_STATE, statement: SELECT java_method('java.lang.Thread', 'sleep', 10000l) as res FROM range(1, 3, 1, 2)
22/05/08 08:53:43 INFO ExecuteStatement:
Spark application name: kyuubi_zeuslvs_USER_SPARK_SQL_b_kyuubi_dev_default_636f21eb-349b-47f0-bf3a-beef3b7146da
application ID: application_1644980694928_2371390
application web UI: https://zeus-lvs-rm-1.vip.hadoop.ebay.com:50030/proxy/application_1644980694928_2371390,https://zeus-lvs-rm-2.vip.hadoop.ebay.com:50030/proxy/application_1644980694928_2371390
master: yarn
deploy mode: cluster
version: 3.1.1.0.6.0
Start time: 2022-05-08T08:32:47.574
User: b_kyuubi_dev
22/05/08 08:53:43 INFO ExecuteStatement: Execute in full collect mode
22/05/08 08:53:43 INFO SparkContext: Starting job: collect at ExecuteStatement.scala:135
22/05/08 08:53:43 INFO SQLOperationListener: Query [ae64608d-9ed6-42ab-bd62-32886ca24482]: Job 36 started with 1 stages, 1 active jobs running
22/05/08 08:53:43 INFO SQLOperationListener: Query [ae64608d-9ed6-42ab-bd62-32886ca24482]: Stage 47 started with 2 tasks, 1 active stages running
08:53:48.154 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[cd3aec1e-4fd1-44cc-a849-d6fa1e829904] in RUNNING_STATE
----------------------------------------------------------------------------------------------
08:53:53.157 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[cd3aec1e-4fd1-44cc-a849-d6fa1e829904] in RUNNING_STATE
22/05/08 08:53:53 INFO SQLOperationListener: Finished stage: Stage(47, 0); Name: 'collect at ExecuteStatement.scala:135'; Status: succeeded; numTasks: 2; Took: 10035 msec
22/05/08 08:53:53 INFO StatsReportListener: task runtime:(count: 2, mean: 10026.500000, stdev: 0.500000, max: 10027.000000, min: 10026.000000)
22/05/08 08:53:53 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
22/05/08 08:53:53 INFO StatsReportListener: 10.0 s 10.0 s 10.0 s 10.0 s 10.0 s 10.0 s 10.0 s 10.0 s 10.0 s
22/05/08 08:53:53 INFO DAGScheduler: Job 36 finished: collect at ExecuteStatement.scala:135, took 10.037578 s
22/05/08 08:53:53 INFO StatsReportListener: shuffle bytes written:(count: 2, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
22/05/08 08:53:53 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
22/05/08 08:53:53 INFO StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B
22/05/08 08:53:53 INFO StatsReportListener: fetch wait time:(count: 2, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
22/05/08 08:53:53 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
22/05/08 08:53:53 INFO StatsReportListener: 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms 0.0 ms
22/05/08 08:53:53 INFO StatsReportListener: remote bytes read:(count: 2, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
22/05/08 08:53:53 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
22/05/08 08:53:53 INFO StatsReportListener: 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B 0.0 B
22/05/08 08:53:53 INFO StatsReportListener: task result size:(count: 2, mean: 1257.000000, stdev: 0.000000, max: 1257.000000, min: 1257.000000)
22/05/08 08:53:53 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
22/05/08 08:53:53 INFO StatsReportListener: 1257.0 B 1257.0 B 1257.0 B 1257.0 B 1257.0 B 1257.0 B 1257.0 B 1257.0 B 1257.0 B
22/05/08 08:53:53 INFO StatsReportListener: executor (non-fetch) time pct: (count: 2, mean: 99.790555, stdev: 0.000010, max: 99.790565, min: 99.790545)
22/05/08 08:53:53 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
22/05/08 08:53:53 INFO StatsReportListener: 100 % 100 % 100 % 100 % 100 % 100 % 100 % 100 % 100 %
22/05/08 08:53:53 INFO StatsReportListener: fetch wait time pct: (count: 2, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
22/05/08 08:53:53 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
22/05/08 08:53:53 INFO StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 %
22/05/08 08:53:53 INFO StatsReportListener: other time pct: (count: 2, mean: 0.209445, stdev: 0.000010, max: 0.209455, min: 0.209435)
22/05/08 08:53:53 INFO StatsReportListener: 0% 5% 10% 25% 50% 75% 90% 95% 100%
22/05/08 08:53:53 INFO StatsReportListener: 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 %
22/05/08 08:53:53 INFO SQLOperationListener: Query [ae64608d-9ed6-42ab-bd62-32886ca24482]: Job 36 succeeded, 0 active jobs running
22/05/08 08:53:53 INFO ExecuteStatement: Processing b_kyuubi_dev's query[ae64608d-9ed6-42ab-bd62-32886ca24482]: RUNNING_STATE -> FINISHED_STATE, statement: SELECT java_method('java.lang.Thread', 'sleep', 10000l) as res FROM range(1, 3, 1, 2), time taken: 10.067 seconds
08:53:53.218 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[cd3aec1e-4fd1-44cc-a849-d6fa1e829904] in FINISHED_STATE
08:53:53.219 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_kyuubi_dev's query[cd3aec1e-4fd1-44cc-a849-d6fa1e829904]: RUNNING_STATE -> FINISHED_STATE, statement: SELECT java_method('java.lang.Thread', 'sleep', 10000l) as res FROM range(1, 3, 1, 2), time taken: 10.069 seconds
Error: java.util.MissingFormatArgumentException: Format specifier '%6s' (state=08S01,code=0)
0: jdbc:hive2://kyuubidev-lb.hadoopmaster-dev>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r865576102
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/JobProgressUpdate.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.operation
+
+import java.util
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.hive.service.rpc.thrift.TJobExecutionStatus
+
+case class JobProgressUpdate(
+ headers: util.List[String] = null,
+ rows: util.List[util.List[String]] = null,
+ footerSummary: String = null,
+ progressedPercentage: Double = 0.0,
+ startTimeMillis: Long = 0L,
+ status: String = null)
+
+object KyuubiProgressMonitorStatusMapper {
+
+ def forStatus(status: String): TJobExecutionStatus = {
+ if (StringUtils.isEmpty(status)) {
+ return TJobExecutionStatus.NOT_AVAILABLE
+ }
+ KyuubiStatus.withName(status) match {
+ case KyuubiStatus.NOT_AVAILABLE =>
+ TJobExecutionStatus.NOT_AVAILABLE
+ case KyuubiStatus.IN_PROGRESS | KyuubiStatus.PENDING | KyuubiStatus.RUNNING =>
+ TJobExecutionStatus.IN_PROGRESS
+ case KyuubiStatus.COMPLETE => TJobExecutionStatus.COMPLETE
+ case _ => TJobExecutionStatus.COMPLETE
+ }
+ }
+}
+
+object KyuubiStatus extends Enumeration {
+ type KyuubiStatus = Value
+ val PENDING, RUNNING, FINISHED, IN_PROGRESS, COMPLETE, NOT_AVAILABLE = Value
Review Comment:
> why is the status num more than TJobExecutionStatus
I mixed the status of Spark Stage with the status of TJobExecutionStatus.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r865574560
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/JobProgressUpdate.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.operation
+
+import java.util
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.hive.service.rpc.thrift.TJobExecutionStatus
+
+case class JobProgressUpdate(
+ headers: util.List[String] = null,
+ rows: util.List[util.List[String]] = null,
+ footerSummary: String = null,
+ progressedPercentage: Double = 0.0,
+ startTimeMillis: Long = 0L,
+ status: String = null)
+
+object KyuubiProgressMonitorStatusMapper {
+
+ def forStatus(status: String): TJobExecutionStatus = {
+ if (StringUtils.isEmpty(status)) {
+ return TJobExecutionStatus.NOT_AVAILABLE
+ }
+ KyuubiStatus.withName(status) match {
+ case KyuubiStatus.NOT_AVAILABLE =>
+ TJobExecutionStatus.NOT_AVAILABLE
+ case KyuubiStatus.IN_PROGRESS | KyuubiStatus.PENDING | KyuubiStatus.RUNNING =>
+ TJobExecutionStatus.IN_PROGRESS
+ case KyuubiStatus.COMPLETE => TJobExecutionStatus.COMPLETE
Review Comment:
There is no FINISHED status in `TJobExecutionStatus`, so we ignore it. refer to https://github.com/apache/hive/blob/branch-4.0.0-alpha-1/service/src/java/org/apache/hive/service/cli/SparkProgressMonitorStatusMapper.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r866435940
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/progress/SparkProgressMonitor.scala:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kyuubi.engine.spark.operation.progress
+
+import java.util
+
+import scala.collection.JavaConverters._
+
+import org.apache.kyuubi.engine.spark.operation.progress.SparkProgressMonitor._
+import org.apache.kyuubi.operation.KyuubiStatus
+
+class SparkProgressMonitor(progressMap: Map[SparkStage, SparkStageProgress], _startTime: Long = 0) {
+
+ def headers: util.List[String] = HEADERS
+
+ def rows: util.List[util.List[String]] = {
+ val progressRows = progressMap.toSeq.sortBy(_._1).map {
+ case (stage, progress) =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ val running = progress.runningTaskCount
+ val failed = progress.failedTaskCount
+ var state =
+ if (total > 0) {
+ KyuubiStatus.PENDING
+ } else {
+ KyuubiStatus.FINISHED
+ }
+ if (complete > 0 || running > 0 || failed > 0) {
+ state = if (complete < total) KyuubiStatus.RUNNING else KyuubiStatus.FINISHED
+ }
+ val attempt = String.valueOf(stage.attemptId)
+ val stageName = "Stage-" + String.valueOf(stage.stageId)
+ val nameWithProgress = getNameWithProgress(stageName, complete, total)
+ val pending = total - complete - running
+ util.Arrays.asList(
+ nameWithProgress,
+ attempt,
+ state.toString,
+ String.valueOf(total),
+ String.valueOf(complete),
+ String.valueOf(running),
+ String.valueOf(pending),
+ String.valueOf(failed),
+ "")
+ }.toList.asJavaCollection
+ new util.ArrayList[util.List[String]](progressRows)
+ }
+
+ def footerSummary: String = {
+ "STAGES: %02d/%02d".format(getCompletedStages, progressMap.keySet.size)
+ }
+
+ def progressedPercentage: Double = {
+ var sumTotal = 0
+ var sumComplete = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ sumTotal += total
+ sumComplete += complete
+ }
+ if (sumTotal == 0) {
+ 1.0f
+ } else {
+ sumComplete.toFloat / sumTotal.toFloat
+ }
+ }
+
+ def startTime: Long = _startTime
+
+ def executionStatus: String =
+ if (getCompletedStages == progressMap.keySet.size) {
+ KyuubiStatus.FINISHED.toString
+ } else {
+ KyuubiStatus.RUNNING.toString
+ }
+
+ private def getNameWithProgress(s: String, complete: Int, total: Int): String = {
+ if (s == null) return ""
+ val percent =
+ if (total == 0) 1.0f
+ else complete.toFloat / total.toFloat
+ // lets use the remaining space in column 1 as progress bar
+ val spaceRemaining = COLUMN_1_WIDTH - s.length - 1
+ var trimmedVName = s
+ // if the vertex name is longer than column 1 width, trim it down
+ if (s.length > COLUMN_1_WIDTH) {
+ trimmedVName = s.substring(0, COLUMN_1_WIDTH - 2)
+ trimmedVName += ".."
+ } else trimmedVName += " "
+ val result = new StringBuilder(trimmedVName)
+ val toFill = (spaceRemaining * percent).toInt
+ for (i <- 0 until toFill) {
+ result.append(".")
+ }
+ result.toString
+ }
+
+ private def getCompletedStages: Int = {
+ var completed = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ if (total > 0 && complete == total) completed += 1
+ }
+ completed
+ }
+
+}
+
+object SparkProgressMonitor {
+
+ private val HEADERS: util.List[String] = util.Arrays.asList(
+ "STAGES",
+ "ATTEMPT",
+ "STATUS",
+ "TOTAL",
+ "COMPLETED",
+ "RUNNING",
+ "PENDING",
+ "FAILED",
+ "")
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1118573508
Irrelevant failed test
```
BackendServiceMetricSuite:
- backend service metric test *** FAILED ***
java.io.FileNotFoundException: /home/runner/work/incubator-kyuubi/incubator-kyuubi/kyuubi-server/target/tmp/kyuubi-73d03542-57cb-41fd-b6a9-9e5d0462f02b/report.json (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:1029)
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3109)
at org.apache.kyuubi.server.BackendServiceMetricSuite.$anonfun$new$2(BackendServiceMetricSuite.scala:67)
at org.apache.kyuubi.server.BackendServiceMetricSuite.$anonfun$new$2$adapted(BackendServiceMetricSuite.scala:45)
at org.apache.kyuubi.operation.JDBCTestHelper.$anonfun$withMultipleConnectionJdbcStatement$3(JDBCTestHelper.scala:60)
at org.apache.kyuubi.operation.JDBCTestHelper.$anonfun$withMultipleConnectionJdbcStatement$3$adapted(JDBCTestHelper.scala:60)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r865602333
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/progress/SparkStageProgress.scala:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kyuubi.engine.spark.operation.progress
+
+case class SparkStage(stageId: Int, attemptId: Int) extends Comparable[SparkStage] {
Review Comment:
> extends Comparable[SparkStage] is unnecessary
I sort progressMap by this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] turboFei commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
turboFei commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1119538409
> enable
> > In addition we may need a configuration to control whether to fetch operation progress.
>
> @yaooqinn @ulysses-you how about named `kyuubi.session.progress.enable` and default to false?
`false` +1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement for spark sql engine
Posted by GitBox <gi...@apache.org>.
wForget commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1120443300
> I met below exception when using kyuubi beeline to execute query if kyuubi.operation.progress.enabled is true.
This may be related to this change https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r865924817, and I will verify it tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1120163296
> > kyuubi.session.engine.spark.showProgress
>
> cc @wForget again `consoleProgressBar` is now duplicated with this feature, shall we remove it?
I think we can keep it, since this progress needs to be rendered by itself on the client side.
There is `KyuubiBeelineInPlaceUpdateStream` in beeline, but it doesn't seem to be implemented in `jdbc driver`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] turboFei commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
turboFei commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r867315319
##########
docs/deployment/settings.md:
##########
@@ -381,6 +381,7 @@ Key | Default | Meaning | Type | Since
<code>kyuubi.session.engine.trino.main.resource</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'><undefined></div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The package used to create Trino engine remote job. If it is undefined, Kyuubi will use the default</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.5.0</div>
<code>kyuubi.session.idle.timeout</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>PT6H</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>session idle timeout, it will be closed when it's not accessed for this duration</div>|<div style='width: 30pt'>duration</div>|<div style='width: 20pt'>1.2.0</div>
<code>kyuubi.session.name</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'><undefined></div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>A human readable name of session and we use empty string by default. This name will be recorded in event. Note that, we only apply this value from session conf.</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.4.0</div>
+<code>kyuubi.session.progress.enable</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>false</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Whether to enable the operation progress. When true, the operation progress will be returned in `GetOperationStatus`.</div>|<div style='width: 30pt'>boolean</div>|<div style='width: 20pt'>1.6.0</div>
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r865533592
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/progress/SparkProgressMonitor.scala:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kyuubi.engine.spark.operation.progress
+
+import java.util
+
+import scala.collection.JavaConverters._
+
+import org.apache.kyuubi.engine.spark.operation.progress.SparkProgressMonitor._
+import org.apache.kyuubi.operation.KyuubiStatus
+
+class SparkProgressMonitor(progressMap: Map[SparkStage, SparkStageProgress], _startTime: Long = 0) {
+
+ def headers: util.List[String] = HEADERS
+
+ def rows: util.List[util.List[String]] = {
+ val progressRows = progressMap.toSeq.sortBy(_._1).map {
+ case (stage, progress) =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ val running = progress.runningTaskCount
+ val failed = progress.failedTaskCount
+ var state =
+ if (total > 0) {
+ KyuubiStatus.PENDING
+ } else {
+ KyuubiStatus.FINISHED
+ }
+ if (complete > 0 || running > 0 || failed > 0) {
+ state = if (complete < total) KyuubiStatus.RUNNING else KyuubiStatus.FINISHED
+ }
+ val attempt = String.valueOf(stage.attemptId)
+ val stageName = "Stage-" + String.valueOf(stage.stageId)
+ val nameWithProgress = getNameWithProgress(stageName, complete, total)
+ val pending = total - complete - running
+ util.Arrays.asList(
+ nameWithProgress,
+ attempt,
+ state.toString,
+ String.valueOf(total),
+ String.valueOf(complete),
+ String.valueOf(running),
+ String.valueOf(pending),
+ String.valueOf(failed),
+ "")
+ }.toList.asJavaCollection
+ new util.ArrayList[util.List[String]](progressRows)
+ }
+
+ def footerSummary: String = {
+ "STAGES: %02d/%02d".format(getCompletedStages, progressMap.keySet.size)
+ }
+
+ def progressedPercentage: Double = {
+ var sumTotal = 0
+ var sumComplete = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ sumTotal += total
+ sumComplete += complete
+ }
+ if (sumTotal == 0) {
+ 1.0f
+ } else {
+ sumComplete.toFloat / sumTotal.toFloat
+ }
+ }
+
+ def startTime: Long = _startTime
+
+ def executionStatus: String =
+ if (getCompletedStages == progressMap.keySet.size) {
Review Comment:
In AQE, it is dynamic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r865531508
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/JobProgressUpdate.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.operation
+
+import java.util
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.hive.service.rpc.thrift.TJobExecutionStatus
+
+case class JobProgressUpdate(
+ headers: util.List[String] = null,
+ rows: util.List[util.List[String]] = null,
+ footerSummary: String = null,
+ progressedPercentage: Double = 0.0,
+ startTimeMillis: Long = 0L,
+ status: String = null)
+
+object KyuubiProgressMonitorStatusMapper {
+
+ def forStatus(status: String): TJobExecutionStatus = {
+ if (StringUtils.isEmpty(status)) {
+ return TJobExecutionStatus.NOT_AVAILABLE
+ }
+ KyuubiStatus.withName(status) match {
+ case KyuubiStatus.NOT_AVAILABLE =>
+ TJobExecutionStatus.NOT_AVAILABLE
+ case KyuubiStatus.IN_PROGRESS | KyuubiStatus.PENDING | KyuubiStatus.RUNNING =>
+ TJobExecutionStatus.IN_PROGRESS
+ case KyuubiStatus.COMPLETE => TJobExecutionStatus.COMPLETE
+ case _ => TJobExecutionStatus.COMPLETE
+ }
+ }
+}
+
+object KyuubiStatus extends Enumeration {
Review Comment:
KyuubiStatus is a overkilling name
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/progress/SparkProgressMonitor.scala:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kyuubi.engine.spark.operation.progress
+
+import java.util
+
+import scala.collection.JavaConverters._
+
+import org.apache.kyuubi.engine.spark.operation.progress.SparkProgressMonitor._
+import org.apache.kyuubi.operation.KyuubiStatus
+
+class SparkProgressMonitor(progressMap: Map[SparkStage, SparkStageProgress], _startTime: Long = 0) {
Review Comment:
unnecessary default value
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/progress/SparkStageProgress.scala:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kyuubi.engine.spark.operation.progress
+
+case class SparkStage(stageId: Int, attemptId: Int) extends Comparable[SparkStage] {
Review Comment:
extends Comparable[SparkStage] is unnecessary
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SparkProgressFetcher.scala:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.kyuubi
+
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.status.api.v1.StageStatus
+
+import org.apache.kyuubi.engine.spark.operation.progress.{SparkProgressMonitor, SparkStage, SparkStageProgress}
+import org.apache.kyuubi.operation.JobProgressUpdate
+
+class SparkProgressFetcher(spark: SparkSession, jobGroup: String) {
+
+ val statusStore = spark.sparkContext.statusStore
+
+ def getJobProgressUpdate(startTime: Long): JobProgressUpdate = {
Review Comment:
the start time can be removed, it's just `operationStatus.start`
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/OperationStatus.scala:
##########
@@ -27,4 +27,5 @@ case class OperationStatus(
lastModified: Long,
completed: Long,
hasResultSet: Boolean,
- exception: Option[KyuubiSQLException] = None)
+ exception: Option[KyuubiSQLException] = None,
+ jobProgressUpdate: Option[JobProgressUpdate] = None)
Review Comment:
may be jobProgressUpdate: Option[TProgressUpdateResp] is OK to use
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/JobProgressUpdate.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.operation
+
+import java.util
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.hive.service.rpc.thrift.TJobExecutionStatus
+
+case class JobProgressUpdate(
Review Comment:
the default values seems useless
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/JobProgressUpdate.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.operation
+
+import java.util
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.hive.service.rpc.thrift.TJobExecutionStatus
+
+case class JobProgressUpdate(
+ headers: util.List[String] = null,
+ rows: util.List[util.List[String]] = null,
+ footerSummary: String = null,
+ progressedPercentage: Double = 0.0,
+ startTimeMillis: Long = 0L,
+ status: String = null)
+
+object KyuubiProgressMonitorStatusMapper {
+
+ def forStatus(status: String): TJobExecutionStatus = {
+ if (StringUtils.isEmpty(status)) {
+ return TJobExecutionStatus.NOT_AVAILABLE
+ }
+ KyuubiStatus.withName(status) match {
+ case KyuubiStatus.NOT_AVAILABLE =>
+ TJobExecutionStatus.NOT_AVAILABLE
+ case KyuubiStatus.IN_PROGRESS | KyuubiStatus.PENDING | KyuubiStatus.RUNNING =>
+ TJobExecutionStatus.IN_PROGRESS
+ case KyuubiStatus.COMPLETE => TJobExecutionStatus.COMPLETE
+ case _ => TJobExecutionStatus.COMPLETE
+ }
+ }
+}
+
+object KyuubiStatus extends Enumeration {
+ type KyuubiStatus = Value
+ val PENDING, RUNNING, FINISHED, IN_PROGRESS, COMPLETE, NOT_AVAILABLE = Value
Review Comment:
why is the status num more than TJobExecutionStatus
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r865933481
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/JobProgressUpdate.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.operation
+
+import java.util
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.hive.service.rpc.thrift.TJobExecutionStatus
+
+case class JobProgressUpdate(
+ headers: util.List[String] = null,
+ rows: util.List[util.List[String]] = null,
+ footerSummary: String = null,
+ progressedPercentage: Double = 0.0,
+ startTimeMillis: Long = 0L,
+ status: String = null)
+
+object KyuubiProgressMonitorStatusMapper {
+
+ def forStatus(status: String): TJobExecutionStatus = {
+ if (StringUtils.isEmpty(status)) {
+ return TJobExecutionStatus.NOT_AVAILABLE
+ }
+ KyuubiStatus.withName(status) match {
+ case KyuubiStatus.NOT_AVAILABLE =>
+ TJobExecutionStatus.NOT_AVAILABLE
+ case KyuubiStatus.IN_PROGRESS | KyuubiStatus.PENDING | KyuubiStatus.RUNNING =>
+ TJobExecutionStatus.IN_PROGRESS
+ case KyuubiStatus.COMPLETE => TJobExecutionStatus.COMPLETE
+ case _ => TJobExecutionStatus.COMPLETE
+ }
+ }
+}
+
+object KyuubiStatus extends Enumeration {
+ type KyuubiStatus = Value
+ val PENDING, RUNNING, FINISHED, IN_PROGRESS, COMPLETE, NOT_AVAILABLE = Value
Review Comment:
> it really confused, seems we can just retain ?
>
> IN_PROGRESS, COMPLETE, NOT_AVAILABLE
The other status is needed in `SparkProgressMonitor#rows`, how about we retain the other status in the spark engine module?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r867333747
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SparkProgressFetcher.scala:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.kyuubi
+
+import org.apache.hive.service.rpc.thrift.TProgressUpdateResp
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.status.api.v1.StageStatus
+
+import org.apache.kyuubi.engine.spark.operation.progress.{SparkProgressMonitor, SparkStage, SparkStageProgress}
+
+class SparkProgressFetcher(spark: SparkSession, jobGroup: String) {
+
+ val statusStore = spark.sparkContext.statusStore
+
+ def getJobProgressUpdate(startTime: Long): TProgressUpdateResp = {
+ val progressMap = getProgressMap
+ val progressMonitor = new SparkProgressMonitor(progressMap)
+ new TProgressUpdateResp(
+ progressMonitor.headers,
+ progressMonitor.rows,
+ progressMonitor.progressedPercentage,
+ progressMonitor.executionStatus,
+ progressMonitor.footerSummary,
+ startTime)
+ }
+
+ private def getProgressMap(): Map[SparkStage, SparkStageProgress] = {
Review Comment:
> How about removing SparkProgressFetcher and merging it into SparkProgressMonitor?
Or just make getProgressMap return SortedMap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r867320213
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecuteStatement.scala:
##########
@@ -51,6 +53,18 @@ class ExecuteStatement(
private val operationListener: SQLOperationListener = new SQLOperationListener(this, spark)
+ private val progressEnable = spark.conf.getOption(SESSION_PROGRESS_ENABLE.key) match {
+ case Some(s) => s.toBoolean
+ case _ => session.sessionManager.getConf.get(SESSION_PROGRESS_ENABLE)
+ }
+
+ private val progressFetcher: Option[SparkProgressFetcher] =
+ if (progressEnable) {
+ Some(new SparkProgressFetcher(spark, statementId))
+ } else {
+ None
+ }
Review Comment:
merge these 2 and make it lazy, getStatus is not always called
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] turboFei commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement for spark sql engine
Posted by GitBox <gi...@apache.org>.
turboFei commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1120440354
I met below exception when using kyuubi beeline to execute query if `kyuubi.operation.progress.enabled` is true.
```
8:35:32.011 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing b_kyuubi_dev's query[17656f79-f8b3-4f46-8a14-d5ce1d146844]: RUNNING_STATE -> FINISHED_STATE, statement: SELECT java_method('java.lang.Thread', 'sleep', 10000l) FROM range(1, 3, 1, 2), time taken: 10.135 seconds
Error: java.util.MissingFormatArgumentException: Format specifier '%6s' (state=08S01,code=0)
java.sql.SQLException: java.util.MissingFormatArgumentException: Format specifier '%6s'
at org.apache.kyuubi.jdbc.hive.KyuubiStatement.waitForOperationToComplete(KyuubiStatement.java:415)
at org.apache.kyuubi.jdbc.hive.KyuubiStatement.executeWithConfOverlay(KyuubiStatement.java:251)
at org.apache.kyuubi.jdbc.hive.KyuubiStatement.execute(KyuubiStatement.java:245)
at org.apache.hive.beeline.KyuubiCommands.executeInternal(KyuubiCommands.java:209)
at org.apache.hive.beeline.KyuubiCommands.execute(KyuubiCommands.java:297)
at org.apache.hive.beeline.KyuubiCommands.sql(KyuubiCommands.java:266)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1314)
at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1178)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1033)
at org.apache.hive.beeline.KyuubiBeeLine.mainWithInputRedirection(KyuubiBeeLine.java:122)
at org.apache.hive.beeline.KyuubiBeeLine.main(KyuubiBeeLine.java:107)
Caused by: java.util.MissingFormatArgumentException: Format specifier '%6s'
at java.util.Formatter.format(Formatter.java:2519)
at java.util.Formatter.format(Formatter.java:2455)
at java.lang.String.format(String.java:2940)
at org.apache.hadoop.hive.common.log.InPlaceUpdate.render(InPlaceUpdate.java:155)
at org.apache.hive.beeline.logs.KyuubiBeelineInPlaceUpdateStream.update(KyuubiBeelineInPlaceUpdateStream.java:52)
at org.apache.kyuubi.jdbc.hive.KyuubiStatement.waitForOperationToComplete(KyuubiStatement.java:382)
... 10 more
0: jdbc:hive2://kyuubidev-lb.hadoopmaster-dev>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r865517163
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/progress/SparkProgressMonitor.scala:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kyuubi.engine.spark.operation.progress
+
+import java.util
+
+import scala.collection.JavaConverters._
+
+import org.apache.kyuubi.engine.spark.operation.progress.SparkProgressMonitor._
+import org.apache.kyuubi.operation.KyuubiStatus
+
+class SparkProgressMonitor(progressMap: Map[SparkStage, SparkStageProgress], _startTime: Long = 0) {
+
+ def headers: util.List[String] = HEADERS
+
+ def rows: util.List[util.List[String]] = {
+ val progressRows = progressMap.toSeq.sortBy(_._1).map {
+ case (stage, progress) =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ val running = progress.runningTaskCount
+ val failed = progress.failedTaskCount
+ var state =
+ if (total > 0) {
+ KyuubiStatus.PENDING
+ } else {
+ KyuubiStatus.FINISHED
+ }
+ if (complete > 0 || running > 0 || failed > 0) {
+ state = if (complete < total) KyuubiStatus.RUNNING else KyuubiStatus.FINISHED
+ }
+ val attempt = String.valueOf(stage.attemptId)
+ val stageName = "Stage-" + String.valueOf(stage.stageId)
+ val nameWithProgress = getNameWithProgress(stageName, complete, total)
+ val pending = total - complete - running
+ util.Arrays.asList(
+ nameWithProgress,
+ attempt,
+ state.toString,
+ String.valueOf(total),
+ String.valueOf(complete),
+ String.valueOf(running),
+ String.valueOf(pending),
+ String.valueOf(failed),
+ "")
+ }.toList.asJavaCollection
+ new util.ArrayList[util.List[String]](progressRows)
+ }
+
+ def footerSummary: String = {
+ "STAGES: %02d/%02d".format(getCompletedStages, progressMap.keySet.size)
+ }
+
+ def progressedPercentage: Double = {
+ var sumTotal = 0
+ var sumComplete = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ sumTotal += total
+ sumComplete += complete
+ }
+ if (sumTotal == 0) {
+ 1.0f
+ } else {
+ sumComplete.toFloat / sumTotal.toFloat
+ }
+ }
+
+ def startTime: Long = _startTime
+
+ def executionStatus: String =
+ if (getCompletedStages == progressMap.keySet.size) {
+ KyuubiStatus.FINISHED.toString
+ } else {
+ KyuubiStatus.RUNNING.toString
+ }
+
+ private def getNameWithProgress(s: String, complete: Int, total: Int): String = {
+ if (s == null) return ""
+ val percent =
+ if (total == 0) 1.0f
+ else complete.toFloat / total.toFloat
+ // lets use the remaining space in column 1 as progress bar
+ val spaceRemaining = COLUMN_1_WIDTH - s.length - 1
+ var trimmedVName = s
+ // if the vertex name is longer than column 1 width, trim it down
+ if (s.length > COLUMN_1_WIDTH) {
+ trimmedVName = s.substring(0, COLUMN_1_WIDTH - 2)
+ trimmedVName += ".."
+ } else trimmedVName += " "
+ val result = new StringBuilder(trimmedVName)
+ val toFill = (spaceRemaining * percent).toInt
+ for (i <- 0 until toFill) {
+ result.append(".")
+ }
+ result.toString
+ }
+
+ private def getCompletedStages: Int = {
+ var completed = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ if (total > 0 && complete == total) completed += 1
+ }
+ completed
+ }
+
+}
+
+object SparkProgressMonitor {
+
+ private val HEADERS: util.List[String] = util.Arrays.asList(
+ "STAGES",
+ "ATTEMPT",
+ "STATUS",
+ "TOTAL",
+ "COMPLETED",
+ "RUNNING",
+ "PENDING",
+ "FAILED",
+ "")
Review Comment:
unnecessary ?
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/progress/SparkStageProgress.scala:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kyuubi.engine.spark.operation.progress
+
+case class SparkStage(stageId: Int, attemptId: Int) extends Comparable[SparkStage] {
+ override def compareTo(o: SparkStage): Int = {
+ if (stageId == o.stageId) {
+ attemptId - o.attemptId
+ } else {
+ stageId - o.stageId
+ }
+ }
+}
+
+case class SparkStageProgress(
+ totalTaskCount: Int,
+ succeededTaskCount: Int,
Review Comment:
unify the name: `completedTasksCount`
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/progress/SparkProgressMonitor.scala:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kyuubi.engine.spark.operation.progress
+
+import java.util
+
+import scala.collection.JavaConverters._
+
+import org.apache.kyuubi.engine.spark.operation.progress.SparkProgressMonitor._
+import org.apache.kyuubi.operation.KyuubiStatus
+
+class SparkProgressMonitor(progressMap: Map[SparkStage, SparkStageProgress], _startTime: Long = 0) {
+
+ def headers: util.List[String] = HEADERS
+
+ def rows: util.List[util.List[String]] = {
+ val progressRows = progressMap.toSeq.sortBy(_._1).map {
+ case (stage, progress) =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ val running = progress.runningTaskCount
+ val failed = progress.failedTaskCount
+ var state =
+ if (total > 0) {
+ KyuubiStatus.PENDING
+ } else {
+ KyuubiStatus.FINISHED
+ }
+ if (complete > 0 || running > 0 || failed > 0) {
+ state = if (complete < total) KyuubiStatus.RUNNING else KyuubiStatus.FINISHED
+ }
+ val attempt = String.valueOf(stage.attemptId)
+ val stageName = "Stage-" + String.valueOf(stage.stageId)
+ val nameWithProgress = getNameWithProgress(stageName, complete, total)
+ val pending = total - complete - running
+ util.Arrays.asList(
+ nameWithProgress,
+ attempt,
+ state.toString,
+ String.valueOf(total),
+ String.valueOf(complete),
+ String.valueOf(running),
+ String.valueOf(pending),
+ String.valueOf(failed),
+ "")
+ }.toList.asJavaCollection
+ new util.ArrayList[util.List[String]](progressRows)
+ }
+
+ def footerSummary: String = {
+ "STAGES: %02d/%02d".format(getCompletedStages, progressMap.keySet.size)
+ }
+
+ def progressedPercentage: Double = {
+ var sumTotal = 0
+ var sumComplete = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ sumTotal += total
+ sumComplete += complete
+ }
+ if (sumTotal == 0) {
+ 1.0f
+ } else {
+ sumComplete.toFloat / sumTotal.toFloat
+ }
+ }
+
+ def startTime: Long = _startTime
+
+ def executionStatus: String =
+ if (getCompletedStages == progressMap.keySet.size) {
Review Comment:
the stage will be generated during execution, so the completed stages may not be the total stages ?
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/progress/SparkProgressMonitor.scala:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kyuubi.engine.spark.operation.progress
+
+import java.util
+
+import scala.collection.JavaConverters._
+
+import org.apache.kyuubi.engine.spark.operation.progress.SparkProgressMonitor._
+import org.apache.kyuubi.operation.KyuubiStatus
+
+class SparkProgressMonitor(progressMap: Map[SparkStage, SparkStageProgress], _startTime: Long = 0) {
+
+ def headers: util.List[String] = HEADERS
+
+ def rows: util.List[util.List[String]] = {
+ val progressRows = progressMap.toSeq.sortBy(_._1).map {
+ case (stage, progress) =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ val running = progress.runningTaskCount
+ val failed = progress.failedTaskCount
+ var state =
+ if (total > 0) {
+ KyuubiStatus.PENDING
+ } else {
+ KyuubiStatus.FINISHED
+ }
+ if (complete > 0 || running > 0 || failed > 0) {
+ state = if (complete < total) KyuubiStatus.RUNNING else KyuubiStatus.FINISHED
+ }
+ val attempt = String.valueOf(stage.attemptId)
+ val stageName = "Stage-" + String.valueOf(stage.stageId)
+ val nameWithProgress = getNameWithProgress(stageName, complete, total)
+ val pending = total - complete - running
+ util.Arrays.asList(
+ nameWithProgress,
+ attempt,
+ state.toString,
+ String.valueOf(total),
+ String.valueOf(complete),
+ String.valueOf(running),
+ String.valueOf(pending),
+ String.valueOf(failed),
+ "")
+ }.toList.asJavaCollection
+ new util.ArrayList[util.List[String]](progressRows)
+ }
+
+ def footerSummary: String = {
+ "STAGES: %02d/%02d".format(getCompletedStages, progressMap.keySet.size)
+ }
+
+ def progressedPercentage: Double = {
+ var sumTotal = 0
+ var sumComplete = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ sumTotal += total
+ sumComplete += complete
+ }
+ if (sumTotal == 0) {
+ 1.0f
+ } else {
+ sumComplete.toFloat / sumTotal.toFloat
+ }
+ }
+
+ def startTime: Long = _startTime
+
+ def executionStatus: String =
+ if (getCompletedStages == progressMap.keySet.size) {
+ KyuubiStatus.FINISHED.toString
+ } else {
+ KyuubiStatus.RUNNING.toString
+ }
+
+ private def getNameWithProgress(s: String, complete: Int, total: Int): String = {
+ if (s == null) return ""
+ val percent =
+ if (total == 0) 1.0f
+ else complete.toFloat / total.toFloat
+ // lets use the remaining space in column 1 as progress bar
+ val spaceRemaining = COLUMN_1_WIDTH - s.length - 1
+ var trimmedVName = s
+ // if the vertex name is longer than column 1 width, trim it down
+ if (s.length > COLUMN_1_WIDTH) {
+ trimmedVName = s.substring(0, COLUMN_1_WIDTH - 2)
+ trimmedVName += ".."
+ } else trimmedVName += " "
+ val result = new StringBuilder(trimmedVName)
+ val toFill = (spaceRemaining * percent).toInt
+ for (i <- 0 until toFill) {
+ result.append(".")
+ }
+ result.toString
+ }
+
+ private def getCompletedStages: Int = {
+ var completed = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
Review Comment:
not see the sort is necessary and we can simplify the code to:
```
progressMap.forall {
...
total > 0 && complete == total
}
```
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/JobProgressUpdate.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.operation
+
+import java.util
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.hive.service.rpc.thrift.TJobExecutionStatus
+
+case class JobProgressUpdate(
+ headers: util.List[String] = null,
+ rows: util.List[util.List[String]] = null,
+ footerSummary: String = null,
+ progressedPercentage: Double = 0.0,
+ startTimeMillis: Long = 0L,
+ status: String = null)
+
+object KyuubiProgressMonitorStatusMapper {
+
+ def forStatus(status: String): TJobExecutionStatus = {
+ if (StringUtils.isEmpty(status)) {
+ return TJobExecutionStatus.NOT_AVAILABLE
+ }
+ KyuubiStatus.withName(status) match {
+ case KyuubiStatus.NOT_AVAILABLE =>
+ TJobExecutionStatus.NOT_AVAILABLE
+ case KyuubiStatus.IN_PROGRESS | KyuubiStatus.PENDING | KyuubiStatus.RUNNING =>
+ TJobExecutionStatus.IN_PROGRESS
+ case KyuubiStatus.COMPLETE => TJobExecutionStatus.COMPLETE
+ case _ => TJobExecutionStatus.COMPLETE
+ }
+ }
+}
+
+object KyuubiStatus extends Enumeration {
Review Comment:
KyuubiStatus is too common.. let's rename it to `OperationProgressStatus`
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/JobProgressUpdate.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.operation
+
+import java.util
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.hive.service.rpc.thrift.TJobExecutionStatus
+
+case class JobProgressUpdate(
+ headers: util.List[String] = null,
+ rows: util.List[util.List[String]] = null,
+ footerSummary: String = null,
+ progressedPercentage: Double = 0.0,
+ startTimeMillis: Long = 0L,
+ status: String = null)
+
+object KyuubiProgressMonitorStatusMapper {
+
+ def forStatus(status: String): TJobExecutionStatus = {
+ if (StringUtils.isEmpty(status)) {
+ return TJobExecutionStatus.NOT_AVAILABLE
+ }
+ KyuubiStatus.withName(status) match {
+ case KyuubiStatus.NOT_AVAILABLE =>
+ TJobExecutionStatus.NOT_AVAILABLE
+ case KyuubiStatus.IN_PROGRESS | KyuubiStatus.PENDING | KyuubiStatus.RUNNING =>
+ TJobExecutionStatus.IN_PROGRESS
+ case KyuubiStatus.COMPLETE => TJobExecutionStatus.COMPLETE
Review Comment:
`FINISHED` ?
what's the difference between `COMPLETE` and `FINISHED` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r865572903
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/progress/SparkProgressMonitor.scala:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kyuubi.engine.spark.operation.progress
+
+import java.util
+
+import scala.collection.JavaConverters._
+
+import org.apache.kyuubi.engine.spark.operation.progress.SparkProgressMonitor._
+import org.apache.kyuubi.operation.KyuubiStatus
+
+class SparkProgressMonitor(progressMap: Map[SparkStage, SparkStageProgress], _startTime: Long = 0) {
+
+ def headers: util.List[String] = HEADERS
+
+ def rows: util.List[util.List[String]] = {
+ val progressRows = progressMap.toSeq.sortBy(_._1).map {
+ case (stage, progress) =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ val running = progress.runningTaskCount
+ val failed = progress.failedTaskCount
+ var state =
+ if (total > 0) {
+ KyuubiStatus.PENDING
+ } else {
+ KyuubiStatus.FINISHED
+ }
+ if (complete > 0 || running > 0 || failed > 0) {
+ state = if (complete < total) KyuubiStatus.RUNNING else KyuubiStatus.FINISHED
+ }
+ val attempt = String.valueOf(stage.attemptId)
+ val stageName = "Stage-" + String.valueOf(stage.stageId)
+ val nameWithProgress = getNameWithProgress(stageName, complete, total)
+ val pending = total - complete - running
+ util.Arrays.asList(
+ nameWithProgress,
+ attempt,
+ state.toString,
+ String.valueOf(total),
+ String.valueOf(complete),
+ String.valueOf(running),
+ String.valueOf(pending),
+ String.valueOf(failed),
+ "")
+ }.toList.asJavaCollection
+ new util.ArrayList[util.List[String]](progressRows)
+ }
+
+ def footerSummary: String = {
+ "STAGES: %02d/%02d".format(getCompletedStages, progressMap.keySet.size)
+ }
+
+ def progressedPercentage: Double = {
+ var sumTotal = 0
+ var sumComplete = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ sumTotal += total
+ sumComplete += complete
+ }
+ if (sumTotal == 0) {
+ 1.0f
+ } else {
+ sumComplete.toFloat / sumTotal.toFloat
+ }
+ }
+
+ def startTime: Long = _startTime
+
+ def executionStatus: String =
+ if (getCompletedStages == progressMap.keySet.size) {
+ KyuubiStatus.FINISHED.toString
+ } else {
+ KyuubiStatus.RUNNING.toString
+ }
+
+ private def getNameWithProgress(s: String, complete: Int, total: Int): String = {
+ if (s == null) return ""
+ val percent =
+ if (total == 0) 1.0f
+ else complete.toFloat / total.toFloat
+ // lets use the remaining space in column 1 as progress bar
+ val spaceRemaining = COLUMN_1_WIDTH - s.length - 1
+ var trimmedVName = s
+ // if the vertex name is longer than column 1 width, trim it down
+ if (s.length > COLUMN_1_WIDTH) {
+ trimmedVName = s.substring(0, COLUMN_1_WIDTH - 2)
+ trimmedVName += ".."
+ } else trimmedVName += " "
+ val result = new StringBuilder(trimmedVName)
+ val toFill = (spaceRemaining * percent).toInt
+ for (i <- 0 until toFill) {
+ result.append(".")
+ }
+ result.toString
+ }
+
+ private def getCompletedStages: Int = {
+ var completed = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ if (total > 0 && complete == total) completed += 1
+ }
+ completed
+ }
+
+}
+
+object SparkProgressMonitor {
+
+ private val HEADERS: util.List[String] = util.Arrays.asList(
+ "STAGES",
+ "ATTEMPT",
+ "STATUS",
+ "TOTAL",
+ "COMPLETED",
+ "RUNNING",
+ "PENDING",
+ "FAILED",
+ "")
Review Comment:
Referring to hive [SparkProgressMonitor.java#L46](https://github.com/apache/hive/blob/357d4906f5c806d585fd84db57cf296e12e6049b/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkProgressMonitor.java#L46), I'm not sure if it's for consistency with tez [TezProgressMonitor.java#L72](https://github.com/apache/hive/blob/357d4906f5c806d585fd84db57cf296e12e6049b/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezProgressMonitor.java#L72).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r865938698
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/OperationStatus.scala:
##########
@@ -27,4 +27,5 @@ case class OperationStatus(
lastModified: Long,
completed: Long,
hasResultSet: Boolean,
- exception: Option[KyuubiSQLException] = None)
+ exception: Option[KyuubiSQLException] = None,
+ jobProgressUpdate: Option[JobProgressUpdate] = None)
Review Comment:
> may be jobProgressUpdate: Option[TProgressUpdateResp] is OK to use
Does this mean I need to remove the JobProgressUpdate class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r866489113
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/progress/SparkProgressMonitor.scala:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kyuubi.engine.spark.operation.progress
+
+import java.util
+
+import scala.collection.JavaConverters._
+
+import org.apache.kyuubi.engine.spark.operation.progress.SparkProgressMonitor._
+import org.apache.kyuubi.operation.KyuubiStatus
+
+class SparkProgressMonitor(progressMap: Map[SparkStage, SparkStageProgress], _startTime: Long = 0) {
+
+ def headers: util.List[String] = HEADERS
+
+ def rows: util.List[util.List[String]] = {
+ val progressRows = progressMap.toSeq.sortBy(_._1).map {
+ case (stage, progress) =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ val running = progress.runningTaskCount
+ val failed = progress.failedTaskCount
+ var state =
+ if (total > 0) {
+ KyuubiStatus.PENDING
+ } else {
+ KyuubiStatus.FINISHED
+ }
+ if (complete > 0 || running > 0 || failed > 0) {
+ state = if (complete < total) KyuubiStatus.RUNNING else KyuubiStatus.FINISHED
+ }
+ val attempt = String.valueOf(stage.attemptId)
+ val stageName = "Stage-" + String.valueOf(stage.stageId)
+ val nameWithProgress = getNameWithProgress(stageName, complete, total)
+ val pending = total - complete - running
+ util.Arrays.asList(
+ nameWithProgress,
+ attempt,
+ state.toString,
+ String.valueOf(total),
+ String.valueOf(complete),
+ String.valueOf(running),
+ String.valueOf(pending),
+ String.valueOf(failed),
+ "")
+ }.toList.asJavaCollection
+ new util.ArrayList[util.List[String]](progressRows)
+ }
+
+ def footerSummary: String = {
+ "STAGES: %02d/%02d".format(getCompletedStages, progressMap.keySet.size)
+ }
+
+ def progressedPercentage: Double = {
+ var sumTotal = 0
+ var sumComplete = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ sumTotal += total
+ sumComplete += complete
+ }
+ if (sumTotal == 0) {
+ 1.0f
+ } else {
+ sumComplete.toFloat / sumTotal.toFloat
+ }
+ }
+
+ def startTime: Long = _startTime
+
+ def executionStatus: String =
+ if (getCompletedStages == progressMap.keySet.size) {
Review Comment:
does it affect the statement status if here return `TJobExecutionStatus.COMPLETE` ? Seems the progress status is isolation with statement status ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on pull request #2529: [WIP][KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1114258671
`SparkApp` is not cleaned up correctly. `SchedulerPoolSuite` and `SparkOperationProgressSuite` use the same SparkApp.
```
13:18:30.777 SparkSQLSessionManager-exec-pool: Thread-2236 INFO ExecuteStatement: Processing runner's query[f1b52565-be19-44f6-85cc-495fffcbd75d]: PENDING_STATE -> RUNNING_STATE, statement: SET kyuubi.operation.scheduler.pool=p0
13:18:30.778 SparkSQLSessionManager-exec-pool: Thread-2235 INFO ExecuteStatement: Processing runner's query[8c9e928c-62bd-4f91-9e38-fabcb9f99729]: PENDING_STATE -> RUNNING_STATE, statement: SET kyuubi.operation.scheduler.pool=p1
13:18:30.778 SparkSQLSessionManager-exec-pool: Thread-2236 INFO ExecuteStatement:
Spark application name: kyuubi_runner_spark_2022-05-01T13:13:24.276956Z
application ID: local-1651410805518
application web UI:
master: local
deploy mode: client
version: 3.1.3
Start time: 2022-05-01T13:13:24.332
User: runner
```
```
13:18:40.563 ScalaTest-main-running-SparkOperationProgressSuite DEBUG TSaslTransport: CLIENT: reading data length: 109
13:18:40.563 ScalaTest-main-running-SparkOperationProgressSuite DEBUG TSaslTransport: writing data length: 176
13:18:40.564 SparkTBinaryFrontendHandler-Pool: Thread-2292 DEBUG TSaslTransport: SERVER: reading data length: 176
13:18:40.564 SparkTBinaryFrontendHandler-Pool: Thread-2292 DEBUG SparkTBinaryFrontendService: TExecuteStatementReq(sessionHandle:TSessionHandle(sessionId:THandleIdentifier(guid:BA 9B B5 17 BE C2 47 11 BA D4 7B 3E C1 AF B1 C4, secret:9B 6C 6F 87 9E 04 40 C6 9A 6F 43 01 62 BC D5 AD)), statement:SELECT java_method('java.lang.Thread', 'sleep', 10000l) FROM range(1, 3, 1, 2), runAsync:true)
13:18:40.564 SparkTBinaryFrontendHandler-Pool: Thread-2292 INFO OperationLog: Creating operation log file /home/runner/work/incubator-kyuubi/incubator-kyuubi/externals/kyuubi-spark-sql-engine/target/work/target/engine_operation_logs/ba9bb517-bec2-4711-bad4-7b3ec1afb1c4/3700a110-6a26-4cee-ba89-cab2ccd07495
13:18:40.564 SparkTBinaryFrontendHandler-Pool: Thread-2292 INFO ExecuteStatement: Processing runner's query[3700a110-6a26-4cee-ba89-cab2ccd07495]: INITIALIZED_STATE -> PENDING_STATE, statement: SELECT java_method('java.lang.Thread', 'sleep', 10000l) FROM range(1, 3, 1, 2)
13:18:40.565 SparkTBinaryFrontendHandler-Pool: Thread-2292 DEBUG TSaslTransport: writing data length: 109
13:18:40.565 SparkSQLSessionManager-exec-pool: Thread-2293 INFO ExecuteStatement: Processing runner's query[3700a110-6a26-4cee-ba89-cab2ccd07495]: PENDING_STATE -> RUNNING_STATE, statement: SELECT java_method('java.lang.Thread', 'sleep', 10000l) FROM range(1, 3, 1, 2)
13:18:40.566 SparkSQLSessionManager-exec-pool: Thread-2293 INFO ExecuteStatement:
Spark application name: kyuubi_runner_spark_2022-05-01T13:13:24.276956Z
application ID: local-1651410805518
application web UI:
master: local
deploy mode: client
version: 3.1.3
Start time: 2022-05-01T13:13:24.332
User: runner
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] turboFei commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement for spark sql engine
Posted by GitBox <gi...@apache.org>.
turboFei commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1120444186
> > I met below exception when using kyuubi beeline to execute query if kyuubi.operation.progress.enabled is true.
>
> This may be related to this change [#2529 (comment)](https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r865924817), and I will verify it tomorrow.
sure, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on pull request #2529: [WIP][KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1114007123
Works fine with `AppStatusStore`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on pull request #2529: [WIP][KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1115723493
In JDK 11, the expected result didn't happen, but I can't seem to explain it, so I have to make it compatible.
expected result:
```
progress: 0.0
rows: [[Stage-1 , 0, RUNNING, 2, 0, 1, 1, 0, ]]
```
actual result:
```
progress: 0.0
rows: [[Stage-1 , 0, PENDING, 2, 0, 0, 2, 0, ]]
progress: 0.0
rows: [[Stage-1 , 0, RUNNING, 2, 0, 2, 0, 0, ]]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1119283459
> Irrelevant failed test
will be fixed in https://github.com/apache/incubator-kyuubi/pull/2557
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1119486983
In addition we may need a configuration to control whether to fetch operation progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r866703191
##########
externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/operation/SparkOperationProgressSuite.scala:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.engine.spark.operation
+
+import scala.collection.JavaConverters._
+
+import org.apache.hive.service.rpc.thrift.{TExecuteStatementReq, TGetOperationStatusReq, TJobExecutionStatus}
+import org.scalatest.concurrent.PatienceConfiguration.Timeout
+import org.scalatest.exceptions.TestFailedException
+import org.scalatest.time.SpanSugar.convertIntToGrainOfTime
+
+import org.apache.kyuubi.engine.spark.WithSparkSQLEngine
+import org.apache.kyuubi.operation.HiveJDBCTestHelper
+
+class SparkOperationProgressSuite extends WithSparkSQLEngine with HiveJDBCTestHelper {
+
+ override protected def jdbcUrl: String = getJdbcUrl
+ override def withKyuubiConf: Map[String, String] = Map(
+ "spark.master" -> "local[1]",
+ "kyuubi.operation.status.polling.timeout" -> "1000")
+
+ test("test operation progress") {
+ withSessionHandle { (client, handle) =>
+ val preReq = new TExecuteStatementReq()
+ preReq.setStatement("show databases")
+ preReq.setSessionHandle(handle)
+ preReq.setRunAsync(false)
+ client.ExecuteStatement(preReq)
+
+ val sql = "SELECT java_method('java.lang.Thread', 'sleep', 10000l) FROM range(1, 3, 1, 2)"
+ val req = new TExecuteStatementReq()
+ req.setStatement(sql)
+ req.setRunAsync(true)
+ req.setSessionHandle(handle)
+ val resp = client.ExecuteStatement(req)
+ var checkFlag1 = false
+ var checkFlag2 = false
+ var checkFlag3 = false
+ val initStageId = initJobId
+ eventually(Timeout(25.seconds)) {
+ val statusReq = new TGetOperationStatusReq(resp.getOperationHandle)
+ val statusResp = client.GetOperationStatus(statusReq)
+ val headers = statusResp.getProgressUpdateResponse.getHeaderNames
+ val progress = statusResp.getProgressUpdateResponse.getProgressedPercentage
+ val rows = statusResp.getProgressUpdateResponse.getRows
+ val footerSummary = statusResp.getProgressUpdateResponse.getFooterSummary
+ val status = statusResp.getProgressUpdateResponse.getStatus
+ assertResult(Seq(
+ "STAGES",
+ "ATTEMPT",
+ "STATUS",
+ "TOTAL",
+ "COMPLETED",
+ "RUNNING",
+ "PENDING",
+ "FAILED"))(headers.asScala)
+ assert(rows.size() == 1)
+ try {
+ progress match {
+ case 0.0 =>
+ assert(Seq(s"Stage-$initStageId ", "0", "PENDING", "2", "0", "0", "2", "0")
+ == rows.get(0).asScala ||
+ Seq(s"Stage-$initStageId ", "0", "RUNNING", "2", "0", "1", "1", "0")
+ == rows.get(0).asScala ||
+ Seq(s"Stage-$initStageId ", "0", "RUNNING", "2", "0", "2", "0", "0")
+ == rows.get(0).asScala)
+ assert("STAGES: 00/01" === footerSummary)
+ assert(TJobExecutionStatus.IN_PROGRESS === status)
+ checkFlag1 = true
+ case 0.5 =>
+ assertResult(Seq(
+ s"Stage-$initStageId ....",
+ "0",
+ "RUNNING",
+ "2",
+ "1",
+ "1",
+ "0",
+ "0"))(
+ rows.get(0).asScala)
+ assert("STAGES: 00/01" === footerSummary)
+ assert(TJobExecutionStatus.IN_PROGRESS === status)
+ checkFlag2 = true
+ case 1.0 =>
+ assertResult(Seq(
+ s"Stage-$initStageId ........",
+ "0",
+ "FINISHED",
+ "2",
+ "2",
+ "0",
+ "0",
+ "0"))(
+ rows.get(0).asScala)
+ assert("STAGES: 01/01" === footerSummary)
+ checkFlag3 = true
+ }
+ } catch {
+ case tfe: TestFailedException =>
+ // scalastyle:off println
+ println(s"progress: $progress")
+ println(s"rows: $rows")
+ println(s"footerSummary: $footerSummary")
+ println(s"status: $status")
+ // scalastyle:on println
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r866704612
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SparkProgressFetcher.scala:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.kyuubi
+
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.status.api.v1.StageStatus
+
+import org.apache.kyuubi.engine.spark.operation.progress.{SparkProgressMonitor, SparkStage, SparkStageProgress}
+import org.apache.kyuubi.operation.JobProgressUpdate
+
+class SparkProgressFetcher(spark: SparkSession, jobGroup: String) {
+
+ val statusStore = spark.sparkContext.statusStore
+
+ def getJobProgressUpdate(startTime: Long): JobProgressUpdate = {
Review Comment:
> the start time can be removed, it's just `operationStatus.start`
It is used to initialize the TProgressUpdateResp object.
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/OperationStatus.scala:
##########
@@ -27,4 +27,5 @@ case class OperationStatus(
lastModified: Long,
completed: Long,
hasResultSet: Boolean,
- exception: Option[KyuubiSQLException] = None)
+ exception: Option[KyuubiSQLException] = None,
+ jobProgressUpdate: Option[JobProgressUpdate] = None)
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r865924817
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/progress/SparkProgressMonitor.scala:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kyuubi.engine.spark.operation.progress
+
+import java.util
+
+import scala.collection.JavaConverters._
+
+import org.apache.kyuubi.engine.spark.operation.progress.SparkProgressMonitor._
+import org.apache.kyuubi.operation.KyuubiStatus
+
+class SparkProgressMonitor(progressMap: Map[SparkStage, SparkStageProgress], _startTime: Long = 0) {
+
+ def headers: util.List[String] = HEADERS
+
+ def rows: util.List[util.List[String]] = {
+ val progressRows = progressMap.toSeq.sortBy(_._1).map {
+ case (stage, progress) =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ val running = progress.runningTaskCount
+ val failed = progress.failedTaskCount
+ var state =
+ if (total > 0) {
+ KyuubiStatus.PENDING
+ } else {
+ KyuubiStatus.FINISHED
+ }
+ if (complete > 0 || running > 0 || failed > 0) {
+ state = if (complete < total) KyuubiStatus.RUNNING else KyuubiStatus.FINISHED
+ }
+ val attempt = String.valueOf(stage.attemptId)
+ val stageName = "Stage-" + String.valueOf(stage.stageId)
+ val nameWithProgress = getNameWithProgress(stageName, complete, total)
+ val pending = total - complete - running
+ util.Arrays.asList(
+ nameWithProgress,
+ attempt,
+ state.toString,
+ String.valueOf(total),
+ String.valueOf(complete),
+ String.valueOf(running),
+ String.valueOf(pending),
+ String.valueOf(failed),
+ "")
+ }.toList.asJavaCollection
+ new util.ArrayList[util.List[String]](progressRows)
+ }
+
+ def footerSummary: String = {
+ "STAGES: %02d/%02d".format(getCompletedStages, progressMap.keySet.size)
+ }
+
+ def progressedPercentage: Double = {
+ var sumTotal = 0
+ var sumComplete = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ sumTotal += total
+ sumComplete += complete
+ }
+ if (sumTotal == 0) {
+ 1.0f
+ } else {
+ sumComplete.toFloat / sumTotal.toFloat
+ }
+ }
+
+ def startTime: Long = _startTime
+
+ def executionStatus: String =
+ if (getCompletedStages == progressMap.keySet.size) {
+ KyuubiStatus.FINISHED.toString
+ } else {
+ KyuubiStatus.RUNNING.toString
+ }
+
+ private def getNameWithProgress(s: String, complete: Int, total: Int): String = {
+ if (s == null) return ""
+ val percent =
+ if (total == 0) 1.0f
+ else complete.toFloat / total.toFloat
+ // lets use the remaining space in column 1 as progress bar
+ val spaceRemaining = COLUMN_1_WIDTH - s.length - 1
+ var trimmedVName = s
+ // if the vertex name is longer than column 1 width, trim it down
+ if (s.length > COLUMN_1_WIDTH) {
+ trimmedVName = s.substring(0, COLUMN_1_WIDTH - 2)
+ trimmedVName += ".."
+ } else trimmedVName += " "
+ val result = new StringBuilder(trimmedVName)
+ val toFill = (spaceRemaining * percent).toInt
+ for (i <- 0 until toFill) {
+ result.append(".")
+ }
+ result.toString
+ }
+
+ private def getCompletedStages: Int = {
+ var completed = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ if (total > 0 && complete == total) completed += 1
+ }
+ completed
+ }
+
+}
+
+object SparkProgressMonitor {
+
+ private val HEADERS: util.List[String] = util.Arrays.asList(
+ "STAGES",
+ "ATTEMPT",
+ "STATUS",
+ "TOTAL",
+ "COMPLETED",
+ "RUNNING",
+ "PENDING",
+ "FAILED",
+ "")
Review Comment:
> I think we can remove it , looks like just a placeholder
It seems so, I will remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r865533062
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/progress/SparkProgressMonitor.scala:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kyuubi.engine.spark.operation.progress
+
+import java.util
+
+import scala.collection.JavaConverters._
+
+import org.apache.kyuubi.engine.spark.operation.progress.SparkProgressMonitor._
+import org.apache.kyuubi.operation.KyuubiStatus
+
+class SparkProgressMonitor(progressMap: Map[SparkStage, SparkStageProgress], _startTime: Long = 0) {
+
+ def headers: util.List[String] = HEADERS
+
+ def rows: util.List[util.List[String]] = {
+ val progressRows = progressMap.toSeq.sortBy(_._1).map {
+ case (stage, progress) =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ val running = progress.runningTaskCount
+ val failed = progress.failedTaskCount
+ var state =
+ if (total > 0) {
+ KyuubiStatus.PENDING
+ } else {
+ KyuubiStatus.FINISHED
+ }
+ if (complete > 0 || running > 0 || failed > 0) {
+ state = if (complete < total) KyuubiStatus.RUNNING else KyuubiStatus.FINISHED
+ }
+ val attempt = String.valueOf(stage.attemptId)
+ val stageName = "Stage-" + String.valueOf(stage.stageId)
+ val nameWithProgress = getNameWithProgress(stageName, complete, total)
+ val pending = total - complete - running
+ util.Arrays.asList(
+ nameWithProgress,
+ attempt,
+ state.toString,
+ String.valueOf(total),
+ String.valueOf(complete),
+ String.valueOf(running),
+ String.valueOf(pending),
+ String.valueOf(failed),
+ "")
+ }.toList.asJavaCollection
+ new util.ArrayList[util.List[String]](progressRows)
+ }
+
+ def footerSummary: String = {
+ "STAGES: %02d/%02d".format(getCompletedStages, progressMap.keySet.size)
+ }
+
+ def progressedPercentage: Double = {
+ var sumTotal = 0
+ var sumComplete = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ sumTotal += total
+ sumComplete += complete
+ }
+ if (sumTotal == 0) {
+ 1.0f
+ } else {
+ sumComplete.toFloat / sumTotal.toFloat
+ }
+ }
+
+ def startTime: Long = _startTime
+
+ def executionStatus: String =
+ if (getCompletedStages == progressMap.keySet.size) {
Review Comment:
> the stage will be generated during execution
The stage seems to be generated in statusStore when job start.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r865833354
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/progress/SparkProgressMonitor.scala:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kyuubi.engine.spark.operation.progress
+
+import java.util
+
+import scala.collection.JavaConverters._
+
+import org.apache.kyuubi.engine.spark.operation.progress.SparkProgressMonitor._
+import org.apache.kyuubi.operation.KyuubiStatus
+
+class SparkProgressMonitor(progressMap: Map[SparkStage, SparkStageProgress], _startTime: Long = 0) {
+
+ def headers: util.List[String] = HEADERS
+
+ def rows: util.List[util.List[String]] = {
+ val progressRows = progressMap.toSeq.sortBy(_._1).map {
+ case (stage, progress) =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ val running = progress.runningTaskCount
+ val failed = progress.failedTaskCount
+ var state =
+ if (total > 0) {
+ KyuubiStatus.PENDING
+ } else {
+ KyuubiStatus.FINISHED
+ }
+ if (complete > 0 || running > 0 || failed > 0) {
+ state = if (complete < total) KyuubiStatus.RUNNING else KyuubiStatus.FINISHED
+ }
+ val attempt = String.valueOf(stage.attemptId)
+ val stageName = "Stage-" + String.valueOf(stage.stageId)
+ val nameWithProgress = getNameWithProgress(stageName, complete, total)
+ val pending = total - complete - running
+ util.Arrays.asList(
+ nameWithProgress,
+ attempt,
+ state.toString,
+ String.valueOf(total),
+ String.valueOf(complete),
+ String.valueOf(running),
+ String.valueOf(pending),
+ String.valueOf(failed),
+ "")
+ }.toList.asJavaCollection
+ new util.ArrayList[util.List[String]](progressRows)
+ }
+
+ def footerSummary: String = {
+ "STAGES: %02d/%02d".format(getCompletedStages, progressMap.keySet.size)
+ }
+
+ def progressedPercentage: Double = {
+ var sumTotal = 0
+ var sumComplete = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ sumTotal += total
+ sumComplete += complete
+ }
+ if (sumTotal == 0) {
+ 1.0f
+ } else {
+ sumComplete.toFloat / sumTotal.toFloat
+ }
+ }
+
+ def startTime: Long = _startTime
+
+ def executionStatus: String =
+ if (getCompletedStages == progressMap.keySet.size) {
+ KyuubiStatus.FINISHED.toString
+ } else {
+ KyuubiStatus.RUNNING.toString
+ }
+
+ private def getNameWithProgress(s: String, complete: Int, total: Int): String = {
+ if (s == null) return ""
+ val percent =
+ if (total == 0) 1.0f
+ else complete.toFloat / total.toFloat
+ // lets use the remaining space in column 1 as progress bar
+ val spaceRemaining = COLUMN_1_WIDTH - s.length - 1
+ var trimmedVName = s
+ // if the vertex name is longer than column 1 width, trim it down
+ if (s.length > COLUMN_1_WIDTH) {
+ trimmedVName = s.substring(0, COLUMN_1_WIDTH - 2)
+ trimmedVName += ".."
+ } else trimmedVName += " "
+ val result = new StringBuilder(trimmedVName)
+ val toFill = (spaceRemaining * percent).toInt
+ for (i <- 0 until toFill) {
+ result.append(".")
+ }
+ result.toString
+ }
+
+ private def getCompletedStages: Int = {
+ var completed = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ if (total > 0 && complete == total) completed += 1
+ }
+ completed
+ }
+
+}
+
+object SparkProgressMonitor {
+
+ private val HEADERS: util.List[String] = util.Arrays.asList(
+ "STAGES",
+ "ATTEMPT",
+ "STATUS",
+ "TOTAL",
+ "COMPLETED",
+ "RUNNING",
+ "PENDING",
+ "FAILED",
+ "")
Review Comment:
I think we can remove it , looks like just a placeholder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r867319415
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SparkProgressFetcher.scala:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.kyuubi
+
+import org.apache.hive.service.rpc.thrift.TProgressUpdateResp
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.status.api.v1.StageStatus
+
+import org.apache.kyuubi.engine.spark.operation.progress.{SparkProgressMonitor, SparkStage, SparkStageProgress}
+
+class SparkProgressFetcher(spark: SparkSession, jobGroup: String) {
+
+ val statusStore = spark.sparkContext.statusStore
+
+ def getJobProgressUpdate(startTime: Long): TProgressUpdateResp = {
+ val progressMap = getProgressMap
+ val progressMonitor = new SparkProgressMonitor(progressMap)
+ new TProgressUpdateResp(
+ progressMonitor.headers,
+ progressMonitor.rows,
+ progressMonitor.progressedPercentage,
+ progressMonitor.executionStatus,
+ progressMonitor.footerSummary,
+ startTime)
+ }
+
+ private def getProgressMap(): Map[SparkStage, SparkStageProgress] = {
Review Comment:
move this to SparkProgressMonitor as a `private val`?
and we sort it once?
https://github.com/apache/incubator-kyuubi/pull/2529/files#diff-441ca5a194460cd9dfe41b5e3db54989c91d9aac23418329b1b3216ba7d8c7eeR32
https://github.com/apache/incubator-kyuubi/pull/2529/files#diff-441ca5a194460cd9dfe41b5e3db54989c91d9aac23418329b1b3216ba7d8c7eeR76
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] turboFei commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
turboFei commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r867310744
##########
docs/deployment/settings.md:
##########
@@ -381,6 +381,7 @@ Key | Default | Meaning | Type | Since
<code>kyuubi.session.engine.trino.main.resource</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'><undefined></div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The package used to create Trino engine remote job. If it is undefined, Kyuubi will use the default</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.5.0</div>
<code>kyuubi.session.idle.timeout</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>PT6H</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>session idle timeout, it will be closed when it's not accessed for this duration</div>|<div style='width: 30pt'>duration</div>|<div style='width: 20pt'>1.2.0</div>
<code>kyuubi.session.name</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'><undefined></div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>A human readable name of session and we use empty string by default. This name will be recorded in event. Note that, we only apply this value from session conf.</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.4.0</div>
+<code>kyuubi.session.progress.enable</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>false</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Whether to enable the operation progress. When true, the operation progress will be returned in `GetOperationStatus`.</div>|<div style='width: 30pt'>boolean</div>|<div style='width: 20pt'>1.6.0</div>
Review Comment:
now it is not enabled for all engines, only for spark now.
So, it is better to identify that it is only for spark engine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] turboFei commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
turboFei commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r867310836
##########
docs/deployment/settings.md:
##########
@@ -381,6 +381,7 @@ Key | Default | Meaning | Type | Since
<code>kyuubi.session.engine.trino.main.resource</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'><undefined></div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The package used to create Trino engine remote job. If it is undefined, Kyuubi will use the default</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.5.0</div>
<code>kyuubi.session.idle.timeout</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>PT6H</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>session idle timeout, it will be closed when it's not accessed for this duration</div>|<div style='width: 30pt'>duration</div>|<div style='width: 20pt'>1.2.0</div>
<code>kyuubi.session.name</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'><undefined></div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>A human readable name of session and we use empty string by default. This name will be recorded in event. Note that, we only apply this value from session conf.</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.4.0</div>
+<code>kyuubi.session.progress.enable</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>false</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Whether to enable the operation progress. When true, the operation progress will be returned in `GetOperationStatus`.</div>|<div style='width: 30pt'>boolean</div>|<div style='width: 20pt'>1.6.0</div>
Review Comment:
FYI: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1119547236
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r865837740
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/JobProgressUpdate.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.operation
+
+import java.util
+
+import org.apache.commons.lang3.StringUtils
+import org.apache.hive.service.rpc.thrift.TJobExecutionStatus
+
+case class JobProgressUpdate(
+ headers: util.List[String] = null,
+ rows: util.List[util.List[String]] = null,
+ footerSummary: String = null,
+ progressedPercentage: Double = 0.0,
+ startTimeMillis: Long = 0L,
+ status: String = null)
+
+object KyuubiProgressMonitorStatusMapper {
+
+ def forStatus(status: String): TJobExecutionStatus = {
+ if (StringUtils.isEmpty(status)) {
+ return TJobExecutionStatus.NOT_AVAILABLE
+ }
+ KyuubiStatus.withName(status) match {
+ case KyuubiStatus.NOT_AVAILABLE =>
+ TJobExecutionStatus.NOT_AVAILABLE
+ case KyuubiStatus.IN_PROGRESS | KyuubiStatus.PENDING | KyuubiStatus.RUNNING =>
+ TJobExecutionStatus.IN_PROGRESS
+ case KyuubiStatus.COMPLETE => TJobExecutionStatus.COMPLETE
+ case _ => TJobExecutionStatus.COMPLETE
+ }
+ }
+}
+
+object KyuubiStatus extends Enumeration {
+ type KyuubiStatus = Value
+ val PENDING, RUNNING, FINISHED, IN_PROGRESS, COMPLETE, NOT_AVAILABLE = Value
Review Comment:
it really confused, seems we can just retain ?
IN_PROGRESS, COMPLETE, NOT_AVAILABLE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on pull request #2529: [WIP][KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1114001829
> Can we distinguish skipped stages with AppStatusStore?
I'm not sure if `StageData#status` for skiped stage is `StageStatus.SKIPPED`, if so we can do that. I will try later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on pull request #2529: [WIP][KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1114519753
> `SparkApp` is not cleaned up correctly. `SchedulerPoolSuite` and `SparkOperationProgressSuite` use the same SparkApp.
This is only because they use the same `KyuubiSparkUtil#diagnostics` object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r867332624
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SparkProgressFetcher.scala:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.kyuubi
+
+import org.apache.hive.service.rpc.thrift.TProgressUpdateResp
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.status.api.v1.StageStatus
+
+import org.apache.kyuubi.engine.spark.operation.progress.{SparkProgressMonitor, SparkStage, SparkStageProgress}
+
+class SparkProgressFetcher(spark: SparkSession, jobGroup: String) {
+
+ val statusStore = spark.sparkContext.statusStore
+
+ def getJobProgressUpdate(startTime: Long): TProgressUpdateResp = {
+ val progressMap = getProgressMap
+ val progressMonitor = new SparkProgressMonitor(progressMap)
+ new TProgressUpdateResp(
+ progressMonitor.headers,
+ progressMonitor.rows,
+ progressMonitor.progressedPercentage,
+ progressMonitor.executionStatus,
+ progressMonitor.footerSummary,
+ startTime)
+ }
+
+ private def getProgressMap(): Map[SparkStage, SparkStageProgress] = {
Review Comment:
How about removing SparkProgressFetcher and merging it into SparkProgressMonitor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1120181023
> So, now `kyuubi.session.engine.spark.showProgress` works for both beeline and other clients ?
It looks like this, it prints progress to operation log and get log in KyuubiStatement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r867334993
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/operation/ExecuteStatement.scala:
##########
@@ -81,6 +81,9 @@ class ExecuteStatement(
while (!isComplete) {
fetchQueryLog()
verifyTStatus(statusResp.getStatus)
+ if (statusResp.getProgressUpdateResponse != null) {
+ setOperationJobProgress(statusResp.getProgressUpdateResponse)
Review Comment:
https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1120181210 here the best place we stringify the ProgressUpdateResponse and do the log
And maybe refine L88 together
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r866524263
##########
externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/operation/SparkOperationProgressSuite.scala:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.engine.spark.operation
+
+import scala.collection.JavaConverters._
+
+import org.apache.hive.service.rpc.thrift.{TExecuteStatementReq, TGetOperationStatusReq, TJobExecutionStatus}
+import org.scalatest.concurrent.PatienceConfiguration.Timeout
+import org.scalatest.exceptions.TestFailedException
+import org.scalatest.time.SpanSugar.convertIntToGrainOfTime
+
+import org.apache.kyuubi.engine.spark.WithSparkSQLEngine
+import org.apache.kyuubi.operation.HiveJDBCTestHelper
+
+class SparkOperationProgressSuite extends WithSparkSQLEngine with HiveJDBCTestHelper {
+
+ override protected def jdbcUrl: String = getJdbcUrl
+ override def withKyuubiConf: Map[String, String] = Map(
+ "spark.master" -> "local[1]",
+ "kyuubi.operation.status.polling.timeout" -> "1000")
+
+ test("test operation progress") {
+ withSessionHandle { (client, handle) =>
+ val preReq = new TExecuteStatementReq()
+ preReq.setStatement("show databases")
+ preReq.setSessionHandle(handle)
+ preReq.setRunAsync(false)
+ client.ExecuteStatement(preReq)
+
+ val sql = "SELECT java_method('java.lang.Thread', 'sleep', 10000l) FROM range(1, 3, 1, 2)"
+ val req = new TExecuteStatementReq()
+ req.setStatement(sql)
+ req.setRunAsync(true)
+ req.setSessionHandle(handle)
+ val resp = client.ExecuteStatement(req)
+ var checkFlag1 = false
+ var checkFlag2 = false
+ var checkFlag3 = false
+ val initStageId = initJobId
+ eventually(Timeout(25.seconds)) {
+ val statusReq = new TGetOperationStatusReq(resp.getOperationHandle)
+ val statusResp = client.GetOperationStatus(statusReq)
+ val headers = statusResp.getProgressUpdateResponse.getHeaderNames
+ val progress = statusResp.getProgressUpdateResponse.getProgressedPercentage
+ val rows = statusResp.getProgressUpdateResponse.getRows
+ val footerSummary = statusResp.getProgressUpdateResponse.getFooterSummary
+ val status = statusResp.getProgressUpdateResponse.getStatus
+ assertResult(Seq(
+ "STAGES",
+ "ATTEMPT",
+ "STATUS",
+ "TOTAL",
+ "COMPLETED",
+ "RUNNING",
+ "PENDING",
+ "FAILED"))(headers.asScala)
+ assert(rows.size() == 1)
+ try {
+ progress match {
+ case 0.0 =>
+ assert(Seq(s"Stage-$initStageId ", "0", "PENDING", "2", "0", "0", "2", "0")
+ == rows.get(0).asScala ||
+ Seq(s"Stage-$initStageId ", "0", "RUNNING", "2", "0", "1", "1", "0")
+ == rows.get(0).asScala ||
+ Seq(s"Stage-$initStageId ", "0", "RUNNING", "2", "0", "2", "0", "0")
+ == rows.get(0).asScala)
+ assert("STAGES: 00/01" === footerSummary)
+ assert(TJobExecutionStatus.IN_PROGRESS === status)
+ checkFlag1 = true
+ case 0.5 =>
+ assertResult(Seq(
+ s"Stage-$initStageId ....",
+ "0",
+ "RUNNING",
+ "2",
+ "1",
+ "1",
+ "0",
+ "0"))(
+ rows.get(0).asScala)
+ assert("STAGES: 00/01" === footerSummary)
+ assert(TJobExecutionStatus.IN_PROGRESS === status)
+ checkFlag2 = true
+ case 1.0 =>
+ assertResult(Seq(
+ s"Stage-$initStageId ........",
+ "0",
+ "FINISHED",
+ "2",
+ "2",
+ "0",
+ "0",
+ "0"))(
+ rows.get(0).asScala)
+ assert("STAGES: 01/01" === footerSummary)
+ checkFlag3 = true
+ }
+ } catch {
+ case tfe: TestFailedException =>
+ // scalastyle:off println
+ println(s"progress: $progress")
+ println(s"rows: $rows")
+ println(s"footerSummary: $footerSummary")
+ println(s"status: $status")
+ // scalastyle:on println
Review Comment:
> remove this, if needed we can use `info`
It is only used to debug this test case, I will remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r866524000
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/OperationStatus.scala:
##########
@@ -27,4 +27,5 @@ case class OperationStatus(
lastModified: Long,
completed: Long,
hasResultSet: Boolean,
- exception: Option[KyuubiSQLException] = None)
+ exception: Option[KyuubiSQLException] = None,
+ jobProgressUpdate: Option[JobProgressUpdate] = None)
Review Comment:
yes, it seem that we don't need wrap/unwrap at engine and server side
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#issuecomment-1119277863
Irrelevant failed test
```
KyuubiBatchYarnClusterSuite:
- open batch session *** FAILED ***
The code passed to eventually never returned normally. Attempted 3585 times over 3.0003451565333332 minutes. Last failure message: FINISHED did not equal ERROR. (KyuubiBatchYarnClusterSuite.scala:87)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r866529260
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/OperationStatus.scala:
##########
@@ -27,4 +27,5 @@ case class OperationStatus(
lastModified: Long,
completed: Long,
hasResultSet: Boolean,
- exception: Option[KyuubiSQLException] = None)
+ exception: Option[KyuubiSQLException] = None,
+ jobProgressUpdate: Option[JobProgressUpdate] = None)
Review Comment:
> yes, it seem that we don't need wrap/unwrap at engine and server side
Thanks, I will remove it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wForget commented on a diff in pull request #2529: [KYUUBI #2493] Implement the progress of statement
Posted by GitBox <gi...@apache.org>.
wForget commented on code in PR #2529:
URL: https://github.com/apache/incubator-kyuubi/pull/2529#discussion_r865821684
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/progress/SparkProgressMonitor.scala:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kyuubi.engine.spark.operation.progress
+
+import java.util
+
+import scala.collection.JavaConverters._
+
+import org.apache.kyuubi.engine.spark.operation.progress.SparkProgressMonitor._
+import org.apache.kyuubi.operation.KyuubiStatus
+
+class SparkProgressMonitor(progressMap: Map[SparkStage, SparkStageProgress], _startTime: Long = 0) {
+
+ def headers: util.List[String] = HEADERS
+
+ def rows: util.List[util.List[String]] = {
+ val progressRows = progressMap.toSeq.sortBy(_._1).map {
+ case (stage, progress) =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ val running = progress.runningTaskCount
+ val failed = progress.failedTaskCount
+ var state =
+ if (total > 0) {
+ KyuubiStatus.PENDING
+ } else {
+ KyuubiStatus.FINISHED
+ }
+ if (complete > 0 || running > 0 || failed > 0) {
+ state = if (complete < total) KyuubiStatus.RUNNING else KyuubiStatus.FINISHED
+ }
+ val attempt = String.valueOf(stage.attemptId)
+ val stageName = "Stage-" + String.valueOf(stage.stageId)
+ val nameWithProgress = getNameWithProgress(stageName, complete, total)
+ val pending = total - complete - running
+ util.Arrays.asList(
+ nameWithProgress,
+ attempt,
+ state.toString,
+ String.valueOf(total),
+ String.valueOf(complete),
+ String.valueOf(running),
+ String.valueOf(pending),
+ String.valueOf(failed),
+ "")
+ }.toList.asJavaCollection
+ new util.ArrayList[util.List[String]](progressRows)
+ }
+
+ def footerSummary: String = {
+ "STAGES: %02d/%02d".format(getCompletedStages, progressMap.keySet.size)
+ }
+
+ def progressedPercentage: Double = {
+ var sumTotal = 0
+ var sumComplete = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
+ val complete = progress.succeededTaskCount
+ val total = progress.totalTaskCount
+ sumTotal += total
+ sumComplete += complete
+ }
+ if (sumTotal == 0) {
+ 1.0f
+ } else {
+ sumComplete.toFloat / sumTotal.toFloat
+ }
+ }
+
+ def startTime: Long = _startTime
+
+ def executionStatus: String =
+ if (getCompletedStages == progressMap.keySet.size) {
+ KyuubiStatus.FINISHED.toString
+ } else {
+ KyuubiStatus.RUNNING.toString
+ }
+
+ private def getNameWithProgress(s: String, complete: Int, total: Int): String = {
+ if (s == null) return ""
+ val percent =
+ if (total == 0) 1.0f
+ else complete.toFloat / total.toFloat
+ // lets use the remaining space in column 1 as progress bar
+ val spaceRemaining = COLUMN_1_WIDTH - s.length - 1
+ var trimmedVName = s
+ // if the vertex name is longer than column 1 width, trim it down
+ if (s.length > COLUMN_1_WIDTH) {
+ trimmedVName = s.substring(0, COLUMN_1_WIDTH - 2)
+ trimmedVName += ".."
+ } else trimmedVName += " "
+ val result = new StringBuilder(trimmedVName)
+ val toFill = (spaceRemaining * percent).toInt
+ for (i <- 0 until toFill) {
+ result.append(".")
+ }
+ result.toString
+ }
+
+ private def getCompletedStages: Int = {
+ var completed = 0
+ progressMap.toSeq.sortBy(_._1).map(_._2).foreach { progress =>
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] turboFei closed pull request #2529: [KYUUBI #2493] Implement the progress of statement for spark sql engine
Posted by GitBox <gi...@apache.org>.
turboFei closed pull request #2529: [KYUUBI #2493] Implement the progress of statement for spark sql engine
URL: https://github.com/apache/incubator-kyuubi/pull/2529
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org