You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "turboFei (via GitHub)" <gi...@apache.org> on 2023/12/19 02:53:09 UTC
[PR] [KYUUBI #5873] Support to get operation progress response with RESTful api [kyuubi]
turboFei opened a new pull request, #5875:
URL: https://github.com/apache/kyuubi/pull/5875
# :mag: Description
## Issue References ๐
<!-- Append the issue number after #. If there is no issue for you to link create one or -->
<!-- If there are no issues to link, please provide details here. -->
As title, support to get operation progress with RESTful api.
This pull request fixes #5873
## Describe Your Solution ๐ง
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes :bookmark:
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan ๐งช
#### Behavior Without This Pull Request :coffin:
#### Behavior With This Pull Request :tada:
#### Related Unit Tests
---
# Checklists
## ๐ Author Self Checklist
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## ๐ Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5873] Support to get operation progress with RESTful API [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #5875:
URL: https://github.com/apache/kyuubi/pull/5875#discussion_r1430909892
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/events/KyuubiOperationEvent.scala:
##########
@@ -60,7 +61,8 @@ case class KyuubiOperationEvent private (
sessionUser: String,
sessionType: String,
kyuubiInstance: String,
- metrics: Map[String, String]) extends KyuubiEvent {
+ metrics: Map[String, String],
+ progress: OperationProgress) extends KyuubiEvent {
Review Comment:
https://github.com/apache/kyuubi/blob/7e96dc7bc9a36cf4e48b3fced3b07e6fe6cd45ab/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/OperationsResource.scala#L43-L63
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5873] Support to get operation progress with RESTful API [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on PR #5875:
URL: https://github.com/apache/kyuubi/pull/5875#issuecomment-1862195292
> LGTM for this pr.
>
> But I still have some confusion.
>
> 1. The API for obtaining session and operation details are inconsistent: `@Path("{sessionHandle}")` and `@Path("{operationHandle}/event")`
> 2. The naming of `dto.KyuubiOperationEvent` and `dto.KyuubiSessionEvent` is confusing, they represent the details of the current session and operation, maybe named `KyuubiXXXDto` is more appropriate.
Thanks, let us followup it in another pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5873] Support to get operation progress with RESTful API [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #5875:
URL: https://github.com/apache/kyuubi/pull/5875#discussion_r1430922462
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/ApiUtils.scala:
##########
@@ -60,7 +60,8 @@ object ApiUtils extends Logging {
opEvent.sessionUser,
opEvent.sessionType,
operation.getSession.asInstanceOf[KyuubiSession].connectionUrl,
- operation.metrics.asJava)
+ operation.metrics.asJava,
+ opEvent.progress)
Review Comment:
addressed, PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5873] Support to get operation progress with RESTful API [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on PR #5875:
URL: https://github.com/apache/kyuubi/pull/5875#issuecomment-1863190666
thanks, merged to branch-1.9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5873] Support to get operation progress with RESTful API [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #5875:
URL: https://github.com/apache/kyuubi/pull/5875#discussion_r1430910169
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/events/KyuubiOperationEvent.scala:
##########
@@ -60,7 +61,8 @@ case class KyuubiOperationEvent private (
sessionUser: String,
sessionType: String,
kyuubiInstance: String,
- metrics: Map[String, String]) extends KyuubiEvent {
+ metrics: Map[String, String],
+ progress: OperationProgress) extends KyuubiEvent {
Review Comment:
Now `KyuubiOperationEvent` is also used for this REST api.
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/events/KyuubiOperationEvent.scala:
##########
@@ -60,7 +61,8 @@ case class KyuubiOperationEvent private (
sessionUser: String,
sessionType: String,
kyuubiInstance: String,
- metrics: Map[String, String]) extends KyuubiEvent {
+ metrics: Map[String, String],
+ progress: OperationProgress) extends KyuubiEvent {
Review Comment:
Now `KyuubiOperationEvent` is also used for this REST API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5873] Support to get operation progress with RESTful API [kyuubi]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #5875:
URL: https://github.com/apache/kyuubi/pull/5875#discussion_r1430908661
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/events/KyuubiOperationEvent.scala:
##########
@@ -60,7 +61,8 @@ case class KyuubiOperationEvent private (
sessionUser: String,
sessionType: String,
kyuubiInstance: String,
- metrics: Map[String, String]) extends KyuubiEvent {
+ metrics: Map[String, String],
+ progress: OperationProgress) extends KyuubiEvent {
Review Comment:
Why do we need to record progress in event? I think progress should be fetched in real time instead of recorded in event.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5873] Support to get operation progress with RESTful API [kyuubi]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #5875:
URL: https://github.com/apache/kyuubi/pull/5875#discussion_r1430914275
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/events/KyuubiOperationEvent.scala:
##########
@@ -60,7 +61,8 @@ case class KyuubiOperationEvent private (
sessionUser: String,
sessionType: String,
kyuubiInstance: String,
- metrics: Map[String, String]) extends KyuubiEvent {
+ metrics: Map[String, String],
+ progress: OperationProgress) extends KyuubiEvent {
Review Comment:
`KyuubiOperationEvent` is only updated when the operation status changes, so it is not real-time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5873] Support to get operation progress with RESTful API [kyuubi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5875:
URL: https://github.com/apache/kyuubi/pull/5875#issuecomment-1862093323
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/5875?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: `26 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`487dd08`)](https://app.codecov.io/gh/apache/kyuubi/commit/487dd0810a818ef037a73515226e2be30070cc4e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.40% compared to head [(`0dbd450`)](https://app.codecov.io/gh/apache/kyuubi/pull/5875?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.38%.
> Report is 5 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/kyuubi/pull/5875?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...he/kyuubi/client/api/v1/dto/OperationProgress.java](https://app.codecov.io/gh/apache/kyuubi/pull/5875?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXJlc3QtY2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reXV1YmkvY2xpZW50L2FwaS92MS9kdG8vT3BlcmF0aW9uUHJvZ3Jlc3MuamF2YQ==) | 56.25% | [17 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/5875?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...kyuubi/client/api/v1/dto/KyuubiOperationEvent.java](https://app.codecov.io/gh/apache/kyuubi/pull/5875?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXJlc3QtY2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reXV1YmkvY2xpZW50L2FwaS92MS9kdG8vS3l1dWJpT3BlcmF0aW9uRXZlbnQuamF2YQ==) | 28.57% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/5875?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #5875 +/- ##
============================================
- Coverage 61.40% 61.38% -0.02%
Complexity 23 23
============================================
Files 612 613 +1
Lines 36239 36310 +71
Branches 4967 4974 +7
============================================
+ Hits 22251 22288 +37
- Misses 11593 11619 +26
- Partials 2395 2403 +8
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/5875?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5873] Support to get operation progress with RESTful API [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #5875:
URL: https://github.com/apache/kyuubi/pull/5875#discussion_r1430909892
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/events/KyuubiOperationEvent.scala:
##########
@@ -60,7 +61,8 @@ case class KyuubiOperationEvent private (
sessionUser: String,
sessionType: String,
kyuubiInstance: String,
- metrics: Map[String, String]) extends KyuubiEvent {
+ metrics: Map[String, String],
+ progress: OperationProgress) extends KyuubiEvent {
Review Comment:
https://github.com/apache/kyuubi/blob/master/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/OperationsResource.scala#L43-L63
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5873] Support to get operation progress with RESTful API [kyuubi]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #5875:
URL: https://github.com/apache/kyuubi/pull/5875#discussion_r1430916746
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/ApiUtils.scala:
##########
@@ -60,7 +60,8 @@ object ApiUtils extends Logging {
opEvent.sessionUser,
opEvent.sessionType,
operation.getSession.asInstanceOf[KyuubiSession].connectionUrl,
- operation.metrics.asJava)
+ operation.metrics.asJava,
+ opEvent.progress)
Review Comment:
How about we use `operation.getOperationJobProgress` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5873] Support to get operation progress with RESTful API [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #5875:
URL: https://github.com/apache/kyuubi/pull/5875#discussion_r1430911091
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/events/KyuubiOperationEvent.scala:
##########
@@ -60,7 +61,8 @@ case class KyuubiOperationEvent private (
sessionUser: String,
sessionType: String,
kyuubiInstance: String,
- metrics: Map[String, String]) extends KyuubiEvent {
+ metrics: Map[String, String],
+ progress: OperationProgress) extends KyuubiEvent {
Review Comment:
Let me try to use the dto. KyuubiOperationEvent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5873] Support to get operation progress with RESTful API [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei closed pull request #5875: [KYUUBI #5873] Support to get operation progress with RESTful API
URL: https://github.com/apache/kyuubi/pull/5875
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org