You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2023/01/07 06:23:39 UTC
[GitHub] [kyuubi] turboFei opened a new pull request, #4119: Open time
turboFei opened a new pull request, #4119:
URL: https://github.com/apache/kyuubi/pull/4119
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/CONTRIBUTING.html
2. If the PR is related to an issue in https://github.com/apache/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] ulysses-you commented on a diff in pull request #4119: Return app submission time for batch
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #4119:
URL: https://github.com/apache/kyuubi/pull/4119#discussion_r1064264875
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala:
##########
@@ -96,7 +99,14 @@ class BatchJobSubmission(
override private[kyuubi] def currentApplicationInfo: Option[ApplicationInfo] = {
// only the ApplicationInfo with non-empty id is valid for the operation
- applicationManager.getApplicationInfo(builder.clusterManager(), batchId).filter(_.id != null)
+ val applicationInfo =
+ applicationManager.getApplicationInfo(builder.clusterManager(), batchId).filter(_.id != null)
+ applicationInfo.foreach { _ =>
+ if (_appSubmissionTime <= 0) {
+ _appSubmissionTime = System.currentTimeMillis()
Review Comment:
I think we should make state machine and the time trace consistent.
If the submitssion time means when the state changes to running, then we should update the time around this code
```
setStateIfNotCanceled(OperationState.RUNNING)
```
Otherwise the client may get the batch info with `submisstion time = 0 but state is running` or `submisstion time != 0 but state is pending`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] turboFei commented on a diff in pull request #4119: Return app submission time for batch
Posted by GitBox <gi...@apache.org>.
turboFei commented on code in PR #4119:
URL: https://github.com/apache/kyuubi/pull/4119#discussion_r1064255071
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala:
##########
@@ -96,7 +99,14 @@ class BatchJobSubmission(
override private[kyuubi] def currentApplicationInfo: Option[ApplicationInfo] = {
// only the ApplicationInfo with non-empty id is valid for the operation
- applicationManager.getApplicationInfo(builder.clusterManager(), batchId).filter(_.id != null)
+ val applicationInfo =
+ applicationManager.getApplicationInfo(builder.clusterManager(), batchId).filter(_.id != null)
+ applicationInfo.foreach { _ =>
+ if (_appSubmissionTime <= 0) {
+ _appSubmissionTime = System.currentTimeMillis()
Review Comment:
> why we donot set it in submitAndMonitorBatchJob method ?
Because in the `submitAndMonitorBatchJob` method, there is interval to track the app state.
And for the `getBatch` api, it will also call currentApplicationState to build the batch report, to keep the state align, it is better to update the app submission time in `currentApplicationState`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] turboFei commented on a diff in pull request #4119: Return app submission time for batch
Posted by GitBox <gi...@apache.org>.
turboFei commented on code in PR #4119:
URL: https://github.com/apache/kyuubi/pull/4119#discussion_r1064282495
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala:
##########
@@ -96,7 +99,14 @@ class BatchJobSubmission(
override private[kyuubi] def currentApplicationInfo: Option[ApplicationInfo] = {
// only the ApplicationInfo with non-empty id is valid for the operation
- applicationManager.getApplicationInfo(builder.clusterManager(), batchId).filter(_.id != null)
+ val applicationInfo =
+ applicationManager.getApplicationInfo(builder.clusterManager(), batchId).filter(_.id != null)
+ applicationInfo.foreach { _ =>
+ if (_appSubmissionTime <= 0) {
+ _appSubmissionTime = System.currentTimeMillis()
Review Comment:
for batch report, it is consistent with the appId/appUrl/appState now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] turboFei commented on a diff in pull request #4119: Return app submission time for batch
Posted by GitBox <gi...@apache.org>.
turboFei commented on code in PR #4119:
URL: https://github.com/apache/kyuubi/pull/4119#discussion_r1064282495
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala:
##########
@@ -96,7 +99,14 @@ class BatchJobSubmission(
override private[kyuubi] def currentApplicationInfo: Option[ApplicationInfo] = {
// only the ApplicationInfo with non-empty id is valid for the operation
- applicationManager.getApplicationInfo(builder.clusterManager(), batchId).filter(_.id != null)
+ val applicationInfo =
+ applicationManager.getApplicationInfo(builder.clusterManager(), batchId).filter(_.id != null)
+ applicationInfo.foreach { _ =>
+ if (_appSubmissionTime <= 0) {
+ _appSubmissionTime = System.currentTimeMillis()
Review Comment:
for batch report, it is consistent with the appId/appState/appState now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] turboFei commented on pull request #4119: Return app submission time for batch
Posted by GitBox <gi...@apache.org>.
turboFei commented on PR #4119:
URL: https://github.com/apache/kyuubi/pull/4119#issuecomment-1374780333
thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] ulysses-you commented on a diff in pull request #4119: Return app submission time for batch
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #4119:
URL: https://github.com/apache/kyuubi/pull/4119#discussion_r1064238143
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala:
##########
@@ -96,7 +99,14 @@ class BatchJobSubmission(
override private[kyuubi] def currentApplicationInfo: Option[ApplicationInfo] = {
// only the ApplicationInfo with non-empty id is valid for the operation
- applicationManager.getApplicationInfo(builder.clusterManager(), batchId).filter(_.id != null)
+ val applicationInfo =
+ applicationManager.getApplicationInfo(builder.clusterManager(), batchId).filter(_.id != null)
+ applicationInfo.foreach { _ =>
+ if (_appSubmissionTime <= 0) {
+ _appSubmissionTime = System.currentTimeMillis()
Review Comment:
why we donot set it in `submitAndMonitorBatchJob` method ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] turboFei commented on a diff in pull request #4119: Return app submission time for batch
Posted by GitBox <gi...@apache.org>.
turboFei commented on code in PR #4119:
URL: https://github.com/apache/kyuubi/pull/4119#discussion_r1064282948
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala:
##########
@@ -96,7 +99,14 @@ class BatchJobSubmission(
override private[kyuubi] def currentApplicationInfo: Option[ApplicationInfo] = {
// only the ApplicationInfo with non-empty id is valid for the operation
- applicationManager.getApplicationInfo(builder.clusterManager(), batchId).filter(_.id != null)
+ val applicationInfo =
+ applicationManager.getApplicationInfo(builder.clusterManager(), batchId).filter(_.id != null)
+ applicationInfo.foreach { _ =>
+ if (_appSubmissionTime <= 0) {
+ _appSubmissionTime = System.currentTimeMillis()
Review Comment:
We can treat it as a filed of batch application info.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] turboFei commented on pull request #4119: Return app submission time for batch
Posted by GitBox <gi...@apache.org>.
turboFei commented on PR #4119:
URL: https://github.com/apache/kyuubi/pull/4119#issuecomment-1374391489
also cc @lightning-L
This pr has metadata schema change, it is blocked by #4022
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] ulysses-you commented on a diff in pull request #4119: Return app submission time for batch
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #4119:
URL: https://github.com/apache/kyuubi/pull/4119#discussion_r1064237556
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala:
##########
@@ -96,7 +99,14 @@ class BatchJobSubmission(
override private[kyuubi] def currentApplicationInfo: Option[ApplicationInfo] = {
// only the ApplicationInfo with non-empty id is valid for the operation
- applicationManager.getApplicationInfo(builder.clusterManager(), batchId).filter(_.id != null)
+ val applicationInfo =
+ applicationManager.getApplicationInfo(builder.clusterManager(), batchId).filter(_.id != null)
+ applicationInfo.foreach { _ =>
+ if (_appSubmissionTime <= 0) {
+ _appSubmissionTime = System.currentTimeMillis()
Review Comment:
Seems it's the `appStartTime` ? Submisstion time is the time before submiting succeed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] turboFei commented on a diff in pull request #4119: Return app submission time for batch
Posted by GitBox <gi...@apache.org>.
turboFei commented on code in PR #4119:
URL: https://github.com/apache/kyuubi/pull/4119#discussion_r1064297750
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala:
##########
@@ -96,7 +99,14 @@ class BatchJobSubmission(
override private[kyuubi] def currentApplicationInfo: Option[ApplicationInfo] = {
// only the ApplicationInfo with non-empty id is valid for the operation
- applicationManager.getApplicationInfo(builder.clusterManager(), batchId).filter(_.id != null)
+ val applicationInfo =
+ applicationManager.getApplicationInfo(builder.clusterManager(), batchId).filter(_.id != null)
+ applicationInfo.foreach { _ =>
+ if (_appSubmissionTime <= 0) {
+ _appSubmissionTime = System.currentTimeMillis()
Review Comment:
here is the pr for followup https://github.com/apache/kyuubi/pull/4129
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] turboFei closed pull request #4119: Return app submission time for batch
Posted by GitBox <gi...@apache.org>.
turboFei closed pull request #4119: Return app submission time for batch
URL: https://github.com/apache/kyuubi/pull/4119
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org