You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "yanghua (via GitHub)" <gi...@apache.org> on 2024/04/10 06:44:39 UTC

[PR] [WIP][KYUUBI-6278] Update DB state when the query fallback to yarn an… [kyuubi]

yanghua opened a new pull request, #6284:
URL: https://github.com/apache/kyuubi/pull/6284

   …d the batch app is terminal
   
   # :mag: Description
   ## Issue References 🔗
   <!-- Append the issue number after #. If there is no issue for you to link create one or -->
   <!-- If there are no issues to link, please provide details here. -->
   
   This pull request fixes #
   
   ## Describe Your Solution 🔧
   
   Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
   
   
   ## Types of changes :bookmark:
   <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
   - [ ] Bugfix (non-breaking change which fixes an issue)
   - [ ] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing functionality to change)
   
   ## Test Plan 🧪
   
   #### Behavior Without This Pull Request :coffin:
   
   
   #### Behavior With This Pull Request :tada:
   
   
   #### Related Unit Tests
   
   
   ---
   
   # Checklist 📝
   <!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
   <!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
   
   - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
   
   **Be nice. Be informative.**
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] [KYUUBI-6278] Update DB state when the query fallback to yarn an… [kyuubi]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6284:
URL: https://github.com/apache/kyuubi/pull/6284#discussion_r1561381709


##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala:
##########
@@ -315,8 +315,14 @@ private[v1] class BatchesResource extends ApiRequestContext with Logging {
       buildBatch(batchSession)
     }.getOrElse {
       sessionManager.getBatchMetadata(batchId).map { metadata =>
+        val isOperationTerminated = (metadata.state.nonEmpty
+          && OperationState.isTerminal(OperationState.withName(metadata.state)))
+        val isApplicationTerminated = (metadata.engineState.nonEmpty

Review Comment:
   ```
   StringUtils.isNotBlank(metadata.engineState)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] [KYUUBI-6278] Update DB state when the query fallback to yarn an… [kyuubi]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6284:
URL: https://github.com/apache/kyuubi/pull/6284#discussion_r1562023677


##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala:
##########
@@ -332,6 +339,12 @@ private[v1] class BatchesResource extends ApiRequestContext with Logging {
                 Some(userName),
                 // prevent that the batch be marked as terminated if application state is NOT_FOUND
                 Some(metadata.engineOpenTime).filter(_ > 0).orElse(Some(System.currentTimeMillis)))
+              // if the batch app is terminated, update the metadata in db.
+              if (BatchJobSubmission.applicationTerminated(batchAppStatus)) {
+                sessionManager.updateMetadata(Metadata(
+                  identifier = batchId,
+                  engineState = batchAppStatus.get.state.toString))

Review Comment:
   should we update all the batchAppStatus into metadata?
   
   Including appId, name, state, error.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] [KYUUBI-6278] Update DB state when the query fallback to yarn an… [kyuubi]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #6284:
URL: https://github.com/apache/kyuubi/pull/6284#discussion_r1561013363


##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala:
##########
@@ -317,6 +317,7 @@ private[v1] class BatchesResource extends ApiRequestContext with Logging {
       sessionManager.getBatchMetadata(batchId).map { metadata =>
         if (batchV2Enabled(metadata.requestConf) ||
           OperationState.isTerminal(OperationState.withName(metadata.state)) ||
+          ApplicationState.isTerminated(ApplicationState.withName(metadata.engineState)) ||

Review Comment:
   engineState might be null



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] [WIP][KYUUBI-6278] Update DB state when the query fallback to yarn an… [kyuubi]

Posted by "yanghua (via GitHub)" <gi...@apache.org>.
yanghua commented on PR #6284:
URL: https://github.com/apache/kyuubi/pull/6284#issuecomment-2046647667

   WIP.  Regardless of whether to change the code, local tests always fail. Use the community's CI to run.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] [WIP][KYUUBI-6278] Update DB state when the query fallback to yarn an… [kyuubi]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #6284:
URL: https://github.com/apache/kyuubi/pull/6284#issuecomment-2047073496

   ## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/6284?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `0%` with `5 lines` in your changes are missing coverage. Please review.
   > Project coverage is 58.41%. Comparing base [(`f8c7b93`)](https://app.codecov.io/gh/apache/kyuubi/commit/f8c7b93f5544a4784778279e35ad5feb15b3bdbc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`a50828d`)](https://app.codecov.io/gh/apache/kyuubi/pull/6284?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 2 commits behind head on master.
   
   | [Files](https://app.codecov.io/gh/apache/kyuubi/pull/6284?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [.../apache/kyuubi/server/api/v1/BatchesResource.scala](https://app.codecov.io/gh/apache/kyuubi/pull/6284?src=pr&el=tree&filepath=kyuubi-server%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fserver%2Fapi%2Fv1%2FBatchesResource.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvYXBpL3YxL0JhdGNoZXNSZXNvdXJjZS5zY2FsYQ==) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6284?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #6284      +/-   ##
   ============================================
   - Coverage     58.45%   58.41%   -0.05%     
     Complexity       24       24              
   ============================================
     Files           650      652       +2     
     Lines         39543    39608      +65     
     Branches       5448     5452       +4     
   ============================================
   + Hits          23116    23138      +22     
   - Misses        13950    13987      +37     
   - Partials       2477     2483       +6     
   ```
   
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/6284?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] [KYUUBI #6278] Update DB state when the query fallback to resource manager and the batch app is terminal [kyuubi]

Posted by "yanghua (via GitHub)" <gi...@apache.org>.
yanghua merged PR #6284:
URL: https://github.com/apache/kyuubi/pull/6284


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] [KYUUBI-6278] Update DB state when the query fallback to resource manager and the batch app is terminal [kyuubi]

Posted by "yanghua (via GitHub)" <gi...@apache.org>.
yanghua commented on code in PR #6284:
URL: https://github.com/apache/kyuubi/pull/6284#discussion_r1565547785


##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala:
##########
@@ -332,6 +339,12 @@ private[v1] class BatchesResource extends ApiRequestContext with Logging {
                 Some(userName),
                 // prevent that the batch be marked as terminated if application state is NOT_FOUND
                 Some(metadata.engineOpenTime).filter(_ > 0).orElse(Some(System.currentTimeMillis)))
+              // if the batch app is terminated, update the metadata in db.
+              if (BatchJobSubmission.applicationTerminated(batchAppStatus)) {
+                sessionManager.updateMetadata(Metadata(
+                  identifier = batchId,
+                  engineState = batchAppStatus.get.state.toString))

Review Comment:
   +1



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org