You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "terana (via GitHub)" <gi...@apache.org> on 2026/01/29 14:04:35 UTC

[PR] [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring [spark]

terana opened a new pull request, #54062:
URL: https://github.com/apache/spark/pull/54062

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
     8. If you want to add or modify an error type or message, please read the guideline first in
        'common/utils/src/main/resources/error/README.md'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   Add GetStatus proto to support execution status monitoring
   
   Initially, this rpc will provide operations statuses. It's designed to be extensible in two ways:
   - Requesting different status types (operation-level, session-level, some specific stats).
   - Requesting different information in each type.
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   This API will allow users to monitor status (e.g. RUNNING/TERMINATING/TERMINATED) of the operations in their session. Particularly useful for multiple-thread sessions, where one "monitoring" thread has to know operation statuses from the whole session.
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
   
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   Yes. 
   Previously, a thread that didn't initiate an execution wouldn't know about its status. This was a challenge for multi-threaded sessions, since monitoring thread wouldn't know about other thread's execution statuses. The new API allows for such monitoring.
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
   -->
   
   Only proto so far. Handler and tests will be added separately.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   <!--
   If generative AI tooling has been used in the process of authoring this patch, please include the
   phrase: 'Generated-by: ' followed by the name of the tool and its version.
   If no, write 'No'.
   Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
   -->
   
   Yes.
   Generated-by: Claude 4.5 Opus
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring [spark]

Posted by "terana (via GitHub)" <gi...@apache.org>.
terana commented on code in PR #54062:
URL: https://github.com/apache/spark/pull/54062#discussion_r2742748790


##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -1213,11 +1213,105 @@ message CloneSessionResponse {
 
   // Session id of the new cloned session.
   string new_session_id = 3;
-  
+
   // Server-side session ID of the new cloned session.
   string new_server_side_session_id = 4;
 }
 
+// Next ID: 6
+message GetStatusRequest {
+  // (Required)
+  //
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // (Required)
+  //
+  // user_context.user_id and session_id both identify a unique remote spark session on the
+  // server side.
+  UserContext user_context = 2;
+
+  // (Optional)
+  //
+  // Provides optional information about the client sending the request. This field
+  // can be used for language or version specific information and is only intended for
+  // logging purposes and will not be interpreted by the server.
+  optional string client_type = 3;
+
+  // (Optional)
+  //
+  // Server-side generated idempotency key from the previous responses (if any). Server
+  // can use this to validate that the server side session has not changed.
+  optional string client_observed_server_side_session_id = 4;
+
+  // The type of status being requested.
+  // Implies mutually exlusive status types, e.g. session-level, operation-level, other custom
+  // stats.
+  oneof status_type {
+    // Get status of operations in the session.
+    OperationStatusRequest operation_status = 5;
+
+    // Extension point for custom status request types.
+    google.protobuf.Any extension = 999;
+  }
+
+  // Request to get status of operations.
+  message OperationStatusRequest {
+    // Get status of operations with these operation_ids.
+    // If unset or empty, returns status of all operations in the session.
+    repeated string operation_ids = 1;
+
+    // Extension point for custom operation-level status requests.
+    repeated google.protobuf.Any extensions = 999;
+  }
+}
+
+// Next ID: 4
+message GetStatusResponse {
+  // Session id of the session for which the status was requested.
+  string session_id = 1;
+
+  // Server-side generated idempotency key that the client can use to assert that the server side
+  // session has not changed.
+  string server_side_session_id = 2;
+
+  // The response type corresponding to the request.
+  oneof result {
+    // Status information about requested operations.
+    OperationStatusResult operation_status = 3;
+
+    // Extension point for custom status response types.
+    google.protobuf.Any extension = 999;
+  }
+
+  // Result containing status of operations.
+  message OperationStatusResult {
+    // Status information about requested operations.
+    repeated OperationStatus operation_statuses = 1;
+  }
+
+  // Status information for a single operation.
+  message OperationStatus {
+    // The operation_id of the operation.
+    string operation_id = 1;
+
+    // The current status of the operation.
+    OperationState status = 2;
+
+    enum OperationState {
+      OPERATION_STATE_UNSPECIFIED = 0;
+      OPERATION_STATE_UNKNOWN = 1;
+      OPERATION_STATE_RUNNING = 2;
+      OPERATION_STATE_TERMINATING = 3;
+      OPERATION_STATE_TERMINATED = 4;

Review Comment:
   with abandoned operations counted as cancelled? fine with me, will add. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring [spark]

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #54062:
URL: https://github.com/apache/spark/pull/54062#discussion_r2743911426


##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -1213,11 +1213,105 @@ message CloneSessionResponse {
 
   // Session id of the new cloned session.
   string new_session_id = 3;
-  
+
   // Server-side session ID of the new cloned session.
   string new_server_side_session_id = 4;
 }
 
+// Next ID: 6
+message GetStatusRequest {
+  // (Required)
+  //
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // (Required)
+  //
+  // user_context.user_id and session_id both identify a unique remote spark session on the
+  // server side.
+  UserContext user_context = 2;
+
+  // (Optional)
+  //
+  // Provides optional information about the client sending the request. This field
+  // can be used for language or version specific information and is only intended for
+  // logging purposes and will not be interpreted by the server.
+  optional string client_type = 3;
+
+  // (Optional)
+  //
+  // Server-side generated idempotency key from the previous responses (if any). Server
+  // can use this to validate that the server side session has not changed.
+  optional string client_observed_server_side_session_id = 4;
+
+  // The type of status being requested.
+  // Implies mutually exlusive status types, e.g. session-level, operation-level, other custom
+  // stats.
+  oneof status_type {

Review Comment:
   IMHO, there is no need to guarantee consistency by using oneOf to limit the client to reading only one type of data per request. The important thing to consider is how to let the server side serve these requests while providing a consistent snapshot for reading. That said, a hybrid approach is more flexible because it is more expressive.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring [spark]

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on PR #54062:
URL: https://github.com/apache/spark/pull/54062#issuecomment-3819385887

   If I understand correctly, simply adding a protobuf file doesn’t change the behavior, so the answer to “Does this PR introduce any user-facing change?” should be no, right?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring [spark]

Posted by "asf-gitbox-commits (via GitHub)" <gi...@apache.org>.
asf-gitbox-commits closed pull request #54062: [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring
URL: https://github.com/apache/spark/pull/54062


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring [spark]

Posted by "terana (via GitHub)" <gi...@apache.org>.
terana commented on code in PR #54062:
URL: https://github.com/apache/spark/pull/54062#discussion_r2745486077


##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -1213,11 +1213,105 @@ message CloneSessionResponse {
 
   // Session id of the new cloned session.
   string new_session_id = 3;
-  
+
   // Server-side session ID of the new cloned session.
   string new_server_side_session_id = 4;
 }
 
+// Next ID: 6
+message GetStatusRequest {
+  // (Required)
+  //
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // (Required)
+  //
+  // user_context.user_id and session_id both identify a unique remote spark session on the
+  // server side.
+  UserContext user_context = 2;
+
+  // (Optional)
+  //
+  // Provides optional information about the client sending the request. This field
+  // can be used for language or version specific information and is only intended for
+  // logging purposes and will not be interpreted by the server.
+  optional string client_type = 3;
+
+  // (Optional)
+  //
+  // Server-side generated idempotency key from the previous responses (if any). Server
+  // can use this to validate that the server side session has not changed.
+  optional string client_observed_server_side_session_id = 4;
+
+  // The type of status being requested.
+  // Implies mutually exlusive status types, e.g. session-level, operation-level, other custom
+  // stats.
+  oneof status_type {
+    // Get status of operations in the session.
+    OperationStatusRequest operation_status = 5;
+
+    // Extension point for custom status request types.
+    google.protobuf.Any extension = 999;
+  }
+
+  // Request to get status of operations.
+  message OperationStatusRequest {
+    // Get status of operations with these operation_ids.
+    // If unset or empty, returns status of all operations in the session.
+    repeated string operation_ids = 1;
+
+    // Extension point for custom operation-level status requests.
+    repeated google.protobuf.Any extensions = 999;
+  }
+}
+
+// Next ID: 4
+message GetStatusResponse {
+  // Session id of the session for which the status was requested.
+  string session_id = 1;
+
+  // Server-side generated idempotency key that the client can use to assert that the server side
+  // session has not changed.
+  string server_side_session_id = 2;
+
+  // The response type corresponding to the request.
+  oneof result {
+    // Status information about requested operations.
+    OperationStatusResult operation_status = 3;
+
+    // Extension point for custom status response types.
+    google.protobuf.Any extension = 999;
+  }
+
+  // Result containing status of operations.
+  message OperationStatusResult {
+    // Status information about requested operations.
+    repeated OperationStatus operation_statuses = 1;
+  }
+
+  // Status information for a single operation.
+  message OperationStatus {
+    // The operation_id of the operation.
+    string operation_id = 1;
+
+    // The current status of the operation.
+    OperationState status = 2;
+
+    enum OperationState {
+      OPERATION_STATE_UNSPECIFIED = 0;
+      OPERATION_STATE_UNKNOWN = 1;
+      OPERATION_STATE_RUNNING = 2;
+      OPERATION_STATE_TERMINATING = 3;
+      OPERATION_STATE_TERMINATED = 4;

Review Comment:
   Done.



##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -1272,4 +1366,6 @@ service SparkConnectService {
   // The request can optionally specify a custom session ID for the cloned session (must be
   // a valid UUID). If not provided, a new UUID will be generated automatically.
   rpc CloneSession(CloneSessionRequest) returns (CloneSessionResponse) {}
+  // Get status information of different types.

Review Comment:
   fixed



##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -1213,11 +1213,105 @@ message CloneSessionResponse {
 
   // Session id of the new cloned session.
   string new_session_id = 3;
-  
+
   // Server-side session ID of the new cloned session.
   string new_server_side_session_id = 4;
 }
 
+// Next ID: 6
+message GetStatusRequest {
+  // (Required)
+  //
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // (Required)
+  //
+  // user_context.user_id and session_id both identify a unique remote spark session on the
+  // server side.
+  UserContext user_context = 2;
+
+  // (Optional)
+  //
+  // Provides optional information about the client sending the request. This field
+  // can be used for language or version specific information and is only intended for
+  // logging purposes and will not be interpreted by the server.
+  optional string client_type = 3;
+
+  // (Optional)
+  //
+  // Server-side generated idempotency key from the previous responses (if any). Server
+  // can use this to validate that the server side session has not changed.
+  optional string client_observed_server_side_session_id = 4;
+
+  // The type of status being requested.
+  // Implies mutually exlusive status types, e.g. session-level, operation-level, other custom
+  // stats.
+  oneof status_type {

Review Comment:
   Moved to top level



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring [spark]

Posted by "terana (via GitHub)" <gi...@apache.org>.
terana commented on code in PR #54062:
URL: https://github.com/apache/spark/pull/54062#discussion_r2742722822


##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -1213,11 +1213,105 @@ message CloneSessionResponse {
 
   // Session id of the new cloned session.
   string new_session_id = 3;
-  
+
   // Server-side session ID of the new cloned session.
   string new_server_side_session_id = 4;
 }
 
+// Next ID: 6
+message GetStatusRequest {
+  // (Required)
+  //
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // (Required)
+  //
+  // user_context.user_id and session_id both identify a unique remote spark session on the
+  // server side.
+  UserContext user_context = 2;
+
+  // (Optional)
+  //
+  // Provides optional information about the client sending the request. This field
+  // can be used for language or version specific information and is only intended for
+  // logging purposes and will not be interpreted by the server.
+  optional string client_type = 3;
+
+  // (Optional)
+  //
+  // Server-side generated idempotency key from the previous responses (if any). Server
+  // can use this to validate that the server side session has not changed.
+  optional string client_observed_server_side_session_id = 4;
+
+  // The type of status being requested.
+  // Implies mutually exlusive status types, e.g. session-level, operation-level, other custom
+  // stats.
+  oneof status_type {

Review Comment:
   The worst thing that can happen if we make it top-lvl is if request infos are overlapping, and we return inconsistent data in two response types because of a race. That's also true for two separate oneof requests, but less confusing.
   
   Otherwise, oneof is not strictly needed indeed for read-only requests, as they should not interfere much (other than being inconsistent with each other, affecting response times).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring [spark]

Posted by "terana (via GitHub)" <gi...@apache.org>.
terana commented on code in PR #54062:
URL: https://github.com/apache/spark/pull/54062#discussion_r2742777172


##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -1213,11 +1213,105 @@ message CloneSessionResponse {
 
   // Session id of the new cloned session.
   string new_session_id = 3;
-  
+
   // Server-side session ID of the new cloned session.
   string new_server_side_session_id = 4;
 }
 
+// Next ID: 6
+message GetStatusRequest {
+  // (Required)
+  //
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // (Required)
+  //
+  // user_context.user_id and session_id both identify a unique remote spark session on the
+  // server side.
+  UserContext user_context = 2;
+
+  // (Optional)
+  //
+  // Provides optional information about the client sending the request. This field
+  // can be used for language or version specific information and is only intended for
+  // logging purposes and will not be interpreted by the server.
+  optional string client_type = 3;
+
+  // (Optional)
+  //
+  // Server-side generated idempotency key from the previous responses (if any). Server
+  // can use this to validate that the server side session has not changed.
+  optional string client_observed_server_side_session_id = 4;
+
+  // The type of status being requested.
+  // Implies mutually exlusive status types, e.g. session-level, operation-level, other custom
+  // stats.
+  oneof status_type {

Review Comment:
   @hvanhovell 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring [spark]

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #54062:
URL: https://github.com/apache/spark/pull/54062#discussion_r2743911426


##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -1213,11 +1213,105 @@ message CloneSessionResponse {
 
   // Session id of the new cloned session.
   string new_session_id = 3;
-  
+
   // Server-side session ID of the new cloned session.
   string new_server_side_session_id = 4;
 }
 
+// Next ID: 6
+message GetStatusRequest {
+  // (Required)
+  //
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // (Required)
+  //
+  // user_context.user_id and session_id both identify a unique remote spark session on the
+  // server side.
+  UserContext user_context = 2;
+
+  // (Optional)
+  //
+  // Provides optional information about the client sending the request. This field
+  // can be used for language or version specific information and is only intended for
+  // logging purposes and will not be interpreted by the server.
+  optional string client_type = 3;
+
+  // (Optional)
+  //
+  // Server-side generated idempotency key from the previous responses (if any). Server
+  // can use this to validate that the server side session has not changed.
+  optional string client_observed_server_side_session_id = 4;
+
+  // The type of status being requested.
+  // Implies mutually exlusive status types, e.g. session-level, operation-level, other custom
+  // stats.
+  oneof status_type {

Review Comment:
   IMHO, there is no need to guarantee consistency by using oneOf to limit the client to reading only one type of data per request. The important thing to consider is how to have the server handle these requests using a consistent snapshot for reading. That said, a hybrid approach is more flexible because it is more expressive and makes it easier to guarantee consistency if the server supports it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring [spark]

Posted by "vicennial (via GitHub)" <gi...@apache.org>.
vicennial commented on code in PR #54062:
URL: https://github.com/apache/spark/pull/54062#discussion_r2742446045


##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -1213,11 +1213,105 @@ message CloneSessionResponse {
 
   // Session id of the new cloned session.
   string new_session_id = 3;
-  
+
   // Server-side session ID of the new cloned session.
   string new_server_side_session_id = 4;
 }
 
+// Next ID: 6
+message GetStatusRequest {
+  // (Required)
+  //
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // (Required)
+  //
+  // user_context.user_id and session_id both identify a unique remote spark session on the
+  // server side.
+  UserContext user_context = 2;
+
+  // (Optional)
+  //
+  // Provides optional information about the client sending the request. This field
+  // can be used for language or version specific information and is only intended for
+  // logging purposes and will not be interpreted by the server.
+  optional string client_type = 3;
+
+  // (Optional)
+  //
+  // Server-side generated idempotency key from the previous responses (if any). Server
+  // can use this to validate that the server side session has not changed.
+  optional string client_observed_server_side_session_id = 4;
+
+  // The type of status being requested.
+  // Implies mutually exlusive status types, e.g. session-level, operation-level, other custom
+  // stats.
+  oneof status_type {
+    // Get status of operations in the session.
+    OperationStatusRequest operation_status = 5;
+
+    // Extension point for custom status request types.
+    google.protobuf.Any extension = 999;
+  }
+
+  // Request to get status of operations.
+  message OperationStatusRequest {
+    // Get status of operations with these operation_ids.
+    // If unset or empty, returns status of all operations in the session.
+    repeated string operation_ids = 1;
+
+    // Extension point for custom operation-level status requests.
+    repeated google.protobuf.Any extensions = 999;
+  }
+}
+
+// Next ID: 4
+message GetStatusResponse {
+  // Session id of the session for which the status was requested.
+  string session_id = 1;
+
+  // Server-side generated idempotency key that the client can use to assert that the server side
+  // session has not changed.
+  string server_side_session_id = 2;
+
+  // The response type corresponding to the request.
+  oneof result {
+    // Status information about requested operations.
+    OperationStatusResult operation_status = 3;
+
+    // Extension point for custom status response types.
+    google.protobuf.Any extension = 999;
+  }
+
+  // Result containing status of operations.
+  message OperationStatusResult {
+    // Status information about requested operations.
+    repeated OperationStatus operation_statuses = 1;
+  }
+
+  // Status information for a single operation.
+  message OperationStatus {
+    // The operation_id of the operation.
+    string operation_id = 1;
+
+    // The current status of the operation.
+    OperationState status = 2;
+
+    enum OperationState {
+      OPERATION_STATE_UNSPECIFIED = 0;
+      OPERATION_STATE_UNKNOWN = 1;
+      OPERATION_STATE_RUNNING = 2;
+      OPERATION_STATE_TERMINATING = 3;
+      OPERATION_STATE_TERMINATED = 4;

Review Comment:
   Should we also include cause of termination? (Successfull/Failed/Cancelled)



##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -1272,4 +1366,6 @@ service SparkConnectService {
   // The request can optionally specify a custom session ID for the cloned session (must be
   // a valid UUID). If not provided, a new UUID will be generated automatically.
   rpc CloneSession(CloneSessionRequest) returns (CloneSessionResponse) {}
+  // Get status information of different types.

Review Comment:
   Nit: add empty line above



##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -1213,11 +1213,105 @@ message CloneSessionResponse {
 
   // Session id of the new cloned session.
   string new_session_id = 3;
-  
+
   // Server-side session ID of the new cloned session.
   string new_server_side_session_id = 4;
 }
 
+// Next ID: 6
+message GetStatusRequest {
+  // (Required)
+  //
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // (Required)
+  //
+  // user_context.user_id and session_id both identify a unique remote spark session on the
+  // server side.
+  UserContext user_context = 2;
+
+  // (Optional)
+  //
+  // Provides optional information about the client sending the request. This field
+  // can be used for language or version specific information and is only intended for
+  // logging purposes and will not be interpreted by the server.
+  optional string client_type = 3;
+
+  // (Optional)
+  //
+  // Server-side generated idempotency key from the previous responses (if any). Server
+  // can use this to validate that the server side session has not changed.
+  optional string client_observed_server_side_session_id = 4;
+
+  // The type of status being requested.
+  // Implies mutually exlusive status types, e.g. session-level, operation-level, other custom
+  // stats.
+  oneof status_type {

Review Comment:
   I've been thinking about this and I'm conlficted between a top-level `oneof` here vs a hybrid:
   ```protobuf
   repeated StatusTypeRequest status_types = ...
   message StatusTypeRequest {
     oneof status_type {
       ...
     }
   }
   
   The top-level oneof is simple and follows the principle of YAGNI but with the context of this RPC being used for monitoring/polling, serving multiple status types in a single round-trip would help avoid complexity on the client side where it may need to look at several different points of information to proceed.
   
   @hvanhovell WDYT? 



##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -1213,11 +1213,105 @@ message CloneSessionResponse {
 
   // Session id of the new cloned session.
   string new_session_id = 3;
-  
+
   // Server-side session ID of the new cloned session.
   string new_server_side_session_id = 4;
 }
 
+// Next ID: 6
+message GetStatusRequest {
+  // (Required)
+  //
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // (Required)
+  //
+  // user_context.user_id and session_id both identify a unique remote spark session on the
+  // server side.
+  UserContext user_context = 2;
+
+  // (Optional)
+  //
+  // Provides optional information about the client sending the request. This field
+  // can be used for language or version specific information and is only intended for
+  // logging purposes and will not be interpreted by the server.
+  optional string client_type = 3;
+
+  // (Optional)
+  //
+  // Server-side generated idempotency key from the previous responses (if any). Server
+  // can use this to validate that the server side session has not changed.
+  optional string client_observed_server_side_session_id = 4;
+
+  // The type of status being requested.
+  // Implies mutually exlusive status types, e.g. session-level, operation-level, other custom

Review Comment:
   Nit: Typo
   ```suggestion
     // Implies mutually exclusive status types, e.g. session-level, operation-level, other custom
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring [spark]

Posted by "terana (via GitHub)" <gi...@apache.org>.
terana commented on code in PR #54062:
URL: https://github.com/apache/spark/pull/54062#discussion_r2742748790


##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -1213,11 +1213,105 @@ message CloneSessionResponse {
 
   // Session id of the new cloned session.
   string new_session_id = 3;
-  
+
   // Server-side session ID of the new cloned session.
   string new_server_side_session_id = 4;
 }
 
+// Next ID: 6
+message GetStatusRequest {
+  // (Required)
+  //
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // (Required)
+  //
+  // user_context.user_id and session_id both identify a unique remote spark session on the
+  // server side.
+  UserContext user_context = 2;
+
+  // (Optional)
+  //
+  // Provides optional information about the client sending the request. This field
+  // can be used for language or version specific information and is only intended for
+  // logging purposes and will not be interpreted by the server.
+  optional string client_type = 3;
+
+  // (Optional)
+  //
+  // Server-side generated idempotency key from the previous responses (if any). Server
+  // can use this to validate that the server side session has not changed.
+  optional string client_observed_server_side_session_id = 4;
+
+  // The type of status being requested.
+  // Implies mutually exlusive status types, e.g. session-level, operation-level, other custom
+  // stats.
+  oneof status_type {
+    // Get status of operations in the session.
+    OperationStatusRequest operation_status = 5;
+
+    // Extension point for custom status request types.
+    google.protobuf.Any extension = 999;
+  }
+
+  // Request to get status of operations.
+  message OperationStatusRequest {
+    // Get status of operations with these operation_ids.
+    // If unset or empty, returns status of all operations in the session.
+    repeated string operation_ids = 1;
+
+    // Extension point for custom operation-level status requests.
+    repeated google.protobuf.Any extensions = 999;
+  }
+}
+
+// Next ID: 4
+message GetStatusResponse {
+  // Session id of the session for which the status was requested.
+  string session_id = 1;
+
+  // Server-side generated idempotency key that the client can use to assert that the server side
+  // session has not changed.
+  string server_side_session_id = 2;
+
+  // The response type corresponding to the request.
+  oneof result {
+    // Status information about requested operations.
+    OperationStatusResult operation_status = 3;
+
+    // Extension point for custom status response types.
+    google.protobuf.Any extension = 999;
+  }
+
+  // Result containing status of operations.
+  message OperationStatusResult {
+    // Status information about requested operations.
+    repeated OperationStatus operation_statuses = 1;
+  }
+
+  // Status information for a single operation.
+  message OperationStatus {
+    // The operation_id of the operation.
+    string operation_id = 1;
+
+    // The current status of the operation.
+    OperationState status = 2;
+
+    enum OperationState {
+      OPERATION_STATE_UNSPECIFIED = 0;
+      OPERATION_STATE_UNKNOWN = 1;
+      OPERATION_STATE_RUNNING = 2;
+      OPERATION_STATE_TERMINATING = 3;
+      OPERATION_STATE_TERMINATED = 4;

Review Comment:
   With abandoned operations counted as cancelled? Sounds good, will add. Would stick closer to ExecuteStatus though and do FINISHED, FAILED, CANCELLED. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring [spark]

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #54062:
URL: https://github.com/apache/spark/pull/54062#discussion_r2743911426


##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -1213,11 +1213,105 @@ message CloneSessionResponse {
 
   // Session id of the new cloned session.
   string new_session_id = 3;
-  
+
   // Server-side session ID of the new cloned session.
   string new_server_side_session_id = 4;
 }
 
+// Next ID: 6
+message GetStatusRequest {
+  // (Required)
+  //
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // (Required)
+  //
+  // user_context.user_id and session_id both identify a unique remote spark session on the
+  // server side.
+  UserContext user_context = 2;
+
+  // (Optional)
+  //
+  // Provides optional information about the client sending the request. This field
+  // can be used for language or version specific information and is only intended for
+  // logging purposes and will not be interpreted by the server.
+  optional string client_type = 3;
+
+  // (Optional)
+  //
+  // Server-side generated idempotency key from the previous responses (if any). Server
+  // can use this to validate that the server side session has not changed.
+  optional string client_observed_server_side_session_id = 4;
+
+  // The type of status being requested.
+  // Implies mutually exlusive status types, e.g. session-level, operation-level, other custom
+  // stats.
+  oneof status_type {

Review Comment:
   IMHO, there is no need to guarantee consistency by using oneOf to limit the client to reading only one type of data in a request, while still being able to provide snapshot-like data on the server side. That said, a hybrid approach is more flexible because it is more expressive.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring [spark]

Posted by "vicennial (via GitHub)" <gi...@apache.org>.
vicennial commented on code in PR #54062:
URL: https://github.com/apache/spark/pull/54062#discussion_r2742553529


##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -1213,11 +1213,105 @@ message CloneSessionResponse {
 
   // Session id of the new cloned session.
   string new_session_id = 3;
-  
+
   // Server-side session ID of the new cloned session.
   string new_server_side_session_id = 4;
 }
 
+// Next ID: 6
+message GetStatusRequest {
+  // (Required)
+  //
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // (Required)
+  //
+  // user_context.user_id and session_id both identify a unique remote spark session on the
+  // server side.
+  UserContext user_context = 2;
+
+  // (Optional)
+  //
+  // Provides optional information about the client sending the request. This field
+  // can be used for language or version specific information and is only intended for
+  // logging purposes and will not be interpreted by the server.
+  optional string client_type = 3;
+
+  // (Optional)
+  //
+  // Server-side generated idempotency key from the previous responses (if any). Server
+  // can use this to validate that the server side session has not changed.
+  optional string client_observed_server_side_session_id = 4;
+
+  // The type of status being requested.
+  // Implies mutually exlusive status types, e.g. session-level, operation-level, other custom
+  // stats.
+  oneof status_type {

Review Comment:
   I've been thinking about this and I'm conlficted between a top-level `oneof` here vs a hybrid:
   ```protobuf
   repeated StatusTypeRequest status_types = ...
   message StatusTypeRequest {
     oneof status_type {
       ...
     }
   }
   ```
   
   The top-level oneof is simple and follows the principle of YAGNI but with the context of this RPC being used for monitoring/polling, serving multiple status types in a single round-trip would help avoid complexity on the client side where it may need to look at several different points of information to proceed.
   
   @hvanhovell WDYT? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring [spark]

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #54062:
URL: https://github.com/apache/spark/pull/54062#discussion_r2743911426


##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -1213,11 +1213,105 @@ message CloneSessionResponse {
 
   // Session id of the new cloned session.
   string new_session_id = 3;
-  
+
   // Server-side session ID of the new cloned session.
   string new_server_side_session_id = 4;
 }
 
+// Next ID: 6
+message GetStatusRequest {
+  // (Required)
+  //
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // (Required)
+  //
+  // user_context.user_id and session_id both identify a unique remote spark session on the
+  // server side.
+  UserContext user_context = 2;
+
+  // (Optional)
+  //
+  // Provides optional information about the client sending the request. This field
+  // can be used for language or version specific information and is only intended for
+  // logging purposes and will not be interpreted by the server.
+  optional string client_type = 3;
+
+  // (Optional)
+  //
+  // Server-side generated idempotency key from the previous responses (if any). Server
+  // can use this to validate that the server side session has not changed.
+  optional string client_observed_server_side_session_id = 4;
+
+  // The type of status being requested.
+  // Implies mutually exlusive status types, e.g. session-level, operation-level, other custom
+  // stats.
+  oneof status_type {

Review Comment:
   IMHO, there is no need to guarantee consistency by using oneOf to limit the client to reading only one type of data per request. The important thing to consider is how to let the server side serve these requests while providing a consistent snapshot for reading. That said, a hybrid approach is more flexible because it is more expressive and makes it easier to guarantee consistency if the server supports it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #54062:
URL: https://github.com/apache/spark/pull/54062#issuecomment-3817896891

   ## JIRA Issue Information
   
   === New Feature SPARK-55280 ===
   Summary: [CONNECT] Add support for monitoring execution status
   Assignee: None
   Status: Open
   Affected: ["4.2"]
   
   ---
   *This comment was automatically generated by GitHub Actions*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring [spark]

Posted by "terana (via GitHub)" <gi...@apache.org>.
terana commented on code in PR #54062:
URL: https://github.com/apache/spark/pull/54062#discussion_r2742748790


##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -1213,11 +1213,105 @@ message CloneSessionResponse {
 
   // Session id of the new cloned session.
   string new_session_id = 3;
-  
+
   // Server-side session ID of the new cloned session.
   string new_server_side_session_id = 4;
 }
 
+// Next ID: 6
+message GetStatusRequest {
+  // (Required)
+  //
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // (Required)
+  //
+  // user_context.user_id and session_id both identify a unique remote spark session on the
+  // server side.
+  UserContext user_context = 2;
+
+  // (Optional)
+  //
+  // Provides optional information about the client sending the request. This field
+  // can be used for language or version specific information and is only intended for
+  // logging purposes and will not be interpreted by the server.
+  optional string client_type = 3;
+
+  // (Optional)
+  //
+  // Server-side generated idempotency key from the previous responses (if any). Server
+  // can use this to validate that the server side session has not changed.
+  optional string client_observed_server_side_session_id = 4;
+
+  // The type of status being requested.
+  // Implies mutually exlusive status types, e.g. session-level, operation-level, other custom
+  // stats.
+  oneof status_type {
+    // Get status of operations in the session.
+    OperationStatusRequest operation_status = 5;
+
+    // Extension point for custom status request types.
+    google.protobuf.Any extension = 999;
+  }
+
+  // Request to get status of operations.
+  message OperationStatusRequest {
+    // Get status of operations with these operation_ids.
+    // If unset or empty, returns status of all operations in the session.
+    repeated string operation_ids = 1;
+
+    // Extension point for custom operation-level status requests.
+    repeated google.protobuf.Any extensions = 999;
+  }
+}
+
+// Next ID: 4
+message GetStatusResponse {
+  // Session id of the session for which the status was requested.
+  string session_id = 1;
+
+  // Server-side generated idempotency key that the client can use to assert that the server side
+  // session has not changed.
+  string server_side_session_id = 2;
+
+  // The response type corresponding to the request.
+  oneof result {
+    // Status information about requested operations.
+    OperationStatusResult operation_status = 3;
+
+    // Extension point for custom status response types.
+    google.protobuf.Any extension = 999;
+  }
+
+  // Result containing status of operations.
+  message OperationStatusResult {
+    // Status information about requested operations.
+    repeated OperationStatus operation_statuses = 1;
+  }
+
+  // Status information for a single operation.
+  message OperationStatus {
+    // The operation_id of the operation.
+    string operation_id = 1;
+
+    // The current status of the operation.
+    OperationState status = 2;
+
+    enum OperationState {
+      OPERATION_STATE_UNSPECIFIED = 0;
+      OPERATION_STATE_UNKNOWN = 1;
+      OPERATION_STATE_RUNNING = 2;
+      OPERATION_STATE_TERMINATING = 3;
+      OPERATION_STATE_TERMINATED = 4;

Review Comment:
   With abandoned operations counted as cancelled? Sounds good, will add. 
   We would need to introduce tracking of the termination reason for closed operations. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org