You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ch...@apache.org on 2026/10/28 13:27:24 UTC

(kyuubi-shaded) branch master updated: [KYUUBI-SHADED #72] Sync Spark Connect pb from Spark 4.3.0 RC3 and bump gRPC 1.76.3, protobuf 3.25.9

This is an automated email from the ASF dual-hosted git repository.

pan3793 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi-shaded.git


The following commit(s) were added to refs/heads/master by this push:
     new e285e7b  [KYUUBI-SHADED #72] Sync Spark Connect pb from Spark 4.3.0 RC3 and bump gRPC 1.76.3, protobuf 3.25.9
e285e7b is described below

commit e285e7bc01a997919f413879e092c08264d9d913
Author: Cheng Pan <pa...@gmail.com>
AuthorDate: Wed Oct 28 21:27:15 2026 +0800

    [KYUUBI-SHADED #72] Sync Spark Connect pb from Spark 4.3.0 RC3 and bump gRPC 1.76.3, protobuf 3.25.9
    
    ### _Why are the changes needed?_
    
    Sync pb files from [Spark 4.3.0 RC3](https://github.com/apache/spark/tree/v4.3.0-rc3/sql/connect/common/src/main/protobuf), which adds `pipelines.proto`.
    
    Also bump gRPC from 1.65.1 to 1.76.3 and protobuf from 3.25.1 to 3.25.9.
    
    ```
    $ git log --oneline v4.0.0-rc4..v4.3.0-rc3 -- sql/connect/common/src/main/protobuf/spark/connect
    ```
    ```
    5cdb1a6e908 [SPARK-58247][CONNECT][SDP] Support SCD Type 2 in the Auto CDC Python and Spark Connect APIs
    a5dcd0fa358 [SPARK-57160][CONNECT][FOLLOWUP] Use signed nanos timestamp fields
    b60e72c9fc8 [SPARK-57160][CONNECT] Add Spark Connect protocol support for nanosecond-capable timestamp types and literals
    f30a7a0e650 [SPARK-57247][SQL][CONNECT] Support DataFrame.zip in Spark Connect
    f6312fbfc35 [SPARK-56650][SDP][CONNECT] Add AutoCDC Spark Connect APIs
    39681148eb7 [SPARK-56395][CONNECT][PYTHON] Add NEAREST BY DataFrame API
    1186f023c1d [SPARK-55689][SQL] Add withSchemaEvolution() in dataframe writer API
    190e3b857c1 [SPARK-56254][PYTHON][CONNECT] Make spark.read.xml accept DataFrame input
    a3930d31665 [SPARK-56221][SQL][PYTHON][FOLLOWUP] Rename TablePartition and remove SHOW CACHED TABLES / listCachedTables
    91d62cef365 [SPARK-56221][SQL][PYTHON] Feature parity between spark.catalog.* vs DDL commands
    58841fa416a [SPARK-55949][SQL][CONNECT] Add DataFrame API and Spark Connect support for CDC queries
    bff9dcfbfb8 [SPARK-55945][SDP] Support structured identifiers for flows in SDP eager analysis protos
    7da1356837e [SPARK-55264] Add ExecuteOutput command to Spark Connect pipelines proto
    7de85bad591 [SPARK-55314][CONNECT] Propagate observed metrics errors to client
    a263a5e4ce8 [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring
    d069f733e95 [SPARK-55104][CONNECT][PYTHON][SS] Add Spark Connect support for DataStreamReader.name()
    7009fb58a98 [SPARK-54660][SS] Add RTM trigger to python
    cc72c647b6f [SPARK-54020] Support `spark.sql(...)` Python API inside query functions for Spark Declarative Pipeline
    0ccaacf54ce [SPARK-54194][CONNECT] Spark Connect Proto Plan Compression
    1a724ba6fea [SPARK-54191][SDP] Add once to Defineflow Proto
    a927a14a59e [SPARK-52463][SDP] Add support for cluster_by in Python Pipelines APIs
    32060bf99fc [SPARK-54175][GEO][CONNECT] Add Geography and Geometry types to Spark Connect proto
    4622e8584e3 [SPARK-52510][GEO][CONNECT][FOLLOWUP] Reserve expression proto numbers for Geometry and Geography
    c707f597e9c Add PipelineAnalysisContext message to support pipeline analysis during Spark Connect query execution
    db81309dd80 [SPARK-53917][CONNECT] Support large local relations
    264ca4dc320 [SPARK-53455][CONNECT] Add `CloneSession` RPC
    418cf56ed4d [SPARK-53796][SDP] Add `extension` field to a few pipeline protos to support forward compatibility
    efcc8f69a01 [SPARK-53850][SDP] Define proto for Sinks and Rename DefineDataset to DefineOutput
    22e24dfd7e8 [SPARK-53812][SDP] Refactor DefineDataset and DefineFlow protos to group related properties and future-proof
    9a1c742098c [SPARK-53751][SDP] Explicit Versioned Checkpoint Location
    be1de5b5733 [SPARK-53802][SDP] Support string values for user-specified schema in SDP tables
    65ff85a31fe [SPARK-52640][SDP] Propagate Python Source Code Location
    e04fd595370 [SPARK-52807][SDP] Proto changes to support analysis inside Declarative Pipelines query functions
    e56ab2fdaea [SPARK-53593][SDP] Add response field for DefineDataset and DefineFlow RPC
    db13a38e565 [SPARK-53578][CONNECT] Simplify data type handling in LiteralValueProtoConverter
    010d36f2194 [SPARK-53507][CONNECT] Add breaking change info to errors
    bb41e19ae9e [SPARK-53525][CONNECT] Spark Connect ArrowBatch Result Chunking
    e75ca577923 [SPARK-53402][CONNECT] Support Direct Passthrough Partitioning Dataset API in Spark Connect in Scala
    1d09e5a470f [SPARK-52449][CONNECT][PYTHON][ML] Make datatypes for Expression.Literal.Map/Array optional
    b5840e15603 [SPARK-52930][CONNECT] Use DataType.Array/Map for Array/Map Literals
    12c87cef0d0 [SPARK-53328][ML][CONNECT] Improve debuggability for SparkML-connect
    39fbf594fa7 [SPARK-52448][CONNECT] Add simplified Struct Expression.Literal
    4a450545754 [SPARK-52851][SDP] Remove public APIs for append once flows
    93748ccc597 [SPARK-52511][SDP] Support dry-run mode in spark-pipelines command
    9204b055894 [SPARK--52810][SDP][SQL] Spark Pipelines CLI Selection Options
    fac4433f239 [SPARK-52779][SQL][CONNECT] Support TimeType literal in Connect
    c84629ed173 [SPARK-52770][CONNECT] Support `TIME` type in connect proto
    7208875483d [SPARK-52757][CONNECT] Rename "plan" field in DefineFlow to "relation"
    73ca5e83d20 [SPARK-52510] Reserve proto numbers for Geospatial types
    fd74b5ec6bd [SPARK-52470][ML][CONNECT] Support model summary offloading
    46de57b5511 [SPARK-52423][CONNECT] Add missing golang package name to pipelines.proto
    8c3194fbb8d [SPARK-52348][CONNECT] Add support for Spark Connect handlers for pipeline commands
    da68d10206d [SPARK-52223][CONNECT] Add SDP Spark Connect Protos
    b50690a89a8 [SPARK-52057][ML][CONNECT] Collect Tree size limit warning messages to client
    b557311d439 [SPARK-51845][ML][CONNECT] Add proto messages `CleanCache` and `GetCacheInfo`
    b4faf063e37 [SPARK-50131][SQL] Add IN Subquery DataFrame API
    c83b88a037e [SPARK-51650][ML][CONNECT] Support delete ml cached objects in batch
    89e05a88deb [SPARK-51326][CONNECT] Remove LazyExpression proto message
    93d10c25d5b [SPARK-50856][SS][PYTHON][CONNECT] Spark Connect Support for TransformWithStateInPandas In Python
    6a71f76f8aa [SPARK-51142][ML][CONNECT] ML protobufs clean up
    4953a9cfabe [SPARK-49308][CONNECT] Support UserDefinedAggregateFunction in Spark Connect Scala Client
    ebcec7c02f3 [SPARK-50104][CONNECT] Support SparkSession.executeCommand in Connect
    e037cb119a2 [SPARK-50133][PYTHON][CONNECT] Support DataFrame conversion to table argument in Spark Connect Python Client
    efeb1e01ce4 [SPARK-50893][CONNECT] Mark UDT.DataType optional
    c7df014aab1 [SPARK-49273][CONNECT][SQL] Origin support for Spark Connect Scala client
    ef363b636b2 [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet
    2721a50234b [SPARK-50851][ML][CONNECT][PYTHON] Express ML params with `proto.Expression.Literal`
    ```
    
    ### _How was this patch tested?_
    
    `build/mvn clean package -pl kyuubi-relocated-spark-connect-rpc` passes, and the jar contains the relocated `Pipeline*` classes.
    
    ### _Was this patch assisted by generative AI tooling?_
    
    Assisted-by: Claude Opus 5.5
    
    Closes #72 from pan3793/proto-4.3.0-rc3.
    
    Closes #72
    
    2366e7f [Cheng Pan] Bump gRPC 1.76.3 and protobuf 3.25.9
    fa9fbef [Cheng Pan] Sync Spark Connect pb from Spark 4.3.0 RC3
    
    Authored-by: Cheng Pan <pa...@gmail.com>
    Signed-off-by: Cheng Pan <ch...@apache.org>
---
 kyuubi-relocated-spark-connect-rpc/pom.xml         |   4 +-
 .../src/main/protobuf/README.md                    |   2 +-
 .../src/main/protobuf/spark/connect/base.proto     | 240 ++++++++++++-
 .../src/main/protobuf/spark/connect/catalog.proto  |  81 +++++
 .../src/main/protobuf/spark/connect/commands.proto |   9 +
 .../src/main/protobuf/spark/connect/common.proto   |   6 +
 .../main/protobuf/spark/connect/expressions.proto  |  89 ++++-
 .../src/main/protobuf/spark/connect/ml.proto       |  27 ++
 .../main/protobuf/spark/connect/pipelines.proto    | 394 +++++++++++++++++++++
 .../main/protobuf/spark/connect/relations.proto    | 108 +++++-
 .../src/main/protobuf/spark/connect/types.proto    |  41 +++
 11 files changed, 989 insertions(+), 12 deletions(-)

diff --git a/kyuubi-relocated-spark-connect-rpc/pom.xml b/kyuubi-relocated-spark-connect-rpc/pom.xml
index b2b2c99..413794d 100644
--- a/kyuubi-relocated-spark-connect-rpc/pom.xml
+++ b/kyuubi-relocated-spark-connect-rpc/pom.xml
@@ -32,8 +32,8 @@ under the License.
     <description>Relocated Spark Connect RPC classes used by Kyuubi internally.</description>
 
     <properties>
-        <grpc.version>1.65.1</grpc.version>
-        <protobuf.version>3.25.1</protobuf.version>
+        <grpc.version>1.76.3</grpc.version>
+        <protobuf.version>3.25.9</protobuf.version>
     </properties>
 
     <dependencyManagement>
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/README.md b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/README.md
index 500b4b1..e1c1c51 100644
--- a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/README.md
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/README.md
@@ -1,5 +1,5 @@
 proto files are copied from
-https://github.com/apache/spark/tree/v4.0.0-rc4/sql/connect/common/src/main/protobuf
+https://github.com/apache/spark/tree/v4.3.0-rc3/sql/connect/common/src/main/protobuf
 
 and with one additional change in each proto file
 ```patch
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/base.proto b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/base.proto
index 38819f4..decc7dd 100644
--- a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/base.proto
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/base.proto
@@ -26,23 +26,42 @@ import "spark/connect/expressions.proto";
 import "spark/connect/relations.proto";
 import "spark/connect/types.proto";
 import "spark/connect/ml.proto";
+import "spark/connect/pipelines.proto";
 
 option java_multiple_files = true;
 option java_package = "org.apache.kyuubi.shaded.spark.connect.proto";
 option go_package = "internal/generated";
 
 // A [[Plan]] is the structure that carries the runtime information for the execution from the
-// client to the server. A [[Plan]] can either be of the type [[Relation]] which is a reference
-// to the underlying logical plan or it can be of the [[Command]] type that is used to execute
-// commands on the server.
+// client to the server. A [[Plan]] can be one of the following:
+// - [[Relation]]: a reference to the underlying logical plan.
+// - [[Command]]: used to execute commands on the server.
+// - [[CompressedOperation]]: a compressed representation of either a Relation or a Command.
 message Plan {
   oneof op_type {
     Relation root = 1;
     Command command = 2;
+    CompressedOperation compressed_operation = 3;
   }
-}
 
+  message CompressedOperation {
+    bytes data = 1;
+    OpType op_type = 2;
+    CompressionCodec compression_codec = 3;
 
+    enum OpType {
+      OP_TYPE_UNSPECIFIED = 0;
+      OP_TYPE_RELATION = 1;
+      OP_TYPE_COMMAND = 2;
+    }
+  }
+}
+
+// Compression codec for plan compression.
+enum CompressionCodec {
+  COMPRESSION_CODEC_UNSPECIFIED = 0;
+  COMPRESSION_CODEC_ZSTD = 1;
+}
 
 // User Context is used to refer to one particular user session that is executing
 // queries in the backend.
@@ -332,6 +351,7 @@ message ExecutePlanRequest {
   message RequestOption {
     oneof request_option {
       ReattachOptions reattach_options = 1;
+      ResultChunkingOptions result_chunking_options = 2;
       // Extension type for request options
       google.protobuf.Any extension = 999;
     }
@@ -399,6 +419,16 @@ message ExecutePlanResponse {
     // ML command response
     MlCommandResult ml_command_result = 20;
 
+    // Response containing pipeline event that is streamed back to the client during a pipeline run
+    PipelineEventResult pipeline_event_result = 21;
+
+    // Pipeline command response
+    PipelineCommandResult pipeline_command_result = 22;
+
+    // A signal from the server to the client to execute the query function for a flow, and to
+    // register its result with the server.
+    PipelineQueryFunctionExecutionSignal pipeline_query_function_execution_signal = 23;
+
     // Support arbitrary result objects.
     google.protobuf.Any extension = 999;
   }
@@ -428,6 +458,14 @@ message ExecutePlanResponse {
 
     // If set, row offset of the start of this ArrowBatch in execution results.
     optional int64 start_offset = 3;
+
+    // Index of this chunk in the batch if chunking is enabled. The index starts from 0.
+    optional int64 chunk_index = 4;
+
+    // Total number of chunks in this batch if chunking is enabled.
+    // It is missing when chunking is disabled - the batch is returned whole
+    // and client will treat this response as the batch.
+    optional int64 num_chunks_in_batch = 5;
   }
 
   message Metrics {
@@ -453,6 +491,12 @@ message ExecutePlanResponse {
     repeated Expression.Literal values = 2;
     repeated string keys = 3;
     int64 plan_id = 4;
+    // (Optional) The index of the root error in errors.
+    // The field will not be set if there are no errors.
+    optional int32 root_error_idx = 5;
+    // A list of errors that occurred while collecting the observed metrics.
+    // If the length is 0, it means no errors occurred.
+    repeated FetchErrorDetailsResponse.Error errors = 6;
   }
 
   message ResultComplete {
@@ -812,6 +856,24 @@ message ReattachOptions {
   bool reattachable = 1;
 }
 
+message ResultChunkingOptions {
+  // Although Arrow results are split into batches with a size limit according to estimation, the
+  // size of the batches is not guaranteed to be less than the limit, especially when a single row
+  // is larger than the limit, in which case the server will fail to split it further into smaller
+  // batches. As a result, the client may encounter a gRPC error stating “Received message larger
+  // than max” when a batch is too large.
+  // If allow_arrow_batch_chunking=true, the server will split large Arrow batches into smaller chunks,
+  // and the client is expected to handle the chunked Arrow batches.
+  //
+  // If false, the server will not chunk large Arrow batches.
+  bool allow_arrow_batch_chunking = 1;
+
+  // Optional preferred Arrow batch size in bytes for the server to use when sending Arrow results.
+  // The server will attempt to use this size if it is set and within the valid range
+  // ([1KB, max batch size on server]). Otherwise, the server's maximum batch size is used.
+  optional int64 preferred_arrow_chunk_size = 2;
+}
+
 message ReattachExecuteRequest {
   // (Required)
   //
@@ -1061,6 +1123,32 @@ message FetchErrorDetailsResponse {
     // Portable error identifier across SQL engines
     // If null, error class or SQLSTATE is not set.
     optional string sql_state = 4;
+
+    // Additional information if the error was caused by a breaking change.
+    optional BreakingChangeInfo breaking_change_info = 5;
+  }
+
+  // BreakingChangeInfo defines the schema for breaking change information.
+  message BreakingChangeInfo {
+    // A message explaining how the user can migrate their job to work
+    // with the breaking change.
+    repeated string migration_message = 1;
+
+    // A spark config flag that can be used to mitigate the breaking change.
+    optional MitigationConfig mitigation_config = 2;
+
+    // If true, the breaking change should be inspected manually.
+    // If false, the spark job should be retried by setting the mitigationConfig.
+    optional bool needs_audit = 3;
+  }
+
+  // MitigationConfig defines a spark config flag that can be used to mitigate a breaking change.
+  message MitigationConfig {
+    // The spark config key.
+    string key = 1;
+
+    // The spark config value that mitigates the breaking change.
+    string value = 2;
   }
 
   // Error defines the schema for the representing exception.
@@ -1088,6 +1176,137 @@ message CheckpointCommandResult {
   CachedRemoteRelation relation = 1;
 }
 
+message CloneSessionRequest {
+  // (Required)
+  //
+  // The session_id specifies a spark session for a user id (which is specified
+  // by user_context.user_id). The session_id is set by the client to be able to
+  // collate streaming responses from different queries within the dedicated session.
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // (Optional)
+  //
+  // Server-side generated idempotency key from the previous responses (if any). Server
+  // can use this to validate that the server side session has not changed.
+  optional string client_observed_server_side_session_id = 5;
+
+  // (Required) User context
+  //
+  // user_context.user_id and session_id both identify a unique remote spark session on the
+  // server side.
+  UserContext user_context = 2;
+
+  // Provides optional information about the client sending the request. This field
+  // can be used for language or version specific information and is only intended for
+  // logging purposes and will not be interpreted by the server.
+  optional string client_type = 3;
+
+  // (Optional)
+  // The session_id for the new cloned session. If not provided, a new UUID will be generated.
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  optional string new_session_id = 4;
+}
+
+// Next ID: 5
+message CloneSessionResponse {
+  // Session id of the original session that was cloned.
+  string session_id = 1;
+
+  // Server-side generated idempotency key that the client can use to assert that the server side
+  // session (parent of the cloned session) has not changed.
+  string server_side_session_id = 2;
+
+  // Session id of the new cloned session.
+  string new_session_id = 3;
+
+  // Server-side session ID of the new cloned session.
+  string new_server_side_session_id = 4;
+}
+
+// Next ID: 6
+message GetStatusRequest {
+  // (Required)
+  //
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // (Required)
+  //
+  // user_context.user_id and session_id both identify a unique remote spark session on the
+  // server side.
+  UserContext user_context = 2;
+
+  // (Optional)
+  //
+  // Provides optional information about the client sending the request. This field
+  // can be used for language or version specific information and is only intended for
+  // logging purposes and will not be interpreted by the server.
+  optional string client_type = 3;
+
+  // (Optional)
+  //
+  // Server-side generated idempotency key from the previous responses (if any). Server
+  // can use this to validate that the server side session has not changed.
+  optional string client_observed_server_side_session_id = 4;
+
+  // (Optional)
+  //
+  // Get status of operations in the session.
+  optional OperationStatusRequest operation_status = 5;
+
+  // Extension point for custom status request types.
+  repeated google.protobuf.Any extensions = 999;
+
+  message OperationStatusRequest {
+    // Get status of operations with these operation_ids.
+    // If unset or empty, returns status of all operations in the session.
+    repeated string operation_ids = 1;
+
+    // Extension point for custom operation-level status requests.
+    repeated google.protobuf.Any extensions = 999;
+  }
+}
+
+// Next ID: 4
+message GetStatusResponse {
+  // Session id of the session for which the status was requested.
+  string session_id = 1;
+
+  // Server-side generated idempotency key that the client can use to assert that the server side
+  // session has not changed.
+  string server_side_session_id = 2;
+
+  // Status information about requested operations.
+  repeated OperationStatus operation_statuses = 3;
+
+  // Extension point for custom status response types.
+  repeated google.protobuf.Any extensions = 999;
+
+  // Status information for a single operation.
+  message OperationStatus {
+    // The operation_id of the operation.
+    string operation_id = 1;
+
+    // The current status of the operation.
+    OperationState state = 2;
+
+    // Extension point for custom operation-level status fields.
+    repeated google.protobuf.Any extensions = 999;
+
+    enum OperationState {
+      OPERATION_STATE_UNSPECIFIED = 0;
+      OPERATION_STATE_UNKNOWN = 1;
+      OPERATION_STATE_RUNNING = 2;
+      OPERATION_STATE_TERMINATING = 3;
+      OPERATION_STATE_SUCCEEDED = 4;
+      OPERATION_STATE_FAILED = 5;
+      OPERATION_STATE_CANCELLED = 6;
+    }
+  }
+}
+
 // Main interface for the SparkConnect service.
 service SparkConnectService {
 
@@ -1132,4 +1351,17 @@ service SparkConnectService {
 
   // FetchErrorDetails retrieves the matched exception with details based on a provided error id.
   rpc FetchErrorDetails(FetchErrorDetailsRequest) returns (FetchErrorDetailsResponse) {}
+
+  // Create a clone of a Spark Connect session on the server side. The server-side session
+  // is cloned with all its current state (SQL configurations, temporary views, registered
+  // functions, catalog state) copied over to a new independent session. The cloned session
+  // is isolated from the source session - any subsequent changes to either session's
+  // server-side state will not be reflected in the other.
+  //
+  // The request can optionally specify a custom session ID for the cloned session (must be
+  // a valid UUID). If not provided, a new UUID will be generated automatically.
+  rpc CloneSession(CloneSessionRequest) returns (CloneSessionResponse) {}
+
+  // Get status information of different types.
+  rpc GetStatus(GetStatusRequest) returns (GetStatusResponse) {}
 }
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/catalog.proto b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/catalog.proto
index 06e6282..91e6674 100644
--- a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/catalog.proto
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/catalog.proto
@@ -55,6 +55,16 @@ message Catalog {
     CurrentCatalog current_catalog = 24;
     SetCurrentCatalog set_current_catalog = 25;
     ListCatalogs list_catalogs = 26;
+    DropTable drop_table = 27;
+    DropView drop_view = 28;
+    CreateDatabase create_database = 29;
+    DropDatabase drop_database = 30;
+    ListPartitions list_partitions = 31;
+    ListViews list_views = 32;
+    GetTableProperties get_table_properties = 33;
+    GetCreateTableString get_create_table_string = 34;
+    TruncateTable truncate_table = 35;
+    AnalyzeTable analyze_table = 36;
   }
 }
 
@@ -241,3 +251,74 @@ message ListCatalogs {
   // (Optional) The pattern that the catalog name needs to match
   optional string pattern = 1;
 }
+
+// See `spark.catalog.dropTable`
+message DropTable {
+  // (Required)
+  string table_name = 1;
+  bool if_exists = 2;
+  bool purge = 3;
+}
+
+// See `spark.catalog.dropView`
+message DropView {
+  // (Required)
+  string view_name = 1;
+  bool if_exists = 2;
+}
+
+// See `spark.catalog.createDatabase`
+message CreateDatabase {
+  // (Required)
+  string db_name = 1;
+  bool if_not_exists = 2;
+  map<string, string> properties = 3;
+}
+
+// See `spark.catalog.dropDatabase`
+message DropDatabase {
+  // (Required)
+  string db_name = 1;
+  bool if_exists = 2;
+  bool cascade = 3;
+}
+
+// See `spark.catalog.listPartitions`
+message ListPartitions {
+  // (Required)
+  string table_name = 1;
+}
+
+// See `spark.catalog.listViews`
+message ListViews {
+  // (Optional)
+  optional string db_name = 1;
+  // (Optional) The pattern that the view name needs to match
+  optional string pattern = 2;
+}
+
+// See `spark.catalog.getTableProperties`
+message GetTableProperties {
+  // (Required)
+  string table_name = 1;
+}
+
+// See `spark.catalog.getCreateTableString`
+message GetCreateTableString {
+  // (Required)
+  string table_name = 1;
+  bool as_serde = 2;
+}
+
+// See `spark.catalog.truncateTable`
+message TruncateTable {
+  // (Required)
+  string table_name = 1;
+}
+
+// See `spark.catalog.analyzeTable`
+message AnalyzeTable {
+  // (Required)
+  string table_name = 1;
+  bool no_scan = 2;
+}
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/commands.proto b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/commands.proto
index e0fb306..38f06fd 100644
--- a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/commands.proto
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/commands.proto
@@ -22,6 +22,7 @@ import "spark/connect/common.proto";
 import "spark/connect/expressions.proto";
 import "spark/connect/relations.proto";
 import "spark/connect/ml.proto";
+import "spark/connect/pipelines.proto";
 
 package spark.connect;
 
@@ -51,6 +52,7 @@ message Command {
     MergeIntoTableCommand merge_into_table_command = 16;
     MlCommand ml_command = 17;
     ExecuteExternalCommand execute_external_command = 18;
+    PipelineCommand pipeline_command = 19;
 
     // This field is used to mark extensions to the protocol. When plugins generate arbitrary
     // Commands they can add them here. During the planning the correct resolution is done.
@@ -142,6 +144,9 @@ message WriteOperation {
   // (Optional) Columns used for clustering the table.
   repeated string clustering_columns = 10;
 
+  // (Optional) Whether schema evolution is enabled for the write.
+  bool with_schema_evolution = 11;
+
   message SaveTable {
     // (Required) The table name.
     string table_name = 1;
@@ -209,6 +214,9 @@ message WriteOperationV2 {
 
   // (Optional) Columns used for clustering the table.
   repeated string clustering_columns = 9;
+
+  // (Optional) Whether schema evolution is enabled for the write.
+  bool with_schema_evolution = 10;
 }
 
 // Starts write stream operation as streaming query. Query ID and Run ID of the streaming
@@ -230,6 +238,7 @@ message WriteStreamOperationStart {
     bool available_now = 6;
     bool once = 7;
     string continuous_checkpoint_interval = 8;
+    string real_time_batch_duration = 100;
   }
 
   string output_mode = 9;
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/common.proto b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/common.proto
index c8e9f25..3079df6 100644
--- a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/common.proto
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/common.proto
@@ -148,6 +148,12 @@ message StackTraceElement {
   int32 line_number = 7;
 }
 
+message ResolvedIdentifier {
+  string catalog_name = 1;
+  repeated string namespace = 2;
+  string table_name = 3;
+}
+
 message Bools {
   repeated bool values = 1;
 }
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/expressions.proto b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/expressions.proto
index b300c3a..8ce150a 100644
--- a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/expressions.proto
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/expressions.proto
@@ -53,6 +53,7 @@ message Expression {
     MergeAction merge_action = 19;
     TypedAggregateExpression typed_aggregate_expression = 20;
     SubqueryExpression subquery_expression = 21;
+    DirectShufflePartitionID direct_shuffle_partition_id = 22;
 
     // This field is used to mark extensions to the protocol. When plugins generate arbitrary
     // relations they can add them here. During the planning the correct resolution is done.
@@ -142,6 +143,14 @@ message Expression {
     }
   }
 
+  // Expression that takes a partition ID value and passes it through directly for use in
+  // shuffle partitioning. This is used with RepartitionByExpression to allow users to
+  // directly specify target partition IDs.
+  message DirectShufflePartitionID {
+    // (Required) The expression that evaluates to the partition ID.
+    Expression child = 1;
+  }
+
   message Cast {
     // (Required) the expression to be casted.
     Expression expr = 1;
@@ -195,8 +204,24 @@ message Expression {
       Struct struct = 24;
 
       SpecializedArray specialized_array = 25;
+      Time time = 26;
+
+      // Nanosecond-capable timestamp literals (precision 7..9). NTZ and LTZ are distinct
+      // arms so the literal kind is self-describing.
+      TimestampNTZNanos timestamp_ntz_nanos = 29;
+      TimestampLTZNanos timestamp_ltz_nanos = 30;
     }
 
+    // Reserved for Geometry and Geography.
+    reserved 27, 28;
+
+    // Data type information for the literal.
+    // This field is required only in the root literal message for null values or
+    // for data types (e.g., array, map, or struct) with non-trivial information.
+    // If the data_type field is not set at the root level, the data type will be
+    // inferred or retrieved from the deprecated data type fields using best efforts.
+    DataType data_type = 100;
+
     message Decimal {
       // the string representation.
       string value = 1;
@@ -214,19 +239,42 @@ message Expression {
     }
 
     message Array {
-      DataType element_type = 1;
+      // (Deprecated) The element type of the array.
+      //
+      // This field is deprecated since Spark 4.1+. Use data_type field instead.
+      DataType element_type = 1 [deprecated = true];
+
+      // The literal values that make up the array elements.
       repeated Literal elements = 2;
     }
 
     message Map {
-      DataType key_type = 1;
-      DataType value_type = 2;
+      // (Deprecated) The key type of the map.
+      //
+      // This field is deprecated since Spark 4.1+. Use data_type field instead.
+      DataType key_type = 1 [deprecated = true];
+
+      // (Deprecated) The value type of the map.
+      //
+      // This field is deprecated since Spark 4.1+ and should only be set
+      // if the data_type field is not set. Use data_type field instead.
+      DataType value_type = 2 [deprecated = true];
+
+      // The literal keys that make up the map.
       repeated Literal keys = 3;
+
+      // The literal values that make up the map.
       repeated Literal values = 4;
     }
 
     message Struct {
-      DataType struct_type = 1;
+      // (Deprecated) The type of the struct.
+      //
+      // This field is deprecated since Spark 4.1+ because using DataType as the type of a struct
+      // is ambiguous. Use data_type field instead.
+      DataType struct_type = 1 [deprecated = true];
+
+      // The literal values that make up the struct elements.
       repeated Literal elements = 2;
     }
 
@@ -240,6 +288,35 @@ message Expression {
         Strings strings = 6;
       }
     }
+
+    message Time {
+      int64 nano = 1;
+      // The precision of this time, if omitted, uses the default value of MICROS_PRECISION.
+      optional int32 precision = 2;
+    }
+
+    // A TIMESTAMP_NTZ literal with nanosecond-capable precision. The physical value is carried
+    // as microseconds since the UNIX epoch plus the extra nanoseconds within that microsecond,
+    // because a single int64 of nanoseconds cannot span the supported year range.
+    message TimestampNTZNanos {
+      // Microseconds since the UNIX epoch (without timezone information).
+      int64 epoch_micros = 1;
+      // Additional nanoseconds within epoch_micros, in [0, 999].
+      int32 nanos_within_micro = 2;
+      // Number of fractional-second digits (7, 8, or 9). If omitted, defaults to 9 (nanoseconds).
+      optional int32 precision = 3;
+    }
+
+    // A TIMESTAMP_LTZ literal with nanosecond-capable precision. See TimestampNTZNanos for the
+    // rationale behind the two-component physical value.
+    message TimestampLTZNanos {
+      // Microseconds since the UNIX epoch.
+      int64 epoch_micros = 1;
+      // Additional nanoseconds within epoch_micros, in [0, 999].
+      int32 nanos_within_micro = 2;
+      // Number of fractional-second digits (7, 8, or 9). If omitted, defaults to 9 (nanoseconds).
+      optional int32 precision = 3;
+    }
   }
 
   // An unresolved attribute that is not explicitly bound to a specific column, but the column
@@ -483,11 +560,15 @@ message SubqueryExpression {
   // (Optional) Options specific to table arguments.
   optional TableArgOptions table_arg_options = 3;
 
+  // (Optional) IN subquery values.
+  repeated Expression in_subquery_values = 4;
+
   enum SubqueryType {
     SUBQUERY_TYPE_UNKNOWN = 0;
     SUBQUERY_TYPE_SCALAR = 1;
     SUBQUERY_TYPE_EXISTS = 2;
     SUBQUERY_TYPE_TABLE_ARG = 3;
+    SUBQUERY_TYPE_IN = 4;
   }
 
   // Nested message for table argument options.
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/ml.proto b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/ml.proto
index d73101f..26da301 100644
--- a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/ml.proto
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/ml.proto
@@ -36,6 +36,10 @@ message MlCommand {
     Write write = 4;
     Read read = 5;
     Evaluate evaluate = 6;
+    CleanCache clean_cache = 7;
+    GetCacheInfo get_cache_info = 8;
+    CreateSummary create_summary = 9;
+    GetModelSize get_model_size = 10;
   }
 
   // Command for estimator.fit(dataset)
@@ -52,8 +56,17 @@ message MlCommand {
   // or summary evaluated by a model
   message Delete {
     repeated ObjectRef obj_refs = 1;
+    // if set `evict_only` to true, only evict the cached model from memory,
+    // but keep the offloaded model in Spark driver local disk.
+    optional bool evict_only = 2;
   }
 
+  // Force to clean up all the ML cached objects
+  message CleanCache { }
+
+  // Get the information of all the ML cached objects
+  message GetCacheInfo { }
+
   // Command to write ML operator
   message Write {
     // It could be an estimator/evaluator or the cached model
@@ -90,6 +103,18 @@ message MlCommand {
     // (Required) the evaluating dataset
     Relation dataset = 3;
   }
+
+  // This is for re-creating the model summary when the model summary is lost
+  // (model summary is lost when the model is offloaded and then loaded back)
+  message CreateSummary {
+    ObjectRef model_ref = 1;
+    Relation dataset = 2;
+  }
+
+  // This is for query the model estimated in-memory size
+  message GetModelSize {
+    ObjectRef model_ref = 1;
+  }
 }
 
 // The result of MlCommand
@@ -116,5 +141,7 @@ message MlCommandResult {
     optional string uid = 3;
     // (Optional) parameters
     optional MlParams params = 4;
+    // (Optional) warning message generated during the ML command execution
+    optional string warning_message = 5;
   }
 }
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/pipelines.proto b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/pipelines.proto
new file mode 100644
index 0000000..df597dc
--- /dev/null
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/pipelines.proto
@@ -0,0 +1,394 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+package spark.connect;
+
+import "google/protobuf/any.proto";
+import "google/protobuf/timestamp.proto";
+import "spark/connect/common.proto";
+import "spark/connect/expressions.proto";
+import "spark/connect/relations.proto";
+import "spark/connect/types.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.kyuubi.shaded.spark.connect.proto";
+option go_package = "internal/generated";
+
+// Dispatch object for pipelines commands. See each individual command for documentation.
+message PipelineCommand {
+  oneof command_type {
+    CreateDataflowGraph create_dataflow_graph = 1;
+    DefineOutput define_output = 2;
+    DefineFlow define_flow = 3;
+    DropDataflowGraph drop_dataflow_graph = 4;
+    StartRun start_run = 5;
+    DefineSqlGraphElements define_sql_graph_elements = 6;
+    GetQueryFunctionExecutionSignalStream get_query_function_execution_signal_stream = 7;
+    DefineFlowQueryFunctionResult define_flow_query_function_result = 8;
+    ExecuteOutputFlows execute_output_flows = 9;
+
+    // Reserved field for protocol extensions.
+    // Used to support forward-compatibility by carrying additional command types
+    // that are not yet defined in this version of the proto. During planning, the
+    // engine will resolve and dispatch the concrete command contained in this field.
+    google.protobuf.Any extension = 999;
+  }
+
+  // Request to create a new dataflow graph.
+  message CreateDataflowGraph {
+    // The default catalog.
+    optional string default_catalog = 1;
+
+    // The default database.
+    optional string default_database = 2;
+
+    // SQL configurations for all flows in this graph.
+    map<string, string> sql_conf = 5;
+  }
+
+  // Drops the graph and stops any running attached flows.
+  message DropDataflowGraph {
+    // The graph to drop.
+    optional string dataflow_graph_id = 1;
+  }
+
+  // Request to define an output: a table, a materialized view, a temporary view or a sink.
+  message DefineOutput {
+    // The graph to attach this output to.
+    optional string dataflow_graph_id = 1;
+
+    // Name of the output. Can be partially or fully qualified.
+    optional string output_name = 2;
+
+    // The type of the output.
+    optional OutputType output_type = 3;
+
+    // Optional comment for the output.
+    optional string comment = 4;
+
+    // The location in source code that this output was defined.
+    optional SourceCodeLocation source_code_location = 5;
+
+    oneof details {
+      TableDetails table_details = 6;
+      SinkDetails sink_details = 7;
+      google.protobuf.Any extension = 999;
+    }
+
+    // Metadata that's only applicable to tables and materialized views.
+    message TableDetails {
+      // Optional table properties.
+      map<string, string> table_properties = 1;
+
+      // Optional partition columns for the table.
+      repeated string partition_cols = 2;
+
+      // The output table format for the table.
+      optional string format = 3;
+
+      // Schema for the table. If unset, this will be inferred from incoming flows.
+      oneof schema {
+        spark.connect.DataType schema_data_type = 4;
+        string schema_string = 5;
+      }
+
+      // Optional cluster columns for the table.
+      repeated string clustering_columns = 6;
+    }
+
+    // Metadata that's only applicable to sinks.
+    message SinkDetails {
+      // Streaming write options
+      map<string, string> options = 1;
+
+      // Streaming write format
+      optional string format = 2;
+    }
+  }
+
+  // Request to define a flow targeting a dataset.
+  message DefineFlow {
+    // The graph to attach this flow to.
+    optional string dataflow_graph_id = 1;
+
+    // Name of the flow. For standalone flows, this must be a single-part name.
+    optional string flow_name = 2;
+
+    // Name of the dataset this flow writes to. Can be partially or fully qualified.
+    optional string target_dataset_name = 3;
+
+    // SQL configurations set when running this flow.
+    map<string, string> sql_conf = 4;
+
+    // Identifier for the client making the request. The server uses this to determine what flow
+    // evaluation request stream to dispatch evaluation requests to for this flow.
+    optional string client_id = 5;
+
+    // The location in source code that this flow was defined.
+    optional SourceCodeLocation source_code_location = 6;
+
+    oneof details {
+      WriteRelationFlowDetails relation_flow_details = 7;
+      AutoCdcFlowDetails auto_cdc_flow_details = 10;
+      google.protobuf.Any extension = 999;
+    }
+
+    // A flow that is that takes the contents of a relation and writes it to the target dataset.
+    message WriteRelationFlowDetails {
+      // An unresolved relation that defines the dataset's flow. Empty if the query function
+      // that defines the flow cannot be analyzed at the time of flow definition.
+      optional spark.connect.Relation relation = 1;
+    }
+
+    // Details for Auto CDC flows.
+    message AutoCdcFlowDetails {
+      // The name of the CDC source to stream from.
+      optional string source = 1;
+
+      // Column(s) that uniquely identify a row in source and target data.
+      repeated Expression keys = 2;
+
+      // Expression to order the source data.
+      optional Expression sequence_by = 3;
+
+      // Delete condition for the merged operation.
+      optional Expression apply_as_deletes = 6;
+
+      // Truncate condition for the merged operation.
+      optional Expression apply_as_truncates = 7;
+
+      // Columns included in the output table.
+      repeated Expression column_list = 8;
+
+      // Columns excluded from the output table.
+      repeated Expression except_column_list = 9;
+
+      // SCD Type for target table.
+      SCDType stored_as_scd_type = 10;
+
+      // SCD2 only. Columns whose value change opens a new history record. When empty, every
+      // eligible selected user column is tracked.
+      repeated Expression track_history_column_list = 11;
+
+      // SCD2 only. Columns excluded from history tracking. Mutually exclusive with
+      // track_history_column_list.
+      repeated Expression track_history_except_column_list = 12;
+
+      // Subset of columns to ignore null in updates.
+      repeated Expression ignore_null_updates_column_list = 14;
+
+      // Subset of columns excluded from ignoring null in updates.
+      repeated Expression ignore_null_updates_except_column_list = 15;
+
+    }
+
+    // SCD Type for Auto CDC target tables.
+    enum SCDType {
+      SCD_TYPE_UNSPECIFIED = 0;
+      SCD_TYPE_1 = 1;
+      SCD_TYPE_2 = 2;
+    }
+
+    // If true, define the flow as a one-time flow, such as for backfill.
+    // Set to true changes the flow in two ways:
+    //   - The flow is run one time by default. If the pipeline is ran with a full refresh,
+    //     the flow will run again.
+    //   - The flow function must be a batch DataFrame, not a streaming DataFrame.
+    optional bool once = 8;
+
+    message Response {
+      // Fully qualified flow name that uniquely identify a flow in the Dataflow graph.
+      optional string flow_name = 1;
+    }
+  }
+
+  // Request to execute all flows for a single output (dataset or sink) remotely.
+  message ExecuteOutputFlows {
+
+    // The output (table or materialized view or sink) definition.
+    optional DefineOutput define_output = 1;
+
+    // The flows to execute for this table.
+    repeated DefineFlow define_flows = 2;
+
+    // Whether to perform a full refresh instead of an incremental update.
+    optional bool full_refresh = 3;
+
+    // Storage location for pipeline checkpoints and metadata.
+    optional string storage = 4;
+
+    // Reserved field for protocol extensions.
+    repeated google.protobuf.Any extension = 999;
+  }
+
+  // Resolves all datasets and flows and start a pipeline update. Should be called after all
+  // graph elements are registered.
+  message StartRun {
+    // The graph to start.
+    optional string dataflow_graph_id = 1;
+
+    // List of dataset to reset and recompute.
+    repeated string full_refresh_selection = 2;
+
+    // Perform a full graph reset and recompute.
+    optional bool full_refresh_all = 3;
+
+    // List of dataset to update.
+    repeated string refresh_selection = 4;
+
+    // If true, the run will not actually execute any flows, but will only validate the graph and
+    // check for any errors. This is useful for testing and validation purposes.
+    optional bool dry = 5;
+
+    // storage location for pipeline checkpoints and metadata.
+    optional string storage = 6;
+  }
+
+  // Parses the SQL file and registers all datasets and flows.
+  message DefineSqlGraphElements {
+    // The graph to attach this dataset to.
+    optional string dataflow_graph_id = 1;
+
+    // The full path to the SQL file. Can be relative or absolute.
+    optional string sql_file_path = 2;
+
+    // The contents of the SQL file.
+    optional string sql_text = 3;
+  }
+
+  // Request to get the stream of query function execution signals for a graph. Responses should
+  // be a stream of PipelineQueryFunctionExecutionSignal messages.
+  message GetQueryFunctionExecutionSignalStream {
+    // The graph to get the query function execution signal stream for.
+    optional string dataflow_graph_id = 1;
+
+    // Identifier for the client that is requesting the stream.
+    optional string client_id = 2;
+  }
+
+  // Request from the client to update the flow function evaluation result
+  // for a previously un-analyzed flow.
+  message DefineFlowQueryFunctionResult {
+    // (Deprecated) The fully qualified name of the flow being updated.
+    //
+    // This field is deprecated since Spark 4.2+. Use flow_identifier field instead.
+    optional string flow_name = 1 [deprecated = true];
+
+    // The fully qualified identifier of the flow being updated.
+    optional ResolvedIdentifier flow_identifier = 4;
+
+    // The ID of the graph this flow belongs to.
+    optional string dataflow_graph_id = 2;
+
+    // An unresolved relation that defines the dataset's flow.
+    optional spark.connect.Relation relation = 3;
+  }
+}
+
+// Dispatch object for pipelines command results.
+message PipelineCommandResult {
+  oneof result_type {
+    CreateDataflowGraphResult create_dataflow_graph_result = 1;
+    DefineOutputResult define_output_result = 2;
+    DefineFlowResult define_flow_result = 3;
+  }
+  message CreateDataflowGraphResult {
+    // The ID of the created graph.
+    optional string dataflow_graph_id = 1;
+  }
+  message DefineOutputResult {
+    // Resolved identifier of the output
+    optional ResolvedIdentifier resolved_identifier = 1;
+  }
+  message DefineFlowResult {
+    // Resolved identifier of the flow
+    optional ResolvedIdentifier resolved_identifier = 1;
+  }
+}
+
+// The type of output.
+enum OutputType {
+  // Safe default value. Should not be used.
+  OUTPUT_TYPE_UNSPECIFIED = 0;
+  // A materialized view which is published to the catalog
+  MATERIALIZED_VIEW = 1;
+  // A table which is published to the catalog
+  TABLE = 2;
+  // A view which is not published to the catalog
+  TEMPORARY_VIEW = 3;
+  // A sink which is not published to the catalog
+  SINK = 4;
+}
+
+// A response containing an event emitted during the run of a pipeline.
+message PipelineEventResult {
+  PipelineEvent event = 1;
+}
+
+message PipelineEvent {
+  // The timestamp corresponding to when the event occurred.
+  google.protobuf.Timestamp timestamp = 1;
+  // The message that should be displayed to users.
+  optional string message = 2;
+}
+
+// Source code location information associated with a particular dataset or flow.
+message SourceCodeLocation {
+  // The file that this pipeline source code was defined in.
+  optional string file_name = 1;
+  // The specific line number that this pipeline source code is located at, if applicable.
+  optional int32 line_number = 2;
+  // The path of the top-level pipeline file determined at runtime during pipeline initialization.
+  optional string definition_path = 3;
+
+  // Reserved field for protocol extensions.
+  // Used to support forward-compatibility by carrying additional fields
+  // that are not yet defined in this version of the proto. During planning, the
+  // engine will resolve and dispatch the concrete command contained in this field.
+  repeated google.protobuf.Any extension = 999;
+}
+
+// A signal from the server to the client to execute the query function for one or more flows, and
+// to register their results with the server.
+message PipelineQueryFunctionExecutionSignal {
+  // (Deprecated) The name of flows that are ready to be re-evaluated.
+  //
+  // This field is deprecated since Spark 4.2+. Use flow_identifiers field instead.
+  repeated string flow_names = 1 [deprecated = true];
+
+  // The identifier of flows that are ready to be re-evaluated
+  repeated ResolvedIdentifier flow_identifiers = 2;
+}
+
+// Metadata providing context about the pipeline during Spark Connect query analysis.
+message PipelineAnalysisContext {
+  // Unique identifier of the dataflow graph associated with this pipeline.
+  optional string dataflow_graph_id = 1;
+  // The path of the top-level pipeline file determined at runtime during pipeline initialization.
+  optional string definition_path = 2;
+  // (Deprecated) The name of the Flow involved in this analysis
+  //
+  // This field is deprecated since Spark 4.2+. Use flow_identifier field instead.
+  optional string flow_name = 3 [deprecated = true];
+  // The identifier of the Flow involved in this analysis
+  optional ResolvedIdentifier flow_identifier = 4;
+
+  // Reserved field for protocol extensions.
+  repeated google.protobuf.Any extension = 999;
+}
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/relations.proto b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/relations.proto
index b401f7f..c03c7d5 100644
--- a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/relations.proto
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/relations.proto
@@ -80,6 +80,10 @@ message Relation {
     Transpose transpose = 42;
     UnresolvedTableValuedFunction unresolved_table_valued_function = 43;
     LateralJoin lateral_join = 44;
+    ChunkedCachedLocalRelation chunked_cached_local_relation = 45;
+    RelationChanges relation_changes = 46;
+    NearestByJoin nearest_by_join = 47;
+    Zip zip = 48;
 
     // NA functions
     NAFill fill_na = 90;
@@ -115,6 +119,9 @@ message MlRelation {
     Transform transform = 1;
     Fetch fetch = 2;
   }
+  // (Optional) the dataset for restoring the model summary
+  optional Relation model_summary_dataset = 3;
+
   // Relation to represent transform(input) of the operator
   // which could be a cached model or a new transformer
   message Transform {
@@ -252,9 +259,33 @@ message Read {
     //
     // This is only supported by the JDBC data source.
     repeated string predicates = 5;
+
+    // (Optional) A user-provided name for the streaming source.
+    // This name is used in checkpoint metadata and enables stable checkpoint locations
+    // for source evolution.
+    optional string source_name = 6;
   }
 }
 
+// Reads Change Data Capture (CDC) changes for a named table.
+//
+// This corresponds to the `DataFrameReader.changes()` or `DataStreamReader.changes()` API.
+// CDC-specific options (startingVersion, endingVersion, startingTimestamp, endingTimestamp,
+// deduplicationMode, computeUpdates, etc.) are passed in the options map.
+message RelationChanges {
+  // (Required) Unparsed identifier for the table.
+  string unparsed_identifier = 1;
+
+  // Options for the CDC query. The map key is case insensitive.
+  // Supported keys include: startingVersion, endingVersion, startingTimestamp,
+  // endingTimestamp, deduplicationMode, computeUpdates, startingBoundInclusive,
+  // endingBoundInclusive.
+  map<string, string> options = 2;
+
+  // (Optional) Indicates if this is a streaming CDC read.
+  bool is_streaming = 3;
+}
+
 // Projection of a bag of expressions for a given input relation.
 //
 // The input relation must be specified.
@@ -496,6 +527,7 @@ message LocalRelation {
 }
 
 // A local relation that has been cached already.
+// CachedLocalRelation doesn't support LocalRelations of size over 2GB.
 message CachedLocalRelation {
   // `userId` and `sessionId` fields are deleted since the server must always use the active
   // session/user rather than arbitrary values provided by the client. It is never valid to access
@@ -507,6 +539,20 @@ message CachedLocalRelation {
   string hash = 3;
 }
 
+// A local relation that has been cached already.
+message ChunkedCachedLocalRelation {
+  // (Required) A list of sha-256 hashes for representing LocalRelation.data.
+  // Data is serialized in Arrow IPC streaming format, each batch is cached on the server as
+  // a separate artifact. Each hash represents one batch stored on the server.
+  // Hashes are hex-encoded strings (e.g., "a3b2c1d4...").
+  repeated string dataHashes = 1;
+
+  // (Optional) A sha-256 hash of the serialized LocalRelation.schema.
+  // Scala clients always provide the schema, Python clients can omit it.
+  // Hash is a hex-encoded string (e.g., "a3b2c1d4...").
+  optional string schemaHash = 2;
+}
+
 // Represents a remote relation that has been cached on server.
 message CachedRemoteRelation {
   // (Required) ID of the remote related (assigned by the service).
@@ -1031,6 +1077,23 @@ message GroupMap {
 
   // (Optional) The schema for the grouped state.
   optional DataType state_schema = 10;
+
+  // Below fields are used by TransformWithState and TransformWithStateInPandas
+  // (Optional) TransformWithState related parameters.
+  optional TransformWithStateInfo transform_with_state_info = 11;
+}
+
+// Additional input parameters used for TransformWithState operator.
+message TransformWithStateInfo {
+  // (Required) Time mode string for transformWithState.
+  string time_mode = 1;
+
+  // (Optional) Event time column name.
+  optional string event_time_column_name = 2;
+
+  // (Optional) Schema for the output DataFrame.
+  // Only required used for TransformWithStateInPandas.
+  optional DataType output_schema = 3;
 }
 
 message CoGroupMap {
@@ -1148,12 +1211,13 @@ message Parse {
   // (Optional) DataType representing the schema. If not set, Spark will infer the schema.
   optional DataType schema = 3;
 
-  // Options for the csv/json parser. The map key is case insensitive.
+  // Options for the csv/json/xml parser. The map key is case insensitive.
   map<string, string> options = 4;
   enum ParseFormat {
     PARSE_FORMAT_UNSPECIFIED = 0;
     PARSE_FORMAT_CSV = 1;
     PARSE_FORMAT_JSON = 2;
+    PARSE_FORMAT_XML = 3;
   }
 }
 
@@ -1214,3 +1278,45 @@ message LateralJoin {
   // (Required) The join type.
   Join.JoinType join_type = 4;
 }
+
+// Relation of type [[NearestByJoin]].
+//
+// For each row on the left side, returns up to `num_results` rows from the right side ranked
+// by `ranking_expression`.
+message NearestByJoin {
+  // (Required) Left (query) input relation.
+  Relation left = 1;
+
+  // (Required) Right (base) input relation.
+  Relation right = 2;
+
+  // (Required) Scalar expression used to rank candidate rows on the right side.
+  Expression ranking_expression = 3;
+
+  // (Required) Maximum number of matches per left row. Must be between 1 and 100000.
+  int32 num_results = 4;
+
+  // The following three fields use `string` (not typed enums) for parity with `AsOfJoin`,
+  // which models analogous fields the same way. Validation happens server-side at planning time.
+
+  // (Required) The join type. Must be one of: "inner", "leftouter".
+  string join_type = 5;
+
+  // (Required) Search algorithm contract. Must be one of: "approx", "exact".
+  string mode = 6;
+
+  // (Required) Ranking direction. Must be one of: "distance", "similarity".
+  string direction = 7;
+}
+
+// Relation of type [[Zip]].
+//
+// Combines the columns of two DataFrames side-by-side. Both DataFrames must produce the same
+// canonicalized plan after stripping outer Project chains.
+message Zip {
+  // (Required) Left input relation.
+  Relation left = 1;
+
+  // (Required) Right input relation.
+  Relation right = 2;
+}
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/types.proto b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/types.proto
index 6b4ee84..90ec5d0 100644
--- a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/types.proto
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/types.proto
@@ -67,8 +67,20 @@ message DataType {
     // UserDefinedType
     UDT udt = 23;
 
+    // Geospatial types
+    Geometry geometry = 26;
+
+    Geography geography = 27;
+
     // UnparsedDataType
     Unparsed unparsed = 24;
+
+    Time time = 28;
+
+    // Nanosecond-capable timestamp types (precision 7..9). NTZ and LTZ are distinct kinds
+    // even though their physical value is identical, mirroring timestamp vs timestamp_ntz.
+    TimestampNTZNanos timestamp_ntz_nanos = 29;
+    TimestampLTZNanos timestamp_ltz_nanos = 30;
   }
 
   message Boolean {
@@ -124,6 +136,25 @@ message DataType {
     uint32 type_variation_reference = 1;
   }
 
+  message Time {
+    optional int32 precision = 1;
+    uint32 type_variation_reference = 2;
+  }
+
+  // Timestamp without time zone with nanosecond-capable fractional-second precision.
+  message TimestampNTZNanos {
+    // Number of fractional-second digits. Valid values are 7, 8, and 9. If omitted, defaults to 9.
+    optional int32 precision = 1;
+    uint32 type_variation_reference = 2;
+  }
+
+  // Timestamp with local time zone with nanosecond-capable fractional-second precision.
+  message TimestampLTZNanos {
+    // Number of fractional-second digits. Valid values are 7, 8, and 9. If omitted, defaults to 9.
+    optional int32 precision = 1;
+    uint32 type_variation_reference = 2;
+  }
+
   message CalendarInterval {
     uint32 type_variation_reference = 1;
   }
@@ -182,6 +213,16 @@ message DataType {
     uint32 type_variation_reference = 4;
   }
 
+  message Geometry {
+    int32 srid = 1;
+    uint32 type_variation_reference = 2;
+  }
+
+  message Geography {
+    int32 srid = 1;
+    uint32 type_variation_reference = 2;
+  }
+
   message Variant {
     uint32 type_variation_reference = 1;
   }