You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ch...@apache.org on 2026/10/28 13:27:24 UTC
(kyuubi-shaded) branch master updated: [KYUUBI-SHADED #72] Sync Spark Connect pb from Spark 4.3.0 RC3 and bump gRPC 1.76.3, protobuf 3.25.9
This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi-shaded.git
The following commit(s) were added to refs/heads/master by this push:
new e285e7b [KYUUBI-SHADED #72] Sync Spark Connect pb from Spark 4.3.0 RC3 and bump gRPC 1.76.3, protobuf 3.25.9
e285e7b is described below
commit e285e7bc01a997919f413879e092c08264d9d913
Author: Cheng Pan <pa...@gmail.com>
AuthorDate: Wed Oct 28 21:27:15 2026 +0800
[KYUUBI-SHADED #72] Sync Spark Connect pb from Spark 4.3.0 RC3 and bump gRPC 1.76.3, protobuf 3.25.9
### _Why are the changes needed?_
Sync pb files from [Spark 4.3.0 RC3](https://github.com/apache/spark/tree/v4.3.0-rc3/sql/connect/common/src/main/protobuf), which adds `pipelines.proto`.
Also bump gRPC from 1.65.1 to 1.76.3 and protobuf from 3.25.1 to 3.25.9.
```
$ git log --oneline v4.0.0-rc4..v4.3.0-rc3 -- sql/connect/common/src/main/protobuf/spark/connect
```
```
5cdb1a6e908 [SPARK-58247][CONNECT][SDP] Support SCD Type 2 in the Auto CDC Python and Spark Connect APIs
a5dcd0fa358 [SPARK-57160][CONNECT][FOLLOWUP] Use signed nanos timestamp fields
b60e72c9fc8 [SPARK-57160][CONNECT] Add Spark Connect protocol support for nanosecond-capable timestamp types and literals
f30a7a0e650 [SPARK-57247][SQL][CONNECT] Support DataFrame.zip in Spark Connect
f6312fbfc35 [SPARK-56650][SDP][CONNECT] Add AutoCDC Spark Connect APIs
39681148eb7 [SPARK-56395][CONNECT][PYTHON] Add NEAREST BY DataFrame API
1186f023c1d [SPARK-55689][SQL] Add withSchemaEvolution() in dataframe writer API
190e3b857c1 [SPARK-56254][PYTHON][CONNECT] Make spark.read.xml accept DataFrame input
a3930d31665 [SPARK-56221][SQL][PYTHON][FOLLOWUP] Rename TablePartition and remove SHOW CACHED TABLES / listCachedTables
91d62cef365 [SPARK-56221][SQL][PYTHON] Feature parity between spark.catalog.* vs DDL commands
58841fa416a [SPARK-55949][SQL][CONNECT] Add DataFrame API and Spark Connect support for CDC queries
bff9dcfbfb8 [SPARK-55945][SDP] Support structured identifiers for flows in SDP eager analysis protos
7da1356837e [SPARK-55264] Add ExecuteOutput command to Spark Connect pipelines proto
7de85bad591 [SPARK-55314][CONNECT] Propagate observed metrics errors to client
a263a5e4ce8 [SPARK-55280][CONNECT] Add GetStatus proto to support execution status monitoring
d069f733e95 [SPARK-55104][CONNECT][PYTHON][SS] Add Spark Connect support for DataStreamReader.name()
7009fb58a98 [SPARK-54660][SS] Add RTM trigger to python
cc72c647b6f [SPARK-54020] Support `spark.sql(...)` Python API inside query functions for Spark Declarative Pipeline
0ccaacf54ce [SPARK-54194][CONNECT] Spark Connect Proto Plan Compression
1a724ba6fea [SPARK-54191][SDP] Add once to Defineflow Proto
a927a14a59e [SPARK-52463][SDP] Add support for cluster_by in Python Pipelines APIs
32060bf99fc [SPARK-54175][GEO][CONNECT] Add Geography and Geometry types to Spark Connect proto
4622e8584e3 [SPARK-52510][GEO][CONNECT][FOLLOWUP] Reserve expression proto numbers for Geometry and Geography
c707f597e9c Add PipelineAnalysisContext message to support pipeline analysis during Spark Connect query execution
db81309dd80 [SPARK-53917][CONNECT] Support large local relations
264ca4dc320 [SPARK-53455][CONNECT] Add `CloneSession` RPC
418cf56ed4d [SPARK-53796][SDP] Add `extension` field to a few pipeline protos to support forward compatibility
efcc8f69a01 [SPARK-53850][SDP] Define proto for Sinks and Rename DefineDataset to DefineOutput
22e24dfd7e8 [SPARK-53812][SDP] Refactor DefineDataset and DefineFlow protos to group related properties and future-proof
9a1c742098c [SPARK-53751][SDP] Explicit Versioned Checkpoint Location
be1de5b5733 [SPARK-53802][SDP] Support string values for user-specified schema in SDP tables
65ff85a31fe [SPARK-52640][SDP] Propagate Python Source Code Location
e04fd595370 [SPARK-52807][SDP] Proto changes to support analysis inside Declarative Pipelines query functions
e56ab2fdaea [SPARK-53593][SDP] Add response field for DefineDataset and DefineFlow RPC
db13a38e565 [SPARK-53578][CONNECT] Simplify data type handling in LiteralValueProtoConverter
010d36f2194 [SPARK-53507][CONNECT] Add breaking change info to errors
bb41e19ae9e [SPARK-53525][CONNECT] Spark Connect ArrowBatch Result Chunking
e75ca577923 [SPARK-53402][CONNECT] Support Direct Passthrough Partitioning Dataset API in Spark Connect in Scala
1d09e5a470f [SPARK-52449][CONNECT][PYTHON][ML] Make datatypes for Expression.Literal.Map/Array optional
b5840e15603 [SPARK-52930][CONNECT] Use DataType.Array/Map for Array/Map Literals
12c87cef0d0 [SPARK-53328][ML][CONNECT] Improve debuggability for SparkML-connect
39fbf594fa7 [SPARK-52448][CONNECT] Add simplified Struct Expression.Literal
4a450545754 [SPARK-52851][SDP] Remove public APIs for append once flows
93748ccc597 [SPARK-52511][SDP] Support dry-run mode in spark-pipelines command
9204b055894 [SPARK--52810][SDP][SQL] Spark Pipelines CLI Selection Options
fac4433f239 [SPARK-52779][SQL][CONNECT] Support TimeType literal in Connect
c84629ed173 [SPARK-52770][CONNECT] Support `TIME` type in connect proto
7208875483d [SPARK-52757][CONNECT] Rename "plan" field in DefineFlow to "relation"
73ca5e83d20 [SPARK-52510] Reserve proto numbers for Geospatial types
fd74b5ec6bd [SPARK-52470][ML][CONNECT] Support model summary offloading
46de57b5511 [SPARK-52423][CONNECT] Add missing golang package name to pipelines.proto
8c3194fbb8d [SPARK-52348][CONNECT] Add support for Spark Connect handlers for pipeline commands
da68d10206d [SPARK-52223][CONNECT] Add SDP Spark Connect Protos
b50690a89a8 [SPARK-52057][ML][CONNECT] Collect Tree size limit warning messages to client
b557311d439 [SPARK-51845][ML][CONNECT] Add proto messages `CleanCache` and `GetCacheInfo`
b4faf063e37 [SPARK-50131][SQL] Add IN Subquery DataFrame API
c83b88a037e [SPARK-51650][ML][CONNECT] Support delete ml cached objects in batch
89e05a88deb [SPARK-51326][CONNECT] Remove LazyExpression proto message
93d10c25d5b [SPARK-50856][SS][PYTHON][CONNECT] Spark Connect Support for TransformWithStateInPandas In Python
6a71f76f8aa [SPARK-51142][ML][CONNECT] ML protobufs clean up
4953a9cfabe [SPARK-49308][CONNECT] Support UserDefinedAggregateFunction in Spark Connect Scala Client
ebcec7c02f3 [SPARK-50104][CONNECT] Support SparkSession.executeCommand in Connect
e037cb119a2 [SPARK-50133][PYTHON][CONNECT] Support DataFrame conversion to table argument in Spark Connect Python Client
efeb1e01ce4 [SPARK-50893][CONNECT] Mark UDT.DataType optional
c7df014aab1 [SPARK-49273][CONNECT][SQL] Origin support for Spark Connect Scala client
ef363b636b2 [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet
2721a50234b [SPARK-50851][ML][CONNECT][PYTHON] Express ML params with `proto.Expression.Literal`
```
### _How was this patch tested?_
`build/mvn clean package -pl kyuubi-relocated-spark-connect-rpc` passes, and the jar contains the relocated `Pipeline*` classes.
### _Was this patch assisted by generative AI tooling?_
Assisted-by: Claude Opus 5.5
Closes #72 from pan3793/proto-4.3.0-rc3.
Closes #72
2366e7f [Cheng Pan] Bump gRPC 1.76.3 and protobuf 3.25.9
fa9fbef [Cheng Pan] Sync Spark Connect pb from Spark 4.3.0 RC3
Authored-by: Cheng Pan <pa...@gmail.com>
Signed-off-by: Cheng Pan <ch...@apache.org>
---
kyuubi-relocated-spark-connect-rpc/pom.xml | 4 +-
.../src/main/protobuf/README.md | 2 +-
.../src/main/protobuf/spark/connect/base.proto | 240 ++++++++++++-
.../src/main/protobuf/spark/connect/catalog.proto | 81 +++++
.../src/main/protobuf/spark/connect/commands.proto | 9 +
.../src/main/protobuf/spark/connect/common.proto | 6 +
.../main/protobuf/spark/connect/expressions.proto | 89 ++++-
.../src/main/protobuf/spark/connect/ml.proto | 27 ++
.../main/protobuf/spark/connect/pipelines.proto | 394 +++++++++++++++++++++
.../main/protobuf/spark/connect/relations.proto | 108 +++++-
.../src/main/protobuf/spark/connect/types.proto | 41 +++
11 files changed, 989 insertions(+), 12 deletions(-)
diff --git a/kyuubi-relocated-spark-connect-rpc/pom.xml b/kyuubi-relocated-spark-connect-rpc/pom.xml
index b2b2c99..413794d 100644
--- a/kyuubi-relocated-spark-connect-rpc/pom.xml
+++ b/kyuubi-relocated-spark-connect-rpc/pom.xml
@@ -32,8 +32,8 @@ under the License.
<description>Relocated Spark Connect RPC classes used by Kyuubi internally.</description>
<properties>
- <grpc.version>1.65.1</grpc.version>
- <protobuf.version>3.25.1</protobuf.version>
+ <grpc.version>1.76.3</grpc.version>
+ <protobuf.version>3.25.9</protobuf.version>
</properties>
<dependencyManagement>
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/README.md b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/README.md
index 500b4b1..e1c1c51 100644
--- a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/README.md
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/README.md
@@ -1,5 +1,5 @@
proto files are copied from
-https://github.com/apache/spark/tree/v4.0.0-rc4/sql/connect/common/src/main/protobuf
+https://github.com/apache/spark/tree/v4.3.0-rc3/sql/connect/common/src/main/protobuf
and with one additional change in each proto file
```patch
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/base.proto b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/base.proto
index 38819f4..decc7dd 100644
--- a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/base.proto
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/base.proto
@@ -26,23 +26,42 @@ import "spark/connect/expressions.proto";
import "spark/connect/relations.proto";
import "spark/connect/types.proto";
import "spark/connect/ml.proto";
+import "spark/connect/pipelines.proto";
option java_multiple_files = true;
option java_package = "org.apache.kyuubi.shaded.spark.connect.proto";
option go_package = "internal/generated";
// A [[Plan]] is the structure that carries the runtime information for the execution from the
-// client to the server. A [[Plan]] can either be of the type [[Relation]] which is a reference
-// to the underlying logical plan or it can be of the [[Command]] type that is used to execute
-// commands on the server.
+// client to the server. A [[Plan]] can be one of the following:
+// - [[Relation]]: a reference to the underlying logical plan.
+// - [[Command]]: used to execute commands on the server.
+// - [[CompressedOperation]]: a compressed representation of either a Relation or a Command.
message Plan {
oneof op_type {
Relation root = 1;
Command command = 2;
+ CompressedOperation compressed_operation = 3;
}
-}
+ message CompressedOperation {
+ bytes data = 1;
+ OpType op_type = 2;
+ CompressionCodec compression_codec = 3;
+ enum OpType {
+ OP_TYPE_UNSPECIFIED = 0;
+ OP_TYPE_RELATION = 1;
+ OP_TYPE_COMMAND = 2;
+ }
+ }
+}
+
+// Compression codec for plan compression.
+enum CompressionCodec {
+ COMPRESSION_CODEC_UNSPECIFIED = 0;
+ COMPRESSION_CODEC_ZSTD = 1;
+}
// User Context is used to refer to one particular user session that is executing
// queries in the backend.
@@ -332,6 +351,7 @@ message ExecutePlanRequest {
message RequestOption {
oneof request_option {
ReattachOptions reattach_options = 1;
+ ResultChunkingOptions result_chunking_options = 2;
// Extension type for request options
google.protobuf.Any extension = 999;
}
@@ -399,6 +419,16 @@ message ExecutePlanResponse {
// ML command response
MlCommandResult ml_command_result = 20;
+ // Response containing pipeline event that is streamed back to the client during a pipeline run
+ PipelineEventResult pipeline_event_result = 21;
+
+ // Pipeline command response
+ PipelineCommandResult pipeline_command_result = 22;
+
+ // A signal from the server to the client to execute the query function for a flow, and to
+ // register its result with the server.
+ PipelineQueryFunctionExecutionSignal pipeline_query_function_execution_signal = 23;
+
// Support arbitrary result objects.
google.protobuf.Any extension = 999;
}
@@ -428,6 +458,14 @@ message ExecutePlanResponse {
// If set, row offset of the start of this ArrowBatch in execution results.
optional int64 start_offset = 3;
+
+ // Index of this chunk in the batch if chunking is enabled. The index starts from 0.
+ optional int64 chunk_index = 4;
+
+ // Total number of chunks in this batch if chunking is enabled.
+ // It is missing when chunking is disabled - the batch is returned whole
+ // and client will treat this response as the batch.
+ optional int64 num_chunks_in_batch = 5;
}
message Metrics {
@@ -453,6 +491,12 @@ message ExecutePlanResponse {
repeated Expression.Literal values = 2;
repeated string keys = 3;
int64 plan_id = 4;
+ // (Optional) The index of the root error in errors.
+ // The field will not be set if there are no errors.
+ optional int32 root_error_idx = 5;
+ // A list of errors that occurred while collecting the observed metrics.
+ // If the length is 0, it means no errors occurred.
+ repeated FetchErrorDetailsResponse.Error errors = 6;
}
message ResultComplete {
@@ -812,6 +856,24 @@ message ReattachOptions {
bool reattachable = 1;
}
+message ResultChunkingOptions {
+ // Although Arrow results are split into batches with a size limit according to estimation, the
+ // size of the batches is not guaranteed to be less than the limit, especially when a single row
+ // is larger than the limit, in which case the server will fail to split it further into smaller
+ // batches. As a result, the client may encounter a gRPC error stating “Received message larger
+ // than max” when a batch is too large.
+ // If allow_arrow_batch_chunking=true, the server will split large Arrow batches into smaller chunks,
+ // and the client is expected to handle the chunked Arrow batches.
+ //
+ // If false, the server will not chunk large Arrow batches.
+ bool allow_arrow_batch_chunking = 1;
+
+ // Optional preferred Arrow batch size in bytes for the server to use when sending Arrow results.
+ // The server will attempt to use this size if it is set and within the valid range
+ // ([1KB, max batch size on server]). Otherwise, the server's maximum batch size is used.
+ optional int64 preferred_arrow_chunk_size = 2;
+}
+
message ReattachExecuteRequest {
// (Required)
//
@@ -1061,6 +1123,32 @@ message FetchErrorDetailsResponse {
// Portable error identifier across SQL engines
// If null, error class or SQLSTATE is not set.
optional string sql_state = 4;
+
+ // Additional information if the error was caused by a breaking change.
+ optional BreakingChangeInfo breaking_change_info = 5;
+ }
+
+ // BreakingChangeInfo defines the schema for breaking change information.
+ message BreakingChangeInfo {
+ // A message explaining how the user can migrate their job to work
+ // with the breaking change.
+ repeated string migration_message = 1;
+
+ // A spark config flag that can be used to mitigate the breaking change.
+ optional MitigationConfig mitigation_config = 2;
+
+ // If true, the breaking change should be inspected manually.
+ // If false, the spark job should be retried by setting the mitigationConfig.
+ optional bool needs_audit = 3;
+ }
+
+ // MitigationConfig defines a spark config flag that can be used to mitigate a breaking change.
+ message MitigationConfig {
+ // The spark config key.
+ string key = 1;
+
+ // The spark config value that mitigates the breaking change.
+ string value = 2;
}
// Error defines the schema for the representing exception.
@@ -1088,6 +1176,137 @@ message CheckpointCommandResult {
CachedRemoteRelation relation = 1;
}
+message CloneSessionRequest {
+ // (Required)
+ //
+ // The session_id specifies a spark session for a user id (which is specified
+ // by user_context.user_id). The session_id is set by the client to be able to
+ // collate streaming responses from different queries within the dedicated session.
+ // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+ string session_id = 1;
+
+ // (Optional)
+ //
+ // Server-side generated idempotency key from the previous responses (if any). Server
+ // can use this to validate that the server side session has not changed.
+ optional string client_observed_server_side_session_id = 5;
+
+ // (Required) User context
+ //
+ // user_context.user_id and session_id both identify a unique remote spark session on the
+ // server side.
+ UserContext user_context = 2;
+
+ // Provides optional information about the client sending the request. This field
+ // can be used for language or version specific information and is only intended for
+ // logging purposes and will not be interpreted by the server.
+ optional string client_type = 3;
+
+ // (Optional)
+ // The session_id for the new cloned session. If not provided, a new UUID will be generated.
+ // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+ optional string new_session_id = 4;
+}
+
+// Next ID: 5
+message CloneSessionResponse {
+ // Session id of the original session that was cloned.
+ string session_id = 1;
+
+ // Server-side generated idempotency key that the client can use to assert that the server side
+ // session (parent of the cloned session) has not changed.
+ string server_side_session_id = 2;
+
+ // Session id of the new cloned session.
+ string new_session_id = 3;
+
+ // Server-side session ID of the new cloned session.
+ string new_server_side_session_id = 4;
+}
+
+// Next ID: 6
+message GetStatusRequest {
+ // (Required)
+ //
+ // The session_id specifies a Spark session for a user identified by user_context.user_id.
+ // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+ string session_id = 1;
+
+ // (Required)
+ //
+ // user_context.user_id and session_id both identify a unique remote spark session on the
+ // server side.
+ UserContext user_context = 2;
+
+ // (Optional)
+ //
+ // Provides optional information about the client sending the request. This field
+ // can be used for language or version specific information and is only intended for
+ // logging purposes and will not be interpreted by the server.
+ optional string client_type = 3;
+
+ // (Optional)
+ //
+ // Server-side generated idempotency key from the previous responses (if any). Server
+ // can use this to validate that the server side session has not changed.
+ optional string client_observed_server_side_session_id = 4;
+
+ // (Optional)
+ //
+ // Get status of operations in the session.
+ optional OperationStatusRequest operation_status = 5;
+
+ // Extension point for custom status request types.
+ repeated google.protobuf.Any extensions = 999;
+
+ message OperationStatusRequest {
+ // Get status of operations with these operation_ids.
+ // If unset or empty, returns status of all operations in the session.
+ repeated string operation_ids = 1;
+
+ // Extension point for custom operation-level status requests.
+ repeated google.protobuf.Any extensions = 999;
+ }
+}
+
+// Next ID: 4
+message GetStatusResponse {
+ // Session id of the session for which the status was requested.
+ string session_id = 1;
+
+ // Server-side generated idempotency key that the client can use to assert that the server side
+ // session has not changed.
+ string server_side_session_id = 2;
+
+ // Status information about requested operations.
+ repeated OperationStatus operation_statuses = 3;
+
+ // Extension point for custom status response types.
+ repeated google.protobuf.Any extensions = 999;
+
+ // Status information for a single operation.
+ message OperationStatus {
+ // The operation_id of the operation.
+ string operation_id = 1;
+
+ // The current status of the operation.
+ OperationState state = 2;
+
+ // Extension point for custom operation-level status fields.
+ repeated google.protobuf.Any extensions = 999;
+
+ enum OperationState {
+ OPERATION_STATE_UNSPECIFIED = 0;
+ OPERATION_STATE_UNKNOWN = 1;
+ OPERATION_STATE_RUNNING = 2;
+ OPERATION_STATE_TERMINATING = 3;
+ OPERATION_STATE_SUCCEEDED = 4;
+ OPERATION_STATE_FAILED = 5;
+ OPERATION_STATE_CANCELLED = 6;
+ }
+ }
+}
+
// Main interface for the SparkConnect service.
service SparkConnectService {
@@ -1132,4 +1351,17 @@ service SparkConnectService {
// FetchErrorDetails retrieves the matched exception with details based on a provided error id.
rpc FetchErrorDetails(FetchErrorDetailsRequest) returns (FetchErrorDetailsResponse) {}
+
+ // Create a clone of a Spark Connect session on the server side. The server-side session
+ // is cloned with all its current state (SQL configurations, temporary views, registered
+ // functions, catalog state) copied over to a new independent session. The cloned session
+ // is isolated from the source session - any subsequent changes to either session's
+ // server-side state will not be reflected in the other.
+ //
+ // The request can optionally specify a custom session ID for the cloned session (must be
+ // a valid UUID). If not provided, a new UUID will be generated automatically.
+ rpc CloneSession(CloneSessionRequest) returns (CloneSessionResponse) {}
+
+ // Get status information of different types.
+ rpc GetStatus(GetStatusRequest) returns (GetStatusResponse) {}
}
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/catalog.proto b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/catalog.proto
index 06e6282..91e6674 100644
--- a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/catalog.proto
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/catalog.proto
@@ -55,6 +55,16 @@ message Catalog {
CurrentCatalog current_catalog = 24;
SetCurrentCatalog set_current_catalog = 25;
ListCatalogs list_catalogs = 26;
+ DropTable drop_table = 27;
+ DropView drop_view = 28;
+ CreateDatabase create_database = 29;
+ DropDatabase drop_database = 30;
+ ListPartitions list_partitions = 31;
+ ListViews list_views = 32;
+ GetTableProperties get_table_properties = 33;
+ GetCreateTableString get_create_table_string = 34;
+ TruncateTable truncate_table = 35;
+ AnalyzeTable analyze_table = 36;
}
}
@@ -241,3 +251,74 @@ message ListCatalogs {
// (Optional) The pattern that the catalog name needs to match
optional string pattern = 1;
}
+
+// See `spark.catalog.dropTable`
+message DropTable {
+ // (Required)
+ string table_name = 1;
+ bool if_exists = 2;
+ bool purge = 3;
+}
+
+// See `spark.catalog.dropView`
+message DropView {
+ // (Required)
+ string view_name = 1;
+ bool if_exists = 2;
+}
+
+// See `spark.catalog.createDatabase`
+message CreateDatabase {
+ // (Required)
+ string db_name = 1;
+ bool if_not_exists = 2;
+ map<string, string> properties = 3;
+}
+
+// See `spark.catalog.dropDatabase`
+message DropDatabase {
+ // (Required)
+ string db_name = 1;
+ bool if_exists = 2;
+ bool cascade = 3;
+}
+
+// See `spark.catalog.listPartitions`
+message ListPartitions {
+ // (Required)
+ string table_name = 1;
+}
+
+// See `spark.catalog.listViews`
+message ListViews {
+ // (Optional)
+ optional string db_name = 1;
+ // (Optional) The pattern that the view name needs to match
+ optional string pattern = 2;
+}
+
+// See `spark.catalog.getTableProperties`
+message GetTableProperties {
+ // (Required)
+ string table_name = 1;
+}
+
+// See `spark.catalog.getCreateTableString`
+message GetCreateTableString {
+ // (Required)
+ string table_name = 1;
+ bool as_serde = 2;
+}
+
+// See `spark.catalog.truncateTable`
+message TruncateTable {
+ // (Required)
+ string table_name = 1;
+}
+
+// See `spark.catalog.analyzeTable`
+message AnalyzeTable {
+ // (Required)
+ string table_name = 1;
+ bool no_scan = 2;
+}
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/commands.proto b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/commands.proto
index e0fb306..38f06fd 100644
--- a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/commands.proto
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/commands.proto
@@ -22,6 +22,7 @@ import "spark/connect/common.proto";
import "spark/connect/expressions.proto";
import "spark/connect/relations.proto";
import "spark/connect/ml.proto";
+import "spark/connect/pipelines.proto";
package spark.connect;
@@ -51,6 +52,7 @@ message Command {
MergeIntoTableCommand merge_into_table_command = 16;
MlCommand ml_command = 17;
ExecuteExternalCommand execute_external_command = 18;
+ PipelineCommand pipeline_command = 19;
// This field is used to mark extensions to the protocol. When plugins generate arbitrary
// Commands they can add them here. During the planning the correct resolution is done.
@@ -142,6 +144,9 @@ message WriteOperation {
// (Optional) Columns used for clustering the table.
repeated string clustering_columns = 10;
+ // (Optional) Whether schema evolution is enabled for the write.
+ bool with_schema_evolution = 11;
+
message SaveTable {
// (Required) The table name.
string table_name = 1;
@@ -209,6 +214,9 @@ message WriteOperationV2 {
// (Optional) Columns used for clustering the table.
repeated string clustering_columns = 9;
+
+ // (Optional) Whether schema evolution is enabled for the write.
+ bool with_schema_evolution = 10;
}
// Starts write stream operation as streaming query. Query ID and Run ID of the streaming
@@ -230,6 +238,7 @@ message WriteStreamOperationStart {
bool available_now = 6;
bool once = 7;
string continuous_checkpoint_interval = 8;
+ string real_time_batch_duration = 100;
}
string output_mode = 9;
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/common.proto b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/common.proto
index c8e9f25..3079df6 100644
--- a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/common.proto
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/common.proto
@@ -148,6 +148,12 @@ message StackTraceElement {
int32 line_number = 7;
}
+message ResolvedIdentifier {
+ string catalog_name = 1;
+ repeated string namespace = 2;
+ string table_name = 3;
+}
+
message Bools {
repeated bool values = 1;
}
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/expressions.proto b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/expressions.proto
index b300c3a..8ce150a 100644
--- a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/expressions.proto
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/expressions.proto
@@ -53,6 +53,7 @@ message Expression {
MergeAction merge_action = 19;
TypedAggregateExpression typed_aggregate_expression = 20;
SubqueryExpression subquery_expression = 21;
+ DirectShufflePartitionID direct_shuffle_partition_id = 22;
// This field is used to mark extensions to the protocol. When plugins generate arbitrary
// relations they can add them here. During the planning the correct resolution is done.
@@ -142,6 +143,14 @@ message Expression {
}
}
+ // Expression that takes a partition ID value and passes it through directly for use in
+ // shuffle partitioning. This is used with RepartitionByExpression to allow users to
+ // directly specify target partition IDs.
+ message DirectShufflePartitionID {
+ // (Required) The expression that evaluates to the partition ID.
+ Expression child = 1;
+ }
+
message Cast {
// (Required) the expression to be casted.
Expression expr = 1;
@@ -195,8 +204,24 @@ message Expression {
Struct struct = 24;
SpecializedArray specialized_array = 25;
+ Time time = 26;
+
+ // Nanosecond-capable timestamp literals (precision 7..9). NTZ and LTZ are distinct
+ // arms so the literal kind is self-describing.
+ TimestampNTZNanos timestamp_ntz_nanos = 29;
+ TimestampLTZNanos timestamp_ltz_nanos = 30;
}
+ // Reserved for Geometry and Geography.
+ reserved 27, 28;
+
+ // Data type information for the literal.
+ // This field is required only in the root literal message for null values or
+ // for data types (e.g., array, map, or struct) with non-trivial information.
+ // If the data_type field is not set at the root level, the data type will be
+ // inferred or retrieved from the deprecated data type fields using best efforts.
+ DataType data_type = 100;
+
message Decimal {
// the string representation.
string value = 1;
@@ -214,19 +239,42 @@ message Expression {
}
message Array {
- DataType element_type = 1;
+ // (Deprecated) The element type of the array.
+ //
+ // This field is deprecated since Spark 4.1+. Use data_type field instead.
+ DataType element_type = 1 [deprecated = true];
+
+ // The literal values that make up the array elements.
repeated Literal elements = 2;
}
message Map {
- DataType key_type = 1;
- DataType value_type = 2;
+ // (Deprecated) The key type of the map.
+ //
+ // This field is deprecated since Spark 4.1+. Use data_type field instead.
+ DataType key_type = 1 [deprecated = true];
+
+ // (Deprecated) The value type of the map.
+ //
+ // This field is deprecated since Spark 4.1+ and should only be set
+ // if the data_type field is not set. Use data_type field instead.
+ DataType value_type = 2 [deprecated = true];
+
+ // The literal keys that make up the map.
repeated Literal keys = 3;
+
+ // The literal values that make up the map.
repeated Literal values = 4;
}
message Struct {
- DataType struct_type = 1;
+ // (Deprecated) The type of the struct.
+ //
+ // This field is deprecated since Spark 4.1+ because using DataType as the type of a struct
+ // is ambiguous. Use data_type field instead.
+ DataType struct_type = 1 [deprecated = true];
+
+ // The literal values that make up the struct elements.
repeated Literal elements = 2;
}
@@ -240,6 +288,35 @@ message Expression {
Strings strings = 6;
}
}
+
+ message Time {
+ int64 nano = 1;
+ // The precision of this time, if omitted, uses the default value of MICROS_PRECISION.
+ optional int32 precision = 2;
+ }
+
+ // A TIMESTAMP_NTZ literal with nanosecond-capable precision. The physical value is carried
+ // as microseconds since the UNIX epoch plus the extra nanoseconds within that microsecond,
+ // because a single int64 of nanoseconds cannot span the supported year range.
+ message TimestampNTZNanos {
+ // Microseconds since the UNIX epoch (without timezone information).
+ int64 epoch_micros = 1;
+ // Additional nanoseconds within epoch_micros, in [0, 999].
+ int32 nanos_within_micro = 2;
+ // Number of fractional-second digits (7, 8, or 9). If omitted, defaults to 9 (nanoseconds).
+ optional int32 precision = 3;
+ }
+
+ // A TIMESTAMP_LTZ literal with nanosecond-capable precision. See TimestampNTZNanos for the
+ // rationale behind the two-component physical value.
+ message TimestampLTZNanos {
+ // Microseconds since the UNIX epoch.
+ int64 epoch_micros = 1;
+ // Additional nanoseconds within epoch_micros, in [0, 999].
+ int32 nanos_within_micro = 2;
+ // Number of fractional-second digits (7, 8, or 9). If omitted, defaults to 9 (nanoseconds).
+ optional int32 precision = 3;
+ }
}
// An unresolved attribute that is not explicitly bound to a specific column, but the column
@@ -483,11 +560,15 @@ message SubqueryExpression {
// (Optional) Options specific to table arguments.
optional TableArgOptions table_arg_options = 3;
+ // (Optional) IN subquery values.
+ repeated Expression in_subquery_values = 4;
+
enum SubqueryType {
SUBQUERY_TYPE_UNKNOWN = 0;
SUBQUERY_TYPE_SCALAR = 1;
SUBQUERY_TYPE_EXISTS = 2;
SUBQUERY_TYPE_TABLE_ARG = 3;
+ SUBQUERY_TYPE_IN = 4;
}
// Nested message for table argument options.
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/ml.proto b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/ml.proto
index d73101f..26da301 100644
--- a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/ml.proto
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/ml.proto
@@ -36,6 +36,10 @@ message MlCommand {
Write write = 4;
Read read = 5;
Evaluate evaluate = 6;
+ CleanCache clean_cache = 7;
+ GetCacheInfo get_cache_info = 8;
+ CreateSummary create_summary = 9;
+ GetModelSize get_model_size = 10;
}
// Command for estimator.fit(dataset)
@@ -52,8 +56,17 @@ message MlCommand {
// or summary evaluated by a model
message Delete {
repeated ObjectRef obj_refs = 1;
+ // if set `evict_only` to true, only evict the cached model from memory,
+ // but keep the offloaded model in Spark driver local disk.
+ optional bool evict_only = 2;
}
+ // Force to clean up all the ML cached objects
+ message CleanCache { }
+
+ // Get the information of all the ML cached objects
+ message GetCacheInfo { }
+
// Command to write ML operator
message Write {
// It could be an estimator/evaluator or the cached model
@@ -90,6 +103,18 @@ message MlCommand {
// (Required) the evaluating dataset
Relation dataset = 3;
}
+
+ // This is for re-creating the model summary when the model summary is lost
+ // (model summary is lost when the model is offloaded and then loaded back)
+ message CreateSummary {
+ ObjectRef model_ref = 1;
+ Relation dataset = 2;
+ }
+
+ // This is for query the model estimated in-memory size
+ message GetModelSize {
+ ObjectRef model_ref = 1;
+ }
}
// The result of MlCommand
@@ -116,5 +141,7 @@ message MlCommandResult {
optional string uid = 3;
// (Optional) parameters
optional MlParams params = 4;
+ // (Optional) warning message generated during the ML command execution
+ optional string warning_message = 5;
}
}
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/pipelines.proto b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/pipelines.proto
new file mode 100644
index 0000000..df597dc
--- /dev/null
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/pipelines.proto
@@ -0,0 +1,394 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+package spark.connect;
+
+import "google/protobuf/any.proto";
+import "google/protobuf/timestamp.proto";
+import "spark/connect/common.proto";
+import "spark/connect/expressions.proto";
+import "spark/connect/relations.proto";
+import "spark/connect/types.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.kyuubi.shaded.spark.connect.proto";
+option go_package = "internal/generated";
+
+// Dispatch object for pipelines commands. See each individual command for documentation.
+message PipelineCommand {
+ oneof command_type {
+ CreateDataflowGraph create_dataflow_graph = 1;
+ DefineOutput define_output = 2;
+ DefineFlow define_flow = 3;
+ DropDataflowGraph drop_dataflow_graph = 4;
+ StartRun start_run = 5;
+ DefineSqlGraphElements define_sql_graph_elements = 6;
+ GetQueryFunctionExecutionSignalStream get_query_function_execution_signal_stream = 7;
+ DefineFlowQueryFunctionResult define_flow_query_function_result = 8;
+ ExecuteOutputFlows execute_output_flows = 9;
+
+ // Reserved field for protocol extensions.
+ // Used to support forward-compatibility by carrying additional command types
+ // that are not yet defined in this version of the proto. During planning, the
+ // engine will resolve and dispatch the concrete command contained in this field.
+ google.protobuf.Any extension = 999;
+ }
+
+ // Request to create a new dataflow graph.
+ message CreateDataflowGraph {
+ // The default catalog.
+ optional string default_catalog = 1;
+
+ // The default database.
+ optional string default_database = 2;
+
+ // SQL configurations for all flows in this graph.
+ map<string, string> sql_conf = 5;
+ }
+
+ // Drops the graph and stops any running attached flows.
+ message DropDataflowGraph {
+ // The graph to drop.
+ optional string dataflow_graph_id = 1;
+ }
+
+ // Request to define an output: a table, a materialized view, a temporary view or a sink.
+ message DefineOutput {
+ // The graph to attach this output to.
+ optional string dataflow_graph_id = 1;
+
+ // Name of the output. Can be partially or fully qualified.
+ optional string output_name = 2;
+
+ // The type of the output.
+ optional OutputType output_type = 3;
+
+ // Optional comment for the output.
+ optional string comment = 4;
+
+ // The location in source code that this output was defined.
+ optional SourceCodeLocation source_code_location = 5;
+
+ oneof details {
+ TableDetails table_details = 6;
+ SinkDetails sink_details = 7;
+ google.protobuf.Any extension = 999;
+ }
+
+ // Metadata that's only applicable to tables and materialized views.
+ message TableDetails {
+ // Optional table properties.
+ map<string, string> table_properties = 1;
+
+ // Optional partition columns for the table.
+ repeated string partition_cols = 2;
+
+ // The output table format for the table.
+ optional string format = 3;
+
+ // Schema for the table. If unset, this will be inferred from incoming flows.
+ oneof schema {
+ spark.connect.DataType schema_data_type = 4;
+ string schema_string = 5;
+ }
+
+ // Optional cluster columns for the table.
+ repeated string clustering_columns = 6;
+ }
+
+ // Metadata that's only applicable to sinks.
+ message SinkDetails {
+ // Streaming write options
+ map<string, string> options = 1;
+
+ // Streaming write format
+ optional string format = 2;
+ }
+ }
+
+ // Request to define a flow targeting a dataset.
+ message DefineFlow {
+ // The graph to attach this flow to.
+ optional string dataflow_graph_id = 1;
+
+ // Name of the flow. For standalone flows, this must be a single-part name.
+ optional string flow_name = 2;
+
+ // Name of the dataset this flow writes to. Can be partially or fully qualified.
+ optional string target_dataset_name = 3;
+
+ // SQL configurations set when running this flow.
+ map<string, string> sql_conf = 4;
+
+ // Identifier for the client making the request. The server uses this to determine what flow
+ // evaluation request stream to dispatch evaluation requests to for this flow.
+ optional string client_id = 5;
+
+ // The location in source code that this flow was defined.
+ optional SourceCodeLocation source_code_location = 6;
+
+ oneof details {
+ WriteRelationFlowDetails relation_flow_details = 7;
+ AutoCdcFlowDetails auto_cdc_flow_details = 10;
+ google.protobuf.Any extension = 999;
+ }
+
+ // A flow that is that takes the contents of a relation and writes it to the target dataset.
+ message WriteRelationFlowDetails {
+ // An unresolved relation that defines the dataset's flow. Empty if the query function
+ // that defines the flow cannot be analyzed at the time of flow definition.
+ optional spark.connect.Relation relation = 1;
+ }
+
+ // Details for Auto CDC flows.
+ message AutoCdcFlowDetails {
+ // The name of the CDC source to stream from.
+ optional string source = 1;
+
+ // Column(s) that uniquely identify a row in source and target data.
+ repeated Expression keys = 2;
+
+ // Expression to order the source data.
+ optional Expression sequence_by = 3;
+
+ // Delete condition for the merged operation.
+ optional Expression apply_as_deletes = 6;
+
+ // Truncate condition for the merged operation.
+ optional Expression apply_as_truncates = 7;
+
+ // Columns included in the output table.
+ repeated Expression column_list = 8;
+
+ // Columns excluded from the output table.
+ repeated Expression except_column_list = 9;
+
+ // SCD Type for target table.
+ SCDType stored_as_scd_type = 10;
+
+ // SCD2 only. Columns whose value change opens a new history record. When empty, every
+ // eligible selected user column is tracked.
+ repeated Expression track_history_column_list = 11;
+
+ // SCD2 only. Columns excluded from history tracking. Mutually exclusive with
+ // track_history_column_list.
+ repeated Expression track_history_except_column_list = 12;
+
+ // Subset of columns to ignore null in updates.
+ repeated Expression ignore_null_updates_column_list = 14;
+
+ // Subset of columns excluded from ignoring null in updates.
+ repeated Expression ignore_null_updates_except_column_list = 15;
+
+ }
+
+ // SCD Type for Auto CDC target tables.
+ enum SCDType {
+ SCD_TYPE_UNSPECIFIED = 0;
+ SCD_TYPE_1 = 1;
+ SCD_TYPE_2 = 2;
+ }
+
+ // If true, define the flow as a one-time flow, such as for backfill.
+ // Set to true changes the flow in two ways:
+ // - The flow is run one time by default. If the pipeline is ran with a full refresh,
+ // the flow will run again.
+ // - The flow function must be a batch DataFrame, not a streaming DataFrame.
+ optional bool once = 8;
+
+ message Response {
+ // Fully qualified flow name that uniquely identify a flow in the Dataflow graph.
+ optional string flow_name = 1;
+ }
+ }
+
+ // Request to execute all flows for a single output (dataset or sink) remotely.
+ message ExecuteOutputFlows {
+
+ // The output (table or materialized view or sink) definition.
+ optional DefineOutput define_output = 1;
+
+ // The flows to execute for this table.
+ repeated DefineFlow define_flows = 2;
+
+ // Whether to perform a full refresh instead of an incremental update.
+ optional bool full_refresh = 3;
+
+ // Storage location for pipeline checkpoints and metadata.
+ optional string storage = 4;
+
+ // Reserved field for protocol extensions.
+ repeated google.protobuf.Any extension = 999;
+ }
+
+ // Resolves all datasets and flows and start a pipeline update. Should be called after all
+ // graph elements are registered.
+ message StartRun {
+ // The graph to start.
+ optional string dataflow_graph_id = 1;
+
+ // List of dataset to reset and recompute.
+ repeated string full_refresh_selection = 2;
+
+ // Perform a full graph reset and recompute.
+ optional bool full_refresh_all = 3;
+
+ // List of dataset to update.
+ repeated string refresh_selection = 4;
+
+ // If true, the run will not actually execute any flows, but will only validate the graph and
+ // check for any errors. This is useful for testing and validation purposes.
+ optional bool dry = 5;
+
+ // storage location for pipeline checkpoints and metadata.
+ optional string storage = 6;
+ }
+
+ // Parses the SQL file and registers all datasets and flows.
+ message DefineSqlGraphElements {
+ // The graph to attach this dataset to.
+ optional string dataflow_graph_id = 1;
+
+ // The full path to the SQL file. Can be relative or absolute.
+ optional string sql_file_path = 2;
+
+ // The contents of the SQL file.
+ optional string sql_text = 3;
+ }
+
+ // Request to get the stream of query function execution signals for a graph. Responses should
+ // be a stream of PipelineQueryFunctionExecutionSignal messages.
+ message GetQueryFunctionExecutionSignalStream {
+ // The graph to get the query function execution signal stream for.
+ optional string dataflow_graph_id = 1;
+
+ // Identifier for the client that is requesting the stream.
+ optional string client_id = 2;
+ }
+
+ // Request from the client to update the flow function evaluation result
+ // for a previously un-analyzed flow.
+ message DefineFlowQueryFunctionResult {
+ // (Deprecated) The fully qualified name of the flow being updated.
+ //
+ // This field is deprecated since Spark 4.2+. Use flow_identifier field instead.
+ optional string flow_name = 1 [deprecated = true];
+
+ // The fully qualified identifier of the flow being updated.
+ optional ResolvedIdentifier flow_identifier = 4;
+
+ // The ID of the graph this flow belongs to.
+ optional string dataflow_graph_id = 2;
+
+ // An unresolved relation that defines the dataset's flow.
+ optional spark.connect.Relation relation = 3;
+ }
+}
+
+// Dispatch object for pipelines command results.
+message PipelineCommandResult {
+ oneof result_type {
+ CreateDataflowGraphResult create_dataflow_graph_result = 1;
+ DefineOutputResult define_output_result = 2;
+ DefineFlowResult define_flow_result = 3;
+ }
+ message CreateDataflowGraphResult {
+ // The ID of the created graph.
+ optional string dataflow_graph_id = 1;
+ }
+ message DefineOutputResult {
+ // Resolved identifier of the output
+ optional ResolvedIdentifier resolved_identifier = 1;
+ }
+ message DefineFlowResult {
+ // Resolved identifier of the flow
+ optional ResolvedIdentifier resolved_identifier = 1;
+ }
+}
+
+// The type of output.
+enum OutputType {
+ // Safe default value. Should not be used.
+ OUTPUT_TYPE_UNSPECIFIED = 0;
+ // A materialized view which is published to the catalog
+ MATERIALIZED_VIEW = 1;
+ // A table which is published to the catalog
+ TABLE = 2;
+ // A view which is not published to the catalog
+ TEMPORARY_VIEW = 3;
+ // A sink which is not published to the catalog
+ SINK = 4;
+}
+
+// A response containing an event emitted during the run of a pipeline.
+message PipelineEventResult {
+ PipelineEvent event = 1;
+}
+
+message PipelineEvent {
+ // The timestamp corresponding to when the event occurred.
+ google.protobuf.Timestamp timestamp = 1;
+ // The message that should be displayed to users.
+ optional string message = 2;
+}
+
+// Source code location information associated with a particular dataset or flow.
+message SourceCodeLocation {
+ // The file that this pipeline source code was defined in.
+ optional string file_name = 1;
+ // The specific line number that this pipeline source code is located at, if applicable.
+ optional int32 line_number = 2;
+ // The path of the top-level pipeline file determined at runtime during pipeline initialization.
+ optional string definition_path = 3;
+
+ // Reserved field for protocol extensions.
+ // Used to support forward-compatibility by carrying additional fields
+ // that are not yet defined in this version of the proto. During planning, the
+ // engine will resolve and dispatch the concrete command contained in this field.
+ repeated google.protobuf.Any extension = 999;
+}
+
+// A signal from the server to the client to execute the query function for one or more flows, and
+// to register their results with the server.
+message PipelineQueryFunctionExecutionSignal {
+ // (Deprecated) The name of flows that are ready to be re-evaluated.
+ //
+ // This field is deprecated since Spark 4.2+. Use flow_identifiers field instead.
+ repeated string flow_names = 1 [deprecated = true];
+
+ // The identifier of flows that are ready to be re-evaluated
+ repeated ResolvedIdentifier flow_identifiers = 2;
+}
+
+// Metadata providing context about the pipeline during Spark Connect query analysis.
+message PipelineAnalysisContext {
+ // Unique identifier of the dataflow graph associated with this pipeline.
+ optional string dataflow_graph_id = 1;
+ // The path of the top-level pipeline file determined at runtime during pipeline initialization.
+ optional string definition_path = 2;
+ // (Deprecated) The name of the Flow involved in this analysis
+ //
+ // This field is deprecated since Spark 4.2+. Use flow_identifier field instead.
+ optional string flow_name = 3 [deprecated = true];
+ // The identifier of the Flow involved in this analysis
+ optional ResolvedIdentifier flow_identifier = 4;
+
+ // Reserved field for protocol extensions.
+ repeated google.protobuf.Any extension = 999;
+}
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/relations.proto b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/relations.proto
index b401f7f..c03c7d5 100644
--- a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/relations.proto
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/relations.proto
@@ -80,6 +80,10 @@ message Relation {
Transpose transpose = 42;
UnresolvedTableValuedFunction unresolved_table_valued_function = 43;
LateralJoin lateral_join = 44;
+ ChunkedCachedLocalRelation chunked_cached_local_relation = 45;
+ RelationChanges relation_changes = 46;
+ NearestByJoin nearest_by_join = 47;
+ Zip zip = 48;
// NA functions
NAFill fill_na = 90;
@@ -115,6 +119,9 @@ message MlRelation {
Transform transform = 1;
Fetch fetch = 2;
}
+ // (Optional) the dataset for restoring the model summary
+ optional Relation model_summary_dataset = 3;
+
// Relation to represent transform(input) of the operator
// which could be a cached model or a new transformer
message Transform {
@@ -252,9 +259,33 @@ message Read {
//
// This is only supported by the JDBC data source.
repeated string predicates = 5;
+
+ // (Optional) A user-provided name for the streaming source.
+ // This name is used in checkpoint metadata and enables stable checkpoint locations
+ // for source evolution.
+ optional string source_name = 6;
}
}
+// Reads Change Data Capture (CDC) changes for a named table.
+//
+// This corresponds to the `DataFrameReader.changes()` or `DataStreamReader.changes()` API.
+// CDC-specific options (startingVersion, endingVersion, startingTimestamp, endingTimestamp,
+// deduplicationMode, computeUpdates, etc.) are passed in the options map.
+message RelationChanges {
+ // (Required) Unparsed identifier for the table.
+ string unparsed_identifier = 1;
+
+ // Options for the CDC query. The map key is case insensitive.
+ // Supported keys include: startingVersion, endingVersion, startingTimestamp,
+ // endingTimestamp, deduplicationMode, computeUpdates, startingBoundInclusive,
+ // endingBoundInclusive.
+ map<string, string> options = 2;
+
+ // (Optional) Indicates if this is a streaming CDC read.
+ bool is_streaming = 3;
+}
+
// Projection of a bag of expressions for a given input relation.
//
// The input relation must be specified.
@@ -496,6 +527,7 @@ message LocalRelation {
}
// A local relation that has been cached already.
+// CachedLocalRelation doesn't support LocalRelations of size over 2GB.
message CachedLocalRelation {
// `userId` and `sessionId` fields are deleted since the server must always use the active
// session/user rather than arbitrary values provided by the client. It is never valid to access
@@ -507,6 +539,20 @@ message CachedLocalRelation {
string hash = 3;
}
+// A local relation that has been cached already.
+message ChunkedCachedLocalRelation {
+ // (Required) A list of sha-256 hashes for representing LocalRelation.data.
+ // Data is serialized in Arrow IPC streaming format, each batch is cached on the server as
+ // a separate artifact. Each hash represents one batch stored on the server.
+ // Hashes are hex-encoded strings (e.g., "a3b2c1d4...").
+ repeated string dataHashes = 1;
+
+ // (Optional) A sha-256 hash of the serialized LocalRelation.schema.
+ // Scala clients always provide the schema, Python clients can omit it.
+ // Hash is a hex-encoded string (e.g., "a3b2c1d4...").
+ optional string schemaHash = 2;
+}
+
// Represents a remote relation that has been cached on server.
message CachedRemoteRelation {
// (Required) ID of the remote related (assigned by the service).
@@ -1031,6 +1077,23 @@ message GroupMap {
// (Optional) The schema for the grouped state.
optional DataType state_schema = 10;
+
+ // Below fields are used by TransformWithState and TransformWithStateInPandas
+ // (Optional) TransformWithState related parameters.
+ optional TransformWithStateInfo transform_with_state_info = 11;
+}
+
+// Additional input parameters used for TransformWithState operator.
+message TransformWithStateInfo {
+ // (Required) Time mode string for transformWithState.
+ string time_mode = 1;
+
+ // (Optional) Event time column name.
+ optional string event_time_column_name = 2;
+
+ // (Optional) Schema for the output DataFrame.
+ // Only required used for TransformWithStateInPandas.
+ optional DataType output_schema = 3;
}
message CoGroupMap {
@@ -1148,12 +1211,13 @@ message Parse {
// (Optional) DataType representing the schema. If not set, Spark will infer the schema.
optional DataType schema = 3;
- // Options for the csv/json parser. The map key is case insensitive.
+ // Options for the csv/json/xml parser. The map key is case insensitive.
map<string, string> options = 4;
enum ParseFormat {
PARSE_FORMAT_UNSPECIFIED = 0;
PARSE_FORMAT_CSV = 1;
PARSE_FORMAT_JSON = 2;
+ PARSE_FORMAT_XML = 3;
}
}
@@ -1214,3 +1278,45 @@ message LateralJoin {
// (Required) The join type.
Join.JoinType join_type = 4;
}
+
+// Relation of type [[NearestByJoin]].
+//
+// For each row on the left side, returns up to `num_results` rows from the right side ranked
+// by `ranking_expression`.
+message NearestByJoin {
+ // (Required) Left (query) input relation.
+ Relation left = 1;
+
+ // (Required) Right (base) input relation.
+ Relation right = 2;
+
+ // (Required) Scalar expression used to rank candidate rows on the right side.
+ Expression ranking_expression = 3;
+
+ // (Required) Maximum number of matches per left row. Must be between 1 and 100000.
+ int32 num_results = 4;
+
+ // The following three fields use `string` (not typed enums) for parity with `AsOfJoin`,
+ // which models analogous fields the same way. Validation happens server-side at planning time.
+
+ // (Required) The join type. Must be one of: "inner", "leftouter".
+ string join_type = 5;
+
+ // (Required) Search algorithm contract. Must be one of: "approx", "exact".
+ string mode = 6;
+
+ // (Required) Ranking direction. Must be one of: "distance", "similarity".
+ string direction = 7;
+}
+
+// Relation of type [[Zip]].
+//
+// Combines the columns of two DataFrames side-by-side. Both DataFrames must produce the same
+// canonicalized plan after stripping outer Project chains.
+message Zip {
+ // (Required) Left input relation.
+ Relation left = 1;
+
+ // (Required) Right input relation.
+ Relation right = 2;
+}
diff --git a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/types.proto b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/types.proto
index 6b4ee84..90ec5d0 100644
--- a/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/types.proto
+++ b/kyuubi-relocated-spark-connect-rpc/src/main/protobuf/spark/connect/types.proto
@@ -67,8 +67,20 @@ message DataType {
// UserDefinedType
UDT udt = 23;
+ // Geospatial types
+ Geometry geometry = 26;
+
+ Geography geography = 27;
+
// UnparsedDataType
Unparsed unparsed = 24;
+
+ Time time = 28;
+
+ // Nanosecond-capable timestamp types (precision 7..9). NTZ and LTZ are distinct kinds
+ // even though their physical value is identical, mirroring timestamp vs timestamp_ntz.
+ TimestampNTZNanos timestamp_ntz_nanos = 29;
+ TimestampLTZNanos timestamp_ltz_nanos = 30;
}
message Boolean {
@@ -124,6 +136,25 @@ message DataType {
uint32 type_variation_reference = 1;
}
+ message Time {
+ optional int32 precision = 1;
+ uint32 type_variation_reference = 2;
+ }
+
+ // Timestamp without time zone with nanosecond-capable fractional-second precision.
+ message TimestampNTZNanos {
+ // Number of fractional-second digits. Valid values are 7, 8, and 9. If omitted, defaults to 9.
+ optional int32 precision = 1;
+ uint32 type_variation_reference = 2;
+ }
+
+ // Timestamp with local time zone with nanosecond-capable fractional-second precision.
+ message TimestampLTZNanos {
+ // Number of fractional-second digits. Valid values are 7, 8, and 9. If omitted, defaults to 9.
+ optional int32 precision = 1;
+ uint32 type_variation_reference = 2;
+ }
+
message CalendarInterval {
uint32 type_variation_reference = 1;
}
@@ -182,6 +213,16 @@ message DataType {
uint32 type_variation_reference = 4;
}
+ message Geometry {
+ int32 srid = 1;
+ uint32 type_variation_reference = 2;
+ }
+
+ message Geography {
+ int32 srid = 1;
+ uint32 type_variation_reference = 2;
+ }
+
message Variant {
uint32 type_variation_reference = 1;
}