You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "David Milicevic (Jira)" <ji...@apache.org> on 2026/02/09 08:49:00 UTC
[jira] [Updated] (SPARK-55441) Phase 1b - Client Integration
[ https://issues.apache.org/jira/browse/SPARK-55441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Milicevic updated SPARK-55441:
------------------------------------
Description:
*Summary:*
Add client-side Ops interfaces and integrate with Spark Connect, Arrow, JDBC, Python, and Thrift
*Description:*
Extend the framework with client-facing operations and integrate with all client communication paths.
*What this includes:*
* New interfaces: {{ProtoTypeOps}} (Spark Connect proto serialization), {{ClientTypeOps}} (JDBC, Arrow, Python, Thrift)
* Extend TimeType Ops with client operation implementations
* Integration in ~13 files: {{{}DataTypeProtoConverter{}}}, {{{}LiteralValueProtoConverter{}}}, {{{}ArrowUtils{}}}, {{{}ArrowWriter{}}}, {{{}ArrowSerializer{}}}, {{{}ArrowVectorReader{}}}, {{{}JdbcTypeUtils{}}}, {{{}SparkExecuteStatementOperation{}}}, {{{}HiveResult{}}}, {{{}EvaluatePython{}}}, {{{}DeserializerBuildHelper{}}}, {{{}SerializerBuildHelper{}}}, {{udaf.scala}}
*Note:*
Client integration has many non-obvious patterns beyond {{case _: TimeType =>}} - Arrow uses {{{}TimeNanoVector{}}}, proto uses {{{}.hasTime(){}}}/{{{}.getTime(){}}}, Thrift uses {{{}TTypeId.STRING_TYPE{}}}. The framework consolidates all of these behind a single Ops interface.
*Design doc:*
Linked in the parent work item.
was:
*Summary:*
Add client-side Ops interfaces and integrate with Spark Connect, Arrow, JDBC, Python, and Thrift
*Description:*
Extend the framework with client-facing operations and integrate with all client communication paths.
*What this includes:*
* New interfaces: {{ProtoTypeOps}} (Spark Connect proto serialization), {{ClientTypeOps}} (JDBC, Arrow, Python, Thrift)
* Extend TimeType Ops with client operation implementations
* Integration in ~13 files: {{{}DataTypeProtoConverter{}}}, {{{}LiteralValueProtoConverter{}}}, {{{}ArrowUtils{}}}, {{{}ArrowWriter{}}}, {{{}ArrowSerializer{}}}, {{{}ArrowVectorReader{}}}, {{{}JdbcTypeUtils{}}}, {{{}SparkExecuteStatementOperation{}}}, {{{}HiveResult{}}}, {{{}EvaluatePython{}}}, {{{}DeserializerBuildHelper{}}}, {{{}SerializerBuildHelper{}}}, {{udaf.scala}}
*Note:*
Client integration has many non-obvious patterns beyond {{case _: TimeType =>}} - Arrow uses {{{}TimeNanoVector{}}}, proto uses {{{}.hasTime(){}}}/{{{}.getTime(){}}}, Thrift uses {{{}TTypeId.STRING_TYPE{}}}. The framework consolidates all of these behind a single Ops interface.
> Phase 1b - Client Integration
> -----------------------------
>
> Key: SPARK-55441
> URL: https://issues.apache.org/jira/browse/SPARK-55441
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: David Milicevic
> Priority: Major
>
> *Summary:*
> Add client-side Ops interfaces and integrate with Spark Connect, Arrow, JDBC, Python, and Thrift
> *Description:*
> Extend the framework with client-facing operations and integrate with all client communication paths.
> *What this includes:*
> * New interfaces: {{ProtoTypeOps}} (Spark Connect proto serialization), {{ClientTypeOps}} (JDBC, Arrow, Python, Thrift)
> * Extend TimeType Ops with client operation implementations
> * Integration in ~13 files: {{{}DataTypeProtoConverter{}}}, {{{}LiteralValueProtoConverter{}}}, {{{}ArrowUtils{}}}, {{{}ArrowWriter{}}}, {{{}ArrowSerializer{}}}, {{{}ArrowVectorReader{}}}, {{{}JdbcTypeUtils{}}}, {{{}SparkExecuteStatementOperation{}}}, {{{}HiveResult{}}}, {{{}EvaluatePython{}}}, {{{}DeserializerBuildHelper{}}}, {{{}SerializerBuildHelper{}}}, {{udaf.scala}}
> *Note:*
> Client integration has many non-obvious patterns beyond {{case _: TimeType =>}} - Arrow uses {{{}TimeNanoVector{}}}, proto uses {{{}.hasTime(){}}}/{{{}.getTime(){}}}, Thrift uses {{{}TTypeId.STRING_TYPE{}}}. The framework consolidates all of these behind a single Ops interface.
> *Design doc:*
> Linked in the parent work item.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org