You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2025/04/07 01:16:44 UTC

(spark) branch master updated: [SPARK-51079] Added constructor for ArrowSerializer for backward compatibility

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 75d80c7795ca [SPARK-51079] Added constructor for ArrowSerializer for backward compatibility
75d80c7795ca is described below

commit 75d80c7795ca71d24229010ab04ae740473126aa
Author: Haiyang Sun <75...@users.noreply.github.com>
AuthorDate: Mon Apr 7 10:16:34 2025 +0900

    [SPARK-51079] Added constructor for ArrowSerializer for backward compatibility
    
    ### What changes were proposed in this pull request?
    
    Previously, [PR](https://github.com/apache/spark/pull/49790) extends ArrowSerializer class to support large variable types in pandas UDF. This PR makes it more backward-compatible by adding a new constructor for cases where the flag for large variable is irrelevant (default to false).
    
    ### Why are the changes needed?
    To achieve versionless in spark connect, we need to be careful about changing a client API in Scala. For example, the change of an interface like this could affect Scala UDF referring to this class (as it is part of the spark connect client package) but running with an older spark connect client.
    
    ### How was this patch tested?
    N/A. It's a new constructor.
    
    Closes #50513 from haiyangsun-db/SPARK-51079.
    
    Authored-by: Haiyang Sun <75...@users.noreply.github.com>
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
 .../org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala
index a74fce3a8aba..d79fb25ec1a0 100644
--- a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala
+++ b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala
@@ -52,6 +52,12 @@ class ArrowSerializer[T](
     private[this] val allocator: BufferAllocator,
     private[this] val timeZoneId: String,
     private[this] val largeVarTypes: Boolean) {
+
+  // SPARK-51079: keep the old constructor for backward-compatibility.
+  def this(enc: AgnosticEncoder[T], allocator: BufferAllocator, timeZoneId: String) = {
+    this(enc, allocator, timeZoneId, false)
+  }
+
   private val (root, serializer) =
     ArrowSerializer.serializerFor(enc, allocator, timeZoneId, largeVarTypes)
   private val vectors = root.getFieldVectors.asScala


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org