[jira] [Created] (SPARK-34588) Support int64 buffer lengths in Java for pyspark Pandas UDF as buffer expanding

["Dmitry Kravchuk (Jira)" <ji...@apache.org>]

Dmitry Kravchuk created SPARK-34588:
---------------------------------------

             Summary: Support int64 buffer lengths in Java for pyspark Pandas UDF as buffer expanding
                 Key: SPARK-34588
                 URL: https://issues.apache.org/jira/browse/SPARK-34588
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 3.0.2
         Environment: Hadoop part:
 * spark 3.0.2
 * java 1.8.0_77
 * scala 2.12.10

Python part:
 * cython 0.29.22
 * numpy 1.19.5
 * pandas 1.1.5
 * pyarrow 2.0.0
            Reporter: Dmitry Kravchuk
             Fix For: 3.1.0, 3.2.0, 3.1.1, 3.1.2, 3.0.3


This issue is an extention of [arrow issue|https://issues.apache.org/jira/browse/ARROW-10957#] for making possible using pyspark Pandas UDF functions for data more than 2gb per data group.

Here is the deal - arrow [supports |https://github.com/apache/arrow/commit/9742007c463e253e2b916e65f668146953456a00#diff-2e086b32ec292aae20695dd4341c647c9a9d7d3d77816bf849f7fbf68e9fa6cfR209]long type for data serialization between java and python but spark doesn't. It gives a lot of problem when somebody is trying to apply Pandas UDF for dataset where any group is more than 2^32(-1) bytes what is equal to 2gb. Solving this problem will help to use more data per Pandas UDF groupping - 2^64(-1) bytes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org