You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Ruifeng Zheng (Jira)" <ji...@apache.org> on 2026/01/07 07:48:00 UTC

[jira] [Created] (SPARK-54936) Monitor upstream behaviour changes

Ruifeng Zheng created SPARK-54936:
-------------------------------------

             Summary: Monitor upstream behaviour changes
                 Key: SPARK-54936
                 URL: https://issues.apache.org/jira/browse/SPARK-54936
             Project: Spark
          Issue Type: Umbrella
          Components: PySpark, Tests
    Affects Versions: 4.2.0
            Reporter: Ruifeng Zheng


PySpark suffers a lot from behaviour changes from upstream communities, like Pandas, PyArrow, Numpy.

We should add tests to monitor the behaviour of key functions/features, like:
 * pa.Array.to_pandas
 * pa.Array.from_pandas
 * pa.Array.cast
 * pa.array
 * pa.scalar
 * pd.Series(data=array_data)
 * time zone handling in pyarrow, pandas
 * zero copy in pandas<->pyarrow data conversions
 * etc

 

The new tests should be dedicated for upstream, spark stuffs should not be involved.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org