You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Bryan Cutler (JIRA)" <ji...@apache.org> on 2019/03/25 23:57:00 UTC
[jira] [Commented] (SPARK-27276) Increase the minimum pyarrow
version to 0.12.0
[ https://issues.apache.org/jira/browse/SPARK-27276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16801250#comment-16801250 ]
Bryan Cutler commented on SPARK-27276:
--------------------------------------
[~shaneknapp] this will need an upgrade on Jenkins, so let me know whenever you think is a good time to do this - I know you are working out some kinks with recent upgrades right now so no rush. I will try to do the code changes next week and then we can coordinate anytime after. Thanks! cc [~hyukjin.kwon]
> Increase the minimum pyarrow version to 0.12.0
> ----------------------------------------------
>
> Key: SPARK-27276
> URL: https://issues.apache.org/jira/browse/SPARK-27276
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, SQL
> Affects Versions: 3.0.0
> Reporter: Bryan Cutler
> Priority: Major
>
> The current minimum version is 0.8.0, which is pretty ancient since Arrow has been moving fast and a lot has changed since this version. There are currently many workarounds checking for different versions or disabling specific functionality, and the code is getting ugly and difficult to maintain. Increasing the version will allow cleanup and upgrade the testing environment.
> This involves changing the pyarrow version in setup.py (currently at 0.8.0), updating Jenkins to test against the new version, code cleanup to remove workarounds from older versions. Users would then need to ensure this version is installed on the cluster.
> There is also a 0.12.1 release, so I will need to check what bugs that fixed to see if that will be a better version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org