You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Shane Knapp (Jira)" <ji...@apache.org> on 2021/07/21 17:57:00 UTC
[jira] [Commented] (SPARK-34930) Install PyArrow and pandas on
Jenkins
[ https://issues.apache.org/jira/browse/SPARK-34930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17385030#comment-17385030 ]
Shane Knapp commented on SPARK-34930:
-------------------------------------
pandas is installed, so i'm a little curious as to why the tests aren't running:
{noformat}
jenkins@research-jenkins-worker-01:~$ python
Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> pandas.__version__
'0.24.2'
>>>{noformat}
pyarrow is a much more complex install than just adding the package, and requires manual compilation. i'll revisit pyarrow in the next couple of weeks...
> Install PyArrow and pandas on Jenkins
> -------------------------------------
>
> Key: SPARK-34930
> URL: https://issues.apache.org/jira/browse/SPARK-34930
> Project: Spark
> Issue Type: Test
> Components: Project Infra
> Affects Versions: 3.2.0
> Reporter: Hyukjin Kwon
> Assignee: Shane Knapp
> Priority: Critical
>
> Looks like Jenkins mahcines don't have pandas and PyArrow (ever since it got upgraded?) which result in skipping related tests in PySpark, see also https://github.com/apache/spark/pull/31470#issuecomment-811618571
> It would be great if we can install both in Python 3.6 on Jenkins.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org