You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2015/08/13 22:43:46 UTC
[jira] [Assigned] (SPARK-9942) Broken pandas could crash PySpark
SQL
[ https://issues.apache.org/jira/browse/SPARK-9942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-9942:
-----------------------------------
Assignee: Davies Liu (was: Apache Spark)
> Broken pandas could crash PySpark SQL
> -------------------------------------
>
> Key: SPARK-9942
> URL: https://issues.apache.org/jira/browse/SPARK-9942
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 1.3.1, 1.4.1, 1.5.0
> Reporter: Davies Liu
> Assignee: Davies Liu
> Priority: Blocker
>
> {code}
> Error from python worker:
> Traceback (most recent call last):
> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 151, in _run_module_as_main
> mod_name, loader, code, fname = _get_module_details(mod_name)
> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 101, in _get_module_details
> loader = get_loader(mod_name)
> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pkgutil.py", line 464, in get_loader
> return find_loader(fullname)
> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pkgutil.py", line 474, in find_loader
> for importer in iter_importers(fullname):
> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pkgutil.py", line 430, in iter_importers
> __import__(pkg)
> File "/Users/davies/work/spark/python/pyspark/__init__.py", line 52, in <module>
> from pyspark.sql import SQLContext, HiveContext, SchemaRDD, Row
> File "/Users/davies/work/spark/python/pyspark/sql/__init__.py", line 63, in <module>
> from pyspark.sql.context import SQLContext, HiveContext
> File "/Users/davies/work/spark/python/pyspark/sql/context.py", line 40, in <module>
> import pandas
> File "/Library/Python/2.7/site-packages/pandas/__init__.py", line 7, in <module>
> from pandas import hashtable, tslib, lib
> File "pandas/src/numpy.pxd", line 157, in init pandas.hashtable (pandas/hashtable.c:23654)
> ValueError: numpy.dtype has the wrong size, try recompiling
> PYTHONPATH was:
> /Users/davies/work/spark/python/lib/pyspark.zip:/Users/davies/work/spark/python/lib/py4j-0.8.2.1-src.zip:/Users/davies/work/spark/sbin/../python/lib/py4j-0.8.2.1-src.zip:/Users/davies/work/spark/sbin/../python:/Users/davies/work/spark/sbin/../python/lib/py4j-0.8.2.1-src.zip:/Users/davies/work/spark/sbin/../python:
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org