You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Peter Mac <Pe...@noaa.gov> on 2015/03/27 17:13:38 UTC
Python Example sql.py not working in version
spark-1.3.0-bin-hadoop2.4
I downloaded spark version spark-1.3.0-bin-hadoop2.4.
When the python version of sql.py is run the following error occurs:
[root@nde-dev8-template python]#
/root/spark-1.3.0-bin-hadoop2.4/bin/spark-submit sql.py
Spark assembly has been built with Hive, including Datanucleus jars on
classpath
Traceback (most recent call last):
File "/root/spark-1.3.0-bin-hadoop2.4/examples/src/main/python/sql.py",
line 22, in <module>
from pyspark.sql import Row, StructField, StructType, StringType,
IntegerType
ImportError: cannot import name StructField
----------------------------------------------------------------------
The sql.py version, spark-1.2.1-bin-hadoop2.4, does not throw the error:
[root@nde-dev8-template python]#
/root/spark-1.2.1-bin-hadoop2.4/bin/spark-submit sql.py
Spark assembly has been built with Hive, including Datanucleus jars on
classpath
15/03/27 14:18:44 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/03/27 14:19:41 WARN ThreadLocalRandom: Failed to generate a seed from
SecureRandom within 3 seconds. Not enough entrophy?
root
|-- age: integer (nullable = true)
|-- name: string (nullable = true)
root
|-- person_name: string (nullable = false)
|-- person_age: integer (nullable = false)
root
|-- age: integer (nullable = true)
|-- name: string (nullable = true)
Justin
-------------------------------------------------
The OS/JAVA environments are:
OS: Linux nde-dev8-template 2.6.32-431.17.1.el6.x86_64 #1 SMP Fri Apr 11
17:27:00 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
JAVA: [root@nde-dev8-template bin]# java -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
The same error occurs when using bin/pyspark shell.
>>> from pyspark.sql import StructField
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name StructField
---------------------------------------------------
Any advice for resolving? Thanks in advance.
Peter
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Python-Example-sql-py-not-working-in-version-spark-1-3-0-bin-hadoop2-4-tp22261.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Python Example sql.py not working in version spark-1.3.0-bin-hadoop2.4
Posted by Davies Liu <da...@databricks.com>.
This will be fixed in https://github.com/apache/spark/pull/5230/files
On Fri, Mar 27, 2015 at 9:13 AM, Peter Mac <Pe...@noaa.gov> wrote:
> I downloaded spark version spark-1.3.0-bin-hadoop2.4.
>
> When the python version of sql.py is run the following error occurs:
>
> [root@nde-dev8-template python]#
> /root/spark-1.3.0-bin-hadoop2.4/bin/spark-submit sql.py
> Spark assembly has been built with Hive, including Datanucleus jars on
> classpath
> Traceback (most recent call last):
> File "/root/spark-1.3.0-bin-hadoop2.4/examples/src/main/python/sql.py",
> line 22, in <module>
> from pyspark.sql import Row, StructField, StructType, StringType,
> IntegerType
> ImportError: cannot import name StructField
>
> ----------------------------------------------------------------------
> The sql.py version, spark-1.2.1-bin-hadoop2.4, does not throw the error:
>
> [root@nde-dev8-template python]#
> /root/spark-1.2.1-bin-hadoop2.4/bin/spark-submit sql.py
> Spark assembly has been built with Hive, including Datanucleus jars on
> classpath
> 15/03/27 14:18:44 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 15/03/27 14:19:41 WARN ThreadLocalRandom: Failed to generate a seed from
> SecureRandom within 3 seconds. Not enough entrophy?
> root
> |-- age: integer (nullable = true)
> |-- name: string (nullable = true)
>
> root
> |-- person_name: string (nullable = false)
> |-- person_age: integer (nullable = false)
>
> root
> |-- age: integer (nullable = true)
> |-- name: string (nullable = true)
>
> Justin
>
>
> -------------------------------------------------
>
> The OS/JAVA environments are:
>
> OS: Linux nde-dev8-template 2.6.32-431.17.1.el6.x86_64 #1 SMP Fri Apr 11
> 17:27:00 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
>
> JAVA: [root@nde-dev8-template bin]# java -version
> java version "1.7.0_51"
> Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
>
> The same error occurs when using bin/pyspark shell.
>
>>>> from pyspark.sql import StructField
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ImportError: cannot import name StructField
>
>
> ---------------------------------------------------
>
> Any advice for resolving? Thanks in advance.
>
> Peter
>
>
>
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Python-Example-sql-py-not-working-in-version-spark-1-3-0-bin-hadoop2-4-tp22261.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org