You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Nicholas Chammas (JIRA)" <ji...@apache.org> on 2015/05/09 19:52:59 UTC
[jira] [Created] (SPARK-7506)
pyspark.sql.types.StructType.fromJson() is incorrectly named
Nicholas Chammas created SPARK-7506:
---------------------------------------
Summary: pyspark.sql.types.StructType.fromJson() is incorrectly named
Key: SPARK-7506
URL: https://issues.apache.org/jira/browse/SPARK-7506
Project: Spark
Issue Type: Bug
Components: PySpark, SQL
Affects Versions: 1.3.1, 1.3.0
Reporter: Nicholas Chammas
Priority: Minor
{code}
>>> json_rdd = sqlContext.jsonRDD(sc.parallelize(['{"name": "Nick"}']))
>>> json_rdd.schema
StructType(List(StructField(name,StringType,true)))
>>> type(json_rdd.schema)
<class 'pyspark.sql.types.StructType'>
>>> json_rdd.schema.json()
'{"fields":[{"metadata":{},"name":"name","nullable":true,"type":"string"}],"type":"struct"}'
>>> pyspark.sql.types.StructType.fromJson(json_rdd.schema.json())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Applications/apache-spark/spark-1.3.1-bin-hadoop2.4/python/pyspark/sql/types.py", line 346, in fromJson
return StructType([StructField.fromJson(f) for f in json["fields"]])
TypeError: string indices must be integers, not str
>>> import json
>>> pyspark.sql.types.StructType.fromJson(json.loads(json_rdd.schema.json()))
StructType(List(StructField(name,StringType,true)))
>>>
{code}
So {{fromJson()}} doesn't actually expect JSON, which is a string. It expects a dictionary.
This method should probably be renamed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org