You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Mateusz Buśkiewicz (JIRA)" <ji...@apache.org> on 2015/07/16 15:21:04 UTC
[jira] [Created] (SPARK-9101) Can't use null in selectExpr
Mateusz Buśkiewicz created SPARK-9101:
-----------------------------------------
Summary: Can't use null in selectExpr
Key: SPARK-9101
URL: https://issues.apache.org/jira/browse/SPARK-9101
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 1.4.0
Reporter: Mateusz Buśkiewicz
In 1.3.1 this worked:
{code:python}
df = sqlContext.createDataFrame([[1]], schema=['col'])
df.selectExpr('null as newCol').collect()
{code}
In 1.4.0 it fails with the following stacktrace:
{code}
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/dataframe.py", line 316, in collect
cls = _create_cls(self.schema)
File "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/dataframe.py", line 229, in schema
self._schema = _parse_datatype_json_string(self._jdf.schema().json())
File "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/types.py", line 519, in _parse_datatype_json_string
return _parse_datatype_json_value(json.loads(json_string))
File "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/types.py", line 539, in _parse_datatype_json_value
return _all_complex_types[tpe].fromJson(json_value)
File "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/types.py", line 386, in fromJson
return StructType([StructField.fromJson(f) for f in json["fields"]])
File "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/types.py", line 347, in fromJson
_parse_datatype_json_value(json["type"]),
File "/opt/boxen/homebrew/opt/apache-spark/libexec/python/pyspark/sql/types.py", line 535, in _parse_datatype_json_value
raise ValueError("Could not parse datatype: %s" % json_value)
ValueError: Could not parse datatype: null
{code}
https://github.com/apache/spark/blob/v1.4.0/python/pyspark/sql/types.py#L461
The cause:_atomic_types doesn't contain NullType
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org