[jira] [Commented] (SPARK-3592) applySchema to an RDD of Row

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-3592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14139299#comment-14139299 ] 

Apache Spark commented on SPARK-3592:
-------------------------------------

User 'davies' has created a pull request for this issue:
https://github.com/apache/spark/pull/2448

> applySchema to an RDD of Row
> ----------------------------
>
>                 Key: SPARK-3592
>                 URL: https://issues.apache.org/jira/browse/SPARK-3592
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark, SQL
>            Reporter: Davies Liu
>            Assignee: Davies Liu
>            Priority: Critical
>
> Right now, we can not appy schema to a RDD of Row, this should be a Bug,
> {code}
> >>> srdd = sqlCtx.jsonRDD(sc.parallelize(["""{"a":2}"""]))
> >>> sqlCtx.applySchema(srdd.map(lambda x:x), srdd.schema())
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/Users/daviesliu/work/spark/python/pyspark/sql.py", line 1121,
> in applySchema
>     _verify_type(row, schema)
>   File "/Users/daviesliu/work/spark/python/pyspark/sql.py", line 736,
> in _verify_type
>     % (dataType, type(obj)))
> TypeError: StructType(List(StructField(a,IntegerType,true))) can not
> accept abject in type <class 'pyspark.sql.Row'>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org