You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Hao Wang <bi...@gmail.com> on 2015/09/15 02:06:37 UTC
How to convert dataframe to a nested StructType schema
Hi,
I created a dataframe with 4 string columns (city, state, country, zipcode).
I then applied the following nested schema to it by creating a custom
StructType. When I run df.take(5), it gives the exception below as expected.
The question is how I can convert the Rows in the dataframe to conform to
this nested schema? Thanks!
root
|-- ZipCode: struct (nullable = true)
| |-- zip: string (nullable = true)
|-- Address: struct (nullable = true)
| |-- city: string (nullable = true)
| |-- state: string (nullable = true)
| |-- country: string (nullable = true)
[info] org.apache.spark.SparkException: Job aborted due to stage failure:
Task 0 in stage 6.0 failed 1 times, most recent failure: Lost task 0.0 in
stage 6.0 (TID 6, localhost): scala.MatchError: 95123 (of class
java.lang.String)
[info] at
org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToCatalystConverter$4.apply(CatalystTypeConverters.scala:178)
[info] at
org.apache.spark.sql.catalyst.CatalystTypeConverters$.convertRowWithConverters(CatalystTypeConverters.scala:348)
[info] at
org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToCatalystConverter$4.apply(CatalystTypeConverters.scala:180)
[info] at
org.apache.spark.sql.SQLContext$$anonfun$9.apply(SQLContext.scala:488)
[info] at
org.apache.spark.sql.SQLContext$$anonfun$9.apply(SQLContext.scala:488)
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-dataframe-to-a-nested-StructType-schema-tp24694.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org