You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Divya Gehlot <di...@gmail.com> on 2016/03/04 06:38:13 UTC
Spark 1.5.2 - Read custom schema from file
Hi,
I have defined a custom schema as shown below :
val customSchema = StructType(
> StructField("year", IntegerType, true),
> StructField("make", StringType, true),
> StructField("model", StringType, true),
> StructField("comment", StringType, true),
StructField("blank", StringType, true))
Is there any way instead of defining it spark job file I can read from file.
I am using Spark-csv to read my data file
val df = sqlContext.read
.format("com.databricks.spark.csv")
.option("header", "true") // Use first line of all files as header
.schema(customSchema)
.load("cars.csv")val selectedData = df.select("year", "model")
selectedData.write
.format("com.databricks.spark.csv")
.option("header", "true")
.save("newcars.csv")