You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sunita Arvind <su...@gmail.com> on 2017/01/05 20:47:36 UTC
Writing Parquet from Avro objects - cannot write null value for
numeric fields
Hello Experts,
I am trying to allow null values in numeric fields. Here are the details of
the issue I have:
http://stackoverflow.com/questions/41492344/spark-avro-to-parquet-writing-null-values-in-number-fields
I also tried making all columns nullable by using the below function (from
one of the suggestions on web)
def setNullableStateForAllColumns( df: DataFrame, nullable: Boolean) :
DataFrame = {
df.sqlContext.createDataFrame(df.rdd,
StructType(df.schema.map(_.copy(nullable = nullable))))
}
The printSchema shows that the columns are now nullable, but still I
cannot persist
as parquet with null in the numeric fields.
Is there a workaround to it? I need to be able to allow null values
for numeric fields
Thanks in advance.
regards
Sunita