json to parquet failure

[Deepak Goel <go...@yahoo.com.INVALID>]

Hi,
I am trying to convert json file into parquet format using spark and json file contains a map where key and value are defined and actual key is scriptId. It fails with below exception-

java.lang.ClassCastException: optional binary scriptId (UTF8) is not a group

       at org.apache.parquet.schema.Type.asGroupType(Type.java:207)

       at org.apache.spark.sql.execution.datasources.parquet.ParquetRowConverter$ParquetMapConverter.<init>(ParquetRowConverter.scala:551)

       at org.apache.spark.sql.execution.datasources.parquet.ParquetRowConverter.org$apache$spark$sql$execution$datasources$parquet$ParquetRowConverter$$newConverter(ParquetRowConverter.scala:315)

       at org.apache.spark.sql.execution.datasources.parquet.ParquetRowConverter$$anonfun$7.apply(ParquetRowConverter.scala:188)

       at org.apache.spark.sql.execution.datasources.parquet.ParquetRowConverter$$anonfun$7.apply(ParquetRowConverter.scala:185)

       at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)

       at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)

       at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

       at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)

       at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)

       at scala.collection.AbstractTraversable.map(Traversable.scala:104)







Any idea why its failing with this class cast  exception inside parquet library. There is no schema is given here as spark infers it from json file itself.




Thanks

raj