You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by mayankshete <ma...@yash.com> on 2016/06/13 07:56:43 UTC
Hive 1.0.0 not able to read Spark 1.6.1 parquet output files on EMR
4.7.0
Hello Team ,
I am facing an issue where output files generated by Spark 1.6.1 are not
read by Hive 1.0.0 . It is because Hive 1.0.0 uses older parquet version
than Spark 1.6.1 which is using 1.7.0 parquet .
Is it possible that we can use older parquet version in Spark or newer
parquet version in Hive ?
I have tried adding parquet-hive-bundle : 1.7.0 to Hive but while reading it
throws Failed with exception
java.io.IOException:java.lang.NullPointerException .
Can anyone give us the solution ?
Thanks ,
Mayank
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Hive-1-0-0-not-able-to-read-Spark-1-6-1-parquet-output-files-on-EMR-4-7-0-tp27144.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Hive 1.0.0 not able to read Spark 1.6.1 parquet output files on
EMR 4.7.0
Posted by Cheng Lian <li...@gmail.com>.
Spark 1.6.1 is also using 1.7.0.
Could you please share the schema of your Parquet file as well as the
exact exception stack trace reported by Hive?
Cheng
On 6/13/16 12:56 AM, mayankshete wrote:
> Hello Team ,
>
> I am facing an issue where output files generated by Spark 1.6.1 are not
> read by Hive 1.0.0 . It is because Hive 1.0.0 uses older parquet version
> than Spark 1.6.1 which is using 1.7.0 parquet .
>
> Is it possible that we can use older parquet version in Spark or newer
> parquet version in Hive ?
> I have tried adding parquet-hive-bundle : 1.7.0 to Hive but while reading it
> throws Failed with exception
> java.io.IOException:java.lang.NullPointerException .
>
> Can anyone give us the solution ?
>
> Thanks ,
> Mayank
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Hive-1-0-0-not-able-to-read-Spark-1-6-1-parquet-output-files-on-EMR-4-7-0-tp27144.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org