Re: Spark data quality bug when reading parquet files from hive metastore

["Long, Andrew" <lo...@amazon.com.INVALID>]

Thanks Fokko,

I will definitely take a look at this.

Cheers Andrew

From: "Driesprong, Fokko" <fo...@driesprong.frl>
Date: Friday, August 24, 2018 at 2:39 AM
To: "reubensawyer@hotmail.com" <re...@hotmail.com>
Cc: "dev@spark.apache.org" <de...@spark.apache.org>
Subject: Re: Spark data quality bug when reading parquet files from hive metastore

Hi Andrew,

This blog gives an idea how to schema is resolved: https://blog.godatadriven.com/multiformat-spark-partition There is some optimisation going on when reading Parquet using Spark. Hope this helps.

Cheers, Fokko


Op wo 22 aug. 2018 om 23:59 schreef t4 <re...@hotmail.com>>:
https://issues.apache.org/jira/browse/SPARK-23576 ?



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org<ma...@spark.apache.org>