Re: How to disable parquet schema merging in 1.4?

[Cheng Lian <li...@gmail.com>]

With Spark 1.4, you may use data source option "mergeSchema" to control it:

   sqlContext.read.option("mergeSchema", "false").parquet("some/path")

or

   CREATE TABLE t USING parquet OPTIONS (mergeSchema false, path 
"some/path")

We're considering to disable schema merging by default in 1.5.0 since it 
brings unnecessary performance cost when schema evolution is not a problem.

Cheng

On 6/23/15 2:20 AM, Rex Xiong wrote:
> I remember in a previous PR, schema merging can be disabled by 
> setting spark.sql.hive.convertMetastoreParquet.mergeSchema to false.
> But in 1.4 release, I don't see this config anymore, is there a new 
> way to do it?
>
> Thanks


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org