Met a problem when using spark to load parquet files with different version schemas

[Wei Yan <yw...@gmail.com>]

Hi, devs,

I met a problem when using spark to read to parquet files with two
different versions of schemas. For example, the first file has one field
with "int" type, while the same field in the second file is a "long". I
thought spark would automatically generate a merged schema "long", and use
that schema to process both files. However, the following code cannot work:

DataFrame df = sqlContext.parquetFile(inputPath);
df.registerTempTable("data");
sqlContext.sql("select count(msg.actual_eta) from data").collect();


Exception:
parquet.io.ParquetDecodingException: Can not read value at 0 in block -1 in
file f1.parquet
at
parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:228)
at
parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:201)


BTW, I use spark 1.3.1, and already set
"spark.sql.parquet.useDataSourceApi" to false.

Any help would be appreciated.

-Wei

Re: Met a problem when using spark to load parquet files with different version schemas

[Wei Yan <yw...@gmail.com>]

Creating dataframes and union them looks reasonable.

thanks,
Wei


On Mon, May 11, 2015 at 6:39 PM, Michael Armbrust <mi...@databricks.com>
wrote:

> Ah, yeah sorry.  I should have read closer and realized that what you are
> asking for is not supported.  It might be possible to add simple coercions
> such as this one, but today, compatible schemas must only add/remove
> columns and cannot change types.
>
> You could try creating different dataframes and unionAll them.  Coercions
> should be inserted automatically in that case.
>
> On Mon, May 11, 2015 at 3:37 PM, Wei Yan <yw...@gmail.com> wrote:
>
>> Thanks for the reply, Michael.
>>
>> The problem is, if I set "spark.sql.parquet.useDataSourceApi" to true,
>> spark cannot create a DataFrame. The exception shows it "failed to merge
>> incompatible schemas". I think here it means that, the "int" schema cannot
>> be merged with the "long" one.
>> Does it mean that the schema merging doesn't support the same field with
>> different types?
>>
>> -Wei
>>
>> On Mon, May 11, 2015 at 3:10 PM, Michael Armbrust <michael@databricks.com
>> > wrote:
>>
>>> BTW, I use spark 1.3.1, and already set
>>>> "spark.sql.parquet.useDataSourceApi" to false.
>>>>
>>>
>>> Schema merging is only supported when this flag is set to true (setting
>>> it to false uses old code that will be removed once the new code is
>>> proven).
>>>
>>
>>
>

Re: Met a problem when using spark to load parquet files with different version schemas

[Michael Armbrust <mi...@databricks.com>]

Ah, yeah sorry.  I should have read closer and realized that what you are
asking for is not supported.  It might be possible to add simple coercions
such as this one, but today, compatible schemas must only add/remove
columns and cannot change types.

You could try creating different dataframes and unionAll them.  Coercions
should be inserted automatically in that case.

On Mon, May 11, 2015 at 3:37 PM, Wei Yan <yw...@gmail.com> wrote:

> Thanks for the reply, Michael.
>
> The problem is, if I set "spark.sql.parquet.useDataSourceApi" to true,
> spark cannot create a DataFrame. The exception shows it "failed to merge
> incompatible schemas". I think here it means that, the "int" schema cannot
> be merged with the "long" one.
> Does it mean that the schema merging doesn't support the same field with
> different types?
>
> -Wei
>
> On Mon, May 11, 2015 at 3:10 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> BTW, I use spark 1.3.1, and already set
>>> "spark.sql.parquet.useDataSourceApi" to false.
>>>
>>
>> Schema merging is only supported when this flag is set to true (setting
>> it to false uses old code that will be removed once the new code is
>> proven).
>>
>
>

Re: Met a problem when using spark to load parquet files with different version schemas

[Wei Yan <yw...@gmail.com>]

Thanks for the reply, Michael.

The problem is, if I set "spark.sql.parquet.useDataSourceApi" to true,
spark cannot create a DataFrame. The exception shows it "failed to merge
incompatible schemas". I think here it means that, the "int" schema cannot
be merged with the "long" one.
Does it mean that the schema merging doesn't support the same field with
different types?

-Wei

On Mon, May 11, 2015 at 3:10 PM, Michael Armbrust <mi...@databricks.com>
wrote:

> BTW, I use spark 1.3.1, and already set
>> "spark.sql.parquet.useDataSourceApi" to false.
>>
>
> Schema merging is only supported when this flag is set to true (setting it
> to false uses old code that will be removed once the new code is proven).
>

Re: Met a problem when using spark to load parquet files with different version schemas

[Michael Armbrust <mi...@databricks.com>]

>
> BTW, I use spark 1.3.1, and already set
> "spark.sql.parquet.useDataSourceApi" to false.
>

Schema merging is only supported when this flag is set to true (setting it
to false uses old code that will be removed once the new code is proven).