Re: ORC native in Spark 2.3, with zlib, gives java.nio.BufferUnderflowException during read

[Eirik Thorsnes <ei...@uni.no>]

On 28. mars 2018 03:26, Dongjoon Hyun wrote:
> You may hit SPARK-23355 (convertMetastore should not ignore table properties).
> 
> Since it's a known Spark issue for all Hive tables (Parquet/ORC), could you check that too?
> 
> Bests,
> Dongjoon.
> 

Hi,

I think you might be right, I can run your example from the other email
OK ( spark.range(10).write.orc("/tmp/zlib_test") +
spark.read.orc("/tmp/zlib_test").show )

I can also do:

spark.range(10).write.format("orc").option("compression","zlib").saveAsTable("zlib_test3")

with a corresponding read. Trying to read a more complicated and
partitioned table fails. Could be because of partitioning perhaps?
Looking more into it now.

Regards,
Eirik

-- 
Eirik Thorsnes


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org