You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by 李铖 <li...@gmail.com> on 2015/09/09 09:29:16 UTC
How to read compressed parquet file
I think too many parquet files may be affect reading capability,so I use
hadoop archive to combine them,but sql_context.read.parquet(output_path)
does not work on the file.
How to fix it ,please help me.
：）
Re: How to read compressed parquet file
Posted by 李铖 <li...@gmail.com>.
It works. at spark 1.4
Thanks a lot.
2015-09-09 17:21 GMT+08:00 Cheng Lian <li...@gmail.com>:
> You need to use "har://" instead of "hdfs://" to read HAR files. Just
> tested against Spark 1.5, and it works as expected.
>
> Cheng
>
>
> On 9/9/15 3:29 PM, 李铖 wrote:
>
> I think too many parquet files may be affect reading capability,so I use
> hadoop archive to combine them,but sql_context.read.parquet(output_path)
> does not work on the file.
> How to fix it ,please help me.
> ：）
>
>
>
Re: How to read compressed parquet file
Posted by Cheng Lian <li...@gmail.com>.
You need to use "har://" instead of "hdfs://" to read HAR files. Just
tested against Spark 1.5, and it works as expected.
Cheng
On 9/9/15 3:29 PM, 李铖 wrote:
> I think too many parquet files may be affect reading capability,so I
> use hadoop archive to combine them,but
> sql_context.read.parquet(output_path) does not work on the file.
> How to fix it ,please help me.
> ：）