You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by JF Chen <da...@gmail.com> on 2018/05/17 08:59:32 UTC

Snappy file compatible problem with spark

I made some snappy compressed json file with normal snappy codec(
https://github.com/xerial/snappy-java ) , which seems cannot be read by
Spark correctly.
So how to make existed snappy file recognized by spark? Any tools to
convert them?

Thanks@!

Regard,
Junfeng Chen

Re: Snappy file compatible problem with spark

Posted by JF Chen <da...@gmail.com>.

Yes. The JSON files compressed by Flume or Spark work well with Spark. But
the json files compressed by myself cannot be read by spark due to codec
problem. It seems sparking can read files compressed by hadoop snappy(
https://code.google.com/archive/p/hadoop-snappy/) only

Regard,
Junfeng Chen

On Thu, May 17, 2018 at 5:47 PM, Victor Noagbodji <
vnoagbodji@amplify-nation.com> wrote:

> Hey, Sorry if I misunderstood. Are you feeding the compressed JSON file to
> Spark directly?
>
> On May 17, 2018, at 4:59 AM, JF Chen <da...@gmail.com> wrote:
>
> I made some snappy compressed json file with normal snappy codec(
> https://github.com/xerial/snappy-java ) , which seems cannot be read by
> Spark correctly.
> So how to make existed snappy file recognized by spark? Any tools to
> convert them?
>
> Thanks@!
>
> Regard,
> Junfeng Chen
>
>
>