You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@orc.apache.org by xndai <gi...@git.apache.org> on 2018/05/14 21:46:08 UTC

[GitHub] orc pull request #268: ORC-363 Enable zstd decompression in ORC Java reader

GitHub user xndai opened a pull request:

    https://github.com/apache/orc/pull/268

    ORC-363 Enable zstd decompression in ORC Java reader

    1. Upgrade aircompressor lib to 0.11
    2. Enable Zstd decompression in Java reader
    3. Zstd compression is still not availiable. Will throw illegal argument
    exception if writer uses zstd compression.
    
    Change-Id: Ic5492b09af4e1e51215e62ed29233729857c45ac

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/xndai/orc ORC-363

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/orc/pull/268.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #268
    
----
commit 3401dd556ccce1492b12f0cc9f3e2f2cdfd9b848
Author: Xiening.Dai <xi...@...>
Date:   2018-05-14T21:42:46Z

    ORC-363 Enable zstd decompression in ORC Java reader
    
    1. Upgrade aircompressor lib to 0.11
    2. Enable Zstd decompression in Java reader
    3. Zstd compression is still not availiable. Will throw illegal argument
    exception if writer uses zstd compression.
    
    Change-Id: Ic5492b09af4e1e51215e62ed29233729857c45ac

----


---

[GitHub] orc issue #268: ORC-363 Enable zstd decompression in ORC Java reader

Posted by xndai <gi...@git.apache.org>.

Github user xndai commented on the issue:

    https://github.com/apache/orc/pull/268
  
    Hi all, I'd like to bring this up again. It's almost another three months, and we haven't seen the zstd java library yet. I would suggest we move forward to add zstd support in C++ reader/writer. And at the same time, enable Java reader by this change. 


---

[GitHub] orc issue #268: ORC-363 Enable zstd decompression in ORC Java reader

Posted by omalley <gi...@git.apache.org>.

Github user omalley commented on the issue:

    https://github.com/apache/orc/pull/268
  
    I'd rather hold off on this until we can do an end-to-end test.


---

[GitHub] orc issue #268: ORC-363 Enable zstd decompression in ORC Java reader

Posted by xndai <gi...@git.apache.org>.

Github user xndai commented on the issue:

    https://github.com/apache/orc/pull/268
  
    The current solution is not perfect. But at least it gives us some ability to read zstd Orc files, which I believe is important from the compatibility perspective - our in-house system has zstd Orc that would like to be consumed by Hive, Spark, etc. I am not sure when the zstd compressor will be available. It's probably another 6 months or a year.
    
    If we enable zstd on C++ reader/writer first. Then we enable Java reader to consume zstd Orc from C++ writer. Would you consider that as end-to-end test?


---

[GitHub] orc issue #268: ORC-363 Enable zstd decompression in ORC Java reader

Posted by t3rmin4t0r <gi...@git.apache.org>.

Github user t3rmin4t0r commented on the issue:

    https://github.com/apache/orc/pull/268
  
    I'm in favour of using the org.apache.hadoop.io.compress.ZStandardCodec if it is available from Hadoop.


---