You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@orc.apache.org by wgtmac <gi...@git.apache.org> on 2018/08/15 07:06:26 UTC

[GitHub] orc pull request #301: ORC-395: Support ZSTD in C++ writer/reader

GitHub user wgtmac opened a pull request:

    https://github.com/apache/orc/pull/301

    ORC-395: Support ZSTD in C++ writer/reader

    Support ZSTD block compression/decompression for C++ writer/reader. Test cases have been added to TestCompression.cc

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/wgtmac/orc ORC-395

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/orc/pull/301.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #301
    
----

----


---

[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader

Posted by majetideepak <gi...@git.apache.org>.
Github user majetideepak commented on the issue:

    https://github.com/apache/orc/pull/301
  
    @wgtmac  thanks for these changes! I will take a look at both patches by end of tomorrow.


---

[GitHub] orc pull request #301: ORC-395: Support ZSTD in C++ writer/reader

Posted by xndai <gi...@git.apache.org>.
Github user xndai commented on a diff in the pull request:

    https://github.com/apache/orc/pull/301#discussion_r215486726
  
    --- Diff: c++/src/Compression.cc ---
    @@ -899,6 +907,166 @@ DIAGNOSTIC_POP
         return static_cast<uint64_t>(result);
       }
     
    +  /**
    +   * Block compression base class
    +   */
    +  class BlockCompressionStream: public CompressionStreamBase {
    --- End diff --
    
    Stream compression is not in this change yet?


---

[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader

Posted by wgtmac <gi...@git.apache.org>.
Github user wgtmac commented on the issue:

    https://github.com/apache/orc/pull/301
  
    @majetideepak From the links you provided, it looks like they only added the meta for ZSTD but not use it. Also, it uses hadoop 2.7.3 which does not have ZStandardCodec implementation yet.


---

[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader

Posted by wgtmac <gi...@git.apache.org>.
Github user wgtmac commented on the issue:

    https://github.com/apache/orc/pull/301
  
    @omalley @majetideepak Can you take a look at this? Thanks!


---

[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader

Posted by wgtmac <gi...@git.apache.org>.
Github user wgtmac commented on the issue:

    https://github.com/apache/orc/pull/301
  
    @majetideepak There is a PR https://github.com/apache/orc/pull/268 from @xndai to support Java reader to decompress ZSTD. It uses aircompressor which has the java implementation of zstd decompressor while zstd compressor is not released yet. Combining that PR with this, we can confirm end-to-end functionality and consistency. Though hadoop 2.9.1 has added ZStandardCodec for both compressor and decompressor, we don't have enough incentive to leverage it because of its JNI implementation.


---

[GitHub] orc pull request #301: ORC-395: Support ZSTD in C++ writer/reader

Posted by xndai <gi...@git.apache.org>.
Github user xndai commented on a diff in the pull request:

    https://github.com/apache/orc/pull/301#discussion_r215486619
  
    --- Diff: c++/src/Compression.cc ---
    @@ -899,6 +907,166 @@ DIAGNOSTIC_POP
         return static_cast<uint64_t>(result);
       }
     
    +  /**
    +   * Block compression base class
    +   */
    +  class BlockCompressionStream: public CompressionStreamBase {
    +  public:
    +    BlockCompressionStream(OutputStream * outStream,
    +                           int compressionLevel,
    +                           uint64_t capacity,
    +                           uint64_t blockSize,
    +                           MemoryPool& pool)
    +                           : CompressionStreamBase(outStream,
    +                                                   compressionLevel,
    +                                                   capacity,
    +                                                   blockSize,
    +                                                   pool)
    +                           , compressorBuffer(pool) {
    +      // PASS
    +    }
    +
    +    virtual bool Next(void** data, int*size) override;
    +    virtual std::string getName() const override = 0;
    +
    +  protected:
    +    // compresses a block and returns the compressed size
    +    virtual uint64_t doBlockCompression() = 0;
    +
    +    // return maximum possible compression size for allocating space for
    +    // compressorBuffer below
    +    virtual uint64_t estimateMaxCompressionSize() = 0;
    +
    +    // should allocate max possible compressed size
    +    DataBuffer<unsigned char> compressorBuffer;
    +  };
    +
    +  bool BlockCompressionStream::Next(void** data, int*size) {
    +    if (bufferSize != 0) {
    +      ensureHeader();
    +
    +      // perform compression
    +      size_t totalCompressedSize = doBlockCompression();
    +
    +      const unsigned char * dataToWrite = nullptr;
    +      int totalSizeToWrite = 0;
    +      char * header = outputBuffer + outputPosition - 3;
    +
    +      if (totalCompressedSize >= static_cast<size_t>(bufferSize)) {
    +        writeHeader(header, static_cast<size_t>(bufferSize), true);
    +        dataToWrite = rawInputBuffer.data();
    +        totalSizeToWrite = bufferSize;
    +      } else {
    +        writeHeader(header, totalCompressedSize, false);
    +        dataToWrite = compressorBuffer.data();
    +        totalSizeToWrite = static_cast<int>(totalCompressedSize);
    +      }
    +
    +      char * dst = header + 3;
    +      while (totalSizeToWrite > 0) {
    +        if (outputPosition >= outputSize) {
    --- End diff --
    
    assert outputPosition not larger than outputSize.


---

[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader

Posted by wgtmac <gi...@git.apache.org>.
Github user wgtmac commented on the issue:

    https://github.com/apache/orc/pull/301
  
    I have created a PR to port this feature to the java side: https://github.com/apache/orc/pull/306. However, hadoop 2.9.0 (which is the first version has ZStandardCodec) is not built with zstd native support by default. So I cannot add any test inside to avoid test break and that's why parquet-mr skips its test as well. parquet-mr uses hadoop v2.7.3 because it uses reflection to create the codec. @majetideepak 


---

[GitHub] orc pull request #301: ORC-395: Support ZSTD in C++ writer/reader

Posted by wgtmac <gi...@git.apache.org>.
Github user wgtmac commented on a diff in the pull request:

    https://github.com/apache/orc/pull/301#discussion_r211793792
  
    --- Diff: c++/src/Compression.cc ---
    @@ -899,6 +900,177 @@ DIAGNOSTIC_POP
         return static_cast<uint64_t>(result);
       }
     
    +  /**
    +   * Block compression base class
    +   */
    +  class BlockCompressionStream: public CompressionStreamBase {
    +  public:
    +    BlockCompressionStream(OutputStream * outStream,
    +                           int compressionLevel,
    +                           uint64_t capacity,
    +                           uint64_t blockSize,
    +                           MemoryPool& pool)
    +                           : CompressionStreamBase(outStream,
    +                                                   compressionLevel,
    +                                                   capacity,
    +                                                   blockSize,
    +                                                   pool)
    +                           , compressorBuffer(pool) {
    +      // PASS
    +    }
    +
    +    virtual bool Next(void** data, int*size) override;
    +    virtual std::string getName() const override = 0;
    +
    +  protected:
    +    // compresses a block and returns the compressed size
    +    virtual uint64_t doBlockCompression() = 0;
    +
    +    // return maximum possible compression size for allocating space for
    +    // compressorBuffer below
    +    virtual uint64_t estimateMaxCompressionSize() = 0;
    +
    +    // should allocate max possible compressed size
    +    DataBuffer<unsigned char> compressorBuffer;
    +  };
    +
    +  bool BlockCompressionStream::Next(void** data, int*size) {
    +    if (bufferSize != 0) {
    --- End diff --
    
    Done. Please check it again when you have time. Thanks!


---

[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader

Posted by wgtmac <gi...@git.apache.org>.
Github user wgtmac commented on the issue:

    https://github.com/apache/orc/pull/301
  
    @majetideepak Can you take a look at this again? Thanks!


---

[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader

Posted by majetideepak <gi...@git.apache.org>.
Github user majetideepak commented on the issue:

    https://github.com/apache/orc/pull/301
  
    @wgtmac  I will take a look at this today. Do you know what is the current behavior on the Java side with this compression format?


---

[GitHub] orc pull request #301: ORC-395: Support ZSTD in C++ writer/reader

Posted by majetideepak <gi...@git.apache.org>.
Github user majetideepak commented on a diff in the pull request:

    https://github.com/apache/orc/pull/301#discussion_r211036369
  
    --- Diff: c++/src/Compression.cc ---
    @@ -899,6 +900,177 @@ DIAGNOSTIC_POP
         return static_cast<uint64_t>(result);
       }
     
    +  /**
    +   * Block compression base class
    +   */
    +  class BlockCompressionStream: public CompressionStreamBase {
    +  public:
    +    BlockCompressionStream(OutputStream * outStream,
    +                           int compressionLevel,
    +                           uint64_t capacity,
    +                           uint64_t blockSize,
    +                           MemoryPool& pool)
    +                           : CompressionStreamBase(outStream,
    +                                                   compressionLevel,
    +                                                   capacity,
    +                                                   blockSize,
    +                                                   pool)
    +                           , compressorBuffer(pool) {
    +      // PASS
    +    }
    +
    +    virtual bool Next(void** data, int*size) override;
    +    virtual std::string getName() const override = 0;
    +
    +  protected:
    +    // compresses a block and returns the compressed size
    +    virtual uint64_t doBlockCompression() = 0;
    +
    +    // return maximum possible compression size for allocating space for
    +    // compressorBuffer below
    +    virtual uint64_t estimateMaxCompressionSize() = 0;
    +
    +    // should allocate max possible compressed size
    +    DataBuffer<unsigned char> compressorBuffer;
    +  };
    +
    +  bool BlockCompressionStream::Next(void** data, int*size) {
    +    if (bufferSize != 0) {
    --- End diff --
    
    Can we move this common compression header code to `CompressionStreamBase` ?


---

[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader

Posted by xndai <gi...@git.apache.org>.
Github user xndai commented on the issue:

    https://github.com/apache/orc/pull/301
  
    @majetideepak I think it makes sense to use org.apache.hadoop.io.compress.ZStandardCodec so compression and decompression can work end to end. In the future, if we have a native java implementation of zstd, hopefully we can just update the hadoop library. I will find a time to update my PR. By the way, since this codec class is only available in the latest hadoop library, there might additional work involved to upgrade the library.


---

[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader

Posted by wgtmac <gi...@git.apache.org>.
Github user wgtmac commented on the issue:

    https://github.com/apache/orc/pull/301
  
    To provide some benchmark results, I did some tests on my laptop using TPC-H 1GB dataset and C++ tools csv-import and orc-scan were used with default configuration.
    
    **Writer CPU Time (unit: second)**
    
    name | zlib | zstd
    -- | -- | --
    customer | 1.976 | 0.777
    lineitem | 50.754 | 19.990
    nation | 0.002 | 0.003
    orders | 11.054 | 4.895
    part | 1.893 | 0.771
    partsupp | 8.791 | 3.512
    region | 0.002 | 0.002
    supplier | 0.130 | 0.056
    
    **Reader CPU Time (unit: second)**
    
    name | zlib | zstd
    -- | -- | --
    customer | 0.084 | 0.063
    lineitem | 2.263 | 2.094
    nation | 0.001 | 0.001
    orders | 0.454 | 0.340
    part | 0.071 | 0.061
    partsupp | 0.343 | 0.253
    region | 0.000 | 0.001
    supplier | 0.006 | 0.005
    
    **File Size (unit: byte)**
    
    name | zlib | zstd
    -- | -- | --
    customer | 7494965 | 7670751
    lineitem | 162544602 | 178904712
    nation | 1760 | 1882
    orders | 34599561 | 38028670
    part | 4273944 | 4676560
    partsupp | 25766380 | 29498151
    region | 1026 | 1097
    supplier | 474099 | 478017
    
    In total, ZSTD writer time has 148.6% saving and reader time has 14.4% saving. File size is 9.4% bigger for ZSTD. The result provides a basic idea of performance comparison between them. As we use default configuration (ZLIB default level is -1 and ZSTD is 3), it may be unfair because ZSTD has 22 levels while ZLIB has 9 in total. If we choose different levels or different datasets, the result can vary a lot and ZSTD can beat ZLIB on file sizes. Overall, ZSTD seems to be a good compression option.



---

[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader

Posted by majetideepak <gi...@git.apache.org>.
Github user majetideepak commented on the issue:

    https://github.com/apache/orc/pull/301
  
    @xndai I am not very familiar with the Java side. Looking at the ZSTD support in parquet-mr, it looks like they only added the ZSTD support but skipped the tests. The parquet-mr library has not upgraded the hadoop library as well.
    https://github.com/apache/parquet-mr/blob/9fa86cca1af7dabc21701247efd89f6085945bd2/pom.xml#L80
     https://github.com/apache/parquet-mr/commit/132b2a8c553bdcfd445e88680beac6f225c50ac4#diff-6a038e86a0fc009909af954b3589cd95R159
    Can we do something like that here?


---