You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@orc.apache.org by wgtmac <gi...@git.apache.org> on 2018/08/15 07:06:26 UTC
[GitHub] orc pull request #301: ORC-395: Support ZSTD in C++ writer/reader
GitHub user wgtmac opened a pull request:
https://github.com/apache/orc/pull/301
ORC-395: Support ZSTD in C++ writer/reader
Support ZSTD block compression/decompression for C++ writer/reader. Test cases have been added to TestCompression.cc
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wgtmac/orc ORC-395
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/orc/pull/301.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #301
----
----
---
[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader
Posted by majetideepak <gi...@git.apache.org>.
Github user majetideepak commented on the issue:
https://github.com/apache/orc/pull/301
@wgtmac thanks for these changes! I will take a look at both patches by end of tomorrow.
---
[GitHub] orc pull request #301: ORC-395: Support ZSTD in C++ writer/reader
Posted by xndai <gi...@git.apache.org>.
Github user xndai commented on a diff in the pull request:
https://github.com/apache/orc/pull/301#discussion_r215486726
--- Diff: c++/src/Compression.cc ---
@@ -899,6 +907,166 @@ DIAGNOSTIC_POP
return static_cast<uint64_t>(result);
}
+ /**
+ * Block compression base class
+ */
+ class BlockCompressionStream: public CompressionStreamBase {
--- End diff --
Stream compression is not in this change yet?
---
[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader
Posted by wgtmac <gi...@git.apache.org>.
Github user wgtmac commented on the issue:
https://github.com/apache/orc/pull/301
@majetideepak From the links you provided, it looks like they only added the meta for ZSTD but not use it. Also, it uses hadoop 2.7.3 which does not have ZStandardCodec implementation yet.
---
[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader
Posted by wgtmac <gi...@git.apache.org>.
Github user wgtmac commented on the issue:
https://github.com/apache/orc/pull/301
@omalley @majetideepak Can you take a look at this? Thanks!
---
[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader
Posted by wgtmac <gi...@git.apache.org>.
Github user wgtmac commented on the issue:
https://github.com/apache/orc/pull/301
@majetideepak There is a PR https://github.com/apache/orc/pull/268 from @xndai to support Java reader to decompress ZSTD. It uses aircompressor which has the java implementation of zstd decompressor while zstd compressor is not released yet. Combining that PR with this, we can confirm end-to-end functionality and consistency. Though hadoop 2.9.1 has added ZStandardCodec for both compressor and decompressor, we don't have enough incentive to leverage it because of its JNI implementation.
---
[GitHub] orc pull request #301: ORC-395: Support ZSTD in C++ writer/reader
Posted by xndai <gi...@git.apache.org>.
Github user xndai commented on a diff in the pull request:
https://github.com/apache/orc/pull/301#discussion_r215486619
--- Diff: c++/src/Compression.cc ---
@@ -899,6 +907,166 @@ DIAGNOSTIC_POP
return static_cast<uint64_t>(result);
}
+ /**
+ * Block compression base class
+ */
+ class BlockCompressionStream: public CompressionStreamBase {
+ public:
+ BlockCompressionStream(OutputStream * outStream,
+ int compressionLevel,
+ uint64_t capacity,
+ uint64_t blockSize,
+ MemoryPool& pool)
+ : CompressionStreamBase(outStream,
+ compressionLevel,
+ capacity,
+ blockSize,
+ pool)
+ , compressorBuffer(pool) {
+ // PASS
+ }
+
+ virtual bool Next(void** data, int*size) override;
+ virtual std::string getName() const override = 0;
+
+ protected:
+ // compresses a block and returns the compressed size
+ virtual uint64_t doBlockCompression() = 0;
+
+ // return maximum possible compression size for allocating space for
+ // compressorBuffer below
+ virtual uint64_t estimateMaxCompressionSize() = 0;
+
+ // should allocate max possible compressed size
+ DataBuffer<unsigned char> compressorBuffer;
+ };
+
+ bool BlockCompressionStream::Next(void** data, int*size) {
+ if (bufferSize != 0) {
+ ensureHeader();
+
+ // perform compression
+ size_t totalCompressedSize = doBlockCompression();
+
+ const unsigned char * dataToWrite = nullptr;
+ int totalSizeToWrite = 0;
+ char * header = outputBuffer + outputPosition - 3;
+
+ if (totalCompressedSize >= static_cast<size_t>(bufferSize)) {
+ writeHeader(header, static_cast<size_t>(bufferSize), true);
+ dataToWrite = rawInputBuffer.data();
+ totalSizeToWrite = bufferSize;
+ } else {
+ writeHeader(header, totalCompressedSize, false);
+ dataToWrite = compressorBuffer.data();
+ totalSizeToWrite = static_cast<int>(totalCompressedSize);
+ }
+
+ char * dst = header + 3;
+ while (totalSizeToWrite > 0) {
+ if (outputPosition >= outputSize) {
--- End diff --
assert outputPosition not larger than outputSize.
---
[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader
Posted by wgtmac <gi...@git.apache.org>.
Github user wgtmac commented on the issue:
https://github.com/apache/orc/pull/301
I have created a PR to port this feature to the java side: https://github.com/apache/orc/pull/306. However, hadoop 2.9.0 (which is the first version has ZStandardCodec) is not built with zstd native support by default. So I cannot add any test inside to avoid test break and that's why parquet-mr skips its test as well. parquet-mr uses hadoop v2.7.3 because it uses reflection to create the codec. @majetideepak
---
[GitHub] orc pull request #301: ORC-395: Support ZSTD in C++ writer/reader
Posted by wgtmac <gi...@git.apache.org>.
Github user wgtmac commented on a diff in the pull request:
https://github.com/apache/orc/pull/301#discussion_r211793792
--- Diff: c++/src/Compression.cc ---
@@ -899,6 +900,177 @@ DIAGNOSTIC_POP
return static_cast<uint64_t>(result);
}
+ /**
+ * Block compression base class
+ */
+ class BlockCompressionStream: public CompressionStreamBase {
+ public:
+ BlockCompressionStream(OutputStream * outStream,
+ int compressionLevel,
+ uint64_t capacity,
+ uint64_t blockSize,
+ MemoryPool& pool)
+ : CompressionStreamBase(outStream,
+ compressionLevel,
+ capacity,
+ blockSize,
+ pool)
+ , compressorBuffer(pool) {
+ // PASS
+ }
+
+ virtual bool Next(void** data, int*size) override;
+ virtual std::string getName() const override = 0;
+
+ protected:
+ // compresses a block and returns the compressed size
+ virtual uint64_t doBlockCompression() = 0;
+
+ // return maximum possible compression size for allocating space for
+ // compressorBuffer below
+ virtual uint64_t estimateMaxCompressionSize() = 0;
+
+ // should allocate max possible compressed size
+ DataBuffer<unsigned char> compressorBuffer;
+ };
+
+ bool BlockCompressionStream::Next(void** data, int*size) {
+ if (bufferSize != 0) {
--- End diff --
Done. Please check it again when you have time. Thanks!
---
[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader
Posted by wgtmac <gi...@git.apache.org>.
Github user wgtmac commented on the issue:
https://github.com/apache/orc/pull/301
@majetideepak Can you take a look at this again? Thanks!
---
[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader
Posted by majetideepak <gi...@git.apache.org>.
Github user majetideepak commented on the issue:
https://github.com/apache/orc/pull/301
@wgtmac I will take a look at this today. Do you know what is the current behavior on the Java side with this compression format?
---
[GitHub] orc pull request #301: ORC-395: Support ZSTD in C++ writer/reader
Posted by majetideepak <gi...@git.apache.org>.
Github user majetideepak commented on a diff in the pull request:
https://github.com/apache/orc/pull/301#discussion_r211036369
--- Diff: c++/src/Compression.cc ---
@@ -899,6 +900,177 @@ DIAGNOSTIC_POP
return static_cast<uint64_t>(result);
}
+ /**
+ * Block compression base class
+ */
+ class BlockCompressionStream: public CompressionStreamBase {
+ public:
+ BlockCompressionStream(OutputStream * outStream,
+ int compressionLevel,
+ uint64_t capacity,
+ uint64_t blockSize,
+ MemoryPool& pool)
+ : CompressionStreamBase(outStream,
+ compressionLevel,
+ capacity,
+ blockSize,
+ pool)
+ , compressorBuffer(pool) {
+ // PASS
+ }
+
+ virtual bool Next(void** data, int*size) override;
+ virtual std::string getName() const override = 0;
+
+ protected:
+ // compresses a block and returns the compressed size
+ virtual uint64_t doBlockCompression() = 0;
+
+ // return maximum possible compression size for allocating space for
+ // compressorBuffer below
+ virtual uint64_t estimateMaxCompressionSize() = 0;
+
+ // should allocate max possible compressed size
+ DataBuffer<unsigned char> compressorBuffer;
+ };
+
+ bool BlockCompressionStream::Next(void** data, int*size) {
+ if (bufferSize != 0) {
--- End diff --
Can we move this common compression header code to `CompressionStreamBase` ?
---
[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader
Posted by xndai <gi...@git.apache.org>.
Github user xndai commented on the issue:
https://github.com/apache/orc/pull/301
@majetideepak I think it makes sense to use org.apache.hadoop.io.compress.ZStandardCodec so compression and decompression can work end to end. In the future, if we have a native java implementation of zstd, hopefully we can just update the hadoop library. I will find a time to update my PR. By the way, since this codec class is only available in the latest hadoop library, there might additional work involved to upgrade the library.
---
[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader
Posted by wgtmac <gi...@git.apache.org>.
Github user wgtmac commented on the issue:
https://github.com/apache/orc/pull/301
To provide some benchmark results, I did some tests on my laptop using TPC-H 1GB dataset and C++ tools csv-import and orc-scan were used with default configuration.
**Writer CPU Time (unit: second)**
name | zlib | zstd
-- | -- | --
customer | 1.976 | 0.777
lineitem | 50.754 | 19.990
nation | 0.002 | 0.003
orders | 11.054 | 4.895
part | 1.893 | 0.771
partsupp | 8.791 | 3.512
region | 0.002 | 0.002
supplier | 0.130 | 0.056
**Reader CPU Time (unit: second)**
name | zlib | zstd
-- | -- | --
customer | 0.084 | 0.063
lineitem | 2.263 | 2.094
nation | 0.001 | 0.001
orders | 0.454 | 0.340
part | 0.071 | 0.061
partsupp | 0.343 | 0.253
region | 0.000 | 0.001
supplier | 0.006 | 0.005
**File Size (unit: byte)**
name | zlib | zstd
-- | -- | --
customer | 7494965 | 7670751
lineitem | 162544602 | 178904712
nation | 1760 | 1882
orders | 34599561 | 38028670
part | 4273944 | 4676560
partsupp | 25766380 | 29498151
region | 1026 | 1097
supplier | 474099 | 478017
In total, ZSTD writer time has 148.6% saving and reader time has 14.4% saving. File size is 9.4% bigger for ZSTD. The result provides a basic idea of performance comparison between them. As we use default configuration (ZLIB default level is -1 and ZSTD is 3), it may be unfair because ZSTD has 22 levels while ZLIB has 9 in total. If we choose different levels or different datasets, the result can vary a lot and ZSTD can beat ZLIB on file sizes. Overall, ZSTD seems to be a good compression option.
---
[GitHub] orc issue #301: ORC-395: Support ZSTD in C++ writer/reader
Posted by majetideepak <gi...@git.apache.org>.
Github user majetideepak commented on the issue:
https://github.com/apache/orc/pull/301
@xndai I am not very familiar with the Java side. Looking at the ZSTD support in parquet-mr, it looks like they only added the ZSTD support but skipped the tests. The parquet-mr library has not upgraded the hadoop library as well.
https://github.com/apache/parquet-mr/blob/9fa86cca1af7dabc21701247efd89f6085945bd2/pom.xml#L80
https://github.com/apache/parquet-mr/commit/132b2a8c553bdcfd445e88680beac6f225c50ac4#diff-6a038e86a0fc009909af954b3589cd95R159
Can we do something like that here?
---