You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@orc.apache.org by GitBox <gi...@apache.org> on 2021/08/31 19:09:40 UTC
[GitHub] [orc] dongjoon-hyun opened a new pull request #890: ORC-965: Fix ZSTD 'Overflow detected' failure
dongjoon-hyun opened a new pull request #890:
URL: https://github.com/apache/orc/pull/890
### What changes were proposed in this pull request?
This PR aims to fix `Overflow detected` failures in ZSTD compression by bringing new aircompressor version.
- https://github.com/airlift/aircompressor/commit/1e364f713390008eada1daa451e7b42cd6647250
### Why are the changes needed?
`Overflow detected` happens not only for the small buffer, but also for the incompressible data.
### How was this patch tested?
Pass the UTs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun commented on pull request #890: ORC-965: Fix ZSTD 'Overflow detected' failure
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #890:
URL: https://github.com/apache/orc/pull/890#issuecomment-909708133
Thank you, @williamhyun ! `branch-1.7/1.6` needs this too. I'll backport this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun commented on a change in pull request #890: ORC-965: Fix ZSTD 'Overflow detected' failure
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #890:
URL: https://github.com/apache/orc/pull/890#discussion_r699602594
##########
File path: java/core/src/test/org/apache/orc/TestVectorOrcFile.java
##########
@@ -2281,7 +2281,7 @@ public void testZstd(Version fileFormat) throws Exception {
OrcFile.writerOptions(conf)
.setSchema(schema)
.compress(CompressionKind.ZSTD)
- .bufferSize(2000)
+ .bufferSize(1000)
Review comment:
With aircompressor 0.19/0.20, this raises `Overflow detected` error.
```
Caused by: java.lang.IllegalStateException: Overflow detected
at io.airlift.compress.zstd.Util.checkState(Util.java:59)
at io.airlift.compress.zstd.BitOutputStream.close(BitOutputStream.java:85)
at io.airlift.compress.zstd.HuffmanCompressor.compressSingleStream(HuffmanCompressor.java:130)
at io.airlift.compress.zstd.HuffmanCompressor.compress4streams(HuffmanCompressor.java:75)
```
##########
File path: java/core/src/test/org/apache/orc/TestVectorOrcFile.java
##########
@@ -2281,7 +2281,7 @@ public void testZstd(Version fileFormat) throws Exception {
OrcFile.writerOptions(conf)
.setSchema(schema)
.compress(CompressionKind.ZSTD)
- .bufferSize(2000)
+ .bufferSize(1000)
Review comment:
With aircompressor 0.19/0.20, this causes `Overflow detected` errors.
```
Caused by: java.lang.IllegalStateException: Overflow detected
at io.airlift.compress.zstd.Util.checkState(Util.java:59)
at io.airlift.compress.zstd.BitOutputStream.close(BitOutputStream.java:85)
at io.airlift.compress.zstd.HuffmanCompressor.compressSingleStream(HuffmanCompressor.java:130)
at io.airlift.compress.zstd.HuffmanCompressor.compress4streams(HuffmanCompressor.java:75)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] williamhyun commented on pull request #890: ORC-965: Fix ZSTD 'Overflow detected' failure
Posted by GitBox <gi...@apache.org>.
williamhyun commented on pull request #890:
URL: https://github.com/apache/orc/pull/890#issuecomment-909705673
+1 LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun commented on pull request #890: ORC-965: Fix ZSTD 'Overflow detected' failure
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #890:
URL: https://github.com/apache/orc/pull/890#issuecomment-909526855
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun commented on pull request #890: ORC-965: Fix ZSTD 'Overflow detected' failure
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #890:
URL: https://github.com/apache/orc/pull/890#issuecomment-909526855
cc @williamhyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun commented on a change in pull request #890: ORC-965: Fix ZSTD 'Overflow detected' failure
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #890:
URL: https://github.com/apache/orc/pull/890#discussion_r699602594
##########
File path: java/core/src/test/org/apache/orc/TestVectorOrcFile.java
##########
@@ -2281,7 +2281,7 @@ public void testZstd(Version fileFormat) throws Exception {
OrcFile.writerOptions(conf)
.setSchema(schema)
.compress(CompressionKind.ZSTD)
- .bufferSize(2000)
+ .bufferSize(1000)
Review comment:
With aircompressor 0.19/0.20, this raises `Overflow detected` error.
```
Caused by: java.lang.IllegalStateException: Overflow detected
at io.airlift.compress.zstd.Util.checkState(Util.java:59)
at io.airlift.compress.zstd.BitOutputStream.close(BitOutputStream.java:85)
at io.airlift.compress.zstd.HuffmanCompressor.compressSingleStream(HuffmanCompressor.java:130)
at io.airlift.compress.zstd.HuffmanCompressor.compress4streams(HuffmanCompressor.java:75)
```
##########
File path: java/core/src/test/org/apache/orc/TestVectorOrcFile.java
##########
@@ -2281,7 +2281,7 @@ public void testZstd(Version fileFormat) throws Exception {
OrcFile.writerOptions(conf)
.setSchema(schema)
.compress(CompressionKind.ZSTD)
- .bufferSize(2000)
+ .bufferSize(1000)
Review comment:
With aircompressor 0.19/0.20, this causes `Overflow detected` errors.
```
Caused by: java.lang.IllegalStateException: Overflow detected
at io.airlift.compress.zstd.Util.checkState(Util.java:59)
at io.airlift.compress.zstd.BitOutputStream.close(BitOutputStream.java:85)
at io.airlift.compress.zstd.HuffmanCompressor.compressSingleStream(HuffmanCompressor.java:130)
at io.airlift.compress.zstd.HuffmanCompressor.compress4streams(HuffmanCompressor.java:75)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] williamhyun merged pull request #890: ORC-965: Fix ZSTD 'Overflow detected' failure
Posted by GitBox <gi...@apache.org>.
williamhyun merged pull request #890:
URL: https://github.com/apache/orc/pull/890
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] williamhyun merged pull request #890: ORC-965: Fix ZSTD 'Overflow detected' failure
Posted by GitBox <gi...@apache.org>.
williamhyun merged pull request #890:
URL: https://github.com/apache/orc/pull/890
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] williamhyun commented on pull request #890: ORC-965: Fix ZSTD 'Overflow detected' failure
Posted by GitBox <gi...@apache.org>.
williamhyun commented on pull request #890:
URL: https://github.com/apache/orc/pull/890#issuecomment-909705673
+1 LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun commented on pull request #890: ORC-965: Fix ZSTD 'Overflow detected' failure
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #890:
URL: https://github.com/apache/orc/pull/890#issuecomment-909554623
Also, cc @pgaref and @omalley
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org