You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/02/09 06:11:45 UTC
[GitHub] [iceberg] zhongyujiang opened a new pull request #4075: Parquet: Fix parquet zstd compression level conf
zhongyujiang opened a new pull request #4075:
URL: https://github.com/apache/iceberg/pull/4075
Parquet had [replaced Hadoop ZSTD with JNI-ZSTD](https://github.com/apache/parquet-mr/pull/793) in 1.12, the zstd compression level preperty key changed from "io.compression.codec.zstd.level" to "parquet.compression.codec.zstd.level":
https://github.com/apache/parquet-mr/blob/300200eb72b9f16df36d9a68cf762683234aeb08/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/codec/ZstandardCodec.java#L51
@rdblue @kbendick could you help review this? thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4075: Parquet: Fix parquet zstd compression level conf
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4075:
URL: https://github.com/apache/iceberg/pull/4075#discussion_r803214857
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -235,7 +235,7 @@ private WriteBuilder createContextFunc(Function<Map<String, String>, Context> ne
config.put("compression.brotli.quality", compressionLevel);
break;
case ZSTD:
- config.put("io.compression.codec.zstd.level", compressionLevel);
+ config.put("parquet.compression.codec.zstd.level", compressionLevel);
Review comment:
Should we put both just in case? There isn't backward compatibility?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zhongyujiang commented on a change in pull request #4075: Parquet: Fix parquet zstd compression level conf
Posted by GitBox <gi...@apache.org>.
zhongyujiang commented on a change in pull request #4075:
URL: https://github.com/apache/iceberg/pull/4075#discussion_r803754180
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -235,7 +235,7 @@ private WriteBuilder createContextFunc(Function<Map<String, String>, Context> ne
config.put("compression.brotli.quality", compressionLevel);
break;
case ZSTD:
- config.put("io.compression.codec.zstd.level", compressionLevel);
+ config.put("parquet.compression.codec.zstd.level", compressionLevel);
Review comment:
Thanks for your review!
Agreed, both should be put for compatibility with users using older versions of parquet. Will update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zhongyujiang commented on a change in pull request #4075: Parquet: Fix parquet zstd compression level conf
Posted by GitBox <gi...@apache.org>.
zhongyujiang commented on a change in pull request #4075:
URL: https://github.com/apache/iceberg/pull/4075#discussion_r803754180
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -235,7 +235,7 @@ private WriteBuilder createContextFunc(Function<Map<String, String>, Context> ne
config.put("compression.brotli.quality", compressionLevel);
break;
case ZSTD:
- config.put("io.compression.codec.zstd.level", compressionLevel);
+ config.put("parquet.compression.codec.zstd.level", compressionLevel);
Review comment:
Thanks for your review!
Agreed, both should be put in for compatibility with users using older versions of parquet. Will update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #4075: Parquet: Fix parquet zstd compression level conf
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #4075:
URL: https://github.com/apache/iceberg/pull/4075#issuecomment-1043468702
Thanks, @zhongyujiang!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #4075: Parquet: Fix parquet zstd compression level conf
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #4075:
URL: https://github.com/apache/iceberg/pull/4075#discussion_r805039367
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -235,7 +235,7 @@ private WriteBuilder createContextFunc(Function<Map<String, String>, Context> ne
config.put("compression.brotli.quality", compressionLevel);
break;
case ZSTD:
- config.put("io.compression.codec.zstd.level", compressionLevel);
+ config.put("parquet.compression.codec.zstd.level", compressionLevel);
Review comment:
Yeah I'd be happy with both as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zhongyujiang commented on pull request #4075: Parquet: Fix parquet zstd compression level conf
Posted by GitBox <gi...@apache.org>.
zhongyujiang commented on pull request #4075:
URL: https://github.com/apache/iceberg/pull/4075#issuecomment-1037055256
Thanks for you review!
Agreed, that's useful, I'll add that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #4075: Parquet: Fix parquet zstd compression level conf
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #4075:
URL: https://github.com/apache/iceberg/pull/4075
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #4075: Parquet: Fix parquet zstd compression level conf
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #4075:
URL: https://github.com/apache/iceberg/pull/4075#discussion_r805039854
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -235,7 +235,7 @@ private WriteBuilder createContextFunc(Function<Map<String, String>, Context> ne
config.put("compression.brotli.quality", compressionLevel);
break;
case ZSTD:
- config.put("io.compression.codec.zstd.level", compressionLevel);
+ config.put("parquet.compression.codec.zstd.level", compressionLevel);
Review comment:
Maybe putting a comment above the first one to say something like `// keep io.compression.codec.zstd.level for backwards compatibility`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org