You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Fokko (via GitHub)" <gi...@apache.org> on 2023/08/07 10:59:46 UTC

[GitHub] [parquet-mr] Fokko opened a new pull request, #1134: PARQUET-2336: Add caching key to CodecFactory

Fokko opened a new pull request, #1134:
URL: https://github.com/apache/parquet-mr/pull/1134

   Make sure you have checked _all_ steps below.
   
   The `CODEC_BY_NAME` is static and may be used across different configurations. If a codec is initialized it will be re-used no matter what the configuration is. This is a problem when there are different compression levels used.
   
   https://github.com/apache/parquet-mr/blob/515734c373f69b5250e8b63eb3d1c973da893b63/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/CodecFactory.java#L45-L46
   
   Therefore we need to cache per compression level as well.
   
   ### Jira
   
   - [ ] My PR addresses the following [Parquet Jira](https://issues.apache.org/jira/browse/PARQUET/) issues and references them in the PR title. For example, "PARQUET-2336: My Parquet PR"
     - https://issues.apache.org/jira/browse/PARQUET-XXX
     - In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
     1. Subject is separated from body by a blank line
     1. Subject is limited to 50 characters (not including Jira issue reference)
     1. Subject does not end with a period
     1. Subject uses the imperative mood ("add", not "adding")
     1. Body wraps at 72 characters
     1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes how to use it.
     - All the public functions and the classes in the PR contain Javadoc that explain what it does
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [parquet-mr] wgtmac commented on pull request #1134: PARQUET-2336: Add caching key to CodecFactory

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1134:
URL: https://github.com/apache/parquet-mr/pull/1134#issuecomment-1671556700

   > Overall this looks good. It changes the cache, but I can't think of why anyone would use it directly.
   > 
   > @Fokko, do we have a long-term plan for getting off of Hadoop codecs? It seems like that is a good idea. I think the main blocker is that we are still using these codecs. Otherwise we would be able to remove Hadoop dependencies fairly easily.
   
   I have added the aircompressor library when I supported the LZ4_RAW codec. Not sure if this makes it easier. @rdblue 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [parquet-mr] Fokko commented on a diff in pull request #1134: PARQUET-2336: Add caching key to CodecFactory

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1134:
URL: https://github.com/apache/parquet-mr/pull/1134#discussion_r1288681389


##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/CodecFactory.java:
##########
@@ -234,11 +234,11 @@ protected CompressionCodec getCodec(CompressionCodecName codecName) {
     if (codecClassName == null) {
       return null;
     }
-    CompressionCodec codec = CODEC_BY_NAME.get(codecClassName);
+    String codecCacheKey = this.cacheKey(codecName);
+    CompressionCodec codec = CODEC_BY_NAME.get(codecCacheKey);

Review Comment:
   Personally, I'm not too worried about this, I don't see anyone doing this. At least nobody in the Apache org: https://github.com/search?q=org%3Aapache%20CODEC_BY_NAME&type=code :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [parquet-mr] zhongyujiang commented on a diff in pull request #1134: PARQUET-2336: Add caching key to CodecFactory

Posted by "zhongyujiang (via GitHub)" <gi...@apache.org>.
zhongyujiang commented on code in PR #1134:
URL: https://github.com/apache/parquet-mr/pull/1134#discussion_r1298198034


##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/CodecFactory.java:
##########
@@ -248,13 +248,36 @@ protected CompressionCodec getCodec(CompressionCodecName codecName) {
         codecClass = configuration.getClassLoader().loadClass(codecClassName);
       }
       codec = (CompressionCodec) ReflectionUtils.newInstance(codecClass, configuration);
-      CODEC_BY_NAME.put(codecClassName, codec);
+      CODEC_BY_NAME.put(codecCacheKey, codec);
       return codec;
     } catch (ClassNotFoundException e) {
       throw new BadConfigurationException("Class " + codecClassName + " was not found", e);
     }
   }
 
+  private String cacheKey(CompressionCodecName codecName) {
+    String level = null;
+    switch (codecName) {
+      case GZIP:
+        level = configuration.get("zlib.compress.level");
+        break;
+      case BROTLI:
+        level = configuration.get("compression.brotli.quality");
+        break;
+      case ZSTD:
+        level = configuration.get("parquet.compression.codec.zstd.level");
+        if (level == null) {
+          // keep "io.compression.codec.zstd.level" for backwards compatibility
+          level = configuration.get("io.compression.codec.zstd.level");

Review Comment:
   Is it necessary to cache the old config levels ? It's already been deprecated and currently only the [new config](https://github.com/apache/parquet-mr/blob/a3a1ad4e58518a469970b45ccef2fb64695c1894/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/codec/ZstandardCodec.java#L112) is used.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [parquet-mr] zhongyujiang commented on a diff in pull request #1134: PARQUET-2336: Add caching key to CodecFactory

Posted by "zhongyujiang (via GitHub)" <gi...@apache.org>.
zhongyujiang commented on code in PR #1134:
URL: https://github.com/apache/parquet-mr/pull/1134#discussion_r1298198034


##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/CodecFactory.java:
##########
@@ -248,13 +248,36 @@ protected CompressionCodec getCodec(CompressionCodecName codecName) {
         codecClass = configuration.getClassLoader().loadClass(codecClassName);
       }
       codec = (CompressionCodec) ReflectionUtils.newInstance(codecClass, configuration);
-      CODEC_BY_NAME.put(codecClassName, codec);
+      CODEC_BY_NAME.put(codecCacheKey, codec);
       return codec;
     } catch (ClassNotFoundException e) {
       throw new BadConfigurationException("Class " + codecClassName + " was not found", e);
     }
   }
 
+  private String cacheKey(CompressionCodecName codecName) {
+    String level = null;
+    switch (codecName) {
+      case GZIP:
+        level = configuration.get("zlib.compress.level");
+        break;
+      case BROTLI:
+        level = configuration.get("compression.brotli.quality");
+        break;
+      case ZSTD:
+        level = configuration.get("parquet.compression.codec.zstd.level");
+        if (level == null) {
+          // keep "io.compression.codec.zstd.level" for backwards compatibility
+          level = configuration.get("io.compression.codec.zstd.level");

Review Comment:
   Do we need to cache the old config levels ? It's already been deprecated and currently only the [new config](https://github.com/apache/parquet-mr/blob/a3a1ad4e58518a469970b45ccef2fb64695c1894/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/codec/ZstandardCodec.java#L112) is used.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [parquet-mr] wgtmac commented on a diff in pull request #1134: PARQUET-2336: Add caching key to CodecFactory

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #1134:
URL: https://github.com/apache/parquet-mr/pull/1134#discussion_r1288642334


##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/CodecFactory.java:
##########
@@ -234,11 +234,11 @@ protected CompressionCodec getCodec(CompressionCodecName codecName) {
     if (codecClassName == null) {
       return null;
     }
-    CompressionCodec codec = CODEC_BY_NAME.get(codecClassName);
+    String codecCacheKey = this.cacheKey(codecName);
+    CompressionCodec codec = CODEC_BY_NAME.get(codecCacheKey);

Review Comment:
   If that is a concern, we can cache the old key (w/o level) as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [parquet-mr] Fokko commented on a diff in pull request #1134: PARQUET-2336: Add caching key to CodecFactory

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1134:
URL: https://github.com/apache/parquet-mr/pull/1134#discussion_r1324368223


##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/CodecFactory.java:
##########
@@ -248,13 +248,36 @@ protected CompressionCodec getCodec(CompressionCodecName codecName) {
         codecClass = configuration.getClassLoader().loadClass(codecClassName);
       }
       codec = (CompressionCodec) ReflectionUtils.newInstance(codecClass, configuration);
-      CODEC_BY_NAME.put(codecClassName, codec);
+      CODEC_BY_NAME.put(codecCacheKey, codec);
       return codec;
     } catch (ClassNotFoundException e) {
       throw new BadConfigurationException("Class " + codecClassName + " was not found", e);
     }
   }
 
+  private String cacheKey(CompressionCodecName codecName) {
+    String level = null;
+    switch (codecName) {
+      case GZIP:
+        level = configuration.get("zlib.compress.level");
+        break;
+      case BROTLI:
+        level = configuration.get("compression.brotli.quality");
+        break;
+      case ZSTD:
+        level = configuration.get("parquet.compression.codec.zstd.level");
+        if (level == null) {
+          // keep "io.compression.codec.zstd.level" for backwards compatibility
+          level = configuration.get("io.compression.codec.zstd.level");

Review Comment:
   Thanks, let's remove it then 👍🏻 



##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/CodecFactory.java:
##########
@@ -248,13 +248,36 @@ protected CompressionCodec getCodec(CompressionCodecName codecName) {
         codecClass = configuration.getClassLoader().loadClass(codecClassName);
       }
       codec = (CompressionCodec) ReflectionUtils.newInstance(codecClass, configuration);
-      CODEC_BY_NAME.put(codecClassName, codec);
+      CODEC_BY_NAME.put(codecCacheKey, codec);
       return codec;
     } catch (ClassNotFoundException e) {
       throw new BadConfigurationException("Class " + codecClassName + " was not found", e);
     }
   }
 
+  private String cacheKey(CompressionCodecName codecName) {
+    String level = null;
+    switch (codecName) {
+      case GZIP:
+        level = configuration.get("zlib.compress.level");
+        break;
+      case BROTLI:
+        level = configuration.get("compression.brotli.quality");
+        break;
+      case ZSTD:
+        level = configuration.get("parquet.compression.codec.zstd.level");
+        if (level == null) {
+          // keep "io.compression.codec.zstd.level" for backwards compatibility
+          level = configuration.get("io.compression.codec.zstd.level");

Review Comment:
   Thanks, let's remove it then 👍🏻 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [parquet-mr] rdblue commented on a diff in pull request #1134: PARQUET-2336: Add caching key to CodecFactory

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #1134:
URL: https://github.com/apache/parquet-mr/pull/1134#discussion_r1286029520


##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/CodecFactory.java:
##########
@@ -234,11 +234,11 @@ protected CompressionCodec getCodec(CompressionCodecName codecName) {
     if (codecClassName == null) {
       return null;
     }
-    CompressionCodec codec = CODEC_BY_NAME.get(codecClassName);
+    String codecCacheKey = this.cacheKey(codecName);
+    CompressionCodec codec = CODEC_BY_NAME.get(codecCacheKey);

Review Comment:
   Since `CODEC_BY_NAME` is `protected`, I think this _could_ break something that is relying on the cache, although I'm not sure why someone would access it directly. Maybe that visibility is an accident?



##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/CodecFactory.java:
##########
@@ -234,11 +234,11 @@ protected CompressionCodec getCodec(CompressionCodecName codecName) {
     if (codecClassName == null) {
       return null;
     }
-    CompressionCodec codec = CODEC_BY_NAME.get(codecClassName);
+    String codecCacheKey = this.cacheKey(codecName);
+    CompressionCodec codec = CODEC_BY_NAME.get(codecCacheKey);
     if (codec != null) {
       return codec;
     }
-

Review Comment:
   Nit: unnecessary whitespace change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [parquet-mr] Fokko commented on pull request #1134: PARQUET-2336: Add caching key to CodecFactory

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #1134:
URL: https://github.com/apache/parquet-mr/pull/1134#issuecomment-1724265539

   Thanks everyone for the reviews!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [parquet-mr] Fokko merged pull request #1134: PARQUET-2336: Add caching key to CodecFactory

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko merged PR #1134:
URL: https://github.com/apache/parquet-mr/pull/1134


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [parquet-mr] wgtmac commented on pull request #1134: PARQUET-2336: Add caching key to CodecFactory

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1134:
URL: https://github.com/apache/parquet-mr/pull/1134#issuecomment-1671601687

   Just curious, is brotli widely adopted? It seems that it does not have an official java encoder implementation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [parquet-mr] Fokko commented on pull request #1134: PARQUET-2336: Add caching key to CodecFactory

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #1134:
URL: https://github.com/apache/parquet-mr/pull/1134#issuecomment-1671798595

   @wgtmac I think it is quite arcane, maybe we can make the `aircompressor` the default at some point, but I think we should keep support around, otherwise, folks won't be able to access their data.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [parquet-mr] Fokko commented on pull request #1134: PARQUET-2336: Add caching key to CodecFactory

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #1134:
URL: https://github.com/apache/parquet-mr/pull/1134#issuecomment-1671588732

   Thanks @wgtmac for jumping in here
   
   > @Fokko, do we have a long-term plan for getting off of Hadoop codecs? It seems like that is a good idea. I think the main blocker is that we are still using these codecs. Otherwise, we would be able to remove Hadoop dependencies fairly easily.
   
   I have a PR that I need to revisit https://github.com/apache/iceberg/pull/7369. It requires some awkward changes to make it work. One issue with the Aircompressor codec it doesn't provide Brolti due to licensing issues.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [parquet-mr] zhongyujiang commented on a diff in pull request #1134: PARQUET-2336: Add caching key to CodecFactory

Posted by "zhongyujiang (via GitHub)" <gi...@apache.org>.
zhongyujiang commented on code in PR #1134:
URL: https://github.com/apache/parquet-mr/pull/1134#discussion_r1298198034


##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/CodecFactory.java:
##########
@@ -248,13 +248,36 @@ protected CompressionCodec getCodec(CompressionCodecName codecName) {
         codecClass = configuration.getClassLoader().loadClass(codecClassName);
       }
       codec = (CompressionCodec) ReflectionUtils.newInstance(codecClass, configuration);
-      CODEC_BY_NAME.put(codecClassName, codec);
+      CODEC_BY_NAME.put(codecCacheKey, codec);
       return codec;
     } catch (ClassNotFoundException e) {
       throw new BadConfigurationException("Class " + codecClassName + " was not found", e);
     }
   }
 
+  private String cacheKey(CompressionCodecName codecName) {
+    String level = null;
+    switch (codecName) {
+      case GZIP:
+        level = configuration.get("zlib.compress.level");
+        break;
+      case BROTLI:
+        level = configuration.get("compression.brotli.quality");
+        break;
+      case ZSTD:
+        level = configuration.get("parquet.compression.codec.zstd.level");
+        if (level == null) {
+          // keep "io.compression.codec.zstd.level" for backwards compatibility
+          level = configuration.get("io.compression.codec.zstd.level");

Review Comment:
   Is it necessary to cache the old config levels ? It's already been deprecated and currently only the [new config](https://github.com/apache/parquet-mr/blob/a3a1ad4e58518a469970b45ccef2fb64695c1894/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/codec/ZstandardCodec.java#L112) is being considered.



##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/CodecFactory.java:
##########
@@ -248,13 +248,36 @@ protected CompressionCodec getCodec(CompressionCodecName codecName) {
         codecClass = configuration.getClassLoader().loadClass(codecClassName);
       }
       codec = (CompressionCodec) ReflectionUtils.newInstance(codecClass, configuration);
-      CODEC_BY_NAME.put(codecClassName, codec);
+      CODEC_BY_NAME.put(codecCacheKey, codec);
       return codec;
     } catch (ClassNotFoundException e) {
       throw new BadConfigurationException("Class " + codecClassName + " was not found", e);
     }
   }
 
+  private String cacheKey(CompressionCodecName codecName) {
+    String level = null;
+    switch (codecName) {
+      case GZIP:
+        level = configuration.get("zlib.compress.level");
+        break;
+      case BROTLI:
+        level = configuration.get("compression.brotli.quality");
+        break;
+      case ZSTD:
+        level = configuration.get("parquet.compression.codec.zstd.level");
+        if (level == null) {
+          // keep "io.compression.codec.zstd.level" for backwards compatibility
+          level = configuration.get("io.compression.codec.zstd.level");

Review Comment:
   Is it necessary to cache the old config levels ? It's already been deprecated and currently only the [new config](https://github.com/apache/parquet-mr/blob/a3a1ad4e58518a469970b45ccef2fb64695c1894/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/codec/ZstandardCodec.java#L112) is being considered.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org