You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "amousavigourabi (via GitHub)" <gi...@apache.org> on 2023/12/04 12:38:18 UTC

[PR] PARQUET-2405: Include fallback CodecFactory implementation [parquet-mr]

amousavigourabi opened a new pull request, #1230:
URL: https://github.com/apache/parquet-mr/pull/1230

   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Parquet Jira](https://issues.apache.org/jira/browse/PARQUET/) issues and references
     them in the PR title. For example, "PARQUET-1234: My Parquet PR"
       - https://issues.apache.org/jira/browse/PARQUET-XXX
       - In case you are adding a dependency, check if the license complies with
         the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines
     from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
       1. Subject is separated from body by a blank line
       1. Subject is limited to 50 characters (not including Jira issue reference)
       1. Subject does not end with a period
       1. Subject uses the imperative mood ("add", not "adding")
       1. Body wraps at 72 characters
       1. Body explains "what" and "why", not "how"
   
   ### Style
   - [x] My contribution adheres to the code style guidelines and Spotless passes.
       - To apply the necessary changes, run `mvn spotless:apply -Pvector-plugins`
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes how to use it.
       - All the public functions and the classes in the PR contain Javadoc that explain what it does
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PARQUET-2405: Include fallback CodecFactory implementation [parquet-mr]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1230:
URL: https://github.com/apache/parquet-mr/pull/1230#issuecomment-1844103588

   Thanks for explanation! So users can pass the fallback CodecFactory to the parquet reader/writer now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2405: Include fallback CodecFactory implementation [parquet-mr]

Posted by "amousavigourabi (via GitHub)" <gi...@apache.org>.
amousavigourabi closed pull request #1230: PARQUET-2405: Include fallback CodecFactory implementation
URL: https://github.com/apache/parquet-mr/pull/1230


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2405: Include fallback CodecFactory implementation [parquet-mr]

Posted by "amousavigourabi (via GitHub)" <gi...@apache.org>.
amousavigourabi commented on code in PR #1230:
URL: https://github.com/apache/parquet-mr/pull/1230#discussion_r1425901480


##########
parquet-common/src/main/java/org/apache/parquet/compression/FallbackCodecFactory.java:
##########
@@ -0,0 +1,74 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing,
+ *  software distributed under the License is distributed on an
+ *  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  KIND, either express or implied.  See the License for the
+ *  specific language governing permissions and limitations
+ *  under the License.
+ */
+package org.apache.parquet.compression;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+/**
+ * {@link CompressionCodecFactory} implementation that delegates to the {@link CompressionCodecFactory} implementations it was constructed with.
+ */
+public class FallbackCodecFactory implements CompressionCodecFactory {
+
+  private final CompressionCodecFactory[] codecFactories;
+
+  /**
+   * Constructs a FallbackCodecFactory using other {@link CompressionCodecFactory} implementations.
+   * The FallbackCodecFactory recognizes that a codec could not be provided by the {@link CompressionCodecFactory} implementations throwing a {@link CodecNotImplementedException}.
+   *
+   * @param codecFactories the {@link CompressionCodecFactory} implementations to delegate to, in order.
+   */
+  public FallbackCodecFactory(CompressionCodecFactory... codecFactories) {
+    Preconditions.checkArgument(
+        codecFactories != null && codecFactories.length > 0,
+        "FallbackCodecFactory needs to be constructed using at least one CompressionCodecFactory to fall back on.");
+    this.codecFactories = codecFactories;
+  }
+
+  @Override
+  public BytesInputCompressor getCompressor(CompressionCodecName codecName) {
+    for (CompressionCodecFactory codecFactory : codecFactories) {
+      try {
+        return codecFactory.getCompressor(codecName);
+      } catch (CodecNotImplementedException ignored) {

Review Comment:
   The exception is a way for these specialized CompressionCodecFactory implementations to signal they fail to resolve a compressor for this request. The current implementations of CompressionCodecFactory we ship OOTB are pretty universal and do not run into any situations where throwing this would be appropriate.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2405: Include fallback CodecFactory implementation [parquet-mr]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #1230:
URL: https://github.com/apache/parquet-mr/pull/1230#discussion_r1421437724


##########
parquet-common/src/main/java/org/apache/parquet/compression/FallbackCodecFactory.java:
##########
@@ -0,0 +1,74 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing,
+ *  software distributed under the License is distributed on an
+ *  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  KIND, either express or implied.  See the License for the
+ *  specific language governing permissions and limitations
+ *  under the License.
+ */
+package org.apache.parquet.compression;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+/**
+ * {@link CompressionCodecFactory} implementation that delegates to the {@link CompressionCodecFactory} implementations it was constructed with.
+ */
+public class FallbackCodecFactory implements CompressionCodecFactory {

Review Comment:
   I agree with @ConeyLiu. Users can anyway do the same thing all by themselves. It seems sufficient when parquet reader and writer support passing a customized codec factory.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2405: Include fallback CodecFactory implementation [parquet-mr]

Posted by "amousavigourabi (via GitHub)" <gi...@apache.org>.
amousavigourabi commented on PR #1230:
URL: https://github.com/apache/parquet-mr/pull/1230#issuecomment-1859966086

   @wgtmac thanks for looking at it, I'll close the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2405: Include fallback CodecFactory implementation [parquet-mr]

Posted by "amousavigourabi (via GitHub)" <gi...@apache.org>.
amousavigourabi commented on code in PR #1230:
URL: https://github.com/apache/parquet-mr/pull/1230#discussion_r1421022132


##########
parquet-common/src/main/java/org/apache/parquet/compression/FallbackCodecFactory.java:
##########
@@ -0,0 +1,74 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing,
+ *  software distributed under the License is distributed on an
+ *  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  KIND, either express or implied.  See the License for the
+ *  specific language governing permissions and limitations
+ *  under the License.
+ */
+package org.apache.parquet.compression;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+/**
+ * {@link CompressionCodecFactory} implementation that delegates to the {@link CompressionCodecFactory} implementations it was constructed with.
+ */
+public class FallbackCodecFactory implements CompressionCodecFactory {

Review Comment:
   Not necessarily don't care, but who might not have guarantees that any system might be able to run their preferred codecs. If you are to implement one using hardware acceleration there are platforms for which this might not work perfectly. This factory provides an easy and modular way to allow for more stable fallbacks for these scenarios OOTB.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2405: Include fallback CodecFactory implementation [parquet-mr]

Posted by "amousavigourabi (via GitHub)" <gi...@apache.org>.
amousavigourabi commented on code in PR #1230:
URL: https://github.com/apache/parquet-mr/pull/1230#discussion_r1425903622


##########
parquet-common/src/main/java/org/apache/parquet/compression/FallbackCodecFactory.java:
##########
@@ -0,0 +1,74 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing,
+ *  software distributed under the License is distributed on an
+ *  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  KIND, either express or implied.  See the License for the
+ *  specific language governing permissions and limitations
+ *  under the License.
+ */
+package org.apache.parquet.compression;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+/**
+ * {@link CompressionCodecFactory} implementation that delegates to the {@link CompressionCodecFactory} implementations it was constructed with.
+ */
+public class FallbackCodecFactory implements CompressionCodecFactory {
+
+  private final CompressionCodecFactory[] codecFactories;
+
+  /**
+   * Constructs a FallbackCodecFactory using other {@link CompressionCodecFactory} implementations.
+   * The FallbackCodecFactory recognizes that a codec could not be provided by the {@link CompressionCodecFactory} implementations throwing a {@link CodecNotImplementedException}.
+   *
+   * @param codecFactories the {@link CompressionCodecFactory} implementations to delegate to, in order.
+   */
+  public FallbackCodecFactory(CompressionCodecFactory... codecFactories) {
+    Preconditions.checkArgument(
+        codecFactories != null && codecFactories.length > 0,
+        "FallbackCodecFactory needs to be constructed using at least one CompressionCodecFactory to fall back on.");
+    this.codecFactories = codecFactories;
+  }
+
+  @Override
+  public BytesInputCompressor getCompressor(CompressionCodecName codecName) {
+    for (CompressionCodecFactory codecFactory : codecFactories) {
+      try {
+        return codecFactory.getCompressor(codecName);

Review Comment:
   I'm not sure whether that would be a good idea, using the custom exception here seems more idiomatic. Using null returns in codec factories would probably also run into relatively hard-to-diagnose issues when used without such fallback mechanisms, so I'd prefer to discourage null returns as much as possible when (de)compressors cannot be resolved for these specialized user implementations.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2405: Include fallback CodecFactory implementation [parquet-mr]

Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #1230:
URL: https://github.com/apache/parquet-mr/pull/1230#discussion_r1418840220


##########
parquet-common/src/main/java/org/apache/parquet/compression/FallbackCodecFactory.java:
##########
@@ -0,0 +1,74 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing,
+ *  software distributed under the License is distributed on an
+ *  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  KIND, either express or implied.  See the License for the
+ *  specific language governing permissions and limitations
+ *  under the License.
+ */
+package org.apache.parquet.compression;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+/**
+ * {@link CompressionCodecFactory} implementation that delegates to the {@link CompressionCodecFactory} implementations it was constructed with.
+ */
+public class FallbackCodecFactory implements CompressionCodecFactory {

Review Comment:
   Is there any specific reason to add this? This seems more like a personal request.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2405: Include fallback CodecFactory implementation [parquet-mr]

Posted by "amousavigourabi (via GitHub)" <gi...@apache.org>.
amousavigourabi commented on code in PR #1230:
URL: https://github.com/apache/parquet-mr/pull/1230#discussion_r1418866150


##########
parquet-common/src/main/java/org/apache/parquet/compression/FallbackCodecFactory.java:
##########
@@ -0,0 +1,74 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing,
+ *  software distributed under the License is distributed on an
+ *  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  KIND, either express or implied.  See the License for the
+ *  specific language governing permissions and limitations
+ *  under the License.
+ */
+package org.apache.parquet.compression;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+/**
+ * {@link CompressionCodecFactory} implementation that delegates to the {@link CompressionCodecFactory} implementations it was constructed with.
+ */
+public class FallbackCodecFactory implements CompressionCodecFactory {

Review Comment:
   It aims to provide more support for the use-cases set out by [PARQUET-1804](https://issues.apache.org/jira/browse/PARQUET-1804), through custom CodecFactory implementations. This implementation would more easily allow for partial CodecFactory implementations by users, strengthening the usability of this part of the API.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2405: Include fallback CodecFactory implementation [parquet-mr]

Posted by "amousavigourabi (via GitHub)" <gi...@apache.org>.
amousavigourabi commented on PR #1230:
URL: https://github.com/apache/parquet-mr/pull/1230#issuecomment-1841119900

   > The implementation looks good. But it seems not used anywhere except the test. Do you want to include its usage in this PR as well?
   
   Thanks for the review @wgtmac ! The implementation is meant s.t. users can more easily implement and use partial CodecFactories themselves, without worrying about encountering codecs they did not include.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PARQUET-2405: Include fallback CodecFactory implementation [parquet-mr]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1230:
URL: https://github.com/apache/parquet-mr/pull/1230#issuecomment-1841066111

   The implementation looks good. But it seems not used anywhere except the test. Do you want to include its usage in this PR as well?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PARQUET-2405: Include fallback CodecFactory implementation [parquet-mr]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #1230:
URL: https://github.com/apache/parquet-mr/pull/1230#discussion_r1421438240


##########
parquet-common/src/main/java/org/apache/parquet/compression/FallbackCodecFactory.java:
##########
@@ -0,0 +1,74 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing,
+ *  software distributed under the License is distributed on an
+ *  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  KIND, either express or implied.  See the License for the
+ *  specific language governing permissions and limitations
+ *  under the License.
+ */
+package org.apache.parquet.compression;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+/**
+ * {@link CompressionCodecFactory} implementation that delegates to the {@link CompressionCodecFactory} implementations it was constructed with.
+ */
+public class FallbackCodecFactory implements CompressionCodecFactory {
+
+  private final CompressionCodecFactory[] codecFactories;
+
+  /**
+   * Constructs a FallbackCodecFactory using other {@link CompressionCodecFactory} implementations.
+   * The FallbackCodecFactory recognizes that a codec could not be provided by the {@link CompressionCodecFactory} implementations throwing a {@link CodecNotImplementedException}.
+   *
+   * @param codecFactories the {@link CompressionCodecFactory} implementations to delegate to, in order.
+   */
+  public FallbackCodecFactory(CompressionCodecFactory... codecFactories) {
+    Preconditions.checkArgument(
+        codecFactories != null && codecFactories.length > 0,
+        "FallbackCodecFactory needs to be constructed using at least one CompressionCodecFactory to fall back on.");
+    this.codecFactories = codecFactories;
+  }
+
+  @Override
+  public BytesInputCompressor getCompressor(CompressionCodecName codecName) {
+    for (CompressionCodecFactory codecFactory : codecFactories) {
+      try {
+        return codecFactory.getCompressor(codecName);

Review Comment:
   What if it returns null? Should it fallback as well?



##########
parquet-common/src/main/java/org/apache/parquet/compression/FallbackCodecFactory.java:
##########
@@ -0,0 +1,74 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing,
+ *  software distributed under the License is distributed on an
+ *  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  KIND, either express or implied.  See the License for the
+ *  specific language governing permissions and limitations
+ *  under the License.
+ */
+package org.apache.parquet.compression;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+/**
+ * {@link CompressionCodecFactory} implementation that delegates to the {@link CompressionCodecFactory} implementations it was constructed with.
+ */
+public class FallbackCodecFactory implements CompressionCodecFactory {
+
+  private final CompressionCodecFactory[] codecFactories;
+
+  /**
+   * Constructs a FallbackCodecFactory using other {@link CompressionCodecFactory} implementations.
+   * The FallbackCodecFactory recognizes that a codec could not be provided by the {@link CompressionCodecFactory} implementations throwing a {@link CodecNotImplementedException}.
+   *
+   * @param codecFactories the {@link CompressionCodecFactory} implementations to delegate to, in order.
+   */
+  public FallbackCodecFactory(CompressionCodecFactory... codecFactories) {
+    Preconditions.checkArgument(
+        codecFactories != null && codecFactories.length > 0,
+        "FallbackCodecFactory needs to be constructed using at least one CompressionCodecFactory to fall back on.");
+    this.codecFactories = codecFactories;
+  }
+
+  @Override
+  public BytesInputCompressor getCompressor(CompressionCodecName codecName) {
+    for (CompressionCodecFactory codecFactory : codecFactories) {
+      try {
+        return codecFactory.getCompressor(codecName);
+      } catch (CodecNotImplementedException ignored) {

Review Comment:
   Here it won't catch `CodecNotImplementedException` unless the internal codecFactory is also a FallbackCodecFactory, which sounds weird to me.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2405: Include fallback CodecFactory implementation [parquet-mr]

Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #1230:
URL: https://github.com/apache/parquet-mr/pull/1230#discussion_r1420072472


##########
parquet-common/src/main/java/org/apache/parquet/compression/FallbackCodecFactory.java:
##########
@@ -0,0 +1,74 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing,
+ *  software distributed under the License is distributed on an
+ *  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ *  KIND, either express or implied.  See the License for the
+ *  specific language governing permissions and limitations
+ *  under the License.
+ */
+package org.apache.parquet.compression;
+
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+/**
+ * {@link CompressionCodecFactory} implementation that delegates to the {@link CompressionCodecFactory} implementations it was constructed with.
+ */
+public class FallbackCodecFactory implements CompressionCodecFactory {

Review Comment:
   It seems the usage for this is people don't care about which compression codec is used. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org