[GitHub] [arrow] davisusanibar opened a new pull request #11591: [Java] ARROW-12163 - Make compression levels configurable

[GitBox <gi...@apache.org>]

davisusanibar opened a new pull request #11591:
URL: https://github.com/apache/arrow/pull/11591


   - Adding compression level configurable at read/write files to use Lz4CompressionCodec / ZstdCompressionCodec
   - Changes on modules: vector & compression 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] pitrou commented on a change in pull request #11591: ARROW-12163: [Java] Make compression levels configurable.

[GitBox <gi...@apache.org>]

pitrou commented on a change in pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#discussion_r741887094



##########
File path: docs/source/java/ipc.rst
##########
@@ -54,6 +54,15 @@ Now, we can begin writing a stream containing some number of these batches. For
     ArrowStreamWriter writer = new ArrowStreamWriter(root, /*DictionaryProvider=*/null, Channels.newChannel(out));
 
 
+Buffer level compression is also supported by passing a codec:
+Lz4CompressionCodec / ZstdCompressionCodec:
+
+.. code-block:: Java
+
+    ArrowStreamWriter writer = new ArrowStreamWriter(schemaRootWrite, true, new Lz4CompressionCodec(), true,
+        null, fileOutputStream.getChannel());

Review comment:
       "true, null" doesn't say much about what these parameters are. Should you add commented parameter names as above (e.g. `/*DictionaryProvider=*/null`, but I don't know if it's the right name here?).

##########
File path: docs/source/java/ipc.rst
##########
@@ -54,6 +54,15 @@ Now, we can begin writing a stream containing some number of these batches. For
     ArrowStreamWriter writer = new ArrowStreamWriter(root, /*DictionaryProvider=*/null, Channels.newChannel(out));
 
 
+Buffer level compression is also supported by passing a codec:
+Lz4CompressionCodec / ZstdCompressionCodec:

Review comment:
       Looks like there is a missing sentence here? Are these the supported codecs? Something else?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] amol- edited a comment on pull request #11591: ARROW-12163: [Java] Make compression levels configurable.

[GitBox <gi...@apache.org>]

amol- edited a comment on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-959234041






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] amol- edited a comment on pull request #11591: ARROW-12163: [Java] Make compression levels configurable.

[GitBox <gi...@apache.org>]

amol- edited a comment on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-959234041


   > One more note. The JIRA linked is about adjusting the compression level used by compression algorithms (trading off speed for output size). I didnt notice a change here for that apologies if i missed it.
   
   @davisusanibar that seems true, the original goal of the JIRA ticket was to allow to configure the compression by specifying options like the level. Similarly to what we allow in C++ and Python where you can set `compression` and `compression_level` ( https://arrow.apache.org/docs/python/generated/pyarrow.Codec.html?highlight=codec#pyarrow.Codec ).
   
   It seems that the `CompressionCodec` and `CommonsCompressionFactory` do not allow to specify anything else than the `codecType` but not a level for such codecs. 
   
   Should `Lz4CompressionCodec` and `ZstdCompressionCodec` grow a constructor argument to specify the level?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] github-actions[bot] commented on pull request #11591: [Java] ARROW-12163 - Make compression levels configurable

[GitBox <gi...@apache.org>]

github-actions[bot] commented on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-957355018


   <!--
     Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
     regarding copyright ownership.  The ASF licenses this file
     to you under the Apache License, Version 2.0 (the
     "License"); you may not use this file except in compliance
     with the License.  You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICENSE-2.0
   
     Unless required by applicable law or agreed to in writing,
     software distributed under the License is distributed on an
     "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
     KIND, either express or implied.  See the License for the
     specific language governing permissions and limitations
     under the License.
   -->
   
   Thanks for opening a pull request!
   
   If this is not a [minor PR](https://github.com/apache/arrow/blob/master/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW
   
   Opening JIRAs ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
   
   Then could you also rename pull request title in the following format?
   
       ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
   
   or
   
       MINOR: [${COMPONENT}] ${SUMMARY}
   
   See also:
   
     * [Other pull requests](https://github.com/apache/arrow/pulls/)
     * [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] edponce commented on pull request #11591: [Java] ARROW-12163 - Make compression levels configurable

[GitBox <gi...@apache.org>]

edponce commented on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-957988740


   Please update the PR title to *ARROW-12163: [Java] Make compression levels configurable*.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] emkornfield commented on a change in pull request #11591: [Java] ARROW-12163 - Make compression levels configurable

[GitBox <gi...@apache.org>]

emkornfield commented on a change in pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#discussion_r741330445



##########
File path: java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodecFile.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.compression;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.compression.CompressionCodec;
+import org.apache.arrow.vector.compression.CompressionUtil;
+import org.apache.arrow.vector.compression.NoCompressionCodec;
+import org.apache.arrow.vector.ipc.ArrowFileReader;
+import org.apache.arrow.vector.ipc.ArrowFileWriter;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.arrow.vector.ipc.ArrowStreamReader;
+import org.apache.arrow.vector.ipc.ArrowStreamWriter;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+public class TestCompressionCodecFile {
+  private final CompressionCodec codec;
+  private final int vectorLength;
+  private final File file;
+
+  public TestCompressionCodecFile(CompressionUtil.CodecType type, int vectorLength, CompressionCodec codec,
+                                  File file) {
+    this.codec = codec;
+    this.file = file;
+    this.vectorLength = vectorLength;
+  }
+
+  @Parameterized.Parameters(name = "codec = {0}, length = {1}, file = {2}")
+  public static Collection<Object[]> getCodecs() {
+    List<Object[]> params = new ArrayList<>();
+
+    int[] lengths = new int[]{10, 100, 1000};
+    for (int len : lengths) {
+      CompressionCodec dumbCodec = NoCompressionCodec.INSTANCE;
+      File fileNoCompression = new File("target/write_no_compression_" + len + ".arrow");
+      params.add(new Object[]{dumbCodec.getCodecType(), len, dumbCodec, fileNoCompression});
+
+      CompressionCodec lz4Codec = new Lz4CompressionCodec();
+      File fileLZ4Compression = new File("target/write_lz4_compression_" + len + ".arrow");

Review comment:
       ```suggestion
         File fileLz4Compression = new File("target/write_lz4_compression_" + len + ".arrow");
   ```
   nit: generally we use camel case for abbreviates as well

##########
File path: docs/source/java/ipc.rst
##########
@@ -54,6 +54,15 @@ Now, we can begin writing a stream containing some number of these batches. For
     ArrowStreamWriter writer = new ArrowStreamWriter(root, /*DictionaryProvider=*/null, Channels.newChannel(out));
 
 
+Here we are not used compression option, but this could be implemented on this way thru codec option:

Review comment:
       ```suggestion
   Buffer level compression is also supported by passing a codec:
   ```

##########
File path: java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodecFile.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.compression;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.compression.CompressionCodec;
+import org.apache.arrow.vector.compression.CompressionUtil;
+import org.apache.arrow.vector.compression.NoCompressionCodec;
+import org.apache.arrow.vector.ipc.ArrowFileReader;
+import org.apache.arrow.vector.ipc.ArrowFileWriter;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.arrow.vector.ipc.ArrowStreamReader;
+import org.apache.arrow.vector.ipc.ArrowStreamWriter;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+public class TestCompressionCodecFile {
+  private final CompressionCodec codec;
+  private final int vectorLength;
+  private final File file;
+
+  public TestCompressionCodecFile(CompressionUtil.CodecType type, int vectorLength, CompressionCodec codec,
+                                  File file) {
+    this.codec = codec;
+    this.file = file;
+    this.vectorLength = vectorLength;
+  }
+
+  @Parameterized.Parameters(name = "codec = {0}, length = {1}, file = {2}")
+  public static Collection<Object[]> getCodecs() {
+    List<Object[]> params = new ArrayList<>();
+
+    int[] lengths = new int[]{10, 100, 1000};
+    for (int len : lengths) {
+      CompressionCodec dumbCodec = NoCompressionCodec.INSTANCE;
+      File fileNoCompression = new File("target/write_no_compression_" + len + ".arrow");
+      params.add(new Object[]{dumbCodec.getCodecType(), len, dumbCodec, fileNoCompression});
+
+      CompressionCodec lz4Codec = new Lz4CompressionCodec();
+      File fileLZ4Compression = new File("target/write_lz4_compression_" + len + ".arrow");
+      params.add(new Object[]{lz4Codec.getCodecType(), len, lz4Codec, fileLZ4Compression});
+
+      CompressionCodec zstdCodec = new ZstdCompressionCodec();
+      File fileZSTDCompression = new File("target/write_zstd_compression_" + len + ".arrow");

Review comment:
       ```suggestion
         File fileZstdCompression = new File("target/write_zstd_compression_" + len + ".arrow");
   ```

##########
File path: java/vector/src/main/java/org/apache/arrow/vector/ipc/ArrowStreamWriter.java
##########
@@ -65,6 +66,48 @@ public ArrowStreamWriter(VectorSchemaRoot root, DictionaryProvider provider, Wri
     super(root, provider, out, option);
   }
 
+  /**
+   * Construct an ArrowStreamWriter with an optional DictionaryProvider for the OutputStream.
+   *
+   * @param root Existing VectorSchemaRoot with vectors to be written.
+   * @param includeNullCount Controls whether null count is copied to the {@link ArrowRecordBatch}
+   * @param codec the codec for compressing data. If it is null, then no compression is needed.
+   * @param alignBuffers Controls if buffers get aligned to 8-byte boundaries.
+   * @param provider DictionaryProvider for any vectors that are dictionary encoded.
+   *                 (Optional, can be null)
+   * @param out OutputStream for writing.
+   */
+  public ArrowStreamWriter(VectorSchemaRoot root, boolean includeNullCount, CompressionCodec codec,

Review comment:
       same comment about constructor explosion as the file writer.

##########
File path: java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodecFile.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.compression;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.compression.CompressionCodec;
+import org.apache.arrow.vector.compression.CompressionUtil;
+import org.apache.arrow.vector.compression.NoCompressionCodec;
+import org.apache.arrow.vector.ipc.ArrowFileReader;
+import org.apache.arrow.vector.ipc.ArrowFileWriter;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.arrow.vector.ipc.ArrowStreamReader;
+import org.apache.arrow.vector.ipc.ArrowStreamWriter;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+public class TestCompressionCodecFile {
+  private final CompressionCodec codec;
+  private final int vectorLength;
+  private final File file;
+
+  public TestCompressionCodecFile(CompressionUtil.CodecType type, int vectorLength, CompressionCodec codec,
+                                  File file) {
+    this.codec = codec;
+    this.file = file;
+    this.vectorLength = vectorLength;
+  }
+
+  @Parameterized.Parameters(name = "codec = {0}, length = {1}, file = {2}")
+  public static Collection<Object[]> getCodecs() {
+    List<Object[]> params = new ArrayList<>();
+
+    int[] lengths = new int[]{10, 100, 1000};
+    for (int len : lengths) {
+      CompressionCodec dumbCodec = NoCompressionCodec.INSTANCE;
+      File fileNoCompression = new File("target/write_no_compression_" + len + ".arrow");
+      params.add(new Object[]{dumbCodec.getCodecType(), len, dumbCodec, fileNoCompression});
+
+      CompressionCodec lz4Codec = new Lz4CompressionCodec();
+      File fileLZ4Compression = new File("target/write_lz4_compression_" + len + ".arrow");
+      params.add(new Object[]{lz4Codec.getCodecType(), len, lz4Codec, fileLZ4Compression});
+
+      CompressionCodec zstdCodec = new ZstdCompressionCodec();
+      File fileZSTDCompression = new File("target/write_zstd_compression_" + len + ".arrow");
+      params.add(new Object[]{zstdCodec.getCodecType(), len, zstdCodec, fileZSTDCompression});
+
+    }
+    return params;
+  }
+
+  @Test
+  public void writeReadRandomAccessFile() throws IOException {
+    RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+    BitVector bitVector = new BitVector("boolean", allocator);
+    VarCharVector varCharVector = new VarCharVector("varchar", allocator);
+    for (int i = 0; i < vectorLength; i++) {
+      bitVector.setSafe(i, i % 2 == 0 ? 0 : 1);
+      varCharVector.setSafe(i, ("test" + i).getBytes(StandardCharsets.UTF_8));
+    }
+    bitVector.setValueCount(vectorLength);
+    varCharVector.setValueCount(vectorLength);
+
+    List<Field> fields = Arrays.asList(bitVector.getField(), varCharVector.getField());
+    List<FieldVector> vectors = Arrays.asList(bitVector, varCharVector);
+
+    VectorSchemaRoot schemaRootWrite = new VectorSchemaRoot(fields, vectors);

Review comment:
       try-with-resources?

##########
File path: java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodecFile.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.compression;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.compression.CompressionCodec;
+import org.apache.arrow.vector.compression.CompressionUtil;
+import org.apache.arrow.vector.compression.NoCompressionCodec;
+import org.apache.arrow.vector.ipc.ArrowFileReader;
+import org.apache.arrow.vector.ipc.ArrowFileWriter;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.arrow.vector.ipc.ArrowStreamReader;
+import org.apache.arrow.vector.ipc.ArrowStreamWriter;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+public class TestCompressionCodecFile {
+  private final CompressionCodec codec;
+  private final int vectorLength;
+  private final File file;
+
+  public TestCompressionCodecFile(CompressionUtil.CodecType type, int vectorLength, CompressionCodec codec,
+                                  File file) {
+    this.codec = codec;
+    this.file = file;
+    this.vectorLength = vectorLength;
+  }
+
+  @Parameterized.Parameters(name = "codec = {0}, length = {1}, file = {2}")
+  public static Collection<Object[]> getCodecs() {
+    List<Object[]> params = new ArrayList<>();
+
+    int[] lengths = new int[]{10, 100, 1000};
+    for (int len : lengths) {
+      CompressionCodec dumbCodec = NoCompressionCodec.INSTANCE;

Review comment:
       ```suggestion
         CompressionCodec noCompression = NoCompressionCodec.INSTANCE;
   ```
   dumb can have negative connotations.

##########
File path: java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodecFile.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.compression;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.compression.CompressionCodec;
+import org.apache.arrow.vector.compression.CompressionUtil;
+import org.apache.arrow.vector.compression.NoCompressionCodec;
+import org.apache.arrow.vector.ipc.ArrowFileReader;
+import org.apache.arrow.vector.ipc.ArrowFileWriter;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.arrow.vector.ipc.ArrowStreamReader;
+import org.apache.arrow.vector.ipc.ArrowStreamWriter;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+public class TestCompressionCodecFile {
+  private final CompressionCodec codec;
+  private final int vectorLength;
+  private final File file;
+
+  public TestCompressionCodecFile(CompressionUtil.CodecType type, int vectorLength, CompressionCodec codec,
+                                  File file) {
+    this.codec = codec;
+    this.file = file;
+    this.vectorLength = vectorLength;
+  }
+
+  @Parameterized.Parameters(name = "codec = {0}, length = {1}, file = {2}")
+  public static Collection<Object[]> getCodecs() {
+    List<Object[]> params = new ArrayList<>();
+
+    int[] lengths = new int[]{10, 100, 1000};
+    for (int len : lengths) {
+      CompressionCodec dumbCodec = NoCompressionCodec.INSTANCE;
+      File fileNoCompression = new File("target/write_no_compression_" + len + ".arrow");
+      params.add(new Object[]{dumbCodec.getCodecType(), len, dumbCodec, fileNoCompression});
+
+      CompressionCodec lz4Codec = new Lz4CompressionCodec();
+      File fileLZ4Compression = new File("target/write_lz4_compression_" + len + ".arrow");
+      params.add(new Object[]{lz4Codec.getCodecType(), len, lz4Codec, fileLZ4Compression});
+
+      CompressionCodec zstdCodec = new ZstdCompressionCodec();
+      File fileZSTDCompression = new File("target/write_zstd_compression_" + len + ".arrow");
+      params.add(new Object[]{zstdCodec.getCodecType(), len, zstdCodec, fileZSTDCompression});
+
+    }
+    return params;
+  }
+
+  @Test
+  public void writeReadRandomAccessFile() throws IOException {
+    RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+    BitVector bitVector = new BitVector("boolean", allocator);
+    VarCharVector varCharVector = new VarCharVector("varchar", allocator);
+    for (int i = 0; i < vectorLength; i++) {
+      bitVector.setSafe(i, i % 2 == 0 ? 0 : 1);
+      varCharVector.setSafe(i, ("test" + i).getBytes(StandardCharsets.UTF_8));
+    }
+    bitVector.setValueCount(vectorLength);
+    varCharVector.setValueCount(vectorLength);
+
+    List<Field> fields = Arrays.asList(bitVector.getField(), varCharVector.getField());
+    List<FieldVector> vectors = Arrays.asList(bitVector, varCharVector);
+
+    VectorSchemaRoot schemaRootWrite = new VectorSchemaRoot(fields, vectors);
+
+    // write
+    FileOutputStream fileOutputStream = new FileOutputStream(file);
+    ArrowFileWriter writer = new ArrowFileWriter(schemaRootWrite, true, codec, true,
+        null, fileOutputStream.getChannel());
+    writer.start();
+    writer.writeBatch();
+    writer.end();
+
+    // validations
+    Assert.assertEquals(vectorLength, schemaRootWrite.getRowCount());

Review comment:
       can you check style in other tests, I thought we might static import assertEquals

##########
File path: java/vector/src/main/java/org/apache/arrow/vector/ipc/ArrowFileWriter.java
##########
@@ -69,6 +70,31 @@ public ArrowFileWriter(VectorSchemaRoot root, DictionaryProvider provider, Writa
     this.metaData = metaData;
   }
 
+  public ArrowFileWriter(VectorSchemaRoot root, boolean includeNullCount, CompressionCodec codec,

Review comment:
       Instead of making an ever expanding list of constructors, I think it is likely better at this point to create an "Options"  class that can be built using a builder pattern and passed in.  So we can have one new constructor:
   
   ArrowFileWriter(VectorSchemaRoot root, WriteOptions options, WritableByteChannel) and then maybe one private constructor if necessary that explodes the options.

##########
File path: java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodecFile.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.compression;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.compression.CompressionCodec;
+import org.apache.arrow.vector.compression.CompressionUtil;
+import org.apache.arrow.vector.compression.NoCompressionCodec;
+import org.apache.arrow.vector.ipc.ArrowFileReader;
+import org.apache.arrow.vector.ipc.ArrowFileWriter;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.arrow.vector.ipc.ArrowStreamReader;
+import org.apache.arrow.vector.ipc.ArrowStreamWriter;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+public class TestCompressionCodecFile {
+  private final CompressionCodec codec;
+  private final int vectorLength;
+  private final File file;
+
+  public TestCompressionCodecFile(CompressionUtil.CodecType type, int vectorLength, CompressionCodec codec,
+                                  File file) {
+    this.codec = codec;
+    this.file = file;
+    this.vectorLength = vectorLength;
+  }
+
+  @Parameterized.Parameters(name = "codec = {0}, length = {1}, file = {2}")
+  public static Collection<Object[]> getCodecs() {
+    List<Object[]> params = new ArrayList<>();
+
+    int[] lengths = new int[]{10, 100, 1000};
+    for (int len : lengths) {
+      CompressionCodec dumbCodec = NoCompressionCodec.INSTANCE;
+      File fileNoCompression = new File("target/write_no_compression_" + len + ".arrow");
+      params.add(new Object[]{dumbCodec.getCodecType(), len, dumbCodec, fileNoCompression});
+
+      CompressionCodec lz4Codec = new Lz4CompressionCodec();
+      File fileLZ4Compression = new File("target/write_lz4_compression_" + len + ".arrow");
+      params.add(new Object[]{lz4Codec.getCodecType(), len, lz4Codec, fileLZ4Compression});
+
+      CompressionCodec zstdCodec = new ZstdCompressionCodec();
+      File fileZSTDCompression = new File("target/write_zstd_compression_" + len + ".arrow");
+      params.add(new Object[]{zstdCodec.getCodecType(), len, zstdCodec, fileZSTDCompression});
+
+    }
+    return params;
+  }
+
+  @Test
+  public void writeReadRandomAccessFile() throws IOException {
+    RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);

Review comment:
       this should be in a try-with-resources block

##########
File path: java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodecFile.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.compression;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.compression.CompressionCodec;
+import org.apache.arrow.vector.compression.CompressionUtil;
+import org.apache.arrow.vector.compression.NoCompressionCodec;
+import org.apache.arrow.vector.ipc.ArrowFileReader;
+import org.apache.arrow.vector.ipc.ArrowFileWriter;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.arrow.vector.ipc.ArrowStreamReader;
+import org.apache.arrow.vector.ipc.ArrowStreamWriter;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+public class TestCompressionCodecFile {
+  private final CompressionCodec codec;
+  private final int vectorLength;
+  private final File file;
+
+  public TestCompressionCodecFile(CompressionUtil.CodecType type, int vectorLength, CompressionCodec codec,
+                                  File file) {
+    this.codec = codec;
+    this.file = file;
+    this.vectorLength = vectorLength;
+  }
+
+  @Parameterized.Parameters(name = "codec = {0}, length = {1}, file = {2}")
+  public static Collection<Object[]> getCodecs() {
+    List<Object[]> params = new ArrayList<>();
+
+    int[] lengths = new int[]{10, 100, 1000};
+    for (int len : lengths) {
+      CompressionCodec dumbCodec = NoCompressionCodec.INSTANCE;
+      File fileNoCompression = new File("target/write_no_compression_" + len + ".arrow");
+      params.add(new Object[]{dumbCodec.getCodecType(), len, dumbCodec, fileNoCompression});
+
+      CompressionCodec lz4Codec = new Lz4CompressionCodec();
+      File fileLZ4Compression = new File("target/write_lz4_compression_" + len + ".arrow");
+      params.add(new Object[]{lz4Codec.getCodecType(), len, lz4Codec, fileLZ4Compression});
+
+      CompressionCodec zstdCodec = new ZstdCompressionCodec();
+      File fileZSTDCompression = new File("target/write_zstd_compression_" + len + ".arrow");
+      params.add(new Object[]{zstdCodec.getCodecType(), len, zstdCodec, fileZSTDCompression});
+
+    }
+    return params;
+  }
+
+  @Test
+  public void writeReadRandomAccessFile() throws IOException {
+    RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+    BitVector bitVector = new BitVector("boolean", allocator);
+    VarCharVector varCharVector = new VarCharVector("varchar", allocator);
+    for (int i = 0; i < vectorLength; i++) {
+      bitVector.setSafe(i, i % 2 == 0 ? 0 : 1);
+      varCharVector.setSafe(i, ("test" + i).getBytes(StandardCharsets.UTF_8));
+    }
+    bitVector.setValueCount(vectorLength);
+    varCharVector.setValueCount(vectorLength);
+
+    List<Field> fields = Arrays.asList(bitVector.getField(), varCharVector.getField());
+    List<FieldVector> vectors = Arrays.asList(bitVector, varCharVector);
+
+    VectorSchemaRoot schemaRootWrite = new VectorSchemaRoot(fields, vectors);
+
+    // write
+    FileOutputStream fileOutputStream = new FileOutputStream(file);
+    ArrowFileWriter writer = new ArrowFileWriter(schemaRootWrite, true, codec, true,

Review comment:
       please try to comment literal paraemters.  e.g. `schemarRootWrite, /*parameter 1 name=*/ true, /*parameter 2 name=*/`

##########
File path: java/vector/src/main/java/org/apache/arrow/vector/ipc/ArrowFileWriter.java
##########
@@ -69,6 +70,31 @@ public ArrowFileWriter(VectorSchemaRoot root, DictionaryProvider provider, Writa
     this.metaData = metaData;
   }
 
+  public ArrowFileWriter(VectorSchemaRoot root, boolean includeNullCount, CompressionCodec codec,

Review comment:
       Instead of making an ever expanding list of constructors, I think it is likely better at this point to create an "Options"  class that can be built using a builder pattern and passed in.  So we can have one new constructor:
   
   ArrowFileWriter(VectorSchemaRoot root, WriteOptions options, WritableByteChannel) and then maybe one private constructor if necessary that explodes the options.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] amol- edited a comment on pull request #11591: ARROW-12163: [Java] Make compression levels configurable.

[GitBox <gi...@apache.org>]

amol- edited a comment on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-959234041


   > One more note. The JIRA linked is about adjusting the compression level used by compression algorithms (trading off speed for output size). I didnt notice a change here for that apologies if i missed it.
   
   @davisusanibar that seems true, the original goal of the JIRA ticket was to allow to configure the compression by specifying options like the level. Similarly to what we allow in C++ and Python where you can set `compression` and `compression_level` ( https://arrow.apache.org/docs/python/generated/pyarrow.Codec.html?highlight=codec#pyarrow.Codec ).
   
   It seems that the `CompressionCodec` and `CommonsCompressionFactory` do not allow to specify anything else than the `codecType` but not a level for such codecs. 
   
   Should `Lz4CompressionCodec` and `ZstdCompressionCodec` grow a constructor argument to specify the level?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] amol- commented on pull request #11591: ARROW-12163: [Java] Make compression levels configurable.

[GitBox <gi...@apache.org>]

amol- commented on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-959234041


   > One more note. The JIRA linked is about adjusting the compression level used by compression algorithms (trading off speed for output size). I didnt notice a change here for that apologies if i missed it.
   
   @davisusanibar that seems true, the original goal of the JIRA ticket was to allow to configure the compression by specifying options like the level. Similarly to what we allow in C++ and Python where you can set `compression` and `compression_level` ( https://arrow.apache.org/docs/python/generated/pyarrow.Codec.html?highlight=codec#pyarrow.Codec ). It seems that the `CompressionCodec` and `CommonsCompressionFactory` do not allow to specify anything else than the `codecType` but not a level for such codecs. Should `Lz4CompressionCodec` and `ZstdCompressionCodec` grow a constructor argument to specify the level?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] pitrou commented on a change in pull request #11591: ARROW-12163: [Java] Make compression levels configurable.

[GitBox <gi...@apache.org>]

pitrou commented on a change in pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#discussion_r741887094



##########
File path: docs/source/java/ipc.rst
##########
@@ -54,6 +54,15 @@ Now, we can begin writing a stream containing some number of these batches. For
     ArrowStreamWriter writer = new ArrowStreamWriter(root, /*DictionaryProvider=*/null, Channels.newChannel(out));
 
 
+Buffer level compression is also supported by passing a codec:
+Lz4CompressionCodec / ZstdCompressionCodec:
+
+.. code-block:: Java
+
+    ArrowStreamWriter writer = new ArrowStreamWriter(schemaRootWrite, true, new Lz4CompressionCodec(), true,
+        null, fileOutputStream.getChannel());

Review comment:
       "true, null" doesn't say much about what these parameters are. Should you add commented parameter names as above (e.g. `/*DictionaryProvider=*/null`, but I don't know if it's the right name here?).

##########
File path: docs/source/java/ipc.rst
##########
@@ -54,6 +54,15 @@ Now, we can begin writing a stream containing some number of these batches. For
     ArrowStreamWriter writer = new ArrowStreamWriter(root, /*DictionaryProvider=*/null, Channels.newChannel(out));
 
 
+Buffer level compression is also supported by passing a codec:
+Lz4CompressionCodec / ZstdCompressionCodec:

Review comment:
       Looks like there is a missing sentence here? Are these the supported codecs? Something else?

##########
File path: docs/source/java/ipc.rst
##########
@@ -54,6 +54,15 @@ Now, we can begin writing a stream containing some number of these batches. For
     ArrowStreamWriter writer = new ArrowStreamWriter(root, /*DictionaryProvider=*/null, Channels.newChannel(out));
 
 
+Buffer level compression is also supported by passing a codec:
+Lz4CompressionCodec / ZstdCompressionCodec:
+
+.. code-block:: Java
+
+    ArrowStreamWriter writer = new ArrowStreamWriter(schemaRootWrite, true, new Lz4CompressionCodec(), true,
+        null, fileOutputStream.getChannel());

Review comment:
       "true, null" doesn't say much about what these parameters are. Should you add commented parameter names as above (e.g. `/*DictionaryProvider=*/null`, but I don't know if it's the right name here?).

##########
File path: docs/source/java/ipc.rst
##########
@@ -54,6 +54,15 @@ Now, we can begin writing a stream containing some number of these batches. For
     ArrowStreamWriter writer = new ArrowStreamWriter(root, /*DictionaryProvider=*/null, Channels.newChannel(out));
 
 
+Buffer level compression is also supported by passing a codec:
+Lz4CompressionCodec / ZstdCompressionCodec:

Review comment:
       Looks like there is a missing sentence here? Are these the supported codecs? Something else?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] amol- commented on pull request #11591: ARROW-12163: [Java] Make compression levels configurable.

[GitBox <gi...@apache.org>]

amol- commented on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-959234041


   > One more note. The JIRA linked is about adjusting the compression level used by compression algorithms (trading off speed for output size). I didnt notice a change here for that apologies if i missed it.
   
   @davisusanibar that seems true, the original goal of the JIRA ticket was to allow to configure the compression by specifying options like the level. Similarly to what we allow in C++ and Python where you can set `compression` and `compression_level` ( https://arrow.apache.org/docs/python/generated/pyarrow.Codec.html?highlight=codec#pyarrow.Codec ). It seems that the `CompressionCodec` and `CommonsCompressionFactory` do not allow to specify anything else than the `codecType` but not a level for such codecs. Should `Lz4CompressionCodec` and `ZstdCompressionCodec` grow a constructor argument to specify the level?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] emkornfield commented on a change in pull request #11591: [Java] ARROW-12163 - Make compression levels configurable

[GitBox <gi...@apache.org>]

emkornfield commented on a change in pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#discussion_r741330445



##########
File path: java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodecFile.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.compression;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.compression.CompressionCodec;
+import org.apache.arrow.vector.compression.CompressionUtil;
+import org.apache.arrow.vector.compression.NoCompressionCodec;
+import org.apache.arrow.vector.ipc.ArrowFileReader;
+import org.apache.arrow.vector.ipc.ArrowFileWriter;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.arrow.vector.ipc.ArrowStreamReader;
+import org.apache.arrow.vector.ipc.ArrowStreamWriter;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+public class TestCompressionCodecFile {
+  private final CompressionCodec codec;
+  private final int vectorLength;
+  private final File file;
+
+  public TestCompressionCodecFile(CompressionUtil.CodecType type, int vectorLength, CompressionCodec codec,
+                                  File file) {
+    this.codec = codec;
+    this.file = file;
+    this.vectorLength = vectorLength;
+  }
+
+  @Parameterized.Parameters(name = "codec = {0}, length = {1}, file = {2}")
+  public static Collection<Object[]> getCodecs() {
+    List<Object[]> params = new ArrayList<>();
+
+    int[] lengths = new int[]{10, 100, 1000};
+    for (int len : lengths) {
+      CompressionCodec dumbCodec = NoCompressionCodec.INSTANCE;
+      File fileNoCompression = new File("target/write_no_compression_" + len + ".arrow");
+      params.add(new Object[]{dumbCodec.getCodecType(), len, dumbCodec, fileNoCompression});
+
+      CompressionCodec lz4Codec = new Lz4CompressionCodec();
+      File fileLZ4Compression = new File("target/write_lz4_compression_" + len + ".arrow");

Review comment:
       ```suggestion
         File fileLz4Compression = new File("target/write_lz4_compression_" + len + ".arrow");
   ```
   nit: generally we use camel case for abbreviates as well

##########
File path: docs/source/java/ipc.rst
##########
@@ -54,6 +54,15 @@ Now, we can begin writing a stream containing some number of these batches. For
     ArrowStreamWriter writer = new ArrowStreamWriter(root, /*DictionaryProvider=*/null, Channels.newChannel(out));
 
 
+Here we are not used compression option, but this could be implemented on this way thru codec option:

Review comment:
       ```suggestion
   Buffer level compression is also supported by passing a codec:
   ```

##########
File path: java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodecFile.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.compression;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.compression.CompressionCodec;
+import org.apache.arrow.vector.compression.CompressionUtil;
+import org.apache.arrow.vector.compression.NoCompressionCodec;
+import org.apache.arrow.vector.ipc.ArrowFileReader;
+import org.apache.arrow.vector.ipc.ArrowFileWriter;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.arrow.vector.ipc.ArrowStreamReader;
+import org.apache.arrow.vector.ipc.ArrowStreamWriter;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+public class TestCompressionCodecFile {
+  private final CompressionCodec codec;
+  private final int vectorLength;
+  private final File file;
+
+  public TestCompressionCodecFile(CompressionUtil.CodecType type, int vectorLength, CompressionCodec codec,
+                                  File file) {
+    this.codec = codec;
+    this.file = file;
+    this.vectorLength = vectorLength;
+  }
+
+  @Parameterized.Parameters(name = "codec = {0}, length = {1}, file = {2}")
+  public static Collection<Object[]> getCodecs() {
+    List<Object[]> params = new ArrayList<>();
+
+    int[] lengths = new int[]{10, 100, 1000};
+    for (int len : lengths) {
+      CompressionCodec dumbCodec = NoCompressionCodec.INSTANCE;
+      File fileNoCompression = new File("target/write_no_compression_" + len + ".arrow");
+      params.add(new Object[]{dumbCodec.getCodecType(), len, dumbCodec, fileNoCompression});
+
+      CompressionCodec lz4Codec = new Lz4CompressionCodec();
+      File fileLZ4Compression = new File("target/write_lz4_compression_" + len + ".arrow");
+      params.add(new Object[]{lz4Codec.getCodecType(), len, lz4Codec, fileLZ4Compression});
+
+      CompressionCodec zstdCodec = new ZstdCompressionCodec();
+      File fileZSTDCompression = new File("target/write_zstd_compression_" + len + ".arrow");

Review comment:
       ```suggestion
         File fileZstdCompression = new File("target/write_zstd_compression_" + len + ".arrow");
   ```

##########
File path: java/vector/src/main/java/org/apache/arrow/vector/ipc/ArrowStreamWriter.java
##########
@@ -65,6 +66,48 @@ public ArrowStreamWriter(VectorSchemaRoot root, DictionaryProvider provider, Wri
     super(root, provider, out, option);
   }
 
+  /**
+   * Construct an ArrowStreamWriter with an optional DictionaryProvider for the OutputStream.
+   *
+   * @param root Existing VectorSchemaRoot with vectors to be written.
+   * @param includeNullCount Controls whether null count is copied to the {@link ArrowRecordBatch}
+   * @param codec the codec for compressing data. If it is null, then no compression is needed.
+   * @param alignBuffers Controls if buffers get aligned to 8-byte boundaries.
+   * @param provider DictionaryProvider for any vectors that are dictionary encoded.
+   *                 (Optional, can be null)
+   * @param out OutputStream for writing.
+   */
+  public ArrowStreamWriter(VectorSchemaRoot root, boolean includeNullCount, CompressionCodec codec,

Review comment:
       same comment about constructor explosion as the file writer.

##########
File path: java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodecFile.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.compression;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.compression.CompressionCodec;
+import org.apache.arrow.vector.compression.CompressionUtil;
+import org.apache.arrow.vector.compression.NoCompressionCodec;
+import org.apache.arrow.vector.ipc.ArrowFileReader;
+import org.apache.arrow.vector.ipc.ArrowFileWriter;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.arrow.vector.ipc.ArrowStreamReader;
+import org.apache.arrow.vector.ipc.ArrowStreamWriter;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+public class TestCompressionCodecFile {
+  private final CompressionCodec codec;
+  private final int vectorLength;
+  private final File file;
+
+  public TestCompressionCodecFile(CompressionUtil.CodecType type, int vectorLength, CompressionCodec codec,
+                                  File file) {
+    this.codec = codec;
+    this.file = file;
+    this.vectorLength = vectorLength;
+  }
+
+  @Parameterized.Parameters(name = "codec = {0}, length = {1}, file = {2}")
+  public static Collection<Object[]> getCodecs() {
+    List<Object[]> params = new ArrayList<>();
+
+    int[] lengths = new int[]{10, 100, 1000};
+    for (int len : lengths) {
+      CompressionCodec dumbCodec = NoCompressionCodec.INSTANCE;
+      File fileNoCompression = new File("target/write_no_compression_" + len + ".arrow");
+      params.add(new Object[]{dumbCodec.getCodecType(), len, dumbCodec, fileNoCompression});
+
+      CompressionCodec lz4Codec = new Lz4CompressionCodec();
+      File fileLZ4Compression = new File("target/write_lz4_compression_" + len + ".arrow");
+      params.add(new Object[]{lz4Codec.getCodecType(), len, lz4Codec, fileLZ4Compression});
+
+      CompressionCodec zstdCodec = new ZstdCompressionCodec();
+      File fileZSTDCompression = new File("target/write_zstd_compression_" + len + ".arrow");
+      params.add(new Object[]{zstdCodec.getCodecType(), len, zstdCodec, fileZSTDCompression});
+
+    }
+    return params;
+  }
+
+  @Test
+  public void writeReadRandomAccessFile() throws IOException {
+    RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+    BitVector bitVector = new BitVector("boolean", allocator);
+    VarCharVector varCharVector = new VarCharVector("varchar", allocator);
+    for (int i = 0; i < vectorLength; i++) {
+      bitVector.setSafe(i, i % 2 == 0 ? 0 : 1);
+      varCharVector.setSafe(i, ("test" + i).getBytes(StandardCharsets.UTF_8));
+    }
+    bitVector.setValueCount(vectorLength);
+    varCharVector.setValueCount(vectorLength);
+
+    List<Field> fields = Arrays.asList(bitVector.getField(), varCharVector.getField());
+    List<FieldVector> vectors = Arrays.asList(bitVector, varCharVector);
+
+    VectorSchemaRoot schemaRootWrite = new VectorSchemaRoot(fields, vectors);

Review comment:
       try-with-resources?

##########
File path: java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodecFile.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.compression;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.compression.CompressionCodec;
+import org.apache.arrow.vector.compression.CompressionUtil;
+import org.apache.arrow.vector.compression.NoCompressionCodec;
+import org.apache.arrow.vector.ipc.ArrowFileReader;
+import org.apache.arrow.vector.ipc.ArrowFileWriter;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.arrow.vector.ipc.ArrowStreamReader;
+import org.apache.arrow.vector.ipc.ArrowStreamWriter;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+public class TestCompressionCodecFile {
+  private final CompressionCodec codec;
+  private final int vectorLength;
+  private final File file;
+
+  public TestCompressionCodecFile(CompressionUtil.CodecType type, int vectorLength, CompressionCodec codec,
+                                  File file) {
+    this.codec = codec;
+    this.file = file;
+    this.vectorLength = vectorLength;
+  }
+
+  @Parameterized.Parameters(name = "codec = {0}, length = {1}, file = {2}")
+  public static Collection<Object[]> getCodecs() {
+    List<Object[]> params = new ArrayList<>();
+
+    int[] lengths = new int[]{10, 100, 1000};
+    for (int len : lengths) {
+      CompressionCodec dumbCodec = NoCompressionCodec.INSTANCE;

Review comment:
       ```suggestion
         CompressionCodec noCompression = NoCompressionCodec.INSTANCE;
   ```
   dumb can have negative connotations.

##########
File path: java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodecFile.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.compression;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.compression.CompressionCodec;
+import org.apache.arrow.vector.compression.CompressionUtil;
+import org.apache.arrow.vector.compression.NoCompressionCodec;
+import org.apache.arrow.vector.ipc.ArrowFileReader;
+import org.apache.arrow.vector.ipc.ArrowFileWriter;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.arrow.vector.ipc.ArrowStreamReader;
+import org.apache.arrow.vector.ipc.ArrowStreamWriter;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+public class TestCompressionCodecFile {
+  private final CompressionCodec codec;
+  private final int vectorLength;
+  private final File file;
+
+  public TestCompressionCodecFile(CompressionUtil.CodecType type, int vectorLength, CompressionCodec codec,
+                                  File file) {
+    this.codec = codec;
+    this.file = file;
+    this.vectorLength = vectorLength;
+  }
+
+  @Parameterized.Parameters(name = "codec = {0}, length = {1}, file = {2}")
+  public static Collection<Object[]> getCodecs() {
+    List<Object[]> params = new ArrayList<>();
+
+    int[] lengths = new int[]{10, 100, 1000};
+    for (int len : lengths) {
+      CompressionCodec dumbCodec = NoCompressionCodec.INSTANCE;
+      File fileNoCompression = new File("target/write_no_compression_" + len + ".arrow");
+      params.add(new Object[]{dumbCodec.getCodecType(), len, dumbCodec, fileNoCompression});
+
+      CompressionCodec lz4Codec = new Lz4CompressionCodec();
+      File fileLZ4Compression = new File("target/write_lz4_compression_" + len + ".arrow");
+      params.add(new Object[]{lz4Codec.getCodecType(), len, lz4Codec, fileLZ4Compression});
+
+      CompressionCodec zstdCodec = new ZstdCompressionCodec();
+      File fileZSTDCompression = new File("target/write_zstd_compression_" + len + ".arrow");
+      params.add(new Object[]{zstdCodec.getCodecType(), len, zstdCodec, fileZSTDCompression});
+
+    }
+    return params;
+  }
+
+  @Test
+  public void writeReadRandomAccessFile() throws IOException {
+    RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+    BitVector bitVector = new BitVector("boolean", allocator);
+    VarCharVector varCharVector = new VarCharVector("varchar", allocator);
+    for (int i = 0; i < vectorLength; i++) {
+      bitVector.setSafe(i, i % 2 == 0 ? 0 : 1);
+      varCharVector.setSafe(i, ("test" + i).getBytes(StandardCharsets.UTF_8));
+    }
+    bitVector.setValueCount(vectorLength);
+    varCharVector.setValueCount(vectorLength);
+
+    List<Field> fields = Arrays.asList(bitVector.getField(), varCharVector.getField());
+    List<FieldVector> vectors = Arrays.asList(bitVector, varCharVector);
+
+    VectorSchemaRoot schemaRootWrite = new VectorSchemaRoot(fields, vectors);
+
+    // write
+    FileOutputStream fileOutputStream = new FileOutputStream(file);
+    ArrowFileWriter writer = new ArrowFileWriter(schemaRootWrite, true, codec, true,
+        null, fileOutputStream.getChannel());
+    writer.start();
+    writer.writeBatch();
+    writer.end();
+
+    // validations
+    Assert.assertEquals(vectorLength, schemaRootWrite.getRowCount());

Review comment:
       can you check style in other tests, I thought we might static import assertEquals

##########
File path: java/vector/src/main/java/org/apache/arrow/vector/ipc/ArrowFileWriter.java
##########
@@ -69,6 +70,31 @@ public ArrowFileWriter(VectorSchemaRoot root, DictionaryProvider provider, Writa
     this.metaData = metaData;
   }
 
+  public ArrowFileWriter(VectorSchemaRoot root, boolean includeNullCount, CompressionCodec codec,

Review comment:
       Instead of making an ever expanding list of constructors, I think it is likely better at this point to create an "Options"  class that can be built using a builder pattern and passed in.  So we can have one new constructor:
   
   ArrowFileWriter(VectorSchemaRoot root, WriteOptions options, WritableByteChannel) and then maybe one private constructor if necessary that explodes the options.

##########
File path: java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodecFile.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.compression;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.compression.CompressionCodec;
+import org.apache.arrow.vector.compression.CompressionUtil;
+import org.apache.arrow.vector.compression.NoCompressionCodec;
+import org.apache.arrow.vector.ipc.ArrowFileReader;
+import org.apache.arrow.vector.ipc.ArrowFileWriter;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.arrow.vector.ipc.ArrowStreamReader;
+import org.apache.arrow.vector.ipc.ArrowStreamWriter;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+public class TestCompressionCodecFile {
+  private final CompressionCodec codec;
+  private final int vectorLength;
+  private final File file;
+
+  public TestCompressionCodecFile(CompressionUtil.CodecType type, int vectorLength, CompressionCodec codec,
+                                  File file) {
+    this.codec = codec;
+    this.file = file;
+    this.vectorLength = vectorLength;
+  }
+
+  @Parameterized.Parameters(name = "codec = {0}, length = {1}, file = {2}")
+  public static Collection<Object[]> getCodecs() {
+    List<Object[]> params = new ArrayList<>();
+
+    int[] lengths = new int[]{10, 100, 1000};
+    for (int len : lengths) {
+      CompressionCodec dumbCodec = NoCompressionCodec.INSTANCE;
+      File fileNoCompression = new File("target/write_no_compression_" + len + ".arrow");
+      params.add(new Object[]{dumbCodec.getCodecType(), len, dumbCodec, fileNoCompression});
+
+      CompressionCodec lz4Codec = new Lz4CompressionCodec();
+      File fileLZ4Compression = new File("target/write_lz4_compression_" + len + ".arrow");
+      params.add(new Object[]{lz4Codec.getCodecType(), len, lz4Codec, fileLZ4Compression});
+
+      CompressionCodec zstdCodec = new ZstdCompressionCodec();
+      File fileZSTDCompression = new File("target/write_zstd_compression_" + len + ".arrow");
+      params.add(new Object[]{zstdCodec.getCodecType(), len, zstdCodec, fileZSTDCompression});
+
+    }
+    return params;
+  }
+
+  @Test
+  public void writeReadRandomAccessFile() throws IOException {
+    RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);

Review comment:
       this should be in a try-with-resources block

##########
File path: java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodecFile.java
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.arrow.compression;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BitVector;
+import org.apache.arrow.vector.FieldVector;
+import org.apache.arrow.vector.VarCharVector;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.compression.CompressionCodec;
+import org.apache.arrow.vector.compression.CompressionUtil;
+import org.apache.arrow.vector.compression.NoCompressionCodec;
+import org.apache.arrow.vector.ipc.ArrowFileReader;
+import org.apache.arrow.vector.ipc.ArrowFileWriter;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.arrow.vector.ipc.ArrowStreamReader;
+import org.apache.arrow.vector.ipc.ArrowStreamWriter;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+@RunWith(Parameterized.class)
+public class TestCompressionCodecFile {
+  private final CompressionCodec codec;
+  private final int vectorLength;
+  private final File file;
+
+  public TestCompressionCodecFile(CompressionUtil.CodecType type, int vectorLength, CompressionCodec codec,
+                                  File file) {
+    this.codec = codec;
+    this.file = file;
+    this.vectorLength = vectorLength;
+  }
+
+  @Parameterized.Parameters(name = "codec = {0}, length = {1}, file = {2}")
+  public static Collection<Object[]> getCodecs() {
+    List<Object[]> params = new ArrayList<>();
+
+    int[] lengths = new int[]{10, 100, 1000};
+    for (int len : lengths) {
+      CompressionCodec dumbCodec = NoCompressionCodec.INSTANCE;
+      File fileNoCompression = new File("target/write_no_compression_" + len + ".arrow");
+      params.add(new Object[]{dumbCodec.getCodecType(), len, dumbCodec, fileNoCompression});
+
+      CompressionCodec lz4Codec = new Lz4CompressionCodec();
+      File fileLZ4Compression = new File("target/write_lz4_compression_" + len + ".arrow");
+      params.add(new Object[]{lz4Codec.getCodecType(), len, lz4Codec, fileLZ4Compression});
+
+      CompressionCodec zstdCodec = new ZstdCompressionCodec();
+      File fileZSTDCompression = new File("target/write_zstd_compression_" + len + ".arrow");
+      params.add(new Object[]{zstdCodec.getCodecType(), len, zstdCodec, fileZSTDCompression});
+
+    }
+    return params;
+  }
+
+  @Test
+  public void writeReadRandomAccessFile() throws IOException {
+    RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+    BitVector bitVector = new BitVector("boolean", allocator);
+    VarCharVector varCharVector = new VarCharVector("varchar", allocator);
+    for (int i = 0; i < vectorLength; i++) {
+      bitVector.setSafe(i, i % 2 == 0 ? 0 : 1);
+      varCharVector.setSafe(i, ("test" + i).getBytes(StandardCharsets.UTF_8));
+    }
+    bitVector.setValueCount(vectorLength);
+    varCharVector.setValueCount(vectorLength);
+
+    List<Field> fields = Arrays.asList(bitVector.getField(), varCharVector.getField());
+    List<FieldVector> vectors = Arrays.asList(bitVector, varCharVector);
+
+    VectorSchemaRoot schemaRootWrite = new VectorSchemaRoot(fields, vectors);
+
+    // write
+    FileOutputStream fileOutputStream = new FileOutputStream(file);
+    ArrowFileWriter writer = new ArrowFileWriter(schemaRootWrite, true, codec, true,

Review comment:
       please try to comment literal paraemters.  e.g. `schemarRootWrite, /*parameter 1 name=*/ true, /*parameter 2 name=*/`

##########
File path: java/vector/src/main/java/org/apache/arrow/vector/ipc/ArrowFileWriter.java
##########
@@ -69,6 +70,31 @@ public ArrowFileWriter(VectorSchemaRoot root, DictionaryProvider provider, Writa
     this.metaData = metaData;
   }
 
+  public ArrowFileWriter(VectorSchemaRoot root, boolean includeNullCount, CompressionCodec codec,

Review comment:
       Instead of making an ever expanding list of constructors, I think it is likely better at this point to create an "Options"  class that can be built using a builder pattern and passed in.  So we can have one new constructor:
   
   ArrowFileWriter(VectorSchemaRoot root, WriteOptions options, WritableByteChannel) and then maybe one private constructor if necessary that explodes the options.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] github-actions[bot] commented on pull request #11591: [Java] ARROW-12163 - Make compression levels configurable

[GitBox <gi...@apache.org>]

github-actions[bot] commented on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-957355018


   <!--
     Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
     regarding copyright ownership.  The ASF licenses this file
     to you under the Apache License, Version 2.0 (the
     "License"); you may not use this file except in compliance
     with the License.  You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICENSE-2.0
   
     Unless required by applicable law or agreed to in writing,
     software distributed under the License is distributed on an
     "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
     KIND, either express or implied.  See the License for the
     specific language governing permissions and limitations
     under the License.
   -->
   
   Thanks for opening a pull request!
   
   If this is not a [minor PR](https://github.com/apache/arrow/blob/master/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW
   
   Opening JIRAs ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
   
   Then could you also rename pull request title in the following format?
   
       ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
   
   or
   
       MINOR: [${COMPONENT}] ${SUMMARY}
   
   See also:
   
     * [Other pull requests](https://github.com/apache/arrow/pulls/)
     * [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] emkornfield commented on pull request #11591: [Java] ARROW-12163 - Make compression levels configurable

[GitBox <gi...@apache.org>]

emkornfield commented on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-957998681


   One more note.  The JIRA linked is about adjusting the compression level used by compression algorithms (trading off speed for output size).  I didnt notice a change here for that apologies if i missed it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] emkornfield commented on pull request #11591: ARROW-12163: [Java] Make compression levels configurable.

[GitBox <gi...@apache.org>]

emkornfield commented on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-962181367






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] edponce commented on pull request #11591: [Java] ARROW-12163 - Make compression levels configurable

[GitBox <gi...@apache.org>]

edponce commented on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-957988740


   Please update the PR title to *ARROW-12163: [Java] Make compression levels configurable*.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] amol- removed a comment on pull request #11591: ARROW-12163: [Java] Make compression levels configurable.

[GitBox <gi...@apache.org>]

amol- removed a comment on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-959234041


   > One more note. The JIRA linked is about adjusting the compression level used by compression algorithms (trading off speed for output size). I didnt notice a change here for that apologies if i missed it.
   
   @davisusanibar that seems true, the original goal of the JIRA ticket was to allow to configure the compression by specifying options like the level. Similarly to what we allow in C++ and Python where you can set `compression` and `compression_level` ( https://arrow.apache.org/docs/python/generated/pyarrow.Codec.html?highlight=codec#pyarrow.Codec ).
   
   It seems that the `CompressionCodec` and `CommonsCompressionFactory` do not allow to specify anything else than the `codecType` but not a level for such codecs. 
   
   Should `Lz4CompressionCodec` and `ZstdCompressionCodec` grow a constructor argument to specify the level?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] amol- commented on pull request #11591: ARROW-12163: [Java] Make compression levels configurable.

[GitBox <gi...@apache.org>]

amol- commented on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-959234041


   > One more note. The JIRA linked is about adjusting the compression level used by compression algorithms (trading off speed for output size). I didnt notice a change here for that apologies if i missed it.
   
   @davisusanibar that seems true, the original goal of the JIRA ticket was to allow to configure the compression by specifying options like the level. Similarly to what we allow in C++ and Python where you can set `compression` and `compression_level` ( https://arrow.apache.org/docs/python/generated/pyarrow.Codec.html?highlight=codec#pyarrow.Codec ). It seems that the `CompressionCodec` and `CommonsCompressionFactory` do not allow to specify anything else than the `codecType` but not a level for such codecs. Should `Lz4CompressionCodec` and `ZstdCompressionCodec` grow a constructor argument to specify the level?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] github-actions[bot] commented on pull request #11591: [Java] ARROW-12163 - Make compression levels configurable

[GitBox <gi...@apache.org>]

github-actions[bot] commented on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-957355018






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] pitrou commented on a change in pull request #11591: ARROW-12163: [Java] Make compression levels configurable.

[GitBox <gi...@apache.org>]

pitrou commented on a change in pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#discussion_r741887094



##########
File path: docs/source/java/ipc.rst
##########
@@ -54,6 +54,15 @@ Now, we can begin writing a stream containing some number of these batches. For
     ArrowStreamWriter writer = new ArrowStreamWriter(root, /*DictionaryProvider=*/null, Channels.newChannel(out));
 
 
+Buffer level compression is also supported by passing a codec:
+Lz4CompressionCodec / ZstdCompressionCodec:
+
+.. code-block:: Java
+
+    ArrowStreamWriter writer = new ArrowStreamWriter(schemaRootWrite, true, new Lz4CompressionCodec(), true,
+        null, fileOutputStream.getChannel());

Review comment:
       "true, null" doesn't say much about what these parameters are. Should you add commented parameter names as above (e.g. `/*DictionaryProvider=*/null`, but I don't know if it's the right name here?).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] github-actions[bot] commented on pull request #11591: ARROW-12163: [Java] Make compression levels configurable.

[GitBox <gi...@apache.org>]

github-actions[bot] commented on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-958109374


   https://issues.apache.org/jira/browse/ARROW-12163


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] amol- removed a comment on pull request #11591: ARROW-12163: [Java] Make compression levels configurable.

[GitBox <gi...@apache.org>]

amol- removed a comment on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-959234041


   > One more note. The JIRA linked is about adjusting the compression level used by compression algorithms (trading off speed for output size). I didnt notice a change here for that apologies if i missed it.
   
   @davisusanibar that seems true, the original goal of the JIRA ticket was to allow to configure the compression by specifying options like the level. Similarly to what we allow in C++ and Python where you can set `compression` and `compression_level` ( https://arrow.apache.org/docs/python/generated/pyarrow.Codec.html?highlight=codec#pyarrow.Codec ).
   
   It seems that the `CompressionCodec` and `CommonsCompressionFactory` do not allow to specify anything else than the `codecType` but not a level for such codecs. 
   
   Should `Lz4CompressionCodec` and `ZstdCompressionCodec` grow a constructor argument to specify the level?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] emkornfield commented on pull request #11591: [Java] ARROW-12163 - Make compression levels configurable

[GitBox <gi...@apache.org>]

emkornfield commented on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-957998681


   One more note.  The JIRA linked is about adjusting the compression level used by compression algorithms (trading off speed for output size).  I didnt notice a change here for that apologies if i missed it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] amol- removed a comment on pull request #11591: ARROW-12163: [Java] Make compression levels configurable.

[GitBox <gi...@apache.org>]

amol- removed a comment on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-959234041






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] amol- commented on pull request #11591: ARROW-12163: [Java] Make compression levels configurable.

[GitBox <gi...@apache.org>]

amol- commented on pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#issuecomment-959234041


   > One more note. The JIRA linked is about adjusting the compression level used by compression algorithms (trading off speed for output size). I didnt notice a change here for that apologies if i missed it.
   
   @davisusanibar that seems true, the original goal of the JIRA ticket was to allow to configure the compression by specifying options like the level. Similarly to what we allow in C++ and Python where you can set `compression` and `compression_level` ( https://arrow.apache.org/docs/python/generated/pyarrow.Codec.html?highlight=codec#pyarrow.Codec ). It seems that the `CompressionCodec` and `CommonsCompressionFactory` do not allow to specify anything else than the `codecType` but not a level for such codecs. Should `Lz4CompressionCodec` and `ZstdCompressionCodec` grow a constructor argument to specify the level?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] pitrou commented on a change in pull request #11591: ARROW-12163: [Java] Make compression levels configurable.

[GitBox <gi...@apache.org>]

pitrou commented on a change in pull request #11591:
URL: https://github.com/apache/arrow/pull/11591#discussion_r741887538



##########
File path: docs/source/java/ipc.rst
##########
@@ -54,6 +54,15 @@ Now, we can begin writing a stream containing some number of these batches. For
     ArrowStreamWriter writer = new ArrowStreamWriter(root, /*DictionaryProvider=*/null, Channels.newChannel(out));
 
 
+Buffer level compression is also supported by passing a codec:
+Lz4CompressionCodec / ZstdCompressionCodec:

Review comment:
       Looks like there is a missing sentence here? Are these the supported codecs? Something else?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org