You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/08/08 16:09:02 UTC
[GitHub] [arrow-rs] alamb opened a new pull request, #2369: Alamb/help feature flags
alamb opened a new pull request, #2369:
URL: https://github.com/apache/arrow-rs/pull/2369
# Which issue does this PR close?
Closes https://github.com/apache/arrow-rs/pull/1855
Closes https://github.com/liukun4515/arrow-rs/pull/1
Closes #1709
Closes #70
# Rationale for this change
Not this is a new PR based on @liukun4515 's work in https://github.com/apache/arrow-rs/pull/1855. Github is showing a massive diff in https://github.com/liukun4515/arrow-rs/pull/1 so I wanted to get this PR up so we could see what the actual change is proposed
# What changes are included in this PR?
1. Add support for LZ4_STREAM and ZSTD under `ipc_compression` feature flag
2. Tests for same
# Are there any user-facing changes?
New feature flag, and support for compression
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
alamb commented on PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#issuecomment-1211798532
> Thanks for your effort and push this feature forward, and help me to refactor this model.
It was my pleasure -- thank you for figuring out the tests and the format that was required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r941184959
##########
arrow/src/ipc/compression/codec.rs:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::buffer::Buffer;
+use crate::error::{ArrowError, Result};
+use crate::ipc::CompressionType;
+use std::io::{Read, Write};
+
+const LENGTH_NO_COMPRESSED_DATA: i64 = -1;
+const LENGTH_OF_PREFIX_DATA: i64 = 8;
+
+#[derive(Debug, Clone, Copy, PartialEq)]
+/// Represents compressing a ipc stream using a particular compression algorithm
+pub enum CompressionCodec {
+ Lz4Frame,
+ Zstd,
+}
+
+impl TryFrom<CompressionType> for CompressionCodec {
+ type Error = ArrowError;
+
+ fn try_from(compression_type: CompressionType) -> Result<Self> {
+ match compression_type {
+ CompressionType::ZSTD => Ok(CompressionCodec::Zstd),
+ CompressionType::LZ4_FRAME => Ok(CompressionCodec::Lz4Frame),
+ other_type => Err(ArrowError::NotYetImplemented(format!(
+ "compression type {:?} not supported ",
+ other_type
+ ))),
+ }
+ }
+}
+
+impl CompressionCodec {
+ /// Compresses the data in `input` to `output` and appends the
+ /// data using the specified compression mechanism.
+ ///
+ /// returns the number of bytes written to the stream
+ ///
+ /// Writes this format to output:
+ /// ```text
+ /// [8 bytes]: uncompressed length
+ /// [reminaing bytes]: compressed data stream
+ /// ```
+ pub(crate) fn compress_to_vec(
+ &self,
+ input: &[u8],
+ output: &mut Vec<u8>,
+ ) -> Result<usize> {
+ let uncompressed_data_len = input.len();
+ let original_output_len = output.len();
+
+ if uncompressed_data_len == 0 {
Review Comment:
```suggestion
if input.is_empty() {
```
##########
arrow/src/ipc/writer.rs:
##########
@@ -322,13 +350,28 @@ impl IpcDataGenerator {
&self,
batch: &RecordBatch,
write_options: &IpcWriteOptions,
- ) -> EncodedData {
+ ) -> Result<EncodedData> {
let mut fbb = FlatBufferBuilder::new();
let mut nodes: Vec<ipc::FieldNode> = vec![];
let mut buffers: Vec<ipc::Buffer> = vec![];
let mut arrow_data: Vec<u8> = vec![];
let mut offset = 0;
+
+ // get the type of compression
+ let batch_compression_type = write_options.batch_compression_type;
+
+ let compression = batch_compression_type.map(|batch_compression_type| {
+ let mut c = ipc::BodyCompressionBuilder::new(&mut fbb);
+ c.add_method(ipc::BodyCompressionMethod::BUFFER);
+ c.add_codec(batch_compression_type);
+ c.finish()
+ });
+
+ let compression_codec: Option<CompressionCodec> = batch_compression_type
+ .map(|batch_compression_type| batch_compression_type.try_into())
Review Comment:
```suggestion
.map(TryInto::try_into)
```
##########
arrow/src/ipc/writer.rs:
##########
@@ -339,19 +382,26 @@ impl IpcDataGenerator {
offset,
array.len(),
array.null_count(),
+ &compression_codec,
write_options,
- );
+ )?;
}
+ // pad the tail of body data
Review Comment:
I'm guessing this previously wasn't needed because the buffers were already 8 byte aligned
##########
arrow/src/ipc/writer.rs:
##########
@@ -520,17 +593,20 @@ impl<W: Write> FileWriter<W> {
let data_gen = IpcDataGenerator::default();
let mut writer = BufWriter::new(writer);
// write magic to header
+ let mut header_size: usize = 0;
Review Comment:
```suggestion
let header_size: usize = ARROW_MAGIC.len() + 2;
```
?
##########
arrow/src/ipc/compression/stub.rs:
##########
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Stubs that implement the same interface as ipc_compression
+//! but always error.
+
+use crate::buffer::Buffer;
+use crate::error::{ArrowError, Result};
+use crate::ipc::CompressionType;
+
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub enum CompressionCodec {}
+
+impl TryFrom<CompressionCodec> for CompressionType {
+ type Error = ArrowError;
+ fn try_from(codec: CompressionCodec) -> Result<Self> {
+ return Err(ArrowError::InvalidArgumentError(
+ format!("codec type {:?} not supported because arrow was not compiled with the ipc_compression feature", codec)));
+ }
+}
+
+impl TryFrom<CompressionType> for CompressionCodec {
+ type Error = ArrowError;
+
+ fn try_from(compression_type: CompressionType) -> Result<Self> {
+ Err(ArrowError::InvalidArgumentError(
+ format!("compression type {:?} not supported because arrow was not compiled with the ipc_compression feature", compression_type))
+ )
+ }
+}
+
+impl CompressionCodec {
+ #[allow(clippy::ptr_arg)]
Review Comment:
I might be misunderstanding this lint, but I don't think it should be firing for this method
##########
arrow/src/ipc/compression/codec.rs:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::buffer::Buffer;
+use crate::error::{ArrowError, Result};
+use crate::ipc::CompressionType;
+use std::io::{Read, Write};
+
+const LENGTH_NO_COMPRESSED_DATA: i64 = -1;
+const LENGTH_OF_PREFIX_DATA: i64 = 8;
+
+#[derive(Debug, Clone, Copy, PartialEq)]
+/// Represents compressing a ipc stream using a particular compression algorithm
+pub enum CompressionCodec {
+ Lz4Frame,
+ Zstd,
+}
+
+impl TryFrom<CompressionType> for CompressionCodec {
+ type Error = ArrowError;
+
+ fn try_from(compression_type: CompressionType) -> Result<Self> {
+ match compression_type {
+ CompressionType::ZSTD => Ok(CompressionCodec::Zstd),
+ CompressionType::LZ4_FRAME => Ok(CompressionCodec::Lz4Frame),
+ other_type => Err(ArrowError::NotYetImplemented(format!(
+ "compression type {:?} not supported ",
+ other_type
+ ))),
+ }
+ }
+}
+
+impl CompressionCodec {
+ /// Compresses the data in `input` to `output` and appends the
+ /// data using the specified compression mechanism.
+ ///
+ /// returns the number of bytes written to the stream
+ ///
+ /// Writes this format to output:
+ /// ```text
+ /// [8 bytes]: uncompressed length
+ /// [reminaing bytes]: compressed data stream
+ /// ```
+ pub(crate) fn compress_to_vec(
+ &self,
+ input: &[u8],
+ output: &mut Vec<u8>,
+ ) -> Result<usize> {
+ let uncompressed_data_len = input.len();
+ let original_output_len = output.len();
+
+ if uncompressed_data_len == 0 {
+ // empty input, nothing to do
+ } else {
+ // write compressed data directly into the output buffer
+ output.extend_from_slice(&uncompressed_data_len.to_le_bytes());
+ self.compress(input, output)?;
+
+ let compression_len = output.len();
+ if compression_len > uncompressed_data_len {
+ // length of compressed data was larger than
+ // uncompressed data, use the uncompressed data with
+ // length -1 to indicate that we don't compress the
+ // data
+ output.truncate(original_output_len);
+ output.extend_from_slice(&LENGTH_NO_COMPRESSED_DATA.to_le_bytes());
+ output.extend_from_slice(input);
+ }
+ }
+ Ok(output.len() - original_output_len)
+ }
+
+ /// Decompresses the input into a [`Buffer`]
+ ///
+ /// The input should look like:
+ /// ```text
+ /// [8 bytes]: uncompressed length
+ /// [reminaing bytes]: compressed data stream
+ /// ```
+ pub(crate) fn decompress_to_buffer(&self, input: &[u8]) -> Result<Buffer> {
+ // read the first 8 bytes to determine if the data is
+ // compressed
+ let decompressed_length = read_uncompressed_size(input);
+ let buffer = if decompressed_length == 0 {
+ // emtpy
+ let empty = Vec::<u8>::new();
+ Buffer::from(empty)
+ } else if decompressed_length == LENGTH_NO_COMPRESSED_DATA {
+ // no compression
+ let data = &input[(LENGTH_OF_PREFIX_DATA as usize)..];
+ Buffer::from(data)
Review Comment:
Just an observation, but this copy is kind of unfortunate (although existed before)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r945109122
##########
arrow/src/ipc/compression/stub.rs:
##########
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Stubs that implement the same interface as ipc_compression
+//! but always error.
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r940425363
##########
arrow/src/ipc/compression/stub.rs:
##########
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Stubs that implement the same interface as ipc_compression
Review Comment:
This is what gets run if the `ipc_compression` flag is not enabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] liukun4515 commented on pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#issuecomment-1211951484
> > Thanks for your effort and push this feature forward, and help me to refactor this model.
>
> It was my pleasure -- thank you for figuring out the tests and the format that was required
Also we need to enable the IT test in the arrow.
https://github.com/apache/arrow-rs/issues/1907
And I can try it.
I have not thought about it carefully, but how to enable the `ipc_compression` in the IT ci?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] liukun4515 commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r943044711
##########
arrow/src/ipc/writer.rs:
##########
@@ -1096,29 +1210,56 @@ fn write_array_data(
offset,
data_ref.len(),
data_ref.null_count(),
+ compression_codec,
write_options,
- );
- });
+ )?;
+ }
}
- offset
+ Ok(offset)
}
-/// Write a buffer to a vector of bytes, and add its ipc::Buffer to a vector
+/// Write a buffer into `arrow_data`, a vector of bytes, and adds its
Review Comment:
thanks for your notes for this function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold merged pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
tustvold merged PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r945123669
##########
arrow/src/ipc/compression/stub.rs:
##########
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Stubs that implement the same interface as ipc_compression
+//! but always error.
+
+use crate::buffer::Buffer;
+use crate::error::{ArrowError, Result};
+use crate::ipc::CompressionType;
+
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub enum CompressionCodec {}
+
+impl TryFrom<CompressionCodec> for CompressionType {
+ type Error = ArrowError;
+ fn try_from(codec: CompressionCodec) -> Result<Self> {
+ return Err(ArrowError::InvalidArgumentError(
+ format!("codec type {:?} not supported because arrow was not compiled with the ipc_compression feature", codec)));
+ }
+}
+
+impl TryFrom<CompressionType> for CompressionCodec {
+ type Error = ArrowError;
+
+ fn try_from(compression_type: CompressionType) -> Result<Self> {
+ Err(ArrowError::InvalidArgumentError(
+ format!("compression type {:?} not supported because arrow was not compiled with the ipc_compression feature", compression_type))
+ )
+ }
+}
+
+impl CompressionCodec {
+ #[allow(clippy::ptr_arg)]
Review Comment:
When I remove it and run clippy like
```shell
cargo clippy -p arrow --features=prettyprint,csv,ipc,test_utils --all-targets -- -D warnings
```
Clippy tells me:
```
error: writing `&mut Vec` instead of `&mut [_]` involves a new object where a slice will do
--> arrow/src/ipc/compression/stub.rs:50:18
|
50 | _output: &mut Vec<u8>,
| ^^^^^^^^^^^^ help: change this to: `&mut [u8]`
|
= note: `-D clippy::ptr-arg` implied by `-D warnings`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#ptr_arg
```
Which while true in this case, is not true for the actual codec implementation, and they both need to have the same signature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r945124206
##########
arrow/src/ipc/compression/stub.rs:
##########
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Stubs that implement the same interface as ipc_compression
+//! but always error.
+
+use crate::buffer::Buffer;
+use crate::error::{ArrowError, Result};
+use crate::ipc::CompressionType;
+
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub enum CompressionCodec {}
+
+impl TryFrom<CompressionCodec> for CompressionType {
+ type Error = ArrowError;
+ fn try_from(codec: CompressionCodec) -> Result<Self> {
+ return Err(ArrowError::InvalidArgumentError(
+ format!("codec type {:?} not supported because arrow was not compiled with the ipc_compression feature", codec)));
+ }
+}
+
+impl TryFrom<CompressionType> for CompressionCodec {
+ type Error = ArrowError;
+
+ fn try_from(compression_type: CompressionType) -> Result<Self> {
+ Err(ArrowError::InvalidArgumentError(
+ format!("compression type {:?} not supported because arrow was not compiled with the ipc_compression feature", compression_type))
+ )
+ }
+}
+
+impl CompressionCodec {
+ #[allow(clippy::ptr_arg)]
Review Comment:
That's a very daft lint, `&mut Vec<u8>` is not the same as `&mut [u8]`, in particular the latter must initialize memory... I somewhat assumed clippy wasn't being silly, guess I was wrong :sweat_smile:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r945124499
##########
arrow/src/ipc/writer.rs:
##########
@@ -520,17 +593,20 @@ impl<W: Write> FileWriter<W> {
let data_gen = IpcDataGenerator::default();
let mut writer = BufWriter::new(writer);
// write magic to header
+ let mut header_size: usize = 0;
Review Comment:
Good call -- I have updated in 2ed7ce308 (and added an `assert` to verify the currently implicit assumption that `ARROW_MAGIC` is 6 bytes long.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r945122243
##########
arrow/src/ipc/compression/codec.rs:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::buffer::Buffer;
+use crate::error::{ArrowError, Result};
+use crate::ipc::CompressionType;
+use std::io::{Read, Write};
+
+const LENGTH_NO_COMPRESSED_DATA: i64 = -1;
+const LENGTH_OF_PREFIX_DATA: i64 = 8;
+
+#[derive(Debug, Clone, Copy, PartialEq)]
+/// Represents compressing a ipc stream using a particular compression algorithm
+pub enum CompressionCodec {
+ Lz4Frame,
+ Zstd,
+}
+
+impl TryFrom<CompressionType> for CompressionCodec {
+ type Error = ArrowError;
+
+ fn try_from(compression_type: CompressionType) -> Result<Self> {
+ match compression_type {
+ CompressionType::ZSTD => Ok(CompressionCodec::Zstd),
+ CompressionType::LZ4_FRAME => Ok(CompressionCodec::Lz4Frame),
+ other_type => Err(ArrowError::NotYetImplemented(format!(
+ "compression type {:?} not supported ",
+ other_type
+ ))),
+ }
+ }
+}
+
+impl CompressionCodec {
+ /// Compresses the data in `input` to `output` and appends the
+ /// data using the specified compression mechanism.
+ ///
+ /// returns the number of bytes written to the stream
+ ///
+ /// Writes this format to output:
+ /// ```text
+ /// [8 bytes]: uncompressed length
+ /// [reminaing bytes]: compressed data stream
+ /// ```
+ pub(crate) fn compress_to_vec(
+ &self,
+ input: &[u8],
+ output: &mut Vec<u8>,
+ ) -> Result<usize> {
+ let uncompressed_data_len = input.len();
+ let original_output_len = output.len();
+
+ if uncompressed_data_len == 0 {
+ // empty input, nothing to do
+ } else {
+ // write compressed data directly into the output buffer
+ output.extend_from_slice(&uncompressed_data_len.to_le_bytes());
+ self.compress(input, output)?;
+
+ let compression_len = output.len();
+ if compression_len > uncompressed_data_len {
+ // length of compressed data was larger than
+ // uncompressed data, use the uncompressed data with
+ // length -1 to indicate that we don't compress the
+ // data
+ output.truncate(original_output_len);
+ output.extend_from_slice(&LENGTH_NO_COMPRESSED_DATA.to_le_bytes());
+ output.extend_from_slice(input);
+ }
+ }
+ Ok(output.len() - original_output_len)
+ }
+
+ /// Decompresses the input into a [`Buffer`]
+ ///
+ /// The input should look like:
+ /// ```text
+ /// [8 bytes]: uncompressed length
+ /// [reminaing bytes]: compressed data stream
+ /// ```
+ pub(crate) fn decompress_to_buffer(&self, input: &[u8]) -> Result<Buffer> {
+ // read the first 8 bytes to determine if the data is
+ // compressed
+ let decompressed_length = read_uncompressed_size(input);
+ let buffer = if decompressed_length == 0 {
+ // emtpy
+ let empty = Vec::<u8>::new();
+ Buffer::from(empty)
+ } else if decompressed_length == LENGTH_NO_COMPRESSED_DATA {
+ // no compression
+ let data = &input[(LENGTH_OF_PREFIX_DATA as usize)..];
+ Buffer::from(data)
Review Comment:
Are you thinking the alternate is to create a Buffer initially and write this code in terms of `Buffer` rather than `&[u8]`?
##########
arrow/src/ipc/reader.rs:
##########
@@ -761,6 +797,21 @@ pub struct FileReader<R: Read + Seek> {
projection: Option<(Vec<usize>, Schema)>,
}
+impl<R: Read + Seek> fmt::Debug for FileReader<R> {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> std::result::Result<(), fmt::Error> {
+ f.debug_struct("StreamReader<R>")
Review Comment:
wow nice find 🦅
##########
arrow/src/ipc/writer.rs:
##########
@@ -339,19 +382,26 @@ impl IpcDataGenerator {
offset,
array.len(),
array.null_count(),
+ &compression_codec,
write_options,
- );
+ )?;
}
+ // pad the tail of body data
Review Comment:
Probably -- I am not sure to be honest. This is code that was in @liukun4515 's original PR but it seemed reasonable to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r945124334
##########
arrow/src/ipc/compression/codec.rs:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::buffer::Buffer;
+use crate::error::{ArrowError, Result};
+use crate::ipc::CompressionType;
+use std::io::{Read, Write};
+
+const LENGTH_NO_COMPRESSED_DATA: i64 = -1;
+const LENGTH_OF_PREFIX_DATA: i64 = 8;
+
+#[derive(Debug, Clone, Copy, PartialEq)]
+/// Represents compressing a ipc stream using a particular compression algorithm
+pub enum CompressionCodec {
+ Lz4Frame,
+ Zstd,
+}
+
+impl TryFrom<CompressionType> for CompressionCodec {
+ type Error = ArrowError;
+
+ fn try_from(compression_type: CompressionType) -> Result<Self> {
+ match compression_type {
+ CompressionType::ZSTD => Ok(CompressionCodec::Zstd),
+ CompressionType::LZ4_FRAME => Ok(CompressionCodec::Lz4Frame),
+ other_type => Err(ArrowError::NotYetImplemented(format!(
+ "compression type {:?} not supported ",
+ other_type
+ ))),
+ }
+ }
+}
+
+impl CompressionCodec {
+ /// Compresses the data in `input` to `output` and appends the
+ /// data using the specified compression mechanism.
+ ///
+ /// returns the number of bytes written to the stream
+ ///
+ /// Writes this format to output:
+ /// ```text
+ /// [8 bytes]: uncompressed length
+ /// [reminaing bytes]: compressed data stream
+ /// ```
+ pub(crate) fn compress_to_vec(
+ &self,
+ input: &[u8],
+ output: &mut Vec<u8>,
+ ) -> Result<usize> {
+ let uncompressed_data_len = input.len();
+ let original_output_len = output.len();
+
+ if uncompressed_data_len == 0 {
+ // empty input, nothing to do
+ } else {
+ // write compressed data directly into the output buffer
+ output.extend_from_slice(&uncompressed_data_len.to_le_bytes());
+ self.compress(input, output)?;
+
+ let compression_len = output.len();
+ if compression_len > uncompressed_data_len {
+ // length of compressed data was larger than
+ // uncompressed data, use the uncompressed data with
+ // length -1 to indicate that we don't compress the
+ // data
+ output.truncate(original_output_len);
+ output.extend_from_slice(&LENGTH_NO_COMPRESSED_DATA.to_le_bytes());
+ output.extend_from_slice(input);
+ }
+ }
+ Ok(output.len() - original_output_len)
+ }
+
+ /// Decompresses the input into a [`Buffer`]
+ ///
+ /// The input should look like:
+ /// ```text
+ /// [8 bytes]: uncompressed length
+ /// [reminaing bytes]: compressed data stream
+ /// ```
+ pub(crate) fn decompress_to_buffer(&self, input: &[u8]) -> Result<Buffer> {
+ // read the first 8 bytes to determine if the data is
+ // compressed
+ let decompressed_length = read_uncompressed_size(input);
+ let buffer = if decompressed_length == 0 {
+ // emtpy
+ let empty = Vec::<u8>::new();
+ Buffer::from(empty)
+ } else if decompressed_length == LENGTH_NO_COMPRESSED_DATA {
+ // no compression
+ let data = &input[(LENGTH_OF_PREFIX_DATA as usize)..];
+ Buffer::from(data)
Review Comment:
Yeah... Not important for this PR, but it seems unfortunate the amount of memory copying we are doing, especially when the major design goal of the IPC spec is to avoid this :sweat_smile:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r945121811
##########
.github/workflows/arrow.yml:
##########
@@ -53,7 +53,7 @@ jobs:
cargo test -p arrow
- name: Test --features=force_validate,prettyprint,ffi
Review Comment:
I think it is a good call to keep the comments up to date with the implementation,. Good spot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r945108821
##########
arrow/src/ipc/compression/codec.rs:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::buffer::Buffer;
+use crate::error::{ArrowError, Result};
+use crate::ipc::CompressionType;
+use std::io::{Read, Write};
+
+const LENGTH_NO_COMPRESSED_DATA: i64 = -1;
+const LENGTH_OF_PREFIX_DATA: i64 = 8;
+
+#[derive(Debug, Clone, Copy, PartialEq)]
+/// Represents compressing a ipc stream using a particular compression algorithm
+pub enum CompressionCodec {
+ Lz4Frame,
+ Zstd,
+}
+
+impl TryFrom<CompressionType> for CompressionCodec {
+ type Error = ArrowError;
+
+ fn try_from(compression_type: CompressionType) -> Result<Self> {
+ match compression_type {
+ CompressionType::ZSTD => Ok(CompressionCodec::Zstd),
+ CompressionType::LZ4_FRAME => Ok(CompressionCodec::Lz4Frame),
+ other_type => Err(ArrowError::NotYetImplemented(format!(
+ "compression type {:?} not supported ",
+ other_type
+ ))),
+ }
+ }
+}
+
+impl CompressionCodec {
+ /// Compresses the data in `input` to `output` and appends the
+ /// data using the specified compression mechanism.
+ ///
+ /// returns the number of bytes written to the stream
+ ///
+ /// Writes this format to output:
+ /// ```text
+ /// [8 bytes]: uncompressed length
+ /// [reminaing bytes]: compressed data stream
+ /// ```
+ pub(crate) fn compress_to_vec(
+ &self,
+ input: &[u8],
+ output: &mut Vec<u8>,
+ ) -> Result<usize> {
+ let uncompressed_data_len = input.len();
+ let original_output_len = output.len();
+
+ if uncompressed_data_len == 0 {
+ // empty input, nothing to do
+ } else {
+ // write compressed data directly into the output buffer
+ output.extend_from_slice(&uncompressed_data_len.to_le_bytes());
+ self.compress(input, output)?;
+
+ let compression_len = output.len();
+ if compression_len > uncompressed_data_len {
+ // length of compressed data was larger than
+ // uncompressed data, use the uncompressed data with
+ // length -1 to indicate that we don't compress the
+ // data
+ output.truncate(original_output_len);
+ output.extend_from_slice(&LENGTH_NO_COMPRESSED_DATA.to_le_bytes());
+ output.extend_from_slice(input);
+ }
+ }
+ Ok(output.len() - original_output_len)
+ }
+
+ /// Decompresses the input into a [`Buffer`]
+ ///
+ /// The input should look like:
+ /// ```text
+ /// [8 bytes]: uncompressed length
+ /// [reminaing bytes]: compressed data stream
Review Comment:
```suggestion
/// [remaining bytes]: compressed data stream
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r945109641
##########
arrow/src/ipc/writer.rs:
##########
@@ -1096,29 +1210,56 @@ fn write_array_data(
offset,
data_ref.len(),
data_ref.null_count(),
+ compression_codec,
write_options,
- );
- });
+ )?;
+ }
}
- offset
+ Ok(offset)
}
-/// Write a buffer to a vector of bytes, and add its ipc::Buffer to a vector
+/// Write a buffer into `arrow_data`, a vector of bytes, and adds its
+/// [`ipc::Buffer`] to `buffers`. Returns the new offset in `arrow_data`
+///
+///
+/// From <https://github.com/apache/arrow/blob/6a936c4ff5007045e86f65f1a6b6c3c955ad5103/format/Message.fbs#L58>
+/// Each constituent buffer is first compressed with the indicated
+/// compressor, and then written with the uncompressed length in the first 8
+/// bytes as a 64-bit little-endian signed integer followed by the compressed
+/// buffer bytes (and then padding as required by the protocol). The
+/// uncompressed length may be set to -1 to indicate that the data that
+/// follows is not compressed, which can be useful for cases where
+/// compression does not yield appreciable savings.
fn write_buffer(
- buffer: &[u8],
- buffers: &mut Vec<ipc::Buffer>,
- arrow_data: &mut Vec<u8>,
- offset: i64,
-) -> i64 {
- let len = buffer.len();
- let pad_len = pad_to_8(len as u32);
- let total_len: i64 = (len + pad_len) as i64;
- // assert_eq!(len % 8, 0, "Buffer width not a multiple of 8 bytes");
- buffers.push(ipc::Buffer::new(offset, total_len));
- arrow_data.extend_from_slice(buffer);
- arrow_data.extend_from_slice(&vec![0u8; pad_len][..]);
- offset + total_len
+ buffer: &[u8], // input
+ buffers: &mut Vec<ipc::Buffer>, // output buffer descriptors
+ arrow_data: &mut Vec<u8>, // output stream
+ offset: i64, // current output stream offset
+ compression_codec: &Option<CompressionCodec>,
+) -> Result<i64> {
+ let len: i64 = match compression_codec {
+ Some(compressor) => compressor.compress_to_vec(buffer, arrow_data)?,
+ None => {
+ arrow_data.extend_from_slice(buffer);
+ buffer.len()
+ }
+ }
+ .try_into()
+ .map_err(|e| {
+ ArrowError::InvalidArgumentError(format!(
+ "Could not convert compressed size to i64: {}",
+ e
+ ))
+ })?;
+
+ // make new indx entry
Review Comment:
```suggestion
// make new index entry
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r945109413
##########
arrow/src/ipc/reader.rs:
##########
@@ -761,6 +797,21 @@ pub struct FileReader<R: Read + Seek> {
projection: Option<(Vec<usize>, Schema)>,
}
+impl<R: Read + Seek> fmt::Debug for FileReader<R> {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> std::result::Result<(), fmt::Error> {
+ f.debug_struct("StreamReader<R>")
Review Comment:
```suggestion
f.debug_struct("FileReader<R>")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] liukun4515 commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r943521524
##########
.github/workflows/arrow.yml:
##########
@@ -53,7 +53,7 @@ jobs:
cargo test -p arrow
- name: Test --features=force_validate,prettyprint,ffi
Review Comment:
```suggestion
- name: Test --features=force_validate,prettyprint,ipc_compression,ffi
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] liukun4515 commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r943522404
##########
.github/workflows/arrow.yml:
##########
@@ -53,7 +53,7 @@ jobs:
cargo test -p arrow
- name: Test --features=force_validate,prettyprint,ffi
Review Comment:
check the ci and find this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r945124206
##########
arrow/src/ipc/compression/stub.rs:
##########
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Stubs that implement the same interface as ipc_compression
+//! but always error.
+
+use crate::buffer::Buffer;
+use crate::error::{ArrowError, Result};
+use crate::ipc::CompressionType;
+
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub enum CompressionCodec {}
+
+impl TryFrom<CompressionCodec> for CompressionType {
+ type Error = ArrowError;
+ fn try_from(codec: CompressionCodec) -> Result<Self> {
+ return Err(ArrowError::InvalidArgumentError(
+ format!("codec type {:?} not supported because arrow was not compiled with the ipc_compression feature", codec)));
+ }
+}
+
+impl TryFrom<CompressionType> for CompressionCodec {
+ type Error = ArrowError;
+
+ fn try_from(compression_type: CompressionType) -> Result<Self> {
+ Err(ArrowError::InvalidArgumentError(
+ format!("compression type {:?} not supported because arrow was not compiled with the ipc_compression feature", compression_type))
+ )
+ }
+}
+
+impl CompressionCodec {
+ #[allow(clippy::ptr_arg)]
Review Comment:
That's a very daft lint, `&mut Vec<u8>` is not the same as `&mut [u8]`, in particular the latter must initialize memory... I can understand `&[u8]` vs `&Vec<u8>` but the mutability makes a difference... I somewhat assumed clippy wasn't being silly, guess I was wrong :sweat_smile:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r945107899
##########
arrow/src/ipc/compression/codec.rs:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::buffer::Buffer;
+use crate::error::{ArrowError, Result};
+use crate::ipc::CompressionType;
+use std::io::{Read, Write};
+
+const LENGTH_NO_COMPRESSED_DATA: i64 = -1;
+const LENGTH_OF_PREFIX_DATA: i64 = 8;
+
+#[derive(Debug, Clone, Copy, PartialEq)]
+/// Represents compressing a ipc stream using a particular compression algorithm
+pub enum CompressionCodec {
+ Lz4Frame,
+ Zstd,
+}
+
+impl TryFrom<CompressionType> for CompressionCodec {
+ type Error = ArrowError;
+
+ fn try_from(compression_type: CompressionType) -> Result<Self> {
+ match compression_type {
+ CompressionType::ZSTD => Ok(CompressionCodec::Zstd),
+ CompressionType::LZ4_FRAME => Ok(CompressionCodec::Lz4Frame),
+ other_type => Err(ArrowError::NotYetImplemented(format!(
+ "compression type {:?} not supported ",
+ other_type
+ ))),
+ }
+ }
+}
+
+impl CompressionCodec {
+ /// Compresses the data in `input` to `output` and appends the
+ /// data using the specified compression mechanism.
+ ///
+ /// returns the number of bytes written to the stream
+ ///
+ /// Writes this format to output:
+ /// ```text
+ /// [8 bytes]: uncompressed length
+ /// [reminaing bytes]: compressed data stream
Review Comment:
```suggestion
/// [remaining bytes]: compressed data stream
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r945130857
##########
arrow/src/ipc/compression/codec.rs:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::buffer::Buffer;
+use crate::error::{ArrowError, Result};
+use crate::ipc::CompressionType;
+use std::io::{Read, Write};
+
+const LENGTH_NO_COMPRESSED_DATA: i64 = -1;
+const LENGTH_OF_PREFIX_DATA: i64 = 8;
+
+#[derive(Debug, Clone, Copy, PartialEq)]
+/// Represents compressing a ipc stream using a particular compression algorithm
+pub enum CompressionCodec {
+ Lz4Frame,
+ Zstd,
+}
+
+impl TryFrom<CompressionType> for CompressionCodec {
+ type Error = ArrowError;
+
+ fn try_from(compression_type: CompressionType) -> Result<Self> {
+ match compression_type {
+ CompressionType::ZSTD => Ok(CompressionCodec::Zstd),
+ CompressionType::LZ4_FRAME => Ok(CompressionCodec::Lz4Frame),
+ other_type => Err(ArrowError::NotYetImplemented(format!(
+ "compression type {:?} not supported ",
+ other_type
+ ))),
+ }
+ }
+}
+
+impl CompressionCodec {
+ /// Compresses the data in `input` to `output` and appends the
+ /// data using the specified compression mechanism.
+ ///
+ /// returns the number of bytes written to the stream
+ ///
+ /// Writes this format to output:
+ /// ```text
+ /// [8 bytes]: uncompressed length
+ /// [reminaing bytes]: compressed data stream
+ /// ```
+ pub(crate) fn compress_to_vec(
+ &self,
+ input: &[u8],
+ output: &mut Vec<u8>,
+ ) -> Result<usize> {
+ let uncompressed_data_len = input.len();
+ let original_output_len = output.len();
+
+ if uncompressed_data_len == 0 {
+ // empty input, nothing to do
+ } else {
+ // write compressed data directly into the output buffer
+ output.extend_from_slice(&uncompressed_data_len.to_le_bytes());
+ self.compress(input, output)?;
+
+ let compression_len = output.len();
+ if compression_len > uncompressed_data_len {
+ // length of compressed data was larger than
+ // uncompressed data, use the uncompressed data with
+ // length -1 to indicate that we don't compress the
+ // data
+ output.truncate(original_output_len);
+ output.extend_from_slice(&LENGTH_NO_COMPRESSED_DATA.to_le_bytes());
+ output.extend_from_slice(input);
+ }
+ }
+ Ok(output.len() - original_output_len)
+ }
+
+ /// Decompresses the input into a [`Buffer`]
+ ///
+ /// The input should look like:
+ /// ```text
+ /// [8 bytes]: uncompressed length
+ /// [reminaing bytes]: compressed data stream
+ /// ```
+ pub(crate) fn decompress_to_buffer(&self, input: &[u8]) -> Result<Buffer> {
+ // read the first 8 bytes to determine if the data is
+ // compressed
+ let decompressed_length = read_uncompressed_size(input);
+ let buffer = if decompressed_length == 0 {
+ // emtpy
+ let empty = Vec::<u8>::new();
+ Buffer::from(empty)
+ } else if decompressed_length == LENGTH_NO_COMPRESSED_DATA {
+ // no compression
+ let data = &input[(LENGTH_OF_PREFIX_DATA as usize)..];
+ Buffer::from(data)
Review Comment:
Filed https://github.com/apache/arrow-rs/issues/2437
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
tustvold commented on PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#issuecomment-1208571301
I plan to review this tomorrow morning
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] liukun4515 commented on pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#issuecomment-1211954871
This issue https://github.com/apache/arrow-rs/issues/2342 also can be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r945109059
##########
arrow/src/ipc/compression/codec.rs:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::buffer::Buffer;
+use crate::error::{ArrowError, Result};
+use crate::ipc::CompressionType;
+use std::io::{Read, Write};
+
+const LENGTH_NO_COMPRESSED_DATA: i64 = -1;
+const LENGTH_OF_PREFIX_DATA: i64 = 8;
+
+#[derive(Debug, Clone, Copy, PartialEq)]
+/// Represents compressing a ipc stream using a particular compression algorithm
+pub enum CompressionCodec {
+ Lz4Frame,
+ Zstd,
+}
+
+impl TryFrom<CompressionType> for CompressionCodec {
+ type Error = ArrowError;
+
+ fn try_from(compression_type: CompressionType) -> Result<Self> {
+ match compression_type {
+ CompressionType::ZSTD => Ok(CompressionCodec::Zstd),
+ CompressionType::LZ4_FRAME => Ok(CompressionCodec::Lz4Frame),
+ other_type => Err(ArrowError::NotYetImplemented(format!(
+ "compression type {:?} not supported ",
+ other_type
+ ))),
+ }
+ }
+}
+
+impl CompressionCodec {
+ /// Compresses the data in `input` to `output` and appends the
+ /// data using the specified compression mechanism.
+ ///
+ /// returns the number of bytes written to the stream
+ ///
+ /// Writes this format to output:
+ /// ```text
+ /// [8 bytes]: uncompressed length
+ /// [reminaing bytes]: compressed data stream
+ /// ```
+ pub(crate) fn compress_to_vec(
+ &self,
+ input: &[u8],
+ output: &mut Vec<u8>,
+ ) -> Result<usize> {
+ let uncompressed_data_len = input.len();
+ let original_output_len = output.len();
+
+ if uncompressed_data_len == 0 {
+ // empty input, nothing to do
+ } else {
+ // write compressed data directly into the output buffer
+ output.extend_from_slice(&uncompressed_data_len.to_le_bytes());
+ self.compress(input, output)?;
+
+ let compression_len = output.len();
+ if compression_len > uncompressed_data_len {
+ // length of compressed data was larger than
+ // uncompressed data, use the uncompressed data with
+ // length -1 to indicate that we don't compress the
+ // data
+ output.truncate(original_output_len);
+ output.extend_from_slice(&LENGTH_NO_COMPRESSED_DATA.to_le_bytes());
+ output.extend_from_slice(input);
+ }
+ }
+ Ok(output.len() - original_output_len)
+ }
+
+ /// Decompresses the input into a [`Buffer`]
+ ///
+ /// The input should look like:
+ /// ```text
+ /// [8 bytes]: uncompressed length
+ /// [reminaing bytes]: compressed data stream
+ /// ```
+ pub(crate) fn decompress_to_buffer(&self, input: &[u8]) -> Result<Buffer> {
+ // read the first 8 bytes to determine if the data is
+ // compressed
+ let decompressed_length = read_uncompressed_size(input);
+ let buffer = if decompressed_length == 0 {
+ // emtpy
+ let empty = Vec::<u8>::new();
+ Buffer::from(empty)
+ } else if decompressed_length == LENGTH_NO_COMPRESSED_DATA {
+ // no compression
+ let data = &input[(LENGTH_OF_PREFIX_DATA as usize)..];
+ Buffer::from(data)
+ } else {
+ // decompress data using the codec
+ let mut _uncompressed_buffer =
+ Vec::with_capacity(decompressed_length as usize);
+ let input_data = &input[(LENGTH_OF_PREFIX_DATA as usize)..];
+ self.decompress(input_data, &mut _uncompressed_buffer)?;
+ Buffer::from(_uncompressed_buffer)
+ };
+ Ok(buffer)
+ }
+
+ /// Compress the data in input buffer and write to output buffer
+ /// using the specified compression
+ fn compress(&self, input: &[u8], output: &mut Vec<u8>) -> Result<()> {
+ match self {
+ CompressionCodec::Lz4Frame => {
+ let mut encoder = lz4::EncoderBuilder::new().build(output)?;
+ encoder.write_all(input)?;
+ match encoder.finish().1 {
+ Ok(_) => Ok(()),
+ Err(e) => Err(e.into()),
+ }
+ }
+ CompressionCodec::Zstd => {
+ let mut encoder = zstd::Encoder::new(output, 0)?;
+ encoder.write_all(input)?;
+ match encoder.finish() {
+ Ok(_) => Ok(()),
+ Err(e) => Err(e.into()),
+ }
+ }
+ }
+ }
+
+ /// Decompress the data in input buffer and write to output buffer
+ /// using the specified compression
+ fn decompress(&self, input: &[u8], output: &mut Vec<u8>) -> Result<usize> {
+ let result: Result<usize> = match self {
+ CompressionCodec::Lz4Frame => {
+ let mut decoder = lz4::Decoder::new(input)?;
+ match decoder.read_to_end(output) {
+ Ok(size) => Ok(size),
+ Err(e) => Err(e.into()),
+ }
+ }
+ CompressionCodec::Zstd => {
+ let mut decoder = zstd::Decoder::new(input)?;
+ match decoder.read_to_end(output) {
+ Ok(size) => Ok(size),
+ Err(e) => Err(e.into()),
+ }
+ }
+ };
+ result
+ }
+}
+
+/// Get the uncompressed length
+/// Notes:
+/// -1: indicate that the data that follows is not compressed
Review Comment:
```suggestion
/// LENGTH_NO_COMPRESSED_DATA: indicate that the data that follows is not compressed
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r945107763
##########
arrow/Cargo.toml:
##########
@@ -86,6 +89,8 @@ rand = { version = "0.8", default-features = false, features = ["std", "std_rng"
criterion = { version = "0.3", default-features = false }
flate2 = { version = "1", default-features = false, features = ["rust_backend"] }
tempfile = { version = "3", default-features = false }
+lz4 = { version = "1.23", default-features = false }
+zstd = { version = "0.11", default-features = false }
Review Comment:
The version is different from above?
```suggestion
zstd = { version = "0.11.1", default-features = false }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a diff in pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#discussion_r945121993
##########
arrow/Cargo.toml:
##########
@@ -86,6 +89,8 @@ rand = { version = "0.8", default-features = false, features = ["std", "std_rng"
criterion = { version = "0.3", default-features = false }
flate2 = { version = "1", default-features = false, features = ["rust_backend"] }
tempfile = { version = "3", default-features = false }
+lz4 = { version = "1.23", default-features = false }
+zstd = { version = "0.11", default-features = false }
Review Comment:
🤔 now that I see that it seems like there is no reason to have these dependencies left in both dependencies and dev-dependencies. I will remove them from dev-dependences.
Thanks for the catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] ursabot commented on pull request #2369: support compression for IPC with revamped feature flags
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #2369:
URL: https://github.com/apache/arrow-rs/pull/2369#issuecomment-1214300053
Benchmark runs are scheduled for baseline = 1c879ae4e7f78d2c4b7b9ee35352b2803404cc79 and contender = 5e27d93904660e4f91e94636e6840823c6fb7d4f. 5e27d93904660e4f91e94636e6840823c6fb7d4f is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/45f430debf3c489db1f4c1291f1b95e3...46454fed55064699921614fdc7e8a36a/)
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/e24e8454a33d4ee6bac08b909d7650b2...73e01fa5f37a4e9fba289f6cda24cdab/)
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/8df7e29810f74f2998b15fcdf3faa179...7585df76d05a4dad99b60a1e4b9a761a/)
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/ca774879a1fe44d3ba282e32864cfa09...5cbd826da8c34cb4af5516bdafcbdd6a/)
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org