You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "albertlockett (via GitHub)" <gi...@apache.org> on 2025/09/18 19:31:28 UTC
[I] Reuse zstd context in arrow IPC writer [arrow-rs]
albertlockett opened a new issue, #8386:
URL: https://github.com/apache/arrow-rs/issues/8386
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
We're developing a system that emits an arrow IPC stream of many record batches, and we noticed that when we enable zstd compression we get quite a high increase in CPU utilization.
Profiling revealed that we spend quite a lot of that time initialising the IPC context:
```
__bzero [libsystem_platform.dylib]
ZSTD_cwksp_clean_tables [cargo:index.crates.io-1949cf8c6b5b557f:zstd-sys-2.0.15+zstd.1.5.7:zstd/lib/compress/zstd_cwksp.h]
ZSTD_reset_matchState [cargo:index.crates.io-1949cf8c6b5b557f:zstd-sys-2.0.15+zstd.1.5.7:zstd/lib/compress/zstd_compress.c]
ZSTD_resetCCtx_internal [cargo:index.crates.io-1949cf8c6b5b557f:zstd-sys-2.0.15+zstd.1.5.7:zstd/lib/compress/zstd_compress.c]
ZSTD_compressBegin_internal [cargo:index.crates.io-1949cf8c6b5b557f:zstd-sys-2.0.15+zstd.1.5.7:zstd/lib/compress/zstd_compress.c]
ZSTD_CCtx_init_compressStream2 [cargo:index.crates.io-1949cf8c6b5b557f:zstd-sys-2.0.15+zstd.1.5.7:zstd/lib/compress/zstd_compress.c]
ZSTD_compressStream2 [cargo:index.crates.io-1949cf8c6b5b557f:zstd-sys-2.0.15+zstd.1.5.7:zstd/lib/compress/zstd_compress.c]
ZSTD_compressStream [cargo:index.crates.io-1949cf8c6b5b557f:zstd-sys-2.0.15+zstd.1.5.7:zstd/lib/compress/zstd_compress.c]
zstd_safe::CCtx::compress_stream [zstd-safe-7.2.4/src/lib.rs]
<zstd::stream::raw::Encoder as zstd::stream::raw::Operation>::run [zstd-0.13.3/src/stream/raw.rs]
<zstd::stream::zio::reader::Reader<R,D> as std::io::Read>::read [zstd-0.13.3/src/stream/zio/reader.rs]
<zstd::stream::read::Encoder<R> as std::io::Read>::read [zstd-0.13.3/src/stream/read/mod.rs]
std::io::Read::read_buf::{{closure}} [/Users/a.lockett/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/io/mod.rs]
std::io::default_read_buf [/Users/a.lockett/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/io/mod.rs]
std::io::Read::read_buf [/Users/a.lockett/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/io/mod.rs]
std::io::default_read_to_end [/Users/a.lockett/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/io/mod.rs]
std::io::Read::read_to_end [/Users/a.lockett/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/io/mod.rs]
<alloc::vec::Vec<u8> as std::io::copy::BufferedWriterSpec>::copy_from [/Users/a.lockett/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/io/copy.rs]
std::io::copy::generic_copy [/Users/a.lockett/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/io/copy.rs]
std::io::copy::copy [/Users/a.lockett/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/io/copy.rs]
arrow_ipc::compression::compress_zstd [/Users/a.lockett/Development/arrow-rs/arrow-ipc/src/compression.rs]
arrow_ipc::compression::CompressionCodec::compress [/Users/a.lockett/Development/arrow-rs/arrow-ipc/src/compression.rs]
arrow_ipc::compression::CompressionCodec::compress_to_vec [/Users/a.lockett/Development/arrow-rs/arrow-ipc/src/compression.rs]
arrow_ipc::writer::write_buffer [/Users/a.lockett/Development/arrow-rs/arrow-ipc/src/writer.rs]
arrow_ipc::writer::write_array_data [/Users/a.lockett/Development/arrow-rs/arrow-ipc/src/writer.rs]
arrow_ipc::writer::IpcDataGenerator::record_batch_to_bytes [/Users/a.lockett/Development/arrow-rs/arrow-ipc/src/writer.rs]
arrow_ipc::writer::IpcDataGenerator::encoded_batch [/Users/a.lockett/Development/arrow-rs/arrow-ipc/src/writer.rs]
arrow_ipc::writer::StreamWriter<W>::write [/Users/a.lockett/Development/arrow-rs/arrow-ipc/src/writer.rs]
otel_arrow_rust::encode::producer::StreamProducer::serialize_batch [/Users/a.lockett/Development/otel-arrow/rust/otel-arrow-rust/src/encode/producer.rs]
otel_arrow_rust::encode::producer::Producer::produce_bar [/Users/a.lockett/Development/otel-arrow/rust/otel-arrow-rust/src/encode/producer.rs]
otap_df_otap::otap_exporter::create_req_stream::{{closure}} [/Users/a.lockett/Development/otel-arrow/rust/otap-dataflow/crates/otap/src/otap_exporter.rs]
```
I think this is happening because we initialise a new zstd encoder for each call to `compress_zstd`:
https://github.com/apache/arrow-rs/blob/f4840f6df1c2549ce0947305b7111edad638b445/arrow-ipc/src/compression.rs#L177-L184
The zstd docs claim that a performance improvement can be had by reusing the context:
https://facebook.github.io/zstd/zstd_manual.html
> When compressing many times,
> it is recommended to allocate a context just once,
> and re-use it for each successive compression operation.
> This will make workload friendlier for system's memory.
> Note : re-using context is just a speed / resource optimization.
> It doesn't change the compression ratio, which remains identical.
> Note 2 : In multi-threaded environments,
> use one different context per thread for parallel execution.
I gave this a quick test using using a bulk::Compressor which is initialized only once, and indeed I noticed that my workload was using ~50% less of a CPU core (PoC code is here, but it's a bit of a mess: https://github.com/apache/arrow-rs/commit/4d37b4ba15fa1dd5696d0697f4eeb380b055f669)
**Describe the solution you'd like**
<!--
A clear and concise description of what you want to happen.
-->
I'd like it if we could reus reuse the Zstd Context between batches which would maybe improve CPU performance
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features you've considered.
-->
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
related to https://github.com/open-telemetry/otel-arrow/issues/1129
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Reuse zstd context in arrow IPC writer [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8386:
URL: https://github.com/apache/arrow-rs/issues/8386#issuecomment-3416676130
`label_issue.py` automatically added labels {'next-major-release'} from #8405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Reuse zstd context in arrow IPC writer [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8386:
URL: https://github.com/apache/arrow-rs/issues/8386#issuecomment-3416675932
`label_issue.py` automatically added labels {'arrow-flight'} from #8405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Reuse zstd context in arrow IPC writer [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8386:
URL: https://github.com/apache/arrow-rs/issues/8386#issuecomment-3416675786
`label_issue.py` automatically added labels {'arrow'} from #8405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Reuse zstd context in arrow IPC writer [arrow-rs]
Posted by "albertlockett (via GitHub)" <gi...@apache.org>.
albertlockett commented on issue #8386:
URL: https://github.com/apache/arrow-rs/issues/8386#issuecomment-3309369170
If there aren't any serious qualms about this, I'd be happy to work on it. Figured I'd open the issue for discussion first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Reuse zstd context in arrow IPC writer [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #8386: Reuse zstd context in arrow IPC writer
URL: https://github.com/apache/arrow-rs/issues/8386
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org