You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "peterxcli (via GitHub)" <gi...@apache.org> on 2026/09/11 19:43:38 UTC
[PR] perf: reduce cached long decoding cost [datafusion-comet]
peterxcli opened a new pull request, #5869:
URL: https://github.com/apache/datafusion-comet/pull/5869
## Which issue does this PR close?
Related to #5485. **Depends on #5859.**
Review the [decoding-only diff](https://github.com/peterxcli/datafusion-comet/compare/091eb0020...2e65a02cb) (two commits, five files). This PR targets `main`, so GitHub also shows the parent commits until #5859 merges; it should be rebased afterward.
## Rationale for this change
After #5859 removes intermediate row conversion, compressed numeric cache decoding remains expensive. Bypassing compression reduced six-long executor CPU from 494 to 172 ms, identifying the compression stream as the main remaining cost. Spark already uses delta encoding for these numeric values.
## What changes are included in this PR?
Store suitable plain long columns as deltas before compression and restore their values in the shared cache reader. Keep validity bits and logical statistics unchanged. Require the delta stream to be over 25% smaller, skip a second compression for irregular full-width longs, and reuse codec factories per partition.
## How are these changes tested?
Cache, row-reader, and Kryo suites pass on Spark 3.4.3, 3.5.9, and 4.1.3, including nulls, wrapping long arithmetic, native/Spark readers, projections, and recaching. Splitting the PR preserves the tested source exactly at `2e65a02cb`.
### Benchmark
Spark 4.1.3, Java 17.0.20, AWS AMD EPYC 7282, `local[1]`, 6 GiB heap, 5M rows, 16 partitions, `MEMORY_ONLY`, LZ4. Mixed has three longs and three strings; numeric has six sequential/modulo-derived longs. All consuming operators run on Spark.
Medians of 40 reads per cell across two fresh JVMs, after five warmups per query. Cache creation and planning are excluded. Parent, Spark, and child results come from separate rounds on the same host; these are descriptive comparisons across those runs. Parent code matches `091eb0020`; child is `2e65a02cb`. Answers match across all nine cases.
| Schema | Columns read | Spark (ms) | #5859 (ms) | With this PR (ms) |
| --- | --- | ---: | ---: | ---: |
| Mixed | count(*) | 165.80 | 137.15 | 130.42 |
| Mixed | 1 long | 189.16 | 188.63 | 143.66 |
| Mixed | 1 string | 629.31 | 357.66 | 344.14 |
| Mixed | 3 columns | 1125.97 | 657.88 | 616.77 |
| Mixed | 6 columns | 2086.11 | 1100.20 | 1155.90 |
| Numeric | count(*) | 123.58 | 86.79 | 91.54 |
| Numeric | 1 long | 175.36 | 166.47 | 127.89 |
| Numeric | 3 columns | 300.04 | 299.86 | 211.16 |
| Numeric | 6 columns | 460.73 | 551.01 | 342.98 |

Whiskers show JVM median ranges. Mixed six-column child medians varied from 986–1,336 ms, and its pooled median was higher than the parent. Count-only and string-only differences do not establish benefits from long-delta decoding. These synthetic values favor delta encoding; native-consumer performance and broader data distributions remain unmeasured.
Repeated cache builds increased from **1.62 to 2.26 s (+39%)** for numeric data and **1.49 to 1.61 s (+9%)** for random longs versus #5859. Numeric payload shrank from **107.7 to 2.1 MB**; random payload remained **241.3 MB**. Build figures use six measurements after two warmups per JVM, with reversed case order. Writing needs a temporary Arrow vector and can hold both compressed representations, so low-reuse caches may lose overall.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: add opt-in delta encoding for cached longs [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5869:
URL: https://github.com/apache/datafusion-comet/pull/5869#discussion_r4238240496
##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CachedBatchIpc.scala:
##########
@@ -524,6 +569,45 @@ private[comet] object CachedBatchIpc {
buffers.add(packed)
}
+ deltaColumns.foreach { case (index, column) =>
+ val buffer = batch.getBuffers.get(index)
+ val packed = buffers.get(index)
+ if (batch.getLength > 0) {
+ val deltas = allocator.buffer(buffer.writerIndex())
+ try {
+ var previous = 0L
+ var smallDeltas = 0
+ var row = 0
+ while (row < batch.getLength) {
+ val value = buffer.getLong(row * 8L)
+ val delta = value - previous
+ deltas.setLong(row * 8L, delta)
+ if (delta == delta.toInt.toLong) smallDeltas += 1
+ previous = value
+ row += 1
+ }
+ deltas.writerIndex(buffer.writerIndex())
+ // Skip a second compression for full-width irregular longs. The size comparison
+ // still rejects poorly compressing deltas from narrower distributions.
+ if (smallDeltas.toLong * 2 >= batch.getLength) {
Review Comment:
This check sends every column whose values fit in an `int` through a second zstd pass, whether or not the deltas end up smaller. I cached a 5M-row relation of three such `bigint` columns: uniform below 1e6, uniform below 2^31, and a 1 ms step with up to 1 ms of jitter. With the option on, none of the 512 batches kept deltas for any column and the footprint stayed at 46.9 MiB. The build went from 210 ms to 299 ms, median of 7, and every run with the option on was slower than every run without it. Per 10,000-row column, uniform values below 1000 took 328 us to serialize instead of 169 us. Their deltas compress to 1.38 times the size of the values, so they can never pass the bar.
Could the loop also compare how wide the deltas are with how wide the values are, and only try the second pass when the deltas are clearly narrower? For example, summing `64 - numberOfLeadingZeros` of each value's zigzag form, and trying only when the deltas come out at least four bits narrower on average. That skips every uniform case I tried and still tries all the sequential, constant-step and cyclic ones.
##########
docs/source/user-guide/latest/in-memory-cache.md:
##########
@@ -68,6 +68,14 @@ A relation whose cached plan records observed metrics, from `Dataset.observe`, i
Comet's format but is scanned by Spark's `InMemoryTableScanExec`, because Spark collects those
metrics only through that scan.
+Set `spark.comet.exec.inMemoryCache.deltaEncoding.enabled=true` to try delta encoding
+for top-level `bigint` columns in compressed caches. A column uses deltas only when
+its compressed data buffer is over 25% smaller than the plain representation.
Review Comment:
The code compares the compressed deltas with the compressed values (`encoded.writerIndex() < packed.writerIndex() * 3 / 4`), not with the plain representation, so this sentence describes a different threshold. The `CometConf` doc string has the same gap, since it only says the delta buffer is over 25% smaller. Could both say the comparison is between the two compressed forms?
It would also help to give numbers here, the way the codec table under Storage format does, perhaps from a delta arm in `runCodecBenchmark`. That benchmark's relation is a good case for this option. I measured 54.8 MiB without deltas and 44.2 MiB with them. `id`, `k` and `v` went from 6.3, 0.9 and 6.5 MiB to 1.3, 0.4 and 1.5 MiB, and reads were no slower. The build cost on columns that are not sequential is worth stating too.
##########
spark/src/test/scala/org/apache/spark/sql/comet/execution/arrow/CometCachedBatchHelper.scala:
##########
@@ -122,8 +125,9 @@ object CometCachedBatchHelper {
batch: ColumnarBatch,
codec: CompressionCodec,
allocator: BufferAllocator,
- chunkSize: Int = 1024 * 1024): ChunkedByteBuffer =
- CachedBatchIpc.serialize(batch, codec, allocator, chunkSize)._1
+ chunkSize: Int = 1024 * 1024,
+ deltaEncoding: Boolean = false): ChunkedByteBuffer =
+ CachedBatchIpc.serialize(batch, codec, allocator, chunkSize, deltaEncoding)._1
Review Comment:
With `deltaEncoding = true` this returns the payload without its flags, and `cachedBatch` below then builds a batch that has none. A test that round trips through these and `load` would silently read the deltas back as values. The doc on `cachedBatch` says it is the batch the writer would have stored, which no longer holds. Could `serialize` return the flags as well, or the `CachedBatch` itself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: add opt-in delta encoding for cached longs [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5869:
URL: https://github.com/apache/datafusion-comet/pull/5869#issuecomment-6397248315
Thanks @peterxcli. The project now limits contributors who are not committers to three non-draft PRs at a time, and you had twelve open, so I am moving this one to draft. It is not a judgment on the change. I picked the three closest to merging, which are #5891, #5292 and #5048, and drafted the rest.
When one of those merges, please mark the next PR ready for review. The suggested order is: #6443, #6589, #6594, #5027, #5854, #5869, #5194, #5161, #6595. Drafts are still welcome to get review comments, rebases and CI runs, and you can keep pushing to this branch.
My Oct 10 review comments are still open (the pre-check costs a zstd pass on narrow bigints, and the docs baseline is wrong), and it conflicts with main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: add opt-in delta encoding for cached longs [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5869:
URL: https://github.com/apache/datafusion-comet/pull/5869#issuecomment-6100134445
@peterxcli, a heads-up on related work. While digging into #5485 I found that most of what remains of the Spark-reader gap is `zstd` decompression, so I opened #6859, which adds `lz4` (Arrow's `LZ4_FRAME`, written with lz4-java) as an opt-in cache codec. A full read through the native scan takes less than half as long as with `zstd`, but the cache takes about 2.8 times the memory, and on sequential `bigint` columns more than Spark's own format, which delta-encodes them: the wide relations in `CometInMemoryCacheBenchmark` take about 74 MiB with `lz4`, against 26 to 27 MiB in Spark's format and with `zstd`.
That is the data this PR targets. The delta pass compresses with whatever codec the batch uses, so the two should compose without further changes once both land, and `lz4` with delta-encoded longs looks like the best candidate for a faster default. I'd like to measure that combination after both merge. Both PRs touch `CachedBatchIpc`, the codec docs in `CometConf` and `in-memory-cache.md`, `CometInMemoryCacheSuite` and `CometCachedBatchHelper`, so whichever merges second will need a small rebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org