You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/27 14:48:53 UTC

[PR] feat: add native support for in-memory cache [datafusion-comet]

andygrove opened a new pull request, #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051

   ## Which issue does this PR close?
   
   Closes #2391.
   
   ## Rationale for this change
   
   This PR continues the work started by @pchintar in #4591, who is no longer able to work on it. All credit for the original implementation goes to them. This branch is based on their branch with latest `apache/main` merged in, and picks up the remaining review feedback and CI failures.
   
   Comet currently has limited support for Spark's in-memory cache.
   
   When a table is cached and later read, the cached data cannot be consumed directly by Comet operators. Instead, the execution plan falls back to Spark's cache scan path and introduces an additional `CometSparkColumnarToColumnar` conversion before execution can continue in Comet.
   
   This extra conversion adds overhead to cached table scans and prevents cached data from remaining on a native Comet execution path.
   
   ## What changes are included in this PR?
   
   A native cache path for in-memory cached tables behind a new configuration:
   
   `spark.comet.exec.inMemoryCache.enabled`
   
   When enabled:
   
   * Cached data is stored using a Comet-specific cache serializer (compressed Arrow IPC).
   * Cached data is represented as `CometCachedBatch`.
   * Cached tables are scanned using `CometInMemoryTableScanExec`.
   * Cached data can be consumed directly by Comet operators without introducing a `CometSparkColumnarToColumnar` conversion.
   * Per-batch column statistics (lower bound, upper bound, null count, row count) are written in the format expected by `SimpleMetricsCachedBatchSerializer`, so Spark's `buildFilter` can prune cached batches before they are decoded.
   
   When disabled:
   
   * Spark's existing cache serializer continues to be used.
   * Existing cache scan behavior is preserved.
   
   ## How are these changes tested?
   
   `CometInMemoryCacheSuite` covers:
   
   - Comet-native cache scan over `CometCachedBatch`
   - Fallback behavior when native cache support is disabled
   - Multi-partition cached tables
   - Empty cached tables
   - Projection-only cache reads
   - Shuffle execution after cached table scans
   - Stats-based batch pruning
   - Empty projection scans (`SELECT count(*)`)
   - Floating-point pruning with NaN values
   - Fallback read path for Spark `DefaultCachedBatch`
   
   `CometInMemoryCacheBenchmark` compares the native cache path against the existing fallback path for repeated cached table scans and selective-filter scans exercising cache pruning.
   
   Benchmark results (release build, Apple M3 Ultra, JDK 17, Spark 3.5 profile, 5M-row cached table):
   
   ### Repeated full scan (`SELECT sum(id), sum(k), sum(v)`)
   
   | Case | Best (ms) | Avg (ms) | Relative |
   |------|----------:|---------:|---------:|
   | Comet cache disabled | 180 | 201 | 1.0x |
   | Comet cache enabled | 121 | 128 | **1.5x** |
   
   ### Selective filter (`WHERE id >= 4500000 AND id < 4750000`)
   
   | Case | Best (ms) | Avg (ms) | Relative |
   |------|----------:|---------:|---------:|
   | Comet cache disabled | 46 | 53 | 1.0x |
   | Comet cache enabled | 42 | 48 | **1.1x** |
   
   Performance follow-ups are tracked in #4781.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5432350051

   > It looks similar to https://github.com/apache/spark/pull/56334 too although seems in Comet we store compressed IPC stream and can decode only selected columns.
   
   Thanks @sunchao, that is a useful pointer and I had not seen it. I went through it and filed #5487 with the comparison so it does not get lost. A few things worth saying here.
   
   **On the version.** It is in `branch-4.3` and `master` only, not 4.0, 4.1 or 4.2, so it is not available in any Spark version Comet supports today. It does not remove the need for Comet's serializer, but it is a more mature implementation of the same idea and several of its decisions are better than mine.
   
   **On the difference you spotted**, which is real but slightly the other way round from how it sounds. Both formats decode only the selected columns. Comet gets there by splitting the payload into one compressed stream per column, which is the change I pushed a few commits ago. Spark keeps a single RecordBatch and instead parses the IPC message flatbuffer, which lists every buffer's offset and length within the body, copying out only the byte ranges belonging to the selected columns and letting `VectorLoader.load` decompress just those. That leans on Arrow's native per-buffer compression rather than wrapping the whole stream in a Spark `CompressionCodec` as Comet does.
   
   Spark's is the better design point. Per-column streams pay framing per column per batch and give up cross-column compression: measured against the previous single-stream format, footprint grows 2.5% at 6 columns and 32% at 60. Spark's approach gets the same projection-proportional decode with none of that. Part of the gap is simply that their `ArrowCachedBatch` carries no Schema message at all, reconstructing it from the relation's attributes on read, where Comet writes a full stream schema per column per batch, roughly 30,000 of them in that 60-column test. I have not changed course here, since the buffer-span reader is about 120 lines of fairly intricate code and belongs in its own PR rather than bolted onto this one, but it is written up as the first thing to reconsider.
   
   **Two smaller things from their implementation**, both in #5487: they register `ArrowCachedBatch` for Kryo, and Comet has no Kryo registrator at all, so `CometCachedBatch` would fail under `spark.kryo.registrationRequired=true`. And their row read path builds typed column readers once and writes straight into an `UnsafeRowWriter` instead of going through a generic row iterator, which is a concrete candidate for #5485, where I have measurements but no established cause.
   
   **One point of convergence worth noting**, since it suggests neither of us is off in the weeds: they landed the same empty-projection optimization I did, emitting row counts without touching the payload so `count(*)` stops decoding everything, and they landed it as a follow-up in SPARK-58390 about a week after the main PR. Same gap, found in the same order.
   
   @viirya if you have a moment, the question I would most value your view on is whether the buffer-span projection reader was worth its complexity in practice, or whether you would take the simpler split-by-column route knowing what you know now.
   
   I also checked our collation handling against yours and there is no bug on our side, but you do prune collated string columns where we decline to: a collated `StringType` does not match Comet's `case StringType` in the bounds check, so those columns get null bounds and no predicate pushdown. Filed as part of #5487 rather than fixed here.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "0lai0 (via GitHub)" <gi...@apache.org>.
0lai0 commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3690964474


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowConverters.scala:
##########
@@ -74,4 +76,52 @@ object CometArrowConverters extends Logging {
       }
     }
   }
+
+  /**
+   * Copy `numRows` rows starting at `startRow` from a Spark `ColumnarBatch` into `root`.
+   *
+   * Spark's `ColumnVector` implementations do not expose Arrow buffers, so values are necessarily
+   * copied element-wise. Shared by [[SparkColumnarArrowReader]], which slices into a stable root,
+   * and [[columnarBatchToArrowBatch]], which fills a fresh one.
+   */
+  private[arrow] def writeColumns(
+      root: VectorSchemaRoot,
+      batch: ColumnarBatch,
+      startRow: Int,
+      numRows: Int): Unit = {
+    val writer = ArrowWriter.create(root)
+    var col = 0
+    while (col < batch.numCols()) {
+      val column = batch.column(col)
+      val columnArray = new ColumnarArray(column, startRow, numRows)
+      if (column.hasNull) {
+        writer.writeCol(columnArray, col)
+      } else {
+        writer.writeColNoNull(columnArray, col)
+      }
+      col += 1
+    }
+    writer.finish()
+    // ArrowWriter derives the root row count from its per-column writes, so a zero-column input
+    // batch (Spark's count-from-metadata scan: numRows > 0, numCols == 0) would otherwise produce
+    // a root with rowCount == 0 and silently drop the rows.
+    root.setRowCount(numRows)
+  }
+
+  /**
+   * Copy a Spark `ColumnarBatch` whose columns are not Arrow-backed (e.g.
+   * `On/OffHeapColumnVector` from Spark's vectorized Parquet reader, or a third-party connector's
+   * vectors) into a freshly allocated Arrow `ColumnarBatch` of `CometVector`s.
+   *
+   * The input batch is not consumed or closed; the caller owns the returned batch and must close
+   * it.
+   */
+  def columnarBatchToArrowBatch(
+      batch: ColumnarBatch,
+      arrowSchema: Schema,
+      allocator: BufferAllocator): ColumnarBatch = {
+    val root = VectorSchemaRoot.create(arrowSchema, allocator)

Review Comment:
   Minor Nit: `columnarBatchToArrowBatch` allocates root before `NativeUtil.rootAsBatch` wraps it. 
   If `writeColumns` or the wrap throws, root leaks, since the caller only closes the returned batch. 
   Consider guarding the body with `try { ... } catch { case NonFatal(e) => root.close(); throw e }. 
   But normal path will not lead to



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5209716289

   > Tested on Spark4.0 against current branch. Using PySpark Got this error on the same pipeline that was testing earlier. And I can clearly see a regression in performance. Same job with `.cache()` on default Spark took <10 min, Comet ran >15min.
   > 
   > Here is error:
   > 
   > ```
   > Py4JJavaError: An error occurred while calling o12641.createOrReplace.
   > : org.apache.spark.SparkException: Job aborted due to stage failure: Task 92 in stage 27.0 failed 4 times, most recent failure: Lost task 92.3 in stage 27.0 (TID 15624) (1.1.1.1 executor 76): org.apache.comet.CometNativeException: Invalid argument error: column types must match schema types, expected List(Struct("col_1": Utf8, "col_2": Boolean, "col_3": Boolean)) but found List(Struct("col_1": Utf8, "col_2": non-null Boolean, "col_3": Boolean)) at column index 5
   > 	at org.apache.comet.Native.executePlan(Native Method)
   > 	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2(CometExecIterator.scala:155)
   > 	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2$adapted(CometExecIterator.scala:154)
   > 	at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:189)
   > 	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$1(CometExecIterator.scala:154)
   > 	at org.apache.comet.Tracing$.withTrace(Tracing.scala:31)
   > 	at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:152)
   > 	at org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:196)
   > ```
   > 
   > (obfuscated column names)
   
   I think the crash may get fixed once https://github.com/apache/datafusion-comet/pull/5138 is merged. Looking into perf regression...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5180593513

   @sandugood could you test this PR again when you have time? Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3677934494


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowConverters.scala:
##########
@@ -74,4 +74,45 @@ object CometArrowConverters extends Logging {
       }
     }
   }
+
+  /**
+   * Copy a Spark `ColumnarBatch` whose columns are not Arrow-backed (e.g.
+   * `On/OffHeapColumnVector` from Spark's vectorized Parquet reader, or a third-party connector's
+   * vectors) into a freshly allocated Arrow `ColumnarBatch` of `CometVector`s.
+   *
+   * The input batch is not consumed or closed; the caller owns the returned batch and must close
+   * it. Values are copied element-wise, since Spark's `ColumnVector` implementations do not
+   * expose Arrow buffers.
+   */
+  def columnarBatchToArrowBatch(

Review Comment:
   You are right on both counts, and your analysis of the `ColumnVector` contract matches what I found — `ColumnarArray` + the standard accessors means it works for any correctly-implemented `ColumnVector`, which is why the fix also covers Iceberg and other external columnar sources without naming them.
   
   The duplication is fixed in c7bc49e5: the copy loop is now `CometArrowConverters.writeColumns(root, batch, startRow, numRows)`, and `SparkColumnarArrowReader.loadNextBatch` calls it too. That also collapses the two copies of the zero-column `setRowCount` workaround back into one.
   
   On your `readerBatchIter` sketch — I went a slightly different way and want to flag why, since it is a real tradeoff:
   
   - It converts **unconditionally**, so a Comet-native cached plan (the primary case for this feature) would get a full element-wise copy through the generic accessors instead of the current zero-copy Arrow IPC write. That is the regression I most wanted to avoid, so `encodeBatches` keeps a per-batch `Utils.isArrowBacked` check and only converts foreign vectors.
   - The reader wraps the whole partition iterator, so the Arrow-vs-not decision becomes per-partition rather than per-batch. In practice a partition is homogeneous, but making it per-batch costs nothing.
   
   Your version does have one genuine advantage mine gives up: `SparkColumnarArrowReader` reuses one stable VSR across batches, whereas `columnarBatchToArrowBatch` allocates a fresh root per batch. If that shows up in cache-materialization profiles it would be worth revisiting — happy to file a follow-up.
   
   One thing your sketch surfaced that I did **not** change: your code uses `conf.sessionLocalTimeZone` for the Arrow schema, which matches the pre-existing row path (`convertInternalRowToCachedBatch`), while my columnar path uses `CometArrowStream.NATIVE_TIMEZONE` (UTC) to match `CometSparkToColumnarExec`. So the same `CometCachedBatch` format is currently written with two timezone conventions depending on the input path. Timestamps are UTC micros either way so this is metadata-only today, but it is inconsistent and I would rather fix it deliberately. Do you have a preference for which one the cache format should standardise on?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3679301588


##########
spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala:
##########
@@ -397,6 +397,15 @@ object Utils extends CometTypeShim with Logging {
     }
   }
 
+  /**
+   * Whether every column in `batch` satisfies [[getBatchFieldVectors]]'s precondition, i.e. is an
+   * Arrow-backed `CometVector`. Callers that may receive batches from a plan they did not build
+   * (e.g. Comet's cache serializer, which Spark hands the cached plan's columnar output) use this
+   * to convert foreign vectors to Arrow instead of tripping the exception below.
+   */
+  def isArrowBacked(batch: ColumnarBatch): Boolean =

Review Comment:
   Is this possible to be false?



##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowConverters.scala:
##########
@@ -74,4 +74,45 @@ object CometArrowConverters extends Logging {
       }
     }
   }
+
+  /**
+   * Copy a Spark `ColumnarBatch` whose columns are not Arrow-backed (e.g.
+   * `On/OffHeapColumnVector` from Spark's vectorized Parquet reader, or a third-party connector's
+   * vectors) into a freshly allocated Arrow `ColumnarBatch` of `CometVector`s.
+   *
+   * The input batch is not consumed or closed; the caller owns the returned batch and must close
+   * it. Values are copied element-wise, since Spark's `ColumnVector` implementations do not
+   * expose Arrow buffers.
+   */
+  def columnarBatchToArrowBatch(

Review Comment:
   > Do you have a preference for which one the cache format should standardise on?
   
   Good catch. I checked Spark’s cache implementations. Released Spark’s default cache does not store timezone metadata: both `TimestampType` and `TimestampNTZType` are stored as raw longs, with LTZ versus NTZ preserved by the relation schema.
   
   Current Spark's Arrow cache constructs `TimestampType` using `conf.sessionLocalTimeZone` and `TimestampNTZType` with `timezone=null`, but its cached RecordBatch is deliberately schema-less. The timezone label is therefore not persisted and is reconstructed on read.
   
   `CometCachedBatch` differs because it stores a complete IPC stream including the schema. I would therefore standardize its physical format on `CometArrowStream.NATIVE_TIMEZONE` (`UTC`) for `TimestampType`, while keeping `TimestampNTZType` timezone-free. This avoids embedding the write session’s mutable timezone, matches Comet’s native schema, and avoids an unnecessary native-boundary cast. The row path can simply pass `NATIVE_TIMEZONE`; no timestamp values are converted.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3677859989


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowConverters.scala:
##########
@@ -74,4 +74,45 @@ object CometArrowConverters extends Logging {
       }
     }
   }
+
+  /**
+   * Copy a Spark `ColumnarBatch` whose columns are not Arrow-backed (e.g.
+   * `On/OffHeapColumnVector` from Spark's vectorized Parquet reader, or a third-party connector's
+   * vectors) into a freshly allocated Arrow `ColumnarBatch` of `CometVector`s.
+   *
+   * The input batch is not consumed or closed; the caller owns the returned batch and must close
+   * it. Values are copied element-wise, since Spark's `ColumnVector` implementations do not
+   * expose Arrow buffers.
+   */
+  def columnarBatchToArrowBatch(

Review Comment:
   A `ColumnarBatch` is a container:
   ```scala
   class ColumnarBatch {
     ColumnVector[] columns;
   }
   ```
   
   Spark 4.1.2’s relevant `ColumnVector` implementations are:
   Producer | Vectors inside ColumnarBatch
   -- | --
   Vectorized Parquet | OnHeapColumnVector, OffHeapColumnVector, ConstantColumnVector
   Vectorized ORC | OrcAtomicColumnVector, OrcArrayColumnVector, OrcMapColumnVector, OrcStructColumnVector; sometimes heap/off-heap or constant vectors
   Spark Arrow/Python paths | ArrowColumnVector
   Comet | CometVector
   External columnar sources | Any implementation extending Spark’s ColumnVector
   
   [SparkColumnarArrowReader](https://github.com/apache/datafusion-comet/blob/63c5a916b2e5626b040dc1f097384ab1a443bf91/spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/SparkColumnarArrowReader.scala#L27-L101) does not inspect those concrete classes. It wraps each column in `ColumnarArray`, and `ArrowWriter` reads through Spark’s standard accessors such as `getLong`, `getUTF8String`, `getArray`, and `getStruct`. Therefore, it supports any implementation that correctly follows the [spark's ColumnVector](https://github.com/apache/spark/blob/v4.1.2/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnVector.java)
   
   `ColumnVector` contract for the declared data type.
   The cache writer could reuse it approximately like this:
   
   ```scala
   override def convertColumnarBatchToCachedBatch(
       input: RDD[ColumnarBatch],
       schema: Seq[Attribute],
       storageLevel: StorageLevel,
       conf: SQLConf): RDD[CachedBatch] = {
   
     val sparkSchema = toStructType(schema)
     val batchSize = conf.columnBatchSize
     val timeZoneId = conf.sessionLocalTimeZone
   
     input.mapPartitions { batches =>;
       // Arrow Schema is not serializable, so construct it inside the task.
       val arrowSchema = Utils.toArrowSchema(sparkSchema, timeZoneId)
   
       val arrowBatches = CometArrowStream.readerBatchIter(
         "CometCacheWrite",
         allocator =&gt;
           new SparkColumnarArrowReader(
             allocator,
             arrowSchema,
             batches,
             batchSize))
   
       encodeBatches(arrowBatches, schema)
     }
   }
   ```
   
   That gives this path:
   ```
   Spark ColumnarBatch
     -> SparkColumnarArrowReader
     -> Arrow-backed ColumnarBatch containing CometVector
     -> existing encodeBatches/getBatchFieldVectors
     -> CometCachedBatch</code>
   ```
   
   _Assist with LLM_



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3694503535


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowConverters.scala:
##########
@@ -64,15 +66,23 @@ object CometArrowConverters extends Logging {
 
       override def next(): ColumnarBatch = {
         val root = VectorSchemaRoot.create(arrowSchema, allocator)
-        val writer = ArrowWriter.create(root)
-        var rowCount = 0L
-        while (rowIter.hasNext &&
-          (maxRecordsPerBatch <= 0 || rowCount < maxRecordsPerBatch)) {
-          writer.write(rowIter.next())
-          rowCount += 1
+        // Same ownership rule as columnarBatchToArrowBatch: the caller only owns the batch that
+        // rootAsBatch returns, so a throw from writing a row has to release the root here.
+        try {
+          val writer = ArrowWriter.create(root)
+          var rowCount = 0L
+          while (rowIter.hasNext &&
+            (maxRecordsPerBatch <= 0 || rowCount < maxRecordsPerBatch)) {
+            writer.write(rowIter.next())
+            rowCount += 1
+          }
+          writer.finish()
+          NativeUtil.rootAsBatch(root)
+        } catch {
+          case NonFatal(e) =>
+            root.close()
+            throw e

Review Comment:
   - https://github.com/apache/spark/blob/78147abefb29096930346c3f9abaf92bdd5de4bb/sql/core/src/main/scala/org/apache/spark/sql/execution/r/ArrowRRunner.scala#L95-L117
   - https://github.com/apache/spark/blob/78147abefb29096930346c3f9abaf92bdd5de4bb/common/utils/src/main/scala/org/apache/spark/util/SparkErrorUtils.scala#L73-L104
   
   looked through spark's arrow writer usage, FYI.



##########
spark/src/test/scala/org/apache/comet/exec/CometInMemoryCacheSuite.scala:
##########
@@ -412,6 +413,114 @@ class CometInMemoryCacheSuite extends CometTestBase {
     }
   }
 
+  test("Comet in-memory cache honors inMemoryColumnarStorage.partitionPruning=false") {
+    // CometInMemoryTableScanExec applies the serializer's stats filter before decoding, the same
+    // way Spark's InMemoryTableScanExec.filteredCachedBatches does. Spark gates that on
+    // spark.sql.inMemoryColumnarStorage.partitionPruning, so Comet must too.
+    //
+    // Pruning is transparent in the results, so it is observed through the scan's numOutputRows:
+    // that counts the rows in the batches actually decoded, so pruning fewer batches means fewer
+    // rows. With pruning off, every cached row must be decoded.
+    def scanRowsFor(pruning: Boolean): (Long, Long) = {
+      var result: (Long, Long) = (0L, 0L)
+      withSQLConf(
+        SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
+        CometConf.COMET_SHUFFLE_MODE.key -> "jvm",
+        SQLConf.CACHE_VECTORIZED_READER_ENABLED.key -> "true",
+        CometConf.COMET_EXEC_IN_MEMORY_CACHE_ENABLED.key -> "true",
+        "spark.comet.sparkToColumnar.enabled" -> "true",
+        "spark.sql.inMemoryColumnarStorage.batchSize" -> "100",
+        SQLConf.IN_MEMORY_PARTITION_PRUNING.key -> pruning.toString) {
+
+        spark.catalog.clearCache()
+        spark
+          .range(0, 1000, 1, 10)
+          .selectExpr("id as key", "id % 7 as value")
+          .createOrReplaceTempView("prune_conf_cache")
+        spark.catalog.cacheTable("prune_conf_cache")
+        val totalRows = spark.table("prune_conf_cache").count()
+
+        val df =
+          spark.sql("SELECT key, value FROM prune_conf_cache WHERE key >= 900 AND key < 905")
+        checkSparkAnswer(df)
+
+        val scans = df.queryExecution.executedPlan.collect {
+          case s: org.apache.spark.sql.comet.CometInMemoryTableScanExec => s
+        }
+        assert(scans.length == 1, s"expected one CometInMemoryTableScan, got ${scans.length}")
+        // scalastyle:off println
+        println(
+          "DIAG rows=" + df.collect().length + " metrics=" + scans.head.metrics
+            .map { case (k, v) => k + "=" + v.value }
+            .mkString(","))
+        println("DIAG plan=" + df.queryExecution.executedPlan.getClass.getName)
+        // scalastyle:on println

Review Comment:
   > reading a metric off `df.queryExecution.executedPlan` after `checkSparkAnswer(df)` gives zero, because checkSparkAnswer executes its own copies. My first version passed vacuously with 0 on both sides of the comparison; it needs the df forced explicitly. The real numbers are 100 rows decoded with pruning on versus 1000 with it off.
   
   
   Remove the `println` and add TODO or new issue link would be easier to followup



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3709229295


##########
spark/src/test/scala/org/apache/comet/exec/CometInMemoryCacheSuite.scala:
##########
@@ -412,6 +413,114 @@ class CometInMemoryCacheSuite extends CometTestBase {
     }
   }
 
+  test("Comet in-memory cache honors inMemoryColumnarStorage.partitionPruning=false") {
+    // CometInMemoryTableScanExec applies the serializer's stats filter before decoding, the same
+    // way Spark's InMemoryTableScanExec.filteredCachedBatches does. Spark gates that on
+    // spark.sql.inMemoryColumnarStorage.partitionPruning, so Comet must too.
+    //
+    // Pruning is transparent in the results, so it is observed through the scan's numOutputRows:
+    // that counts the rows in the batches actually decoded, so pruning fewer batches means fewer
+    // rows. With pruning off, every cached row must be decoded.
+    def scanRowsFor(pruning: Boolean): (Long, Long) = {
+      var result: (Long, Long) = (0L, 0L)
+      withSQLConf(
+        SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
+        CometConf.COMET_SHUFFLE_MODE.key -> "jvm",
+        SQLConf.CACHE_VECTORIZED_READER_ENABLED.key -> "true",
+        CometConf.COMET_EXEC_IN_MEMORY_CACHE_ENABLED.key -> "true",
+        "spark.comet.sparkToColumnar.enabled" -> "true",
+        "spark.sql.inMemoryColumnarStorage.batchSize" -> "100",
+        SQLConf.IN_MEMORY_PARTITION_PRUNING.key -> pruning.toString) {
+
+        spark.catalog.clearCache()
+        spark
+          .range(0, 1000, 1, 10)
+          .selectExpr("id as key", "id % 7 as value")
+          .createOrReplaceTempView("prune_conf_cache")
+        spark.catalog.cacheTable("prune_conf_cache")
+        val totalRows = spark.table("prune_conf_cache").count()
+
+        val df =
+          spark.sql("SELECT key, value FROM prune_conf_cache WHERE key >= 900 AND key < 905")
+        checkSparkAnswer(df)
+
+        val scans = df.queryExecution.executedPlan.collect {
+          case s: org.apache.spark.sql.comet.CometInMemoryTableScanExec => s
+        }
+        assert(scans.length == 1, s"expected one CometInMemoryTableScan, got ${scans.length}")
+        // scalastyle:off println
+        println(
+          "DIAG rows=" + df.collect().length + " metrics=" + scans.head.metrics
+            .map { case (k, v) => k + "=" + v.value }
+            .mkString(","))
+        println("DIAG plan=" + df.queryExecution.executedPlan.getClass.getName)
+        // scalastyle:on println

Review Comment:
   Removed in ebefc1cf4. One thing worth keeping visible: the `df.collect()` buried in that `println` was load-bearing, not diagnostic. `checkSparkAnswer` takes its argument by name and executes its own copies of the query, so without an explicit force this `df`'s plan instance has never run and every metric on it reads zero, which makes the pruned-vs-unpruned comparison pass vacuously as 0 == 0. So the `collect()` is now an explicit call with that reason in a comment above it, rather than a side effect of a debug line.
   
   Same commit fixes an unrelated failure I hit while re-running the suite: "cache a non-Arrow-backed Spark columnar plan with complex types" built its binary column with `cast(id as binary)`, which ANSI mode rejects (`DATATYPE_MISMATCH.CAST_WITH_CONF_SUGGESTION`). It passed when I wrote it on spark-3.5 and would have failed the Spark 4.x jobs. Now cast via string. All 21 tests pass on the default 4.1 profile.



##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowConverters.scala:
##########
@@ -64,15 +66,23 @@ object CometArrowConverters extends Logging {
 
       override def next(): ColumnarBatch = {
         val root = VectorSchemaRoot.create(arrowSchema, allocator)
-        val writer = ArrowWriter.create(root)
-        var rowCount = 0L
-        while (rowIter.hasNext &&
-          (maxRecordsPerBatch <= 0 || rowCount < maxRecordsPerBatch)) {
-          writer.write(rowIter.next())
-          rowCount += 1
+        // Same ownership rule as columnarBatchToArrowBatch: the caller only owns the batch that
+        // rootAsBatch returns, so a throw from writing a row has to release the root here.
+        try {
+          val writer = ArrowWriter.create(root)
+          var rowCount = 0L
+          while (rowIter.hasNext &&
+            (maxRecordsPerBatch <= 0 || rowCount < maxRecordsPerBatch)) {
+            writer.write(rowIter.next())
+            rowCount += 1
+          }
+          writer.finish()
+          NativeUtil.rootAsBatch(root)
+        } catch {
+          case NonFatal(e) =>
+            root.close()
+            throw e

Review Comment:
   Useful reference. `tryWithSafeFinally` itself does not fit here, since ownership of the root transfers to the returned batch on success and it must deliberately stay open, so the release is failure-only rather than a `finally`. But the part of it I was missing does apply: my guard closed the root with a bare `root.close()`, and an Arrow root can throw from `close` (`IllegalStateException` for outstanding child allocations). That exception would have replaced the failure that actually caused the abort, which is the less useful of the two.
   
   4b9a5167a adopts the suppressed-exception semantics: the close error is attached to the original with `addSuppressed` and the original is rethrown. The guard is now one helper shared by `rowToArrowBatchIter` and `columnarBatchToArrowBatch` instead of two copies.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5123240386

   Thanks for the report @sandugood, and sorry for the bad error message — it sent you chasing scan configs when the failure was actually in the new cache serializer. @peterxcli diagnosed it correctly in the review thread above.
   
   What happened: the Comet cache serializer decided it could accept columnar input based on the schema alone. That makes Spark strip the `ColumnarToRow` above the cached plan and hand the serializer whatever that plan produces. When the cached plan falls back to Spark (or is an external columnar source like Iceberg), those are `On/OffHeapColumnVector`, not Arrow vectors, so cache materialization threw. Your "pipeline has lots of fallbacks to default Spark operators" was the trigger — and neither `spark.comet.scan.allowIncompatible` nor `spark.comet.convert.parquet.enabled` could help, because the failure is at cache write time, not in the scan.
   
   **Workaround on current main:** `spark.comet.exec.inMemoryCache.enabled=false` (the default), which keeps Spark's own cache serializer.
   
   **Fixed** in 9dce0d66 + c7bc49e5 on this branch: non-Arrow columnar batches are now copied into Arrow at cache write time, while Comet-native cached plans keep the existing zero-copy path. Two regression tests cover it — primitives/string/decimal/date/timestamp, and nullable array/struct/map/binary.
   
   Since Iceberg MOR is the one input I could not reproduce locally, it would be really helpful if you could retest your pipeline against the branch. Iceberg's vectors go through the same generic `ColumnVector` accessors as Spark's, so it should be covered, but confirmation from a real workload would be worth a lot here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3682024550


##########
spark/src/main/scala/org/apache/spark/sql/comet/CometInMemoryTableScanExec.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.columnar.{CachedBatch, CachedBatchSerializer}
+import org.apache.spark.sql.execution.LeafExecNode
+import org.apache.spark.sql.execution.columnar.InMemoryTableScanExec
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.CometConf
+import org.apache.comet.serde.CometOperatorSerde
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.serializeDataType
+
+/**
+ * Reads Spark cached table data when the cache was written by Comet's cache serializer.
+ *
+ * Spark stores cached data through `CachedBatchSerializer`. This node keeps the scan inside Comet
+ * by asking the serializer to decode cached batches directly into `ColumnarBatch` output,
+ * avoiding the extra Spark columnar-to-Comet columnar conversion used by the default path.
+ *
+ * `relationOutput` is the full schema stored in the cache. `scanOutput` is the subset requested
+ * by this scan after pruning.
+ */
+case class CometInMemoryTableScanExec(
+    originalPlan: InMemoryTableScanExec,
+    serializer: CachedBatchSerializer,
+    cachedBuffers: RDD[CachedBatch],
+    relationOutput: Seq[Attribute],
+    scanOutput: Seq[Attribute])
+    extends CometExec
+    with LeafExecNode {
+
+  override lazy val metrics: Map[String, SQLMetric] = Map(
+    "numOutputRows" -> SQLMetrics.createMetric(sparkContext, "number of output rows"))
+
+  // For an empty-projection scan (`SELECT count(*)`) this is empty while `scanOutput` holds the
+  // full cache schema, so the emitted batches are wider than the declared output. That is safe
+  // because the only consumer of an empty-output scan is a count-style aggregate, which reads
+  // the row count rather than any column; `convert` and `createExec` deliberately fall back to
+  // the cache schema in that case because the native plan still needs a non-empty scan schema.
+  override def output: Seq[Attribute] = originalPlan.output
+
+  // Use the serializer's vector types because the cached batch layout is owned by the serializer.
+  override def vectorTypes: Option[Seq[String]] =
+    serializer.vectorTypes(scanOutput, conf)
+
+  // Apply Spark's cache batch filter before decoding. Spark's InMemoryTableScanExec does this in
+  // filteredCachedBatches(), but that method is private. Reusing the serializer's buildFilter here
+  // keeps Comet on the same stats-based pruning path instead of decoding every cached batch.
+  override def doExecuteColumnar(): RDD[ColumnarBatch] = {

Review Comment:
   pruning configuration ignored: still prunes when `spark.sql.inMemoryColumnarStorage.partitionPruning=false`. 
   
   not sure enable `partitionPruning` is always better than disable? if true, then nvm.



##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet.execution.arrow
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, GenericInternalRow, IsNotNull, IsNull, UnsafeProjection}
+import org.apache.spark.sql.columnar.{CachedBatch, SimpleMetricsCachedBatch, SimpleMetricsCachedBatchSerializer}
+import org.apache.spark.sql.comet.util.Utils
+import org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.unsafe.types.{ByteArray, UTF8String}
+import org.apache.spark.util.io.ChunkedByteBuffer
+
+import org.apache.comet.CometArrowAllocator
+
+/**
+ * Cached batch format used when Comet writes Spark in-memory cache data.
+ *
+ * `bytes` contains compressed Arrow stream data produced by `Utils.serializeBatches`. The cache
+ * manager still owns storage and eviction; this class only changes the cached payload.
+ */
+private case class CometCachedBatch(
+    override val numRows: Int,
+    override val sizeInBytes: Long,
+    override val stats: InternalRow,
+    bytes: ChunkedByteBuffer)
+    extends SimpleMetricsCachedBatch
+
+/**
+ * Cache serializer that stores Comet-compatible Arrow batches in Spark's in-memory cache.
+ *
+ * The cached payload format is decided by the schema alone. A relation whose schema Comet's Arrow
+ * writer supports is stored as `CometCachedBatch`, and every other relation is delegated in full
+ * to Spark's `DefaultCachedBatchSerializer`. The format deliberately does not depend on any
+ * runtime config: `spark.sql.cache.serializer` is a static conf, so installing this serializer is
+ * already a per-application decision, and a relation whose format could flip mid-session cannot
+ * be read back reliably. `spark.comet.exec.inMemoryCache.enabled` still governs whether a scan
+ * over the cache runs natively, and its value at startup is what makes `CometDriverPlugin`
+ * install this serializer in the first place.
+ *
+ * Reads of `CometCachedBatch` keep working when the native scan is disabled, because Spark then
+ * reads the same cached data through the SparkToColumnar fallback path.
+ */
+class ArrowCachedBatchSerializer extends SimpleMetricsCachedBatchSerializer {

Review Comment:
   add early termination handling: eg. `LIMIT`, `take`, or any other cancellation
   
   ref: [spark's TaskCompletionListener of ArrowCachedBatchSerializer](https://github.com/apache/spark/blob/166dbc560441c0c39a8368a82c7b1d451ca71ea9/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala#L1168-L1182)



##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet.execution.arrow
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, GenericInternalRow, IsNotNull, IsNull, UnsafeProjection}
+import org.apache.spark.sql.columnar.{CachedBatch, SimpleMetricsCachedBatch, SimpleMetricsCachedBatchSerializer}
+import org.apache.spark.sql.comet.util.Utils
+import org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.unsafe.types.{ByteArray, UTF8String}
+import org.apache.spark.util.io.ChunkedByteBuffer
+
+import org.apache.comet.CometArrowAllocator
+
+/**
+ * Cached batch format used when Comet writes Spark in-memory cache data.
+ *
+ * `bytes` contains compressed Arrow stream data produced by `Utils.serializeBatches`. The cache
+ * manager still owns storage and eviction; this class only changes the cached payload.
+ */
+private case class CometCachedBatch(
+    override val numRows: Int,
+    override val sizeInBytes: Long,
+    override val stats: InternalRow,
+    bytes: ChunkedByteBuffer)

Review Comment:
   ChunkedByteBuffer is [Externalizable](https://github.com/apache/spark/blob/v4.1.2/core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala#L35-L124), so BlockManager can serialize it to DiskStore. so both storage level (`DISK_ONLY` and `MEMORY_AND_DISK`) should already supported? Could we add a `StorageLevel.DISK_ONLY` regression test?
   
   A deterministic DISK_ONLY test should:
   - Materialize a `CometCachedBatch` using `StorageLevel.DISK_ONLY`.
   - Assert `memSize == 0`, `diskSize > 0`, and all partitions cached.
   - Run a second query and verify the answer plus `CometInMemoryTableScan`, proving disk deserialization and decoding work.
   
   Spark uses the same size assertions in its [BlockManager regression](https://github.com/apache/spark/blob/v4.1.2/core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala#L2528-L2548).



##########
spark/src/main/scala/org/apache/spark/sql/comet/CometInMemoryTableScanExec.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.columnar.{CachedBatch, CachedBatchSerializer}
+import org.apache.spark.sql.execution.LeafExecNode
+import org.apache.spark.sql.execution.columnar.InMemoryTableScanExec
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.CometConf
+import org.apache.comet.serde.CometOperatorSerde
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.serializeDataType
+
+/**
+ * Reads Spark cached table data when the cache was written by Comet's cache serializer.
+ *
+ * Spark stores cached data through `CachedBatchSerializer`. This node keeps the scan inside Comet
+ * by asking the serializer to decode cached batches directly into `ColumnarBatch` output,
+ * avoiding the extra Spark columnar-to-Comet columnar conversion used by the default path.
+ *
+ * `relationOutput` is the full schema stored in the cache. `scanOutput` is the subset requested
+ * by this scan after pruning.
+ */
+case class CometInMemoryTableScanExec(

Review Comment:
   Add AQE test:
   
   - [SPARK-42101](https://github.com/apache/spark/blob/v4.1.2/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala#L3114-L3153): leaves a cached join cold, first-touches it through an AQE aggregation, and checks cold/warm materialization and plan rewrites.
   - [Table-cache stage in an AQE join](https://github.com/apache/spark/blob/v4.1.2/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala#L3156-L3190): verifies TableCacheQueryStageExec and shuffle behavior.
   - [SPARK-37742](https://github.com/apache/spark/blob/v4.1.2/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala#L2780-L2831): verifies AQE does not choose joins using invalid cache runtime statistics.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5432479372

   > Will take a look! It looks similar to [apache/spark#56334](https://github.com/apache/spark/pull/56334) too although seems in Comet we store compressed IPC stream and can decode only selected columns. cc @viirya too
   
   I don't look at the details of this yet. I can do it later this week.
   
   In Spark, the Arrow cache also stores IPC format data and can decode/decompress the selected columns only.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5432632450

   @sunchao thanks, that is a genuinely valuable review and all five findings were real. Fixed in cb07053e3, with a regression test for each. Replies are on the individual threads; the short version:
   
   | Finding | Status | Introduced by this PR? |
   |---|---|---|
   | Join over empty-output cache scan returns wrong results | Fixed | No, predates it |
   | Dictionary provider reused across columns | Fixed | **Yes** |
   | Readers leak when a later column fails to open | Fixed | **Yes** |
   | Cache RDD built during planning | Fixed | No, predates it |
   | Large-offset Arrow vectors accepted then rejected | Fixed | No, predates it |
   
   The P1 fix is better than a repair. The code widened an empty projection to a placeholder column because a comment asserted the native plan needs a non-empty scan schema. That assertion is simply false: passing `op.output` through unchanged, empty and all, plans and runs fine. So the widening is deleted rather than reconciled with `output`, which fixes your join **and** makes `count(*)` decode nothing at all, which is better than the placeholder column I had introduced a commit earlier.
   
   Two of the five are mine, both consequences of splitting the payload into one stream per column: independent readers mean independent dictionary namespaces, and a partial failure mid-construction has no owner yet. Worth saying plainly, since the split is a change I made deliberately and these are its costs.
   
   One process note in case it is useful to you. My first version of the planning regression test asserted on `isCachedColumnBuffersLoaded`, which also requires the blocks to be populated, so it stayed false either way and passed with the fix reverted. I only caught it because I revert each fix and re-run before claiming a test works. The version on the branch asserts the cached plan still reports `isFinalPlan=false` after `explain()`, which is the effect you actually described, and it does fail without the fix.
   
   All 30 tests in `CometInMemoryCacheSuite` pass, plus `UtilsSuite`, `CometNativeColumnarToRowSuite` and `NativeUtilSuite`. Scalafix is clean on the 3.4, 3.5 and 4.0 profiles.
   
   Separately, I went through the Spark work you linked and filed #5487 with the comparison, including a couple of things worth taking from it that are unrelated to these findings.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "sandugood (via GitHub)" <gi...@apache.org>.
sandugood commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5208500029

   Tested on Spark4.0 against current branch. Using PySpark
   Got this error on the same pipeline that was testing earlier. And I can clearly see a regression in performance. Same job with `.cache()` on default Spark took <10 min, Comet ran >15min.
   
   
   Here is error:
   ```
   Py4JJavaError: An error occurred while calling o12641.createOrReplace.
   : org.apache.spark.SparkException: Job aborted due to stage failure: Task 92 in stage 27.0 failed 4 times, most recent failure: Lost task 92.3 in stage 27.0 (TID 15624) (1.1.1.1 executor 76): org.apache.comet.CometNativeException: Invalid argument error: column types must match schema types, expected List(Struct("col_1": Utf8, "col_2": Boolean, "col_3": Boolean)) but found List(Struct("col_1": Utf8, "col_2": non-null Boolean, "col_3": Boolean)) at column index 5
   	at org.apache.comet.Native.executePlan(Native Method)
   	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2(CometExecIterator.scala:155)
   	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2$adapted(CometExecIterator.scala:154)
   	at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:189)
   	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$1(CometExecIterator.scala:154)
   	at org.apache.comet.Tracing$.withTrace(Tracing.scala:31)
   	at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:152)
   	at org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:196)
   ```
   (obfuscated column names)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5147199985

   @peterxcli @0lai0 thanks both — five of the six comments are addressed in 1367665bd, and two were real bugs.
   
   **Pruning config was silently ignored** (@peterxcli). `CometInMemoryTableScanExec` applied the stats filter unconditionally, so `spark.sql.inMemoryColumnarStorage.partitionPruning=false` did nothing. Now gated the same way Spark's `filteredCachedBatches` is. On your "not sure enabling pruning is always better" — pruning is normally the win, but the point is that the config exists so a user *can* turn it off, usually to rule out a stats bug when results look wrong; ignoring it breaks the one knob they are reaching for.
   
   **Readers leaked on early termination** (@peterxcli). Confirmed: `ArrowReaderIterator.close()` is only reached on exhaustion, so LIMIT/`take()`/cancellation left a reader open. Now registers a `TaskCompletionListener` as Spark's implementation does. Since `flatMap` consumes inner iterators sequentially, tracking the current reader is enough, and `close()` is already idempotent and synchronized.
   
   **Root leaked if conversion threw** (@0lai0). Fixed with the `try`/`NonFatal` guard you suggested. You are right the normal path cannot reach it, but `writeColumns` walks arbitrary `ColumnVector` implementations — the new path in this PR — so a misbehaving external vector is exactly the case. `rowToArrowBatchIter` had the same shape and got the same guard.
   
   **DISK_ONLY test** added to your spec: `memSize == 0`, `diskSize > 0`, all partitions cached, payload still `CometCachedBatch`, second query correct through the native scan. Your `Externalizable` reasoning held — it worked with no production change, which is exactly why the regression test is worth having.
   
   **`isArrowBacked` can be false** — that is the case this PR exists for: with `spark.comet.scan.enabled=false`, Spark's vectorized Parquet reader hands us `OnHeapColumnVector`s, and any external connector's `ColumnVector` does the same. The two "not Arrow-backed" tests at the end of the suite cover it.
   
   **AQE tests: not done.** Porting the three scenarios you linked is a real chunk of work on AQE plan-shape assertions rather than the cache format this PR changes, so I left it rather than rushing it. Since you said these could be follow-ups, I suggest a tracking issue for AQE coverage of `CometInMemoryTableScanExec` — I have not filed one, happy to if you want it.
   
   One thing worth passing on from writing the pruning test: reading a metric off `df.queryExecution.executedPlan` after `checkSparkAnswer(df)` gives **zero**, because `checkSparkAnswer` executes its own copies. My first version passed vacuously with 0 on both sides of the comparison; it needs the df forced explicitly. The real numbers are 100 rows decoded with pruning on versus 1000 with it off.
   
   All 19 tests in `CometInMemoryCacheSuite` pass on spark-3.5 (1 canceled, the pre-existing `isSpark40Plus` gate).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3677859989


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowConverters.scala:
##########
@@ -74,4 +74,45 @@ object CometArrowConverters extends Logging {
       }
     }
   }
+
+  /**
+   * Copy a Spark `ColumnarBatch` whose columns are not Arrow-backed (e.g.
+   * `On/OffHeapColumnVector` from Spark's vectorized Parquet reader, or a third-party connector's
+   * vectors) into a freshly allocated Arrow `ColumnarBatch` of `CometVector`s.
+   *
+   * The input batch is not consumed or closed; the caller owns the returned batch and must close
+   * it. Values are copied element-wise, since Spark's `ColumnVector` implementations do not
+   * expose Arrow buffers.
+   */
+  def columnarBatchToArrowBatch(

Review Comment:
   A `ColumnarBatch` is a container:
   ```scala
   class ColumnarBatch {
     ColumnVector[] columns;
   }
   ```
   
   Spark 4.1.2’s relevant `ColumnVector` implementations are:
   Producer | Vectors inside ColumnarBatch
   -- | --
   Vectorized Parquet | OnHeapColumnVector, OffHeapColumnVector, ConstantColumnVector
   Vectorized ORC | OrcAtomicColumnVector, OrcArrayColumnVector, OrcMapColumnVector, OrcStructColumnVector; sometimes heap/off-heap or constant vectors
   Spark Arrow/Python paths | ArrowColumnVector
   Comet | CometVector
   External columnar sources | Any implementation extending Spark’s ColumnVector
   
   [SparkColumnarArrowReader](https://github.com/apache/datafusion-comet/blob/63c5a916b2e5626b040dc1f097384ab1a443bf91/spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/SparkColumnarArrowReader.scala#L27-L101) does not inspect those concrete classes. It wraps each column in `ColumnarArray`, and `ArrowWriter` reads through Spark’s standard accessors such as `getLong`, `getUTF8String`, `getArray`, and `getStruct`. Therefore, it supports any implementation that correctly follows the [spark's ColumnVector](https://github.com/apache/spark/blob/v4.1.2/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnVector.java)
   
   `ColumnVector` contract for the declared data type.
   The cache writer could reuse it approximately like this:
   
   ```scala
   override def convertColumnarBatchToCachedBatch(
       input: RDD[ColumnarBatch],
       schema: Seq[Attribute],
       storageLevel: StorageLevel,
       conf: SQLConf): RDD[CachedBatch] = {
   
     val sparkSchema = toStructType(schema)
     val batchSize = conf.columnBatchSize
     val timeZoneId = conf.sessionLocalTimeZone
   
     input.mapPartitions { batches =&gt;
       // Arrow Schema is not serializable, so construct it inside the task.
       val arrowSchema = Utils.toArrowSchema(sparkSchema, timeZoneId)
   
       val arrowBatches = CometArrowStream.readerBatchIter(
         "CometCacheWrite",
         allocator =&gt;
           new SparkColumnarArrowReader(
             allocator,
             arrowSchema,
             batches,
             batchSize))
   
       encodeBatches(arrowBatches, schema)
     }
   }
   ```
   
   That gives this path:
   ```
   Spark ColumnarBatch
     → SparkColumnarArrowReader
     → Arrow-backed ColumnarBatch containing CometVector
     → existing encodeBatches/getBatchFieldVectors
     → CometCachedBatch</code>
   ```
   
   _Assist with LLM_



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5432497098

   > @viirya if you have a moment, the question I would most value your view on is whether the buffer-span projection reader was worth its complexity in practice, or whether you would take the simpler split-by-column route knowing what you know now.
   
   "the buffer-span projection reader" is what you implemented in this PR?
   
   So your question is about the column pruning (i.e., selection of columns) in the cache?
   
   I did the column pruning in Arrow cache by this Spark PR https://github.com/apache/spark/pull/58177.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5443337945

   @viirya thanks — you were right on both counts, and the Kryo one turned out to be wider than either of us had it. Fixed in 7e19dff20.
   
   **On the blast radius.** Agreed, and your framing is what made me go looking. #5487 framed it as a DISK_ONLY concern and that was under-scoped: the `_SER` levels, replication, cross-executor fetches and the disk half of the default `MEMORY_AND_DISK` all serialize the `CachedBatch`, so a plain `.cache()` that spills is enough. Took your option 1.
   
   **But it is not only the cache.** While writing the test I checked whether the registrator needed anything beyond the cached batch, and `CometBroadcastExchangeExec` broadcasts an `Array[ChunkedByteBuffer]` from `Utils.serializeBatches`. Spark registers `ChunkedByteBuffer` in `KryoSerializer.toRegister` but not an array of them, so a native broadcast join fails under `registrationRequired=true` on `main` today, with no cache involved:
   
   ```
   java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.util.io.ChunkedByteBuffer[]
     at org.apache.spark.sql.comet.CometBroadcastExchangeExec.doExecuteBroadcast(CometBroadcastExchangeExec.scala:232)
   ```
   
   Same registration fixes it, since the cache write path hands back the same type, so I fixed it here rather than filing it — but it is a pre-existing bug rather than one this PR introduces, so say the word if you would rather it were split out.
   
   **One thing I could not do, and it changes the shape of the fix.** Comet cannot install this for the user. `KryoSerializer` reads `spark.kryo.registrator` into a `val` in its constructor, and `SparkEnv.create` builds it at `SparkContext.scala:478`, whereas `PluginContainer` is constructed at line 574. So unlike `spark.sql.cache.serializer` — a `StaticSQLConf` read lazily from the session, which is exactly why `maybeSetCacheSerializer` can inject it — this one is already captured by the time any plugin runs. Setting it from the driver plugin would reach executors but not the driver's own `SparkEnv`, so it would work in cluster mode and fail in local mode, which is worse than a documented requirement.
   
   So the fix is three parts rather than one:
   
   * `org.apache.comet.CometKryoRegistrator`, with the class list split between `Utils.arrowBytesKryoClasses` (the Arrow-bytes carriers, shared by broadcast and cache) and `ArrowCachedBatchSerializer.kryoClasses` (the cached batch and its statistics row).
   * Documented on `spark.comet.exec.inMemoryCache.enabled`, as your option 2, since option 1 does not remove the need to tell people about it.
   * `CometDriverPlugin` warns at startup when `spark.serializer` is Kryo, `registrationRequired` is true, and the registrator is absent, naming the conf to add. That is the part that addresses "hard to attribute" — the warning fires before anything is cached or broadcast.
   
   **Two findings worth recording from building the list.**
   
   `DefaultCachedBatch` had to be registered too. A schema Comet cannot store is delegated to Spark's serializer, and Spark registers that class itself only from 4.1 — 3.4, 3.5 and 4.0 do not, along with `GenericInternalRow`. So the delegated path fails on those versions without Comet registering it, and some entries that look redundant on 4.1 are load-bearing on 3.5.
   
   `java.math.MathContext` cannot be registered at all on JDK 17 — `kryo.register` builds a `FieldSerializer` eagerly and gets `InaccessibleObjectException: module java.base does not "opens java.math"`. Decimal bounds above `Long` precision survive anyway because Chill's `scala.math.BigDecimal` serializer writes the `java.math.BigDecimal` inside as a class-and-object rather than reflecting over the `MathContext`, which is why the list registers the `java.math` types and not the Scala one. Worth knowing before anyone tries to extend this list by intuition.
   
   **Testing.** `CometInMemoryCacheKryoSuite`, four tests, all reverted-and-rechecked rather than assumed: cache round trip at `DISK_ONLY` and `MEMORY_AND_DISK_SER` over a relation carrying every type whose bounds the statistics row records, read back through a predicate so the bounds go through Kryo too; the native broadcast; and the delegated `DefaultCachedBatch` path. Without the registrator they fail with exactly the `Class is not registered` you predicted. Green on the 3.4, 3.5, 4.0 and 4.1 profiles, along with `CometInMemoryCacheSuite`, `UtilsSuite`, `CometArrowStreamSuite`, `NativeUtilSuite` and `CometPluginsSuite`.
   
   I did not test the `MEMORY_AND_DISK` spill case itself. Both levels in the suite serialize deterministically on put; forcing a spill is not cheap to arrange, so the case that motivated your comment is covered by argument rather than by a test.
   
   **On the nit** — confirmed, that is intended and not a bug. Because `spark.sql.cache.serializer` is static, the format is an application-lifetime decision made at startup; flipping `spark.comet.exec.inMemoryCache.enabled` off later only routes scans back to Spark's execution path and the footprint stays Arrow. The config doc says that, and I would rather keep the honest explanation than split the flag, since a relation whose cached format could change mid-session could not be read back reliably.
   
   Agreed on AQE (#5245) being the follow-up to weight most, and SPARK-37742 within it for the reason you and @peterxcli both landed on.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5441403778

   Reviewed the current HEAD (`70f046abf`). This is in good shape — the design boundary is clean (the serializer owns only the cached payload; Spark's cache manager keeps lifecycle, storage, and eviction), the fallback path is careful (`enabled && cometCacheFormat`, with distinct fallback reasons, and the guarantee that turning it on is never worse than off), and the comments explaining the non-obvious decisions — the empty-output/`count(*)` handling, `doCanonicalize`, deferring `cachedColumnBuffers` to execution, the fixed `NATIVE_TIMEZONE` label, the pruning-safety restriction in `buildFilter` — are genuinely helpful and clearly the residue of the earlier review rounds. I re-checked the fixes for @sunchao's findings (zero-column join output, per-column dictionary providers across the cache-write / broadcast / shuffle paths, reader-leak-on-init, canonicalization) and they hold; I didn't find new sibling instances of those bug-classes.
   
   One thing I'd like to see addressed before merge, and one nit.
   
   **`spark.kryo.registrationRequired=true` — the blast radius is wider than "DISK_ONLY".**
   
   This is the Kryo-registration gap you already recorded in #5487 ("caching fails outright under `spark.kryo.registrationRequired=true`"), so the finding itself isn't in dispute — but I think it's under-scoped there, and that changes whether it should be a pre-merge item.
   
   #5487 (and the natural reading of it) frames this as only affecting explicitly serialized storage levels, with DISK_ONLY as the path that exercises it. But a `CachedBatch` goes through the configured serializer in more cases than that:
   
   - the `*_SER` levels (`MEMORY_ONLY_SER`, `MEMORY_AND_DISK_SER`),
   - replicated levels (`_2`) and any cross-executor block fetch,
   - and — the one that matters — **the disk-spill portion of the default `MEMORY_AND_DISK`** that a plain `df.cache()` uses. The in-memory copy is deserialized, but once a partition spills, it's serialized.
   
   So under `spark.kryo.registrationRequired=true`, an ordinary `.cache()` that spills is enough to hit this; the user doesn't have to reach for DISK_ONLY. And because `CometCachedBatch` is unregistered, the failure surfaces as Kryo's "Class is not registered" rather than anything pointing back at this feature, so it's hard to attribute.
   
   The existing DISK_ONLY test doesn't cover it — the suite never sets a Kryo serializer, so that test runs on the default Java serializer.
   
   Suggestion, in order of preference:
   
   1. Register `CometCachedBatch` (and the classes it carries) in a Comet Kryo registrator — Spark does exactly this for its own `ArrowCachedBatch`, so there's a direct precedent — plus a `KryoSerializer` + `registrationRequired=true` DISK_ONLY regression test. Small, and it closes a hard-failure hole.
   2. If you'd rather keep it a follow-up: at minimum document it on `spark.comet.exec.inMemoryCache.enabled`. That doc already lists several limitations but not this one, and the near-zero-cost win is turning a silent hard failure into a documented one.
   
   I'd lean against leaving it purely as #5487 given that plain `.cache()` + spill can trip it, not just DISK_ONLY.
   
   **Nit:** the `enabled` config conflates two things — whether `CometDriverPlugin` installs the serializer (decided at startup, since `spark.sql.cache.serializer` is static) and whether scans run natively. The doc explains this honestly, so this is just a "confirm it's intended": once the feature has been on at startup, every supported relation is cached in Arrow format for the life of the application even if the flag is later flipped off (that only reroutes scans back to Spark). That's a reasonable consequence of the static conf, not a bug — flagging it only because a reader turning the flag off at runtime might expect the cache footprint to revert too.
   
   Overall I think this is mergeable as an experimental, default-off feature along the "merge and iterate" lines you proposed, with the Kryo item as the one I'd want handled (fix or document) first. The AQE gap (#5245) — especially the SPARK-37742 stats-divergence you called out, where Comet's own size accounting feeds AQE's join-strategy choice — is the follow-up I'd weight most after this lands.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3867744580


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala:
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet.execution.arrow
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.TaskContext
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, GenericInternalRow, IsNotNull, IsNull, UnsafeProjection}
+import org.apache.spark.sql.columnar.{CachedBatch, SimpleMetricsCachedBatch, SimpleMetricsCachedBatchSerializer}
+import org.apache.spark.sql.comet.util.Utils
+import org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.unsafe.types.{ByteArray, UTF8String}
+import org.apache.spark.util.io.ChunkedByteBuffer
+
+import org.apache.comet.CometArrowAllocator
+
+/**
+ * Cached batch format used when Comet writes Spark in-memory cache data.
+ *
+ * `columns` holds one compressed Arrow stream per cached column, in cache-schema order, produced
+ * by `Utils.serializeBatchColumns`. Storing columns separately is what lets a scan decode only
+ * the ones it projected; a single stream covering the whole batch would have to be inflated in
+ * full before any projection could be applied. The cache manager still owns storage and eviction;
+ * this class only changes the cached payload.
+ */
+private case class CometCachedBatch(
+    override val numRows: Int,
+    override val sizeInBytes: Long,
+    override val stats: InternalRow,
+    columns: Array[ChunkedByteBuffer])
+    extends SimpleMetricsCachedBatch
+
+/**
+ * Cache serializer that stores Comet-compatible Arrow batches in Spark's in-memory cache.
+ *
+ * The cached payload format is decided by the schema alone. A relation whose schema Comet's Arrow
+ * writer supports is stored as `CometCachedBatch`, and every other relation is delegated in full
+ * to Spark's `DefaultCachedBatchSerializer`. The format deliberately does not depend on any
+ * runtime config: `spark.sql.cache.serializer` is a static conf, so installing this serializer is
+ * already a per-application decision, and a relation whose format could flip mid-session cannot
+ * be read back reliably. `spark.comet.exec.inMemoryCache.enabled` still governs whether a scan
+ * over the cache runs natively, and its value at startup is what makes `CometDriverPlugin`
+ * install this serializer in the first place.
+ *
+ * Reads of `CometCachedBatch` keep working when the native scan is disabled, because Spark then
+ * reads the same cached data through the SparkToColumnar fallback path.
+ */
+class ArrowCachedBatchSerializer extends SimpleMetricsCachedBatchSerializer {
+
+  import ArrowCachedBatchSerializer.supportsSchema
+
+  private val fallback = new DefaultCachedBatchSerializer()
+
+  // Bounds and null counts per column, gathered before the batch is serialized: serializing
+  // clears the batch's vectors, and the per-column byte sizes that complete the statistics row
+  // are only known afterwards. See statsRow.
+  private def gatherColumnStats(
+      batch: ColumnarBatch,
+      attrs: Seq[Attribute]): (Array[Any], Array[Any], Array[Int]) = {
+    val numCols = attrs.length
+    val lower = new Array[Any](numCols)
+    val upper = new Array[Any](numCols)
+    val nulls = Array.fill[Int](numCols)(0)
+    val numRows = batch.numRows()
+
+    var c = 0
+    while (c < numCols) {
+      val dt = attrs(c).dataType
+      val col = batch.column(c)
+      var r = 0
+      while (r < numRows) {
+        if (col.isNullAt(r)) {
+          nulls(c) += 1
+        } else if (tracksBounds(dt)) {
+          val value = readValue(col, dt, r)
+          if (lower(c) == null || compare(dt, value, lower(c)) < 0) {
+            lower(c) = value
+          }
+          if (upper(c) == null || compare(dt, value, upper(c)) > 0) {
+            upper(c) = value
+          }
+        }
+        r += 1
+      }
+      c += 1
+    }
+
+    (lower, upper, nulls)
+  }
+
+  // Build the statistics row expected by SimpleMetricsCachedBatchSerializer.
+  // For each cached column Spark expects five values in this order:
+  // lower bound, upper bound, null count, row count, and size in bytes.
+  private def statsRow(
+      lower: Array[Any],
+      upper: Array[Any],
+      nulls: Array[Int],
+      numRows: Int,
+      columnSizes: Array[Long]): InternalRow = {
+    val numCols = lower.length
+    val values = new Array[Any](numCols * 5)
+    var c = 0
+    while (c < numCols) {
+      val base = c * 5
+      values(base) = lower(c)
+      values(base + 1) = upper(c)
+      values(base + 2) = nulls(c)
+      values(base + 3) = numRows
+      // Each column is its own compressed stream, so its size is known exactly. Cache pruning
+      // uses bounds/null-count/row-count rather than this field, but Spark reserves it and
+      // reports it, so record the real value.
+      values(base + 4) = columnSizes(c)
+      c += 1
+    }
+
+    new GenericInternalRow(values)
+  }
+
+  // Spark can prune cache batches only for types whose bounds can be compared.
+  // Other types still report null count and row count but leave bounds as null.
+  private def tracksBounds(dt: DataType): Boolean = dt match {
+    case BooleanType | ByteType | ShortType | IntegerType | LongType | FloatType | DoubleType |
+        _: DecimalType | StringType | DateType | TimestampType | TimestampNTZType =>
+      true
+    case _ => false
+  }
+
+  // Read a non-null value from a ColumnVector using Spark's internal value type
+  // for the corresponding DataType.
+  private def readValue(col: ColumnVector, dt: DataType, rowId: Int): Any = dt match {
+    case BooleanType => col.getBoolean(rowId)
+    case ByteType => col.getByte(rowId)
+    case ShortType => col.getShort(rowId)
+    case IntegerType | DateType => col.getInt(rowId)
+    case LongType | TimestampType | TimestampNTZType => col.getLong(rowId)
+    case FloatType => col.getFloat(rowId)
+    case DoubleType => col.getDouble(rowId)
+    case d: DecimalType => col.getDecimal(rowId, d.precision, d.scale)
+    case StringType => col.getUTF8String(rowId).copy()
+    case _ => null
+  }
+
+  // Compare values using the same physical representation used in the stats row.
+  private def compare(dt: DataType, left: Any, right: Any): Int = dt match {
+    case BooleanType =>
+      java.lang.Boolean.compare(left.asInstanceOf[Boolean], right.asInstanceOf[Boolean])
+    case ByteType =>
+      java.lang.Byte.compare(left.asInstanceOf[Byte], right.asInstanceOf[Byte])
+    case ShortType =>
+      java.lang.Short.compare(left.asInstanceOf[Short], right.asInstanceOf[Short])
+    case IntegerType | DateType =>
+      java.lang.Integer.compare(left.asInstanceOf[Int], right.asInstanceOf[Int])
+    case LongType | TimestampType | TimestampNTZType =>
+      java.lang.Long.compare(left.asInstanceOf[Long], right.asInstanceOf[Long])
+    case FloatType =>
+      java.lang.Float.compare(left.asInstanceOf[Float], right.asInstanceOf[Float])
+    case DoubleType =>
+      java.lang.Double.compare(left.asInstanceOf[Double], right.asInstanceOf[Double])
+    case _: DecimalType =>
+      left.asInstanceOf[Decimal].compare(right.asInstanceOf[Decimal])
+    case StringType =>
+      ByteArray.compareBinary(
+        left.asInstanceOf[UTF8String].getBytes,
+        right.asInstanceOf[UTF8String].getBytes)
+    case other =>
+      throw new IllegalStateException(s"compare called for unsupported type $other")
+  }
+
+  // Compute Spark-compatible cache stats before serializing each batch to Arrow.
+  // The stats are stored beside the Arrow bytes so Spark's cache filter can prune
+  // CometCachedBatch without decoding the batch first.
+  //
+  // A columnar input batch is not guaranteed to be Arrow-backed; see supportsColumnarInput for
+  // why. Batches that are not get copied into Arrow first, since Utils.serializeBatches only
+  // writes CometVector columns.
+  private def encodeBatches(
+      batches: Iterator[ColumnarBatch],
+      attrs: Seq[Attribute]): Iterator[CachedBatch] = {
+    val arrowSchema =
+      Utils.toArrowSchema(Utils.fromAttributes(attrs), CometArrowStream.NATIVE_TIMEZONE)
+
+    batches.map { batch =>
+      // Bounds and null counts are read from the input batch, which serializing then clears, so
+      // they have to be gathered first. The row is only assembled once the per-column sizes are
+      // known.
+      val (lower, upper, nulls) = gatherColumnStats(batch, attrs)
+      val numRows = batch.numRows()
+
+      val columns = if (Utils.isArrowBacked(batch)) {
+        Utils.serializeBatchColumns(batch)
+      } else {
+        val arrowBatch =
+          CometArrowConverters.columnarBatchToArrowBatch(batch, arrowSchema, CometArrowAllocator)
+        try Utils.serializeBatchColumns(arrowBatch)
+        finally arrowBatch.close()

Review Comment:
   Filed the durable part of this as #5488.
   
   To be precise about what does and does not outlive this PR, since I called it pre-existing above and that was ambiguous. `CometInMemoryTableScanExec` and `ArrowCachedBatchSerializer` are both new files in #5051, so the two findings in the scan node cannot survive it being dropped. This one is different: `getFieldVector` and both `Utils.serializeBatches` consumers, `getByteArrayRdd` in `operators.scala` and `CometBroadcastExchangeExec`, are on `main` today.
   
   And the fix here does not close it. Making `isArrowBacked` reject those vectors only reroutes the cache write path to conversion; the broadcast and collect paths have no conversion fallback and still throw. `isSupportedFieldVector` is a useful building block for a real fix, but it goes away with this PR too. #5488 has the producer trail through `CometMapInBatchExec`, and notes that neither of us ran the mapInArrow path end to end.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5432289472

   Will take a look! It looks similar to https://github.com/apache/spark/pull/56334 too although seems in Comet we store compressed IPC stream and can decode only selected columns. cc @viirya too


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3677720957


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet.execution.arrow
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, GenericInternalRow, IsNotNull, IsNull, UnsafeProjection}
+import org.apache.spark.sql.columnar.{CachedBatch, SimpleMetricsCachedBatch, SimpleMetricsCachedBatchSerializer}
+import org.apache.spark.sql.comet.util.Utils
+import org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.unsafe.types.{ByteArray, UTF8String}
+import org.apache.spark.util.io.ChunkedByteBuffer
+
+import org.apache.comet.CometArrowAllocator
+
+/**
+ * Cached batch format used when Comet writes Spark in-memory cache data.
+ *
+ * `bytes` contains compressed Arrow stream data produced by `Utils.serializeBatches`. The cache
+ * manager still owns storage and eviction; this class only changes the cached payload.
+ */
+private case class CometCachedBatch(
+    override val numRows: Int,
+    override val sizeInBytes: Long,
+    override val stats: InternalRow,
+    bytes: ChunkedByteBuffer)
+    extends SimpleMetricsCachedBatch
+
+/**
+ * Cache serializer that stores Comet-compatible Arrow batches in Spark's in-memory cache.
+ *
+ * The cached payload format is decided by the schema alone. A relation whose schema Comet's Arrow
+ * writer supports is stored as `CometCachedBatch`, and every other relation is delegated in full
+ * to Spark's `DefaultCachedBatchSerializer`. The format deliberately does not depend on any
+ * runtime config: `spark.sql.cache.serializer` is a static conf, so installing this serializer is
+ * already a per-application decision, and a relation whose format could flip mid-session cannot
+ * be read back reliably. `spark.comet.exec.inMemoryCache.enabled` still governs whether a scan
+ * over the cache runs natively, and its value at startup is what makes `CometDriverPlugin`
+ * install this serializer in the first place.
+ *
+ * Reads of `CometCachedBatch` keep working when the native scan is disabled, because Spark then
+ * reads the same cached data through the SparkToColumnar fallback path.
+ */
+class ArrowCachedBatchSerializer extends SimpleMetricsCachedBatchSerializer {
+
+  import ArrowCachedBatchSerializer.supportsSchema
+
+  private val fallback = new DefaultCachedBatchSerializer()
+
+  // Row-to-Arrow conversion needs a StructType, while cache APIs pass attributes.
+  private def toStructType(schema: Seq[Attribute]): StructType = {
+    StructType(schema.map { attr =>
+      StructField(attr.name, attr.dataType, attr.nullable, attr.metadata)
+    })
+  }
+
+  // Build the statistics row expected by SimpleMetricsCachedBatchSerializer.
+  // For each cached column Spark expects five values in this order:
+  // lower bound, upper bound, null count, row count, and size in bytes.
+  private def computeStats(batch: ColumnarBatch, attrs: Seq[Attribute]): InternalRow = {
+    val numCols = attrs.length
+    val lower = new Array[Any](numCols)
+    val upper = new Array[Any](numCols)
+    val nulls = Array.fill[Int](numCols)(0)
+    val numRows = batch.numRows()
+
+    var c = 0
+    while (c < numCols) {
+      val dt = attrs(c).dataType
+      val col = batch.column(c)
+      var r = 0
+      while (r < numRows) {
+        if (col.isNullAt(r)) {
+          nulls(c) += 1
+        } else if (tracksBounds(dt)) {
+          val value = readValue(col, dt, r)
+          if (lower(c) == null || compare(dt, value, lower(c)) < 0) {
+            lower(c) = value
+          }
+          if (upper(c) == null || compare(dt, value, upper(c)) > 0) {
+            upper(c) = value
+          }
+        }
+        r += 1
+      }
+      c += 1
+    }
+
+    val values = new Array[Any](numCols * 5)
+    c = 0
+    while (c < numCols) {
+      val base = c * 5
+      values(base) = lower(c)
+      values(base + 1) = upper(c)
+      values(base + 2) = nulls(c)
+      values(base + 3) = numRows
+      // Spark reserves the fifth field for per-column size. Comet stores the whole
+      // Arrow stream as one compressed buffer, so per-column size is not tracked here.
+      // Cache pruning uses bounds/null-count/row-count, not this size field.
+      values(base + 4) = 0L
+      c += 1
+    }
+
+    new GenericInternalRow(values)
+  }
+
+  // Spark can prune cache batches only for types whose bounds can be compared.
+  // Other types still report null count and row count but leave bounds as null.
+  private def tracksBounds(dt: DataType): Boolean = dt match {
+    case BooleanType | ByteType | ShortType | IntegerType | LongType | FloatType | DoubleType |
+        _: DecimalType | StringType | DateType | TimestampType | TimestampNTZType =>
+      true
+    case _ => false
+  }
+
+  // Read a non-null value from a ColumnVector using Spark's internal value type
+  // for the corresponding DataType.
+  private def readValue(col: ColumnVector, dt: DataType, rowId: Int): Any = dt match {
+    case BooleanType => col.getBoolean(rowId)
+    case ByteType => col.getByte(rowId)
+    case ShortType => col.getShort(rowId)
+    case IntegerType | DateType => col.getInt(rowId)
+    case LongType | TimestampType | TimestampNTZType => col.getLong(rowId)
+    case FloatType => col.getFloat(rowId)
+    case DoubleType => col.getDouble(rowId)
+    case d: DecimalType => col.getDecimal(rowId, d.precision, d.scale)
+    case StringType => col.getUTF8String(rowId).copy()
+    case _ => null
+  }
+
+  // Compare values using the same physical representation used in the stats row.
+  private def compare(dt: DataType, left: Any, right: Any): Int = dt match {
+    case BooleanType =>
+      java.lang.Boolean.compare(left.asInstanceOf[Boolean], right.asInstanceOf[Boolean])
+    case ByteType =>
+      java.lang.Byte.compare(left.asInstanceOf[Byte], right.asInstanceOf[Byte])
+    case ShortType =>
+      java.lang.Short.compare(left.asInstanceOf[Short], right.asInstanceOf[Short])
+    case IntegerType | DateType =>
+      java.lang.Integer.compare(left.asInstanceOf[Int], right.asInstanceOf[Int])
+    case LongType | TimestampType | TimestampNTZType =>
+      java.lang.Long.compare(left.asInstanceOf[Long], right.asInstanceOf[Long])
+    case FloatType =>
+      java.lang.Float.compare(left.asInstanceOf[Float], right.asInstanceOf[Float])
+    case DoubleType =>
+      java.lang.Double.compare(left.asInstanceOf[Double], right.asInstanceOf[Double])
+    case _: DecimalType =>
+      left.asInstanceOf[Decimal].compare(right.asInstanceOf[Decimal])
+    case StringType =>
+      ByteArray.compareBinary(
+        left.asInstanceOf[UTF8String].getBytes,
+        right.asInstanceOf[UTF8String].getBytes)
+    case other =>
+      throw new IllegalStateException(s"compare called for unsupported type $other")
+  }
+
+  // Compute Spark-compatible cache stats before serializing each batch to Arrow.
+  // The stats are stored beside the Arrow bytes so Spark's cache filter can prune
+  // CometCachedBatch without decoding the batch first.
+  private def encodeBatches(
+      batches: Iterator[ColumnarBatch],
+      attrs: Seq[Attribute]): Iterator[CachedBatch] = {
+    batches.flatMap { batch =>
+      val stats = computeStats(batch, attrs)
+
+      Utils.serializeBatches(Iterator.single(batch)).map { case (rows, buffer) =>
+        CometCachedBatch(
+          numRows = rows.toInt,
+          sizeInBytes = buffer.size,
+          stats = stats,
+          bytes = buffer)
+      }
+    }
+  }
+
+  // Resolve requested columns by exprId, not by name, because aliases may reuse names.
+  private def selectedIndices(
+      cacheAttributes: Seq[Attribute],
+      selectedAttributes: Seq[Attribute]): Array[Int] = {
+    if (selectedAttributes.isEmpty) {
+      cacheAttributes.indices.toArray
+    } else {
+      val byExprId = cacheAttributes.zipWithIndex.map { case (attr, idx) =>
+        attr.exprId -> idx
+      }.toMap
+
+      selectedAttributes.map { attr =>
+        byExprId.getOrElse(
+          attr.exprId,
+          throw new IllegalStateException(
+            s"Could not resolve selected attribute ${attr.name} from cache attributes"))
+      }.toArray
+    }
+  }
+
+  // A full-width projection is only an identity projection if every selected index
+  // is already in column order. For example, [1, 0] must still be projected.
+  private def isIdentityProjection(indices: Array[Int], numCols: Int): Boolean =
+    indices.length == numCols && indices.indices.forall(i => indices(i) == i)
+
+  private def projectBatch(batch: ColumnarBatch, indices: Array[Int]): ColumnarBatch = {
+    if (isIdentityProjection(indices, batch.numCols())) {
+      batch
+    } else {
+      val cols = indices.map(i => batch.column(i).asInstanceOf[ColumnVector])
+      new ColumnarBatch(cols, batch.numRows())
+    }
+  }
+
+  // Spark's SimpleMetricsCachedBatchSerializer prunes a batch when the generated partition filter
+  // does not evaluate to true against the stats row. Bounds are only computed for the types
+  // tracksBounds accepts, and for every other column the lower and upper bounds stay null, which
+  // makes a comparison against them evaluate to null and therefore prune the batch. That would
+  // silently drop rows, so predicates over columns without bounds are not pushed down at all.
+  // Null counts and row counts are recorded for every column, so IsNull and IsNotNull stay safe.
+  override def buildFilter(
+      predicates: Seq[Expression],
+      cachedAttributes: Seq[Attribute]): (Int, Iterator[CachedBatch]) => Iterator[CachedBatch] = {
+    val prunable = cachedAttributes.collect {
+      case a if tracksBounds(a.dataType) => a.exprId
+    }.toSet
+
+    val prunablePredicates = predicates.filter {
+      case _: IsNull | _: IsNotNull => true
+      case p => p.references.forall(a => prunable.contains(a.exprId))
+    }
+
+    super.buildFilter(prunablePredicates, cachedAttributes)
+  }
+
+  // Comet's Arrow writer only handles the types listed in supportsSchema. Reporting false here
+  // sends the relation down the row path, where it is delegated to Spark's default serializer,
+  // instead of failing at cache materialization inside Utils.serializeBatches.
+  override def supportsColumnarInput(schema: Seq[Attribute]): Boolean = supportsSchema(schema)

Review Comment:
   probably the cause of https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5122801192



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5123051509

   I pushed a potential fix at the same time @peterxcli was reviewing. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "sandugood (via GitHub)" <gi...@apache.org>.
sandugood commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5128349753

   When the PR is sound I could test it again. Is it ready to be tested @andygrove?
   
   Last time I've built it, got an error while writing an iceberg table:
   `org.apache.comet.CometNativeException: Invalid argument error: column types must match schema types, expected List(Struct("col_1": Utf8, "col_2": Boolean, "col_3": Boolean)) but found List(Struct("col_1": Utf8, "col_2": non-null Boolean, "col_3": Boolean)) at column index 5 `
   
   (note that I obfuscated column names, but kept their order.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3677732055


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet.execution.arrow
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, GenericInternalRow, IsNotNull, IsNull, UnsafeProjection}
+import org.apache.spark.sql.columnar.{CachedBatch, SimpleMetricsCachedBatch, SimpleMetricsCachedBatchSerializer}
+import org.apache.spark.sql.comet.util.Utils
+import org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.unsafe.types.{ByteArray, UTF8String}
+import org.apache.spark.util.io.ChunkedByteBuffer
+
+import org.apache.comet.CometArrowAllocator
+
+/**
+ * Cached batch format used when Comet writes Spark in-memory cache data.
+ *
+ * `bytes` contains compressed Arrow stream data produced by `Utils.serializeBatches`. The cache
+ * manager still owns storage and eviction; this class only changes the cached payload.
+ */
+private case class CometCachedBatch(
+    override val numRows: Int,
+    override val sizeInBytes: Long,
+    override val stats: InternalRow,
+    bytes: ChunkedByteBuffer)
+    extends SimpleMetricsCachedBatch
+
+/**
+ * Cache serializer that stores Comet-compatible Arrow batches in Spark's in-memory cache.
+ *
+ * The cached payload format is decided by the schema alone. A relation whose schema Comet's Arrow
+ * writer supports is stored as `CometCachedBatch`, and every other relation is delegated in full
+ * to Spark's `DefaultCachedBatchSerializer`. The format deliberately does not depend on any
+ * runtime config: `spark.sql.cache.serializer` is a static conf, so installing this serializer is
+ * already a per-application decision, and a relation whose format could flip mid-session cannot
+ * be read back reliably. `spark.comet.exec.inMemoryCache.enabled` still governs whether a scan
+ * over the cache runs natively, and its value at startup is what makes `CometDriverPlugin`
+ * install this serializer in the first place.
+ *
+ * Reads of `CometCachedBatch` keep working when the native scan is disabled, because Spark then
+ * reads the same cached data through the SparkToColumnar fallback path.
+ */
+class ArrowCachedBatchSerializer extends SimpleMetricsCachedBatchSerializer {
+
+  import ArrowCachedBatchSerializer.supportsSchema
+
+  private val fallback = new DefaultCachedBatchSerializer()
+
+  // Row-to-Arrow conversion needs a StructType, while cache APIs pass attributes.
+  private def toStructType(schema: Seq[Attribute]): StructType = {
+    StructType(schema.map { attr =>
+      StructField(attr.name, attr.dataType, attr.nullable, attr.metadata)
+    })
+  }
+
+  // Build the statistics row expected by SimpleMetricsCachedBatchSerializer.
+  // For each cached column Spark expects five values in this order:
+  // lower bound, upper bound, null count, row count, and size in bytes.
+  private def computeStats(batch: ColumnarBatch, attrs: Seq[Attribute]): InternalRow = {
+    val numCols = attrs.length
+    val lower = new Array[Any](numCols)
+    val upper = new Array[Any](numCols)
+    val nulls = Array.fill[Int](numCols)(0)
+    val numRows = batch.numRows()
+
+    var c = 0
+    while (c < numCols) {
+      val dt = attrs(c).dataType
+      val col = batch.column(c)
+      var r = 0
+      while (r < numRows) {
+        if (col.isNullAt(r)) {
+          nulls(c) += 1
+        } else if (tracksBounds(dt)) {
+          val value = readValue(col, dt, r)
+          if (lower(c) == null || compare(dt, value, lower(c)) < 0) {
+            lower(c) = value
+          }
+          if (upper(c) == null || compare(dt, value, upper(c)) > 0) {
+            upper(c) = value
+          }
+        }
+        r += 1
+      }
+      c += 1
+    }
+
+    val values = new Array[Any](numCols * 5)
+    c = 0
+    while (c < numCols) {
+      val base = c * 5
+      values(base) = lower(c)
+      values(base + 1) = upper(c)
+      values(base + 2) = nulls(c)
+      values(base + 3) = numRows
+      // Spark reserves the fifth field for per-column size. Comet stores the whole
+      // Arrow stream as one compressed buffer, so per-column size is not tracked here.
+      // Cache pruning uses bounds/null-count/row-count, not this size field.
+      values(base + 4) = 0L
+      c += 1
+    }
+
+    new GenericInternalRow(values)
+  }
+
+  // Spark can prune cache batches only for types whose bounds can be compared.
+  // Other types still report null count and row count but leave bounds as null.
+  private def tracksBounds(dt: DataType): Boolean = dt match {
+    case BooleanType | ByteType | ShortType | IntegerType | LongType | FloatType | DoubleType |
+        _: DecimalType | StringType | DateType | TimestampType | TimestampNTZType =>
+      true
+    case _ => false
+  }
+
+  // Read a non-null value from a ColumnVector using Spark's internal value type
+  // for the corresponding DataType.
+  private def readValue(col: ColumnVector, dt: DataType, rowId: Int): Any = dt match {
+    case BooleanType => col.getBoolean(rowId)
+    case ByteType => col.getByte(rowId)
+    case ShortType => col.getShort(rowId)
+    case IntegerType | DateType => col.getInt(rowId)
+    case LongType | TimestampType | TimestampNTZType => col.getLong(rowId)
+    case FloatType => col.getFloat(rowId)
+    case DoubleType => col.getDouble(rowId)
+    case d: DecimalType => col.getDecimal(rowId, d.precision, d.scale)
+    case StringType => col.getUTF8String(rowId).copy()
+    case _ => null
+  }
+
+  // Compare values using the same physical representation used in the stats row.
+  private def compare(dt: DataType, left: Any, right: Any): Int = dt match {
+    case BooleanType =>
+      java.lang.Boolean.compare(left.asInstanceOf[Boolean], right.asInstanceOf[Boolean])
+    case ByteType =>
+      java.lang.Byte.compare(left.asInstanceOf[Byte], right.asInstanceOf[Byte])
+    case ShortType =>
+      java.lang.Short.compare(left.asInstanceOf[Short], right.asInstanceOf[Short])
+    case IntegerType | DateType =>
+      java.lang.Integer.compare(left.asInstanceOf[Int], right.asInstanceOf[Int])
+    case LongType | TimestampType | TimestampNTZType =>
+      java.lang.Long.compare(left.asInstanceOf[Long], right.asInstanceOf[Long])
+    case FloatType =>
+      java.lang.Float.compare(left.asInstanceOf[Float], right.asInstanceOf[Float])
+    case DoubleType =>
+      java.lang.Double.compare(left.asInstanceOf[Double], right.asInstanceOf[Double])
+    case _: DecimalType =>
+      left.asInstanceOf[Decimal].compare(right.asInstanceOf[Decimal])
+    case StringType =>
+      ByteArray.compareBinary(
+        left.asInstanceOf[UTF8String].getBytes,
+        right.asInstanceOf[UTF8String].getBytes)
+    case other =>
+      throw new IllegalStateException(s"compare called for unsupported type $other")
+  }
+
+  // Compute Spark-compatible cache stats before serializing each batch to Arrow.
+  // The stats are stored beside the Arrow bytes so Spark's cache filter can prune
+  // CometCachedBatch without decoding the batch first.
+  private def encodeBatches(
+      batches: Iterator[ColumnarBatch],
+      attrs: Seq[Attribute]): Iterator[CachedBatch] = {
+    batches.flatMap { batch =>
+      val stats = computeStats(batch, attrs)
+
+      Utils.serializeBatches(Iterator.single(batch)).map { case (rows, buffer) =>
+        CometCachedBatch(
+          numRows = rows.toInt,
+          sizeInBytes = buffer.size,
+          stats = stats,
+          bytes = buffer)
+      }
+    }
+  }
+
+  // Resolve requested columns by exprId, not by name, because aliases may reuse names.
+  private def selectedIndices(
+      cacheAttributes: Seq[Attribute],
+      selectedAttributes: Seq[Attribute]): Array[Int] = {
+    if (selectedAttributes.isEmpty) {
+      cacheAttributes.indices.toArray
+    } else {
+      val byExprId = cacheAttributes.zipWithIndex.map { case (attr, idx) =>
+        attr.exprId -> idx
+      }.toMap
+
+      selectedAttributes.map { attr =>
+        byExprId.getOrElse(
+          attr.exprId,
+          throw new IllegalStateException(
+            s"Could not resolve selected attribute ${attr.name} from cache attributes"))
+      }.toArray
+    }
+  }
+
+  // A full-width projection is only an identity projection if every selected index
+  // is already in column order. For example, [1, 0] must still be projected.
+  private def isIdentityProjection(indices: Array[Int], numCols: Int): Boolean =
+    indices.length == numCols && indices.indices.forall(i => indices(i) == i)
+
+  private def projectBatch(batch: ColumnarBatch, indices: Array[Int]): ColumnarBatch = {
+    if (isIdentityProjection(indices, batch.numCols())) {
+      batch
+    } else {
+      val cols = indices.map(i => batch.column(i).asInstanceOf[ColumnVector])
+      new ColumnarBatch(cols, batch.numRows())
+    }
+  }
+
+  // Spark's SimpleMetricsCachedBatchSerializer prunes a batch when the generated partition filter
+  // does not evaluate to true against the stats row. Bounds are only computed for the types
+  // tracksBounds accepts, and for every other column the lower and upper bounds stay null, which
+  // makes a comparison against them evaluate to null and therefore prune the batch. That would
+  // silently drop rows, so predicates over columns without bounds are not pushed down at all.
+  // Null counts and row counts are recorded for every column, so IsNull and IsNotNull stay safe.
+  override def buildFilter(
+      predicates: Seq[Expression],
+      cachedAttributes: Seq[Attribute]): (Int, Iterator[CachedBatch]) => Iterator[CachedBatch] = {
+    val prunable = cachedAttributes.collect {
+      case a if tracksBounds(a.dataType) => a.exprId
+    }.toSet
+
+    val prunablePredicates = predicates.filter {
+      case _: IsNull | _: IsNotNull => true
+      case p => p.references.forall(a => prunable.contains(a.exprId))
+    }
+
+    super.buildFilter(prunablePredicates, cachedAttributes)
+  }
+
+  // Comet's Arrow writer only handles the types listed in supportsSchema. Reporting false here
+  // sends the relation down the row path, where it is delegated to Spark's default serializer,
+  // instead of failing at cache materialization inside Utils.serializeBatches.
+  override def supportsColumnarInput(schema: Seq[Attribute]): Boolean = supportsSchema(schema)

Review Comment:
   During initial cache materialization, Spark calls `convertColumnarBatchToCachedBatch` with the cached plan’s original columnar output. If that plan is a Spark vectorized Parquet/ORC scan rather than a Comet scan, its vectors are not `CometVector`, so `Utils.getBatchFieldVectors` throws before any `CometCachedBatch` is created. Please either force the row-input path or convert generic Spark column vectors to Arrow, with a vectorized Parquet cache-fill regression.
   
   _Assist with LLM_



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "sandugood (via GitHub)" <gi...@apache.org>.
sandugood commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5122801192

   Tried out on the pipeline that relies heavily on caching and got an error:
   
   ```
   org.apache.spark.SparkException: Comet execution only takes Arrow Arrays, but got class org.apache.spark.sql.execution.vectorized.OffHeapColumnVector. This typically happens when a Comet scan falls back to Spark due to unsupported data types (e.g., complex types like structs, arrays, or maps). To resolve this, you can: (1) enable spark.comet.scan.allowIncompatible=true to use a compatible native scan variant, or (2) enable spark.comet.convert.parquet.enabled=true to convert Spark Parquet data to Arrow format automatically.
   	at org.apache.spark.sql.comet.util.Utils$.$anonfun$getBatchFieldVectors$1(Utils.scala:417)
   	at org.apache.spark.sql.comet.util.Utils$.$anonfun$getBatchFieldVectors$1$adapted(Utils.scala:403)
   	at scala.collection.immutable.Range.map(Range.scala:60)
   	at org.apache.spark.sql.comet.util.Utils$.getBatchFieldVectors(Utils.scala:403)
   	at org.apache.spark.sql.comet.util.Utils$.$anonfun$serializeBatches$1(Utils.scala:248)
   ```
   
   Note that I've tried to enable the suggested config insertions, but it didn't work either. 
   For context: pipeline has lots of fallbacks to default Spark operators. Also using Iceberg V2 as datasource with MOR tables.
   
   cc @andygrove 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5147440247

   @peterxcli the timezone question is settled in c25c7b32f — thanks for checking Spark's implementations, your recommendation was the right call and for a reason I had missed.
   
   The distinction that matters: Spark's Arrow cache can build `TimestampType` from `conf.sessionLocalTimeZone` **safely** because its RecordBatch is deliberately schema-less, so the label is never persisted and is reconstructed on read. `CometCachedBatch` stores a full IPC stream *including the schema*, so for us that label really does get written down — which makes a mutable session value materially worse here than the analogy with Spark suggests.
   
   So the row write path now passes `CometArrowStream.NATIVE_TIMEZONE`, matching the columnar path, which already did. Both of your predictions held:
   
   - **No values move** — Spark stores timestamps as micros since the Unix epoch regardless of session timezone. The test checks values *and* their `CAST(ts AS STRING)` rendering against Spark under two non-UTC session timezones.
   - **`TimestampNTZType` needed nothing** — `Utils.toArrowType` maps it to `Timestamp(MICROSECOND, null)` whatever is passed in.
   
   Small bonus: the closure no longer needs the session timezone, so it captures nothing derived from `conf`. That removes a live footgun — the original had to hoist `val sessionTz` outside `mapPartitions`, and when I moved the read inside while testing, it NPEd in `ConfigEntry.readString` on the executor.
   
   The new test caches a row-based plan (local `Seq`, so it goes through `convertInternalRowToCachedBatch`) under `America/Los_Angeles` and `Asia/Kolkata`, decodes the cached batches back through the serializer, and asserts the Arrow field timezone. I verified it is not vacuous: reverting the one-line change fails it with `got [America/Los_Angeles]`.
   
   All 20 tests in `CometInMemoryCacheSuite` pass on spark-3.5 (1 canceled, the pre-existing `isSpark40Plus` gate).
   
   That clears every inline thread on this PR except the AQE test coverage, which we agreed can be a follow-up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3677801321


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowConverters.scala:
##########
@@ -74,4 +74,45 @@ object CometArrowConverters extends Logging {
       }
     }
   }
+
+  /**
+   * Copy a Spark `ColumnarBatch` whose columns are not Arrow-backed (e.g.
+   * `On/OffHeapColumnVector` from Spark's vectorized Parquet reader, or a third-party connector's
+   * vectors) into a freshly allocated Arrow `ColumnarBatch` of `CometVector`s.
+   *
+   * The input batch is not consumed or closed; the caller owns the returned batch and must close
+   * it. Values are copied element-wise, since Spark's `ColumnVector` implementations do not
+   * expose Arrow buffers.
+   */
+  def columnarBatchToArrowBatch(

Review Comment:
   https://github.com/apache/datafusion-comet/blob/63c5a916b2e5626b040dc1f097384ab1a443bf91/spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/SparkColumnarArrowReader.scala#L27-L101
   
   I think we already have one?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3677931534


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet.execution.arrow
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, GenericInternalRow, IsNotNull, IsNull, UnsafeProjection}
+import org.apache.spark.sql.columnar.{CachedBatch, SimpleMetricsCachedBatch, SimpleMetricsCachedBatchSerializer}
+import org.apache.spark.sql.comet.util.Utils
+import org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.unsafe.types.{ByteArray, UTF8String}
+import org.apache.spark.util.io.ChunkedByteBuffer
+
+import org.apache.comet.CometArrowAllocator
+
+/**
+ * Cached batch format used when Comet writes Spark in-memory cache data.
+ *
+ * `bytes` contains compressed Arrow stream data produced by `Utils.serializeBatches`. The cache
+ * manager still owns storage and eviction; this class only changes the cached payload.
+ */
+private case class CometCachedBatch(
+    override val numRows: Int,
+    override val sizeInBytes: Long,
+    override val stats: InternalRow,
+    bytes: ChunkedByteBuffer)
+    extends SimpleMetricsCachedBatch
+
+/**
+ * Cache serializer that stores Comet-compatible Arrow batches in Spark's in-memory cache.
+ *
+ * The cached payload format is decided by the schema alone. A relation whose schema Comet's Arrow
+ * writer supports is stored as `CometCachedBatch`, and every other relation is delegated in full
+ * to Spark's `DefaultCachedBatchSerializer`. The format deliberately does not depend on any
+ * runtime config: `spark.sql.cache.serializer` is a static conf, so installing this serializer is
+ * already a per-application decision, and a relation whose format could flip mid-session cannot
+ * be read back reliably. `spark.comet.exec.inMemoryCache.enabled` still governs whether a scan
+ * over the cache runs natively, and its value at startup is what makes `CometDriverPlugin`
+ * install this serializer in the first place.
+ *
+ * Reads of `CometCachedBatch` keep working when the native scan is disabled, because Spark then
+ * reads the same cached data through the SparkToColumnar fallback path.
+ */
+class ArrowCachedBatchSerializer extends SimpleMetricsCachedBatchSerializer {
+
+  import ArrowCachedBatchSerializer.supportsSchema
+
+  private val fallback = new DefaultCachedBatchSerializer()
+
+  // Row-to-Arrow conversion needs a StructType, while cache APIs pass attributes.
+  private def toStructType(schema: Seq[Attribute]): StructType = {
+    StructType(schema.map { attr =>
+      StructField(attr.name, attr.dataType, attr.nullable, attr.metadata)
+    })
+  }
+
+  // Build the statistics row expected by SimpleMetricsCachedBatchSerializer.
+  // For each cached column Spark expects five values in this order:
+  // lower bound, upper bound, null count, row count, and size in bytes.
+  private def computeStats(batch: ColumnarBatch, attrs: Seq[Attribute]): InternalRow = {
+    val numCols = attrs.length
+    val lower = new Array[Any](numCols)
+    val upper = new Array[Any](numCols)
+    val nulls = Array.fill[Int](numCols)(0)
+    val numRows = batch.numRows()
+
+    var c = 0
+    while (c < numCols) {
+      val dt = attrs(c).dataType
+      val col = batch.column(c)
+      var r = 0
+      while (r < numRows) {
+        if (col.isNullAt(r)) {
+          nulls(c) += 1
+        } else if (tracksBounds(dt)) {
+          val value = readValue(col, dt, r)
+          if (lower(c) == null || compare(dt, value, lower(c)) < 0) {
+            lower(c) = value
+          }
+          if (upper(c) == null || compare(dt, value, upper(c)) > 0) {
+            upper(c) = value
+          }
+        }
+        r += 1
+      }
+      c += 1
+    }
+
+    val values = new Array[Any](numCols * 5)
+    c = 0
+    while (c < numCols) {
+      val base = c * 5
+      values(base) = lower(c)
+      values(base + 1) = upper(c)
+      values(base + 2) = nulls(c)
+      values(base + 3) = numRows
+      // Spark reserves the fifth field for per-column size. Comet stores the whole
+      // Arrow stream as one compressed buffer, so per-column size is not tracked here.
+      // Cache pruning uses bounds/null-count/row-count, not this size field.
+      values(base + 4) = 0L
+      c += 1
+    }
+
+    new GenericInternalRow(values)
+  }
+
+  // Spark can prune cache batches only for types whose bounds can be compared.
+  // Other types still report null count and row count but leave bounds as null.
+  private def tracksBounds(dt: DataType): Boolean = dt match {
+    case BooleanType | ByteType | ShortType | IntegerType | LongType | FloatType | DoubleType |
+        _: DecimalType | StringType | DateType | TimestampType | TimestampNTZType =>
+      true
+    case _ => false
+  }
+
+  // Read a non-null value from a ColumnVector using Spark's internal value type
+  // for the corresponding DataType.
+  private def readValue(col: ColumnVector, dt: DataType, rowId: Int): Any = dt match {
+    case BooleanType => col.getBoolean(rowId)
+    case ByteType => col.getByte(rowId)
+    case ShortType => col.getShort(rowId)
+    case IntegerType | DateType => col.getInt(rowId)
+    case LongType | TimestampType | TimestampNTZType => col.getLong(rowId)
+    case FloatType => col.getFloat(rowId)
+    case DoubleType => col.getDouble(rowId)
+    case d: DecimalType => col.getDecimal(rowId, d.precision, d.scale)
+    case StringType => col.getUTF8String(rowId).copy()
+    case _ => null
+  }
+
+  // Compare values using the same physical representation used in the stats row.
+  private def compare(dt: DataType, left: Any, right: Any): Int = dt match {
+    case BooleanType =>
+      java.lang.Boolean.compare(left.asInstanceOf[Boolean], right.asInstanceOf[Boolean])
+    case ByteType =>
+      java.lang.Byte.compare(left.asInstanceOf[Byte], right.asInstanceOf[Byte])
+    case ShortType =>
+      java.lang.Short.compare(left.asInstanceOf[Short], right.asInstanceOf[Short])
+    case IntegerType | DateType =>
+      java.lang.Integer.compare(left.asInstanceOf[Int], right.asInstanceOf[Int])
+    case LongType | TimestampType | TimestampNTZType =>
+      java.lang.Long.compare(left.asInstanceOf[Long], right.asInstanceOf[Long])
+    case FloatType =>
+      java.lang.Float.compare(left.asInstanceOf[Float], right.asInstanceOf[Float])
+    case DoubleType =>
+      java.lang.Double.compare(left.asInstanceOf[Double], right.asInstanceOf[Double])
+    case _: DecimalType =>
+      left.asInstanceOf[Decimal].compare(right.asInstanceOf[Decimal])
+    case StringType =>
+      ByteArray.compareBinary(
+        left.asInstanceOf[UTF8String].getBytes,
+        right.asInstanceOf[UTF8String].getBytes)
+    case other =>
+      throw new IllegalStateException(s"compare called for unsupported type $other")
+  }
+
+  // Compute Spark-compatible cache stats before serializing each batch to Arrow.
+  // The stats are stored beside the Arrow bytes so Spark's cache filter can prune
+  // CometCachedBatch without decoding the batch first.
+  private def encodeBatches(
+      batches: Iterator[ColumnarBatch],
+      attrs: Seq[Attribute]): Iterator[CachedBatch] = {
+    batches.flatMap { batch =>
+      val stats = computeStats(batch, attrs)
+
+      Utils.serializeBatches(Iterator.single(batch)).map { case (rows, buffer) =>
+        CometCachedBatch(
+          numRows = rows.toInt,
+          sizeInBytes = buffer.size,
+          stats = stats,
+          bytes = buffer)
+      }
+    }
+  }
+
+  // Resolve requested columns by exprId, not by name, because aliases may reuse names.
+  private def selectedIndices(
+      cacheAttributes: Seq[Attribute],
+      selectedAttributes: Seq[Attribute]): Array[Int] = {
+    if (selectedAttributes.isEmpty) {
+      cacheAttributes.indices.toArray
+    } else {
+      val byExprId = cacheAttributes.zipWithIndex.map { case (attr, idx) =>
+        attr.exprId -> idx
+      }.toMap
+
+      selectedAttributes.map { attr =>
+        byExprId.getOrElse(
+          attr.exprId,
+          throw new IllegalStateException(
+            s"Could not resolve selected attribute ${attr.name} from cache attributes"))
+      }.toArray
+    }
+  }
+
+  // A full-width projection is only an identity projection if every selected index
+  // is already in column order. For example, [1, 0] must still be projected.
+  private def isIdentityProjection(indices: Array[Int], numCols: Int): Boolean =
+    indices.length == numCols && indices.indices.forall(i => indices(i) == i)
+
+  private def projectBatch(batch: ColumnarBatch, indices: Array[Int]): ColumnarBatch = {
+    if (isIdentityProjection(indices, batch.numCols())) {
+      batch
+    } else {
+      val cols = indices.map(i => batch.column(i).asInstanceOf[ColumnVector])
+      new ColumnarBatch(cols, batch.numRows())
+    }
+  }
+
+  // Spark's SimpleMetricsCachedBatchSerializer prunes a batch when the generated partition filter
+  // does not evaluate to true against the stats row. Bounds are only computed for the types
+  // tracksBounds accepts, and for every other column the lower and upper bounds stay null, which
+  // makes a comparison against them evaluate to null and therefore prune the batch. That would
+  // silently drop rows, so predicates over columns without bounds are not pushed down at all.
+  // Null counts and row counts are recorded for every column, so IsNull and IsNotNull stay safe.
+  override def buildFilter(
+      predicates: Seq[Expression],
+      cachedAttributes: Seq[Attribute]): (Int, Iterator[CachedBatch]) => Iterator[CachedBatch] = {
+    val prunable = cachedAttributes.collect {
+      case a if tracksBounds(a.dataType) => a.exprId
+    }.toSet
+
+    val prunablePredicates = predicates.filter {
+      case _: IsNull | _: IsNotNull => true
+      case p => p.references.forall(a => prunable.contains(a.exprId))
+    }
+
+    super.buildFilter(prunablePredicates, cachedAttributes)
+  }
+
+  // Comet's Arrow writer only handles the types listed in supportsSchema. Reporting false here
+  // sends the relation down the row path, where it is delegated to Spark's default serializer,
+  // instead of failing at cache materialization inside Utils.serializeBatches.
+  override def supportsColumnarInput(schema: Seq[Attribute]): Boolean = supportsSchema(schema)

Review Comment:
   Confirmed, and thanks for catching this — your diagnosis is exactly right, including that it is the cause of the report in #5051 (comment).
   
   I reproduced it: with `spark.comet.scan.enabled=false` and `spark.comet.sparkToColumnar.enabled=false`, caching a Parquet table throws the same exception from `CachedRDDBuilder$$anon$2.next(InMemoryRelation.scala:277)`. The mechanism is slightly worse than "Spark can expose vectorized plans here": because `supportsColumnarInput` returns true, `InMemoryRelation.apply` actively calls `convertToColumnarIfPossible` and *strips* the `ColumnarToRow` above the cached plan (InMemoryRelation.scala:324-328), so any columnar leaf underneath becomes the serializer input.
   
   Of your two options I took the second (generic Spark-vector conversion) rather than returning `false`. Returning `false` is safe but gives up the columnar fast path for Comet-native cached plans, which is most of the point of the feature — the row path would force columnar -> row -> Arrow. Instead `encodeBatches` now checks `Utils.isArrowBacked` per batch and copies only foreign vectors into Arrow.
   
   The regression you asked for is in 9dce0d66, and there is a second one covering nullable array/struct/map/binary through the nested vectorized reader. I also documented the contract at `supportsColumnarInput` itself so the conversion does not read as deletable defensive code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3693347774


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowConverters.scala:
##########
@@ -74,4 +74,45 @@ object CometArrowConverters extends Logging {
       }
     }
   }
+
+  /**
+   * Copy a Spark `ColumnarBatch` whose columns are not Arrow-backed (e.g.
+   * `On/OffHeapColumnVector` from Spark's vectorized Parquet reader, or a third-party connector's
+   * vectors) into a freshly allocated Arrow `ColumnarBatch` of `CometVector`s.
+   *
+   * The input batch is not consumed or closed; the caller owns the returned batch and must close
+   * it. Values are copied element-wise, since Spark's `ColumnVector` implementations do not
+   * expose Arrow buffers.
+   */
+  def columnarBatchToArrowBatch(

Review Comment:
   Thanks for digging into Spark's implementations — that answers it, and I have taken your recommendation in c25c7b32f.
   
   Your framing of the difference is the key point and I had missed it: Spark's Arrow cache can construct `TimestampType` from `conf.sessionLocalTimeZone` safely *because* its RecordBatch is schema-less, so the label is never persisted and gets reconstructed on read. `CometCachedBatch` stores a full IPC stream including the schema, so for us that label really is written down — which makes using a mutable session value materially worse than it looks by analogy with Spark.
   
   So the row path now passes `CometArrowStream.NATIVE_TIMEZONE`, matching the columnar path, which already did. Confirmed the two things you predicted:
   
   - **No values move.** Spark's internal representation is micros since the Unix epoch regardless of session timezone, so this is a label-only change. The test checks both the values and their `CAST(ts AS STRING)` rendering against Spark under two non-UTC session timezones.
   - **`TimestampNTZType` needs nothing.** `Utils.toArrowType` maps it to `Timestamp(MICROSECOND, null)` whatever timezone is passed, so it was already timezone-free.
   
   A small bonus: the closure no longer needs the session timezone, so it no longer captures anything derived from `conf` at all. Worth noting because the original had to hoist `val sessionTz` outside `mapPartitions` — when I moved the read inside while testing, it NPEd in `ConfigEntry.readString` on the executor. That hazard is now gone.
   
   The test caches a row-based plan (local `Seq`, so `convertInternalRowToCachedBatch` rather than the columnar path) under `America/Los_Angeles` and `Asia/Kolkata`, decodes the cached batches back through the serializer, and asserts the Arrow field's timezone. I checked it is not vacuous: reverting the one-line change fails it with `got [America/Los_Angeles]`.



##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowConverters.scala:
##########
@@ -74,4 +74,45 @@ object CometArrowConverters extends Logging {
       }
     }
   }
+
+  /**
+   * Copy a Spark `ColumnarBatch` whose columns are not Arrow-backed (e.g.
+   * `On/OffHeapColumnVector` from Spark's vectorized Parquet reader, or a third-party connector's
+   * vectors) into a freshly allocated Arrow `ColumnarBatch` of `CometVector`s.
+   *
+   * The input batch is not consumed or closed; the caller owns the returned batch and must close
+   * it. Values are copied element-wise, since Spark's `ColumnVector` implementations do not
+   * expose Arrow buffers.
+   */
+  def columnarBatchToArrowBatch(

Review Comment:
   Agreed, and that is what I ended up doing — `SparkColumnarArrowReader.loadNextBatch` and `columnarBatchToArrowBatch` now share `CometArrowConverters.writeColumns` (c7bc49e5), so there is one copy loop rather than two. Marking this thread as settled; the remaining timezone question from it is answered on the sibling thread.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "sandugood (via GitHub)" <gi...@apache.org>.
sandugood commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5368373045

   Any updates?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3867237889


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowConverters.scala:
##########
@@ -74,4 +74,45 @@ object CometArrowConverters extends Logging {
       }
     }
   }
+
+  /**
+   * Copy a Spark `ColumnarBatch` whose columns are not Arrow-backed (e.g.
+   * `On/OffHeapColumnVector` from Spark's vectorized Parquet reader, or a third-party connector's
+   * vectors) into a freshly allocated Arrow `ColumnarBatch` of `CometVector`s.
+   *
+   * The input batch is not consumed or closed; the caller owns the returned batch and must close
+   * it. Values are copied element-wise, since Spark's `ColumnVector` implementations do not
+   * expose Arrow buffers.
+   */
+  def columnarBatchToArrowBatch(

Review Comment:
   Following up here because merging latest `apache/main` changed the outcome of this thread.
   
   The shared copy loop we settled on, `CometArrowConverters.writeColumns`, is gone. #5046 and #5442 landed on main in the meantime and moved that responsibility onto `ArrowWriter` itself as `ArrowWriter.writeColumns(input, startRow, numRows)`, with a bulk-copy path for fixed-width columns rather than the element-wise loop this PR had. So the deduplication you asked for now lives upstream of both callers, and I resolved the conflict by deleting this PR's version rather than keeping a second one: `SparkColumnarArrowReader.loadNextBatch` takes main's version wholesale, and `columnarBatchToArrowBatch` now calls `ArrowWriter.create(root, numRows)`, `writeColumns`, `finish()`.
   
   One detail worth recording, since it was a deliberate deletion rather than an oversight. This PR's `writeColumns` ended with an explicit `root.setRowCount(numRows)`, because `ArrowWriter` derived the root row count from per-column writes and a zero-column batch (Spark's count-from-metadata scan, `numRows > 0` with `numCols == 0`) would otherwise have produced `rowCount == 0` and silently dropped the rows. Main's `writeColumns` sets `count = numRows` directly, so `finish()` gets it right with no columns present and the workaround is no longer needed. `CometInMemoryCacheSuite` has a test for that exact case ("supports empty projection scan"), and it still passes, so this is covered rather than taken on trust.
   
   The leak guard from the other thread is unaffected and still wraps both call sites. All 21 tests in the suite pass on the merged branch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3867700659


##########
spark/src/main/scala/org/apache/spark/sql/comet/CometInMemoryTableScanExec.scala:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.columnar.{CachedBatch, CachedBatchSerializer}
+import org.apache.spark.sql.execution.LeafExecNode
+import org.apache.spark.sql.execution.columnar.InMemoryTableScanExec
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.CometConf
+import org.apache.comet.serde.CometOperatorSerde
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.serializeDataType
+
+/**
+ * Reads Spark cached table data when the cache was written by Comet's cache serializer.
+ *
+ * Spark stores cached data through `CachedBatchSerializer`. This node keeps the scan inside Comet
+ * by asking the serializer to decode cached batches directly into `ColumnarBatch` output,
+ * avoiding the extra Spark columnar-to-Comet columnar conversion used by the default path.
+ *
+ * `relationOutput` is the full schema stored in the cache. `scanOutput` is the subset requested
+ * by this scan after pruning.
+ */
+case class CometInMemoryTableScanExec(
+    originalPlan: InMemoryTableScanExec,
+    serializer: CachedBatchSerializer,
+    cachedBuffers: RDD[CachedBatch],
+    relationOutput: Seq[Attribute],
+    scanOutput: Seq[Attribute])
+    extends CometExec
+    with LeafExecNode {
+
+  override lazy val metrics: Map[String, SQLMetric] = Map(
+    "numOutputRows" -> SQLMetrics.createMetric(sparkContext, "number of output rows"))
+
+  // For an empty-projection scan (`SELECT count(*)`) this is empty while `scanOutput` holds one
+  // placeholder column, so the emitted batches are wider than the declared output. That is safe
+  // because the only consumer of an empty-output scan is a count-style aggregate, which reads the
+  // row count rather than any column; see `scanOutputFor` for why the scan cannot simply be empty.
+  override def output: Seq[Attribute] = originalPlan.output
+
+  // Use the serializer's vector types because the cached batch layout is owned by the serializer.
+  override def vectorTypes: Option[Seq[String]] =
+    serializer.vectorTypes(scanOutput, conf)
+
+  // Apply Spark's cache batch filter before decoding. Spark's InMemoryTableScanExec does this in
+  // filteredCachedBatches(), but that method is private. Reusing the serializer's buildFilter here
+  // keeps Comet on the same stats-based pruning path instead of decoding every cached batch.
+  //
+  // Gated on conf.inMemoryPartitionPruning the same way Spark's filteredCachedBatches is, so
+  // spark.sql.inMemoryColumnarStorage.partitionPruning=false disables pruning here too. Pruning is
+  // normally a win, but the config exists to be able to turn it off -- for debugging a suspected
+  // stats bug, for instance -- and silently ignoring it would make Comet diverge from Spark on a
+  // knob a user reaching for it is specifically trying to control.
+  override def doExecuteColumnar(): RDD[ColumnarBatch] = {
+    val numOutputRows = longMetric("numOutputRows")
+
+    val filteredBuffers =
+      if (originalPlan.predicates.nonEmpty && conf.inMemoryPartitionPruning) {
+        val filter = serializer.buildFilter(originalPlan.predicates, relationOutput)
+        cachedBuffers.mapPartitionsWithIndex(filter)
+      } else {
+        cachedBuffers
+      }
+
+    serializer
+      .convertCachedBatchToColumnarBatch(filteredBuffers, relationOutput, scanOutput, conf)
+      .map { cb =>
+        numOutputRows += cb.numRows()
+        cb
+      }
+  }
+}
+
+object CometInMemoryTableScanExec extends CometOperatorSerde[InMemoryTableScanExec] {
+
+  override def enabledConfig: Option[org.apache.comet.ConfigEntry[Boolean]] =
+    Some(CometConf.COMET_EXEC_IN_MEMORY_CACHE_ENABLED)
+
+  override def convert(
+      op: InMemoryTableScanExec,
+      builder: OperatorOuterClass.Operator.Builder,
+      childOp: Operator*): Option[Operator] = {
+
+    val scanTypes = scanOutputFor(op).flatMap(attr => serializeDataType(attr.dataType))
+
+    val scanBuilder = OperatorOuterClass.Scan
+      .newBuilder()
+      .setSource(op.getClass.getSimpleName)
+      .addAllFields(scanTypes.asJava)
+
+    Some(builder.setScan(scanBuilder).build())
+  }
+
+  // Reuse Spark's InMemoryRelation metadata so cache materialization, pruning, and storage
+  // behavior remain controlled by Spark's cache manager.
+  override def createExec(nativeOp: Operator, op: InMemoryTableScanExec): CometNativeExec = {
+    val relation = op.relation
+
+    CometScanWrapper(
+      nativeOp,
+      CometInMemoryTableScanExec(
+        op,
+        relation.cacheBuilder.serializer,
+        relation.cacheBuilder.cachedColumnBuffers,
+        relation.output,
+        scanOutputFor(op)))
+  }
+
+  /**
+   * Columns the cache scan asks the serializer to decode.
+   *
+   * An empty-output scan (`SELECT count(*)`) still needs a non-empty schema for native planning,
+   * and the batches the node emits have to match that schema. Falling back to the whole cache
+   * schema satisfies both, but the serializer decodes exactly what it is asked for, so the
+   * cheapest query in the workload would decode every cached column. One column is enough: the
+   * aggregate above an empty-output scan reads the row count and never a value, so pick the
+   * cheapest to decode rather than all of them.
+   *
+   * `convert` and `createExec` must choose identically, or the native scan's declared schema and
+   * the batches fed to it disagree.
+   */
+  private def scanOutputFor(op: InMemoryTableScanExec): Seq[Attribute] = {
+    if (op.output.nonEmpty) {
+      op.output
+    } else if (op.relation.output.isEmpty) {
+      Nil
+    } else {
+      Seq(op.relation.output.minBy(a => decodeCostRank(a.dataType)))

Review Comment:
   Confirmed and fixed in cb07053e3. I reproduced your case exactly: **66** with native caching on, **3** with it off, and the direct `r.id` projection failing with the column count mismatch.
   
   The premise the code rested on turned out to be false. The comment claimed the widening was needed because "the native plan still needs a non-empty scan schema", and I carried that forward when I narrowed the fallback from the whole cache schema to a single cheap column. It isn't true: passing `op.output` straight through, empty and all, plans and executes fine. So the fix is to delete the fallback rather than to make the extra column agree with `output`.
   
   Worth noting for the record that this predates my change. The previous code fell back to `relation.output`, and since `spark.range` has exactly one column, your repro produced an identical one-column scan before and after. My change altered how wide the leak was on wider relations, not whether it existed.
   
   Two things improve as a result. The join is correct, and `count(*)` now decodes nothing at all rather than one placeholder column, which is strictly better than what I had: `selectedIndices` keeps an empty selection empty, so the serializer short-circuits on row count alone.
   
   Regression test added as "joins correctly over an empty-output cache scan", covering both the aggregate and the direct projection, and the existing count-only test now asserts `scanOutput.isEmpty` rather than a placeholder column.
   



##########
spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala:
##########
@@ -268,6 +268,37 @@ object Utils extends CometTypeShim with Logging {
     }
   }
 
+  /**
+   * Serializes each column of `batch` into its own compressed Arrow IPC stream, in column order.
+   *
+   * [[serializeBatches]] writes one stream covering every column, so a reader has to inflate all
+   * of them before it can project. Comet's in-memory cache stores columns separately instead, so
+   * a scan decodes only the ones it selected. Each stream is self-contained, including its schema
+   * and any dictionaries the column needs.
+   *
+   * The row count is not recoverable from the result when `batch` has no columns, so callers keep
+   * it alongside. As with [[serializeBatches]], the batch's vectors are cleared once written.
+   */
+  def serializeBatchColumns(batch: ColumnarBatch): Array[ChunkedByteBuffer] = {
+    val (fieldVectors, batchProviderOpt) = getBatchFieldVectors(batch)
+    val provider = batchProviderOpt.getOrElse(new CDataDictionaryProvider)
+    val codec = CompressionCodec.createCodec(SparkEnv.get.conf)
+
+    fieldVectors.map { fieldVector =>
+      val cbbos = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)
+      val out = new DataOutputStream(codec.compressedOutputStream(cbbos))
+
+      val root = new VectorSchemaRoot(Seq(fieldVector).asJava)
+      val writer = new ArrowStreamWriter(root, provider, Channels.newChannel(out))
+      writer.start()
+      writer.writeBatch()

Review Comment:
   Correct, and this one is squarely mine: it became reachable when I split the payload into one stream per column. Before that every column came from a single reader and therefore shared a provider, so collapsing to the first one was harmless. With independent readers the ID namespaces collide.
   
   Fixed in cb07053e3. `Utils` gains `getBatchFieldVectorsWithProviders`, which pairs each field vector with the provider its own column was decoded with, and `serializeBatchColumns` uses that per column. `getBatchFieldVectors` keeps its existing signature and collapses to the first provider, so the shuffle and broadcast callers are unchanged.
   
   Regression test added, using your reproduction shape: cache a repartitioned frame with two low-cardinality string columns, then with `spark.comet.exec.enabled=false` and the native cache scan disabled, `first.union(first).cache().count()`, so Spark's columnar Union hands the decoded batches back to the serializer. I verified it fails without the fix, with the `Could not find dictionary with ID 1` you described.
   



##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala:
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet.execution.arrow
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.TaskContext
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, GenericInternalRow, IsNotNull, IsNull, UnsafeProjection}
+import org.apache.spark.sql.columnar.{CachedBatch, SimpleMetricsCachedBatch, SimpleMetricsCachedBatchSerializer}
+import org.apache.spark.sql.comet.util.Utils
+import org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.unsafe.types.{ByteArray, UTF8String}
+import org.apache.spark.util.io.ChunkedByteBuffer
+
+import org.apache.comet.CometArrowAllocator
+
+/**
+ * Cached batch format used when Comet writes Spark in-memory cache data.
+ *
+ * `columns` holds one compressed Arrow stream per cached column, in cache-schema order, produced
+ * by `Utils.serializeBatchColumns`. Storing columns separately is what lets a scan decode only
+ * the ones it projected; a single stream covering the whole batch would have to be inflated in
+ * full before any projection could be applied. The cache manager still owns storage and eviction;
+ * this class only changes the cached payload.
+ */
+private case class CometCachedBatch(
+    override val numRows: Int,
+    override val sizeInBytes: Long,
+    override val stats: InternalRow,
+    columns: Array[ChunkedByteBuffer])
+    extends SimpleMetricsCachedBatch
+
+/**
+ * Cache serializer that stores Comet-compatible Arrow batches in Spark's in-memory cache.
+ *
+ * The cached payload format is decided by the schema alone. A relation whose schema Comet's Arrow
+ * writer supports is stored as `CometCachedBatch`, and every other relation is delegated in full
+ * to Spark's `DefaultCachedBatchSerializer`. The format deliberately does not depend on any
+ * runtime config: `spark.sql.cache.serializer` is a static conf, so installing this serializer is
+ * already a per-application decision, and a relation whose format could flip mid-session cannot
+ * be read back reliably. `spark.comet.exec.inMemoryCache.enabled` still governs whether a scan
+ * over the cache runs natively, and its value at startup is what makes `CometDriverPlugin`
+ * install this serializer in the first place.
+ *
+ * Reads of `CometCachedBatch` keep working when the native scan is disabled, because Spark then
+ * reads the same cached data through the SparkToColumnar fallback path.
+ */
+class ArrowCachedBatchSerializer extends SimpleMetricsCachedBatchSerializer {
+
+  import ArrowCachedBatchSerializer.supportsSchema
+
+  private val fallback = new DefaultCachedBatchSerializer()
+
+  // Bounds and null counts per column, gathered before the batch is serialized: serializing
+  // clears the batch's vectors, and the per-column byte sizes that complete the statistics row
+  // are only known afterwards. See statsRow.
+  private def gatherColumnStats(
+      batch: ColumnarBatch,
+      attrs: Seq[Attribute]): (Array[Any], Array[Any], Array[Int]) = {
+    val numCols = attrs.length
+    val lower = new Array[Any](numCols)
+    val upper = new Array[Any](numCols)
+    val nulls = Array.fill[Int](numCols)(0)
+    val numRows = batch.numRows()
+
+    var c = 0
+    while (c < numCols) {
+      val dt = attrs(c).dataType
+      val col = batch.column(c)
+      var r = 0
+      while (r < numRows) {
+        if (col.isNullAt(r)) {
+          nulls(c) += 1
+        } else if (tracksBounds(dt)) {
+          val value = readValue(col, dt, r)
+          if (lower(c) == null || compare(dt, value, lower(c)) < 0) {
+            lower(c) = value
+          }
+          if (upper(c) == null || compare(dt, value, upper(c)) > 0) {
+            upper(c) = value
+          }
+        }
+        r += 1
+      }
+      c += 1
+    }
+
+    (lower, upper, nulls)
+  }
+
+  // Build the statistics row expected by SimpleMetricsCachedBatchSerializer.
+  // For each cached column Spark expects five values in this order:
+  // lower bound, upper bound, null count, row count, and size in bytes.
+  private def statsRow(
+      lower: Array[Any],
+      upper: Array[Any],
+      nulls: Array[Int],
+      numRows: Int,
+      columnSizes: Array[Long]): InternalRow = {
+    val numCols = lower.length
+    val values = new Array[Any](numCols * 5)
+    var c = 0
+    while (c < numCols) {
+      val base = c * 5
+      values(base) = lower(c)
+      values(base + 1) = upper(c)
+      values(base + 2) = nulls(c)
+      values(base + 3) = numRows
+      // Each column is its own compressed stream, so its size is known exactly. Cache pruning
+      // uses bounds/null-count/row-count rather than this field, but Spark reserves it and
+      // reports it, so record the real value.
+      values(base + 4) = columnSizes(c)
+      c += 1
+    }
+
+    new GenericInternalRow(values)
+  }
+
+  // Spark can prune cache batches only for types whose bounds can be compared.
+  // Other types still report null count and row count but leave bounds as null.
+  private def tracksBounds(dt: DataType): Boolean = dt match {
+    case BooleanType | ByteType | ShortType | IntegerType | LongType | FloatType | DoubleType |
+        _: DecimalType | StringType | DateType | TimestampType | TimestampNTZType =>
+      true
+    case _ => false
+  }
+
+  // Read a non-null value from a ColumnVector using Spark's internal value type
+  // for the corresponding DataType.
+  private def readValue(col: ColumnVector, dt: DataType, rowId: Int): Any = dt match {
+    case BooleanType => col.getBoolean(rowId)
+    case ByteType => col.getByte(rowId)
+    case ShortType => col.getShort(rowId)
+    case IntegerType | DateType => col.getInt(rowId)
+    case LongType | TimestampType | TimestampNTZType => col.getLong(rowId)
+    case FloatType => col.getFloat(rowId)
+    case DoubleType => col.getDouble(rowId)
+    case d: DecimalType => col.getDecimal(rowId, d.precision, d.scale)
+    case StringType => col.getUTF8String(rowId).copy()
+    case _ => null
+  }
+
+  // Compare values using the same physical representation used in the stats row.
+  private def compare(dt: DataType, left: Any, right: Any): Int = dt match {
+    case BooleanType =>
+      java.lang.Boolean.compare(left.asInstanceOf[Boolean], right.asInstanceOf[Boolean])
+    case ByteType =>
+      java.lang.Byte.compare(left.asInstanceOf[Byte], right.asInstanceOf[Byte])
+    case ShortType =>
+      java.lang.Short.compare(left.asInstanceOf[Short], right.asInstanceOf[Short])
+    case IntegerType | DateType =>
+      java.lang.Integer.compare(left.asInstanceOf[Int], right.asInstanceOf[Int])
+    case LongType | TimestampType | TimestampNTZType =>
+      java.lang.Long.compare(left.asInstanceOf[Long], right.asInstanceOf[Long])
+    case FloatType =>
+      java.lang.Float.compare(left.asInstanceOf[Float], right.asInstanceOf[Float])
+    case DoubleType =>
+      java.lang.Double.compare(left.asInstanceOf[Double], right.asInstanceOf[Double])
+    case _: DecimalType =>
+      left.asInstanceOf[Decimal].compare(right.asInstanceOf[Decimal])
+    case StringType =>
+      ByteArray.compareBinary(
+        left.asInstanceOf[UTF8String].getBytes,
+        right.asInstanceOf[UTF8String].getBytes)
+    case other =>
+      throw new IllegalStateException(s"compare called for unsupported type $other")
+  }
+
+  // Compute Spark-compatible cache stats before serializing each batch to Arrow.
+  // The stats are stored beside the Arrow bytes so Spark's cache filter can prune
+  // CometCachedBatch without decoding the batch first.
+  //
+  // A columnar input batch is not guaranteed to be Arrow-backed; see supportsColumnarInput for
+  // why. Batches that are not get copied into Arrow first, since Utils.serializeBatches only
+  // writes CometVector columns.
+  private def encodeBatches(
+      batches: Iterator[ColumnarBatch],
+      attrs: Seq[Attribute]): Iterator[CachedBatch] = {
+    val arrowSchema =
+      Utils.toArrowSchema(Utils.fromAttributes(attrs), CometArrowStream.NATIVE_TIMEZONE)
+
+    batches.map { batch =>
+      // Bounds and null counts are read from the input batch, which serializing then clears, so
+      // they have to be gathered first. The row is only assembled once the per-column sizes are
+      // known.
+      val (lower, upper, nulls) = gatherColumnStats(batch, attrs)
+      val numRows = batch.numRows()
+
+      val columns = if (Utils.isArrowBacked(batch)) {
+        Utils.serializeBatchColumns(batch)
+      } else {
+        val arrowBatch =
+          CometArrowConverters.columnarBatchToArrowBatch(batch, arrowSchema, CometArrowAllocator)
+        try Utils.serializeBatchColumns(arrowBatch)
+        finally arrowBatch.close()
+      }
+
+      val columnSizes = columns.map(_.size)
+      CometCachedBatch(
+        numRows = numRows,
+        sizeInBytes = columnSizes.sum,
+        stats = statsRow(lower, upper, nulls, numRows, columnSizes),
+        columns = columns)
+    }
+  }
+
+  // Resolve requested columns by exprId, not by name, because aliases may reuse names.
+  //
+  // An empty selection stays empty rather than expanding to every column. Spark asks for no
+  // columns when the query only needs the row count (SELECT count(*)), and since projection now
+  // decides what gets decoded, expanding it would turn the cheapest possible read into the most
+  // expensive one.
+  private def selectedIndices(
+      cacheAttributes: Seq[Attribute],
+      selectedAttributes: Seq[Attribute]): Array[Int] = {
+    val byExprId = cacheAttributes.zipWithIndex.map { case (attr, idx) =>
+      attr.exprId -> idx
+    }.toMap
+
+    selectedAttributes.map { attr =>
+      byExprId.getOrElse(
+        attr.exprId,
+        throw new IllegalStateException(
+          s"Could not resolve selected attribute ${attr.name} from cache attributes"))
+    }.toArray
+  }
+
+  // Spark's SimpleMetricsCachedBatchSerializer prunes a batch when the generated partition filter
+  // does not evaluate to true against the stats row. Bounds are only computed for the types
+  // tracksBounds accepts, and for every other column the lower and upper bounds stay null, which
+  // makes a comparison against them evaluate to null and therefore prune the batch. That would
+  // silently drop rows, so predicates over columns without bounds are not pushed down at all.
+  // Null counts and row counts are recorded for every column, so IsNull and IsNotNull stay safe.
+  override def buildFilter(
+      predicates: Seq[Expression],
+      cachedAttributes: Seq[Attribute]): (Int, Iterator[CachedBatch]) => Iterator[CachedBatch] = {
+    val prunable = cachedAttributes.collect {
+      case a if tracksBounds(a.dataType) => a.exprId
+    }.toSet
+
+    val prunablePredicates = predicates.filter {
+      case _: IsNull | _: IsNotNull => true
+      case p => p.references.forall(a => prunable.contains(a.exprId))
+    }
+
+    super.buildFilter(prunablePredicates, cachedAttributes)
+  }
+
+  // Comet's Arrow writer only handles the types listed in supportsSchema. Reporting false here
+  // sends the relation down the row path, where it is delegated to Spark's default serializer,
+  // instead of failing at cache materialization inside Utils.serializeBatches.
+  //
+  // This answer is schema-only, because attributes are all Spark gives us; it says nothing about
+  // the vectors. Returning true also makes InMemoryRelation strip the ColumnarToRow above the
+  // cached plan, so convertColumnarBatchToCachedBatch then receives whatever that plan produces:
+  // a Comet scan's CometVectors, but equally Spark's vectorized Parquet/ORC reader or a
+  // connector's own vectors. encodeBatches converts the non-Arrow ones; that conversion is load
+  // bearing, not defensive.
+  override def supportsColumnarInput(schema: Seq[Attribute]): Boolean = supportsSchema(schema)
+
+  // A relation Comet stores is always readable as columnar Arrow. Anything else holds
+  // DefaultCachedBatch, so defer to Spark, which only claims columnar output for the primitive
+  // types its ColumnAccessor.decompress path can actually decode.
+  override def supportsColumnarOutput(schema: StructType): Boolean = {
+    if (schema.fields.forall(f => ArrowCachedBatchSerializer.supportsType(f.dataType))) {
+      true
+    } else {
+      fallback.supportsColumnarOutput(schema)
+    }
+  }
+
+  // Columnar Comet output is stored as compressed Arrow stream bytes. Spark only calls this when
+  // supportsColumnarInput returned true, so the schema is known to be Comet-writable here.
+  override def convertColumnarBatchToCachedBatch(
+      input: RDD[ColumnarBatch],
+      schema: Seq[Attribute],
+      storageLevel: StorageLevel,
+      conf: SQLConf): RDD[CachedBatch] = {
+
+    input.mapPartitions { batches =>
+      encodeBatches(batches, schema)
+    }
+  }
+
+  override def convertCachedBatchToColumnarBatch(
+      input: RDD[CachedBatch],
+      cacheAttributes: Seq[Attribute],
+      selectedAttributes: Seq[Attribute],
+      conf: SQLConf): RDD[ColumnarBatch] = {
+    if (!supportsSchema(cacheAttributes)) {
+      return fallback.convertCachedBatchToColumnarBatch(
+        input,
+        cacheAttributes,
+        selectedAttributes,
+        conf)
+    }
+
+    val indices = selectedIndices(cacheAttributes, selectedAttributes)
+
+    input.mapPartitions { it =>
+      // A ColumnReaders closes its readers (releasing the vectors they are holding) only when the
+      // batch it produced has been consumed. A consumer that stops early -- LIMIT, take(), or a
+      // cancelled task -- leaves the readers for the batch in flight open, so close them on task
+      // completion. Spark's own ArrowCachedBatchSerializer registers a listener for the same
+      // reason.
+      //
+      // flatMap consumes each inner iterator fully before building the next, so at most one batch
+      // is open at a time and tracking the current one is enough. close() is idempotent, so
+      // closing one that already released itself is a no-op.
+      @volatile var current: ColumnReaders = null
+      Option(TaskContext.get()).foreach { tc =>
+        tc.addTaskCompletionListener[Unit] { _ =>
+          val readers = current
+          current = null
+          if (readers != null) {
+            readers.close()
+          }
+        }
+      }
+
+      it.flatMap {
+        case cb: CometCachedBatch =>
+          if (indices.isEmpty) {
+            // Nothing to decode: the row count is the whole answer, and it is already here.
+            Iterator.single(new ColumnarBatch(Array.empty[ColumnVector], cb.numRows))
+          } else {
+            val readers = new ColumnReaders(indices.map(i => cb.columns(i)), cb.numRows)
+            current = readers
+            readers.batches
+          }
+
+        case other =>
+          throw new IllegalStateException(
+            s"Unsupported cached batch type ${other.getClass.getName}")
+      }
+    }
+  }
+
+  // Decodes one selected column stream apiece and stitches the results back into a single batch.
+  //
+  // Each stream is self-contained, so the columns a scan did not select are never inflated. The
+  // decoded vectors stay owned by their readers: closing them releases the batch, which is why
+  // this yields a single-element iterator that closes on exhaustion, matching what
+  // ArrowReaderIterator did when the payload was one stream.
+  private class ColumnReaders(buffers: Array[ChunkedByteBuffer], numRows: Int) {
+    private val readers: Array[Iterator[ColumnarBatch]] =
+      buffers.map(Utils.decodeBatches(_, "CometCache"))

Review Comment:
   Confirmed and fixed in cb07053e3. Your reading of the mechanism is exactly right: `current` is assigned only after the constructor returns, so the listener has nothing to close, and `decodeBatches` allocates eagerly because `ArrowReaderIterator` decodes its first batch in its constructor.
   
   The readers are now opened in a loop guarded by `try`/`NonFatal` that closes everything already opened before rethrowing, with a failing `close` attached via `addSuppressed` rather than replacing the original failure, matching the guard used in `CometArrowConverters`.
   
   I tested it without needing an allocator limit, since `CometArrowAllocator` exposes its accounting: corrupt the second of two selected column streams, record `getAllocatedMemory`, attempt the read, and assert the allocation returns to the baseline. Without the fix that leaks 2048 bytes per attempt, which matches the growth you saw across repeated tasks.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3867702018


##########
spark/src/main/scala/org/apache/spark/sql/comet/CometInMemoryTableScanExec.scala:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.columnar.{CachedBatch, CachedBatchSerializer}
+import org.apache.spark.sql.execution.LeafExecNode
+import org.apache.spark.sql.execution.columnar.InMemoryTableScanExec
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.CometConf
+import org.apache.comet.serde.CometOperatorSerde
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.serializeDataType
+
+/**
+ * Reads Spark cached table data when the cache was written by Comet's cache serializer.
+ *
+ * Spark stores cached data through `CachedBatchSerializer`. This node keeps the scan inside Comet
+ * by asking the serializer to decode cached batches directly into `ColumnarBatch` output,
+ * avoiding the extra Spark columnar-to-Comet columnar conversion used by the default path.
+ *
+ * `relationOutput` is the full schema stored in the cache. `scanOutput` is the subset requested
+ * by this scan after pruning.
+ */
+case class CometInMemoryTableScanExec(
+    originalPlan: InMemoryTableScanExec,
+    serializer: CachedBatchSerializer,
+    cachedBuffers: RDD[CachedBatch],
+    relationOutput: Seq[Attribute],
+    scanOutput: Seq[Attribute])
+    extends CometExec
+    with LeafExecNode {
+
+  override lazy val metrics: Map[String, SQLMetric] = Map(
+    "numOutputRows" -> SQLMetrics.createMetric(sparkContext, "number of output rows"))
+
+  // For an empty-projection scan (`SELECT count(*)`) this is empty while `scanOutput` holds one
+  // placeholder column, so the emitted batches are wider than the declared output. That is safe
+  // because the only consumer of an empty-output scan is a count-style aggregate, which reads the
+  // row count rather than any column; see `scanOutputFor` for why the scan cannot simply be empty.
+  override def output: Seq[Attribute] = originalPlan.output
+
+  // Use the serializer's vector types because the cached batch layout is owned by the serializer.
+  override def vectorTypes: Option[Seq[String]] =
+    serializer.vectorTypes(scanOutput, conf)
+
+  // Apply Spark's cache batch filter before decoding. Spark's InMemoryTableScanExec does this in
+  // filteredCachedBatches(), but that method is private. Reusing the serializer's buildFilter here
+  // keeps Comet on the same stats-based pruning path instead of decoding every cached batch.
+  //
+  // Gated on conf.inMemoryPartitionPruning the same way Spark's filteredCachedBatches is, so
+  // spark.sql.inMemoryColumnarStorage.partitionPruning=false disables pruning here too. Pruning is
+  // normally a win, but the config exists to be able to turn it off -- for debugging a suspected
+  // stats bug, for instance -- and silently ignoring it would make Comet diverge from Spark on a
+  // knob a user reaching for it is specifically trying to control.
+  override def doExecuteColumnar(): RDD[ColumnarBatch] = {
+    val numOutputRows = longMetric("numOutputRows")
+
+    val filteredBuffers =
+      if (originalPlan.predicates.nonEmpty && conf.inMemoryPartitionPruning) {
+        val filter = serializer.buildFilter(originalPlan.predicates, relationOutput)
+        cachedBuffers.mapPartitionsWithIndex(filter)
+      } else {
+        cachedBuffers
+      }
+
+    serializer
+      .convertCachedBatchToColumnarBatch(filteredBuffers, relationOutput, scanOutput, conf)
+      .map { cb =>
+        numOutputRows += cb.numRows()
+        cb
+      }
+  }
+}
+
+object CometInMemoryTableScanExec extends CometOperatorSerde[InMemoryTableScanExec] {
+
+  override def enabledConfig: Option[org.apache.comet.ConfigEntry[Boolean]] =
+    Some(CometConf.COMET_EXEC_IN_MEMORY_CACHE_ENABLED)
+
+  override def convert(
+      op: InMemoryTableScanExec,
+      builder: OperatorOuterClass.Operator.Builder,
+      childOp: Operator*): Option[Operator] = {
+
+    val scanTypes = scanOutputFor(op).flatMap(attr => serializeDataType(attr.dataType))
+
+    val scanBuilder = OperatorOuterClass.Scan
+      .newBuilder()
+      .setSource(op.getClass.getSimpleName)
+      .addAllFields(scanTypes.asJava)
+
+    Some(builder.setScan(scanBuilder).build())
+  }
+
+  // Reuse Spark's InMemoryRelation metadata so cache materialization, pruning, and storage
+  // behavior remain controlled by Spark's cache manager.
+  override def createExec(nativeOp: Operator, op: InMemoryTableScanExec): CometNativeExec = {
+    val relation = op.relation
+
+    CometScanWrapper(
+      nativeOp,
+      CometInMemoryTableScanExec(
+        op,
+        relation.cacheBuilder.serializer,
+        relation.cacheBuilder.cachedColumnBuffers,

Review Comment:
   Fixed in cb07053e3. `CometInMemoryTableScanExec` now holds the `CachedRDDBuilder` instead of an `RDD[CachedBatch]`, and resolves `cachedColumnBuffers` inside `doExecuteColumnar()`.
   
   Regression test added. One note on how it asserts, because my first attempt was wrong in a way worth flagging: `isCachedColumnBuffersLoaded` is not the right signal, since it also requires the blocks to be populated, so it stayed false either way and the test passed with the fix reverted. The test now asserts on AQE finalization instead, which is the effect you actually described: the cached plan must still report `isFinalPlan=false` after `explain()`, and the query must still materialize normally when run. That version fails without the fix.
   
   This one predates my change as well, since the original constructor took the RDD too, but it is a real problem and worth fixing here.
   



##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala:
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet.execution.arrow
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.TaskContext
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, GenericInternalRow, IsNotNull, IsNull, UnsafeProjection}
+import org.apache.spark.sql.columnar.{CachedBatch, SimpleMetricsCachedBatch, SimpleMetricsCachedBatchSerializer}
+import org.apache.spark.sql.comet.util.Utils
+import org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.unsafe.types.{ByteArray, UTF8String}
+import org.apache.spark.util.io.ChunkedByteBuffer
+
+import org.apache.comet.CometArrowAllocator
+
+/**
+ * Cached batch format used when Comet writes Spark in-memory cache data.
+ *
+ * `columns` holds one compressed Arrow stream per cached column, in cache-schema order, produced
+ * by `Utils.serializeBatchColumns`. Storing columns separately is what lets a scan decode only
+ * the ones it projected; a single stream covering the whole batch would have to be inflated in
+ * full before any projection could be applied. The cache manager still owns storage and eviction;
+ * this class only changes the cached payload.
+ */
+private case class CometCachedBatch(
+    override val numRows: Int,
+    override val sizeInBytes: Long,
+    override val stats: InternalRow,
+    columns: Array[ChunkedByteBuffer])
+    extends SimpleMetricsCachedBatch
+
+/**
+ * Cache serializer that stores Comet-compatible Arrow batches in Spark's in-memory cache.
+ *
+ * The cached payload format is decided by the schema alone. A relation whose schema Comet's Arrow
+ * writer supports is stored as `CometCachedBatch`, and every other relation is delegated in full
+ * to Spark's `DefaultCachedBatchSerializer`. The format deliberately does not depend on any
+ * runtime config: `spark.sql.cache.serializer` is a static conf, so installing this serializer is
+ * already a per-application decision, and a relation whose format could flip mid-session cannot
+ * be read back reliably. `spark.comet.exec.inMemoryCache.enabled` still governs whether a scan
+ * over the cache runs natively, and its value at startup is what makes `CometDriverPlugin`
+ * install this serializer in the first place.
+ *
+ * Reads of `CometCachedBatch` keep working when the native scan is disabled, because Spark then
+ * reads the same cached data through the SparkToColumnar fallback path.
+ */
+class ArrowCachedBatchSerializer extends SimpleMetricsCachedBatchSerializer {
+
+  import ArrowCachedBatchSerializer.supportsSchema
+
+  private val fallback = new DefaultCachedBatchSerializer()
+
+  // Bounds and null counts per column, gathered before the batch is serialized: serializing
+  // clears the batch's vectors, and the per-column byte sizes that complete the statistics row
+  // are only known afterwards. See statsRow.
+  private def gatherColumnStats(
+      batch: ColumnarBatch,
+      attrs: Seq[Attribute]): (Array[Any], Array[Any], Array[Int]) = {
+    val numCols = attrs.length
+    val lower = new Array[Any](numCols)
+    val upper = new Array[Any](numCols)
+    val nulls = Array.fill[Int](numCols)(0)
+    val numRows = batch.numRows()
+
+    var c = 0
+    while (c < numCols) {
+      val dt = attrs(c).dataType
+      val col = batch.column(c)
+      var r = 0
+      while (r < numRows) {
+        if (col.isNullAt(r)) {
+          nulls(c) += 1
+        } else if (tracksBounds(dt)) {
+          val value = readValue(col, dt, r)
+          if (lower(c) == null || compare(dt, value, lower(c)) < 0) {
+            lower(c) = value
+          }
+          if (upper(c) == null || compare(dt, value, upper(c)) > 0) {
+            upper(c) = value
+          }
+        }
+        r += 1
+      }
+      c += 1
+    }
+
+    (lower, upper, nulls)
+  }
+
+  // Build the statistics row expected by SimpleMetricsCachedBatchSerializer.
+  // For each cached column Spark expects five values in this order:
+  // lower bound, upper bound, null count, row count, and size in bytes.
+  private def statsRow(
+      lower: Array[Any],
+      upper: Array[Any],
+      nulls: Array[Int],
+      numRows: Int,
+      columnSizes: Array[Long]): InternalRow = {
+    val numCols = lower.length
+    val values = new Array[Any](numCols * 5)
+    var c = 0
+    while (c < numCols) {
+      val base = c * 5
+      values(base) = lower(c)
+      values(base + 1) = upper(c)
+      values(base + 2) = nulls(c)
+      values(base + 3) = numRows
+      // Each column is its own compressed stream, so its size is known exactly. Cache pruning
+      // uses bounds/null-count/row-count rather than this field, but Spark reserves it and
+      // reports it, so record the real value.
+      values(base + 4) = columnSizes(c)
+      c += 1
+    }
+
+    new GenericInternalRow(values)
+  }
+
+  // Spark can prune cache batches only for types whose bounds can be compared.
+  // Other types still report null count and row count but leave bounds as null.
+  private def tracksBounds(dt: DataType): Boolean = dt match {
+    case BooleanType | ByteType | ShortType | IntegerType | LongType | FloatType | DoubleType |
+        _: DecimalType | StringType | DateType | TimestampType | TimestampNTZType =>
+      true
+    case _ => false
+  }
+
+  // Read a non-null value from a ColumnVector using Spark's internal value type
+  // for the corresponding DataType.
+  private def readValue(col: ColumnVector, dt: DataType, rowId: Int): Any = dt match {
+    case BooleanType => col.getBoolean(rowId)
+    case ByteType => col.getByte(rowId)
+    case ShortType => col.getShort(rowId)
+    case IntegerType | DateType => col.getInt(rowId)
+    case LongType | TimestampType | TimestampNTZType => col.getLong(rowId)
+    case FloatType => col.getFloat(rowId)
+    case DoubleType => col.getDouble(rowId)
+    case d: DecimalType => col.getDecimal(rowId, d.precision, d.scale)
+    case StringType => col.getUTF8String(rowId).copy()
+    case _ => null
+  }
+
+  // Compare values using the same physical representation used in the stats row.
+  private def compare(dt: DataType, left: Any, right: Any): Int = dt match {
+    case BooleanType =>
+      java.lang.Boolean.compare(left.asInstanceOf[Boolean], right.asInstanceOf[Boolean])
+    case ByteType =>
+      java.lang.Byte.compare(left.asInstanceOf[Byte], right.asInstanceOf[Byte])
+    case ShortType =>
+      java.lang.Short.compare(left.asInstanceOf[Short], right.asInstanceOf[Short])
+    case IntegerType | DateType =>
+      java.lang.Integer.compare(left.asInstanceOf[Int], right.asInstanceOf[Int])
+    case LongType | TimestampType | TimestampNTZType =>
+      java.lang.Long.compare(left.asInstanceOf[Long], right.asInstanceOf[Long])
+    case FloatType =>
+      java.lang.Float.compare(left.asInstanceOf[Float], right.asInstanceOf[Float])
+    case DoubleType =>
+      java.lang.Double.compare(left.asInstanceOf[Double], right.asInstanceOf[Double])
+    case _: DecimalType =>
+      left.asInstanceOf[Decimal].compare(right.asInstanceOf[Decimal])
+    case StringType =>
+      ByteArray.compareBinary(
+        left.asInstanceOf[UTF8String].getBytes,
+        right.asInstanceOf[UTF8String].getBytes)
+    case other =>
+      throw new IllegalStateException(s"compare called for unsupported type $other")
+  }
+
+  // Compute Spark-compatible cache stats before serializing each batch to Arrow.
+  // The stats are stored beside the Arrow bytes so Spark's cache filter can prune
+  // CometCachedBatch without decoding the batch first.
+  //
+  // A columnar input batch is not guaranteed to be Arrow-backed; see supportsColumnarInput for
+  // why. Batches that are not get copied into Arrow first, since Utils.serializeBatches only
+  // writes CometVector columns.
+  private def encodeBatches(
+      batches: Iterator[ColumnarBatch],
+      attrs: Seq[Attribute]): Iterator[CachedBatch] = {
+    val arrowSchema =
+      Utils.toArrowSchema(Utils.fromAttributes(attrs), CometArrowStream.NATIVE_TIMEZONE)
+
+    batches.map { batch =>
+      // Bounds and null counts are read from the input batch, which serializing then clears, so
+      // they have to be gathered first. The row is only assembled once the per-column sizes are
+      // known.
+      val (lower, upper, nulls) = gatherColumnStats(batch, attrs)
+      val numRows = batch.numRows()
+
+      val columns = if (Utils.isArrowBacked(batch)) {
+        Utils.serializeBatchColumns(batch)
+      } else {
+        val arrowBatch =
+          CometArrowConverters.columnarBatchToArrowBatch(batch, arrowSchema, CometArrowAllocator)
+        try Utils.serializeBatchColumns(arrowBatch)
+        finally arrowBatch.close()

Review Comment:
   Fixed in cb07053e3. The mismatch is exactly as you describe: `isArrowBacked` asked only whether every column is a `CometVector`, while `getFieldVector` accepts a specific vector list that excludes both large-offset representations, so `supportsColumnarInput` accepted the schema and materialization then threw.
   
   `Utils` now has `isSupportedFieldVector`, which answers the same question `getFieldVector` does without throwing to find out, and both use it: `getFieldVector` for its check, `isArrowBacked` to report false for a `CometVector` wrapping something it cannot write. Such batches take the conversion path, which rebuilds them against the Arrow schema derived from the Spark schema and so normalizes large offsets to `VarChar`/`VarBinary`.
   
   Regression test added in `UtilsSuite` at the level the fix lives, covering both `LargeVarCharVector` and `LargeVarBinaryVector`. I did not attempt the `mapInArrow` end-to-end path, matching the scope you tested.
   
   This is pre-existing rather than new here, since the previous single-stream writer went through the same `getBatchFieldVectors` call, but it is cheap to fix and the fix is contained.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3867821448


##########
spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala:
##########
@@ -398,33 +430,69 @@ object Utils extends CometTypeShim with Logging {
     }
   }
 
+  /**
+   * Whether every column in `batch` is an Arrow-backed `CometVector`, so [[getBatchFieldVectors]]
+   * can hand out its vectors directly. Callers that may receive batches from a plan they did not
+   * build (e.g. Comet's cache serializer, which Spark hands the cached plan's columnar output)
+   * use this to convert foreign vectors to Arrow instead of tripping the exception below.
+   *
+   * Stricter than what [[getBatchFieldVectors]] accepts: a `ConstantColumnVector` is rejected
+   * here even though that method materializes one, so such a batch takes the conversion path
+   * rather than being materialized column by column.
+   */
+  def isArrowBacked(batch: ColumnarBatch): Boolean =
+    (0 until batch.numCols()).forall { i =>
+      batch.column(i) match {
+        // Not every CometVector can be handed to getFieldVector: a CometPlainVector can wrap a
+        // LargeVarCharVector or LargeVarBinaryVector (an accelerated mapInArrow returning
+        // pa.large_string(), for instance), which it rejects. Answering true for those would
+        // send a batch down the direct write path that then fails, so check the vector itself
+        // and let the caller convert instead.
+        case v: CometVector => isSupportedFieldVector(v.getValueVector)
+        case _ => false
+      }
+    }
+
   def getBatchFieldVectors(
       batch: ColumnarBatch): (Seq[FieldVector], Option[DictionaryProvider]) = {
-    var provider: Option[DictionaryProvider] = None
+    val columns = getBatchFieldVectorsWithProviders(batch)
+    (columns.map(_._1), columns.flatMap(_._2).headOption)

Review Comment:
   **[P2] Preserve each column's dictionary when broadcasting cached batches**
   
   The per-column cache write fix works, but this whole-batch helper still discards every provider except the first. A native broadcast can consume `CometInMemoryTableScanExec` directly after the wrapper is removed, so `CometBroadcastExchangeExec.getByteArrayRdd` passes a batch with independent providers to `Utils.serializeBatches`.
   
   I reproduced this with Comet caching, JVM shuffle and AQE disabled: cache a repartitioned DataFrame with two low-cardinality string columns, then run `SELECT /*+ BROADCAST(c) */ c.a, c.b FROM range(1) r JOIN dictionary_cache c ON true`. It fails with `Could not find dictionary with ID 1`. Disabling only `spark.comet.exec.inMemoryCache.enabled` lets the same query over the same cache return all 10,000 rows. Please normalize or safely combine the dictionaries for whole-batch serialization as well as the per-column writer.



##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala:
##########
@@ -0,0 +1,539 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet.execution.arrow
+
+import scala.collection.JavaConverters._
+import scala.util.control.NonFatal
+
+import org.apache.spark.TaskContext
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, GenericInternalRow, IsNotNull, IsNull, UnsafeProjection}
+import org.apache.spark.sql.columnar.{CachedBatch, SimpleMetricsCachedBatch, SimpleMetricsCachedBatchSerializer}
+import org.apache.spark.sql.comet.util.Utils
+import org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.unsafe.types.{ByteArray, UTF8String}
+import org.apache.spark.util.io.ChunkedByteBuffer
+
+import org.apache.comet.CometArrowAllocator
+
+/**
+ * Cached batch format used when Comet writes Spark in-memory cache data.
+ *
+ * `columns` holds one compressed Arrow stream per cached column, in cache-schema order, produced
+ * by `Utils.serializeBatchColumns`. Storing columns separately is what lets a scan decode only
+ * the ones it projected; a single stream covering the whole batch would have to be inflated in
+ * full before any projection could be applied. The cache manager still owns storage and eviction;
+ * this class only changes the cached payload.
+ */
+private case class CometCachedBatch(
+    override val numRows: Int,
+    override val sizeInBytes: Long,
+    override val stats: InternalRow,
+    columns: Array[ChunkedByteBuffer])
+    extends SimpleMetricsCachedBatch
+
+/**
+ * Cache serializer that stores Comet-compatible Arrow batches in Spark's in-memory cache.
+ *
+ * The cached payload format is decided by the schema alone. A relation whose schema Comet's Arrow
+ * writer supports is stored as `CometCachedBatch`, and every other relation is delegated in full
+ * to Spark's `DefaultCachedBatchSerializer`. The format deliberately does not depend on any
+ * runtime config: `spark.sql.cache.serializer` is a static conf, so installing this serializer is
+ * already a per-application decision, and a relation whose format could flip mid-session cannot
+ * be read back reliably. `spark.comet.exec.inMemoryCache.enabled` still governs whether a scan
+ * over the cache runs natively, and its value at startup is what makes `CometDriverPlugin`
+ * install this serializer in the first place.
+ *
+ * Reads of `CometCachedBatch` keep working when the native scan is disabled, because Spark then
+ * reads the same cached data through the SparkToColumnar fallback path.
+ */
+class ArrowCachedBatchSerializer extends SimpleMetricsCachedBatchSerializer {
+
+  import ArrowCachedBatchSerializer.supportsSchema
+
+  private val fallback = new DefaultCachedBatchSerializer()
+
+  // Bounds and null counts per column, gathered before the batch is serialized: serializing
+  // clears the batch's vectors, and the per-column byte sizes that complete the statistics row
+  // are only known afterwards. See statsRow.
+  private def gatherColumnStats(
+      batch: ColumnarBatch,
+      attrs: Seq[Attribute]): (Array[Any], Array[Any], Array[Int]) = {
+    val numCols = attrs.length
+    val lower = new Array[Any](numCols)
+    val upper = new Array[Any](numCols)
+    val nulls = Array.fill[Int](numCols)(0)
+    val numRows = batch.numRows()
+
+    var c = 0
+    while (c < numCols) {
+      val dt = attrs(c).dataType
+      val col = batch.column(c)
+      var r = 0
+      while (r < numRows) {
+        if (col.isNullAt(r)) {
+          nulls(c) += 1
+        } else if (tracksBounds(dt)) {
+          val value = readValue(col, dt, r)
+          if (lower(c) == null || compare(dt, value, lower(c)) < 0) {
+            lower(c) = value
+          }
+          if (upper(c) == null || compare(dt, value, upper(c)) > 0) {
+            upper(c) = value
+          }
+        }
+        r += 1
+      }
+      c += 1
+    }
+
+    (lower, upper, nulls)
+  }
+
+  // Build the statistics row expected by SimpleMetricsCachedBatchSerializer.
+  // For each cached column Spark expects five values in this order:
+  // lower bound, upper bound, null count, row count, and size in bytes.
+  private def statsRow(
+      lower: Array[Any],
+      upper: Array[Any],
+      nulls: Array[Int],
+      numRows: Int,
+      columnSizes: Array[Long]): InternalRow = {
+    val numCols = lower.length
+    val values = new Array[Any](numCols * 5)
+    var c = 0
+    while (c < numCols) {
+      val base = c * 5
+      values(base) = lower(c)
+      values(base + 1) = upper(c)
+      values(base + 2) = nulls(c)
+      values(base + 3) = numRows
+      // Each column is its own compressed stream, so its size is known exactly. Cache pruning
+      // uses bounds/null-count/row-count rather than this field, but Spark reserves it and
+      // reports it, so record the real value.
+      values(base + 4) = columnSizes(c)
+      c += 1
+    }
+
+    new GenericInternalRow(values)
+  }
+
+  // Spark can prune cache batches only for types whose bounds can be compared.
+  // Other types still report null count and row count but leave bounds as null.
+  private def tracksBounds(dt: DataType): Boolean = dt match {
+    case BooleanType | ByteType | ShortType | IntegerType | LongType | FloatType | DoubleType |
+        _: DecimalType | StringType | DateType | TimestampType | TimestampNTZType =>
+      true
+    case _ => false
+  }
+
+  // Read a non-null value from a ColumnVector using Spark's internal value type
+  // for the corresponding DataType.
+  private def readValue(col: ColumnVector, dt: DataType, rowId: Int): Any = dt match {
+    case BooleanType => col.getBoolean(rowId)
+    case ByteType => col.getByte(rowId)
+    case ShortType => col.getShort(rowId)
+    case IntegerType | DateType => col.getInt(rowId)
+    case LongType | TimestampType | TimestampNTZType => col.getLong(rowId)
+    case FloatType => col.getFloat(rowId)
+    case DoubleType => col.getDouble(rowId)
+    case d: DecimalType => col.getDecimal(rowId, d.precision, d.scale)
+    case StringType => col.getUTF8String(rowId).copy()
+    case _ => null
+  }
+
+  // Compare values using the same physical representation used in the stats row.
+  private def compare(dt: DataType, left: Any, right: Any): Int = dt match {
+    case BooleanType =>
+      java.lang.Boolean.compare(left.asInstanceOf[Boolean], right.asInstanceOf[Boolean])
+    case ByteType =>
+      java.lang.Byte.compare(left.asInstanceOf[Byte], right.asInstanceOf[Byte])
+    case ShortType =>
+      java.lang.Short.compare(left.asInstanceOf[Short], right.asInstanceOf[Short])
+    case IntegerType | DateType =>
+      java.lang.Integer.compare(left.asInstanceOf[Int], right.asInstanceOf[Int])
+    case LongType | TimestampType | TimestampNTZType =>
+      java.lang.Long.compare(left.asInstanceOf[Long], right.asInstanceOf[Long])
+    case FloatType =>
+      java.lang.Float.compare(left.asInstanceOf[Float], right.asInstanceOf[Float])
+    case DoubleType =>
+      java.lang.Double.compare(left.asInstanceOf[Double], right.asInstanceOf[Double])
+    case _: DecimalType =>
+      left.asInstanceOf[Decimal].compare(right.asInstanceOf[Decimal])
+    case StringType =>
+      ByteArray.compareBinary(
+        left.asInstanceOf[UTF8String].getBytes,
+        right.asInstanceOf[UTF8String].getBytes)
+    case other =>
+      throw new IllegalStateException(s"compare called for unsupported type $other")
+  }
+
+  // Compute Spark-compatible cache stats before serializing each batch to Arrow.
+  // The stats are stored beside the Arrow bytes so Spark's cache filter can prune
+  // CometCachedBatch without decoding the batch first.
+  //
+  // A columnar input batch is not guaranteed to be Arrow-backed; see supportsColumnarInput for
+  // why. Batches that are not get copied into Arrow first, since Utils.serializeBatches only
+  // writes CometVector columns.
+  private def encodeBatches(
+      batches: Iterator[ColumnarBatch],
+      attrs: Seq[Attribute]): Iterator[CachedBatch] = {
+    val arrowSchema =
+      Utils.toArrowSchema(Utils.fromAttributes(attrs), CometArrowStream.NATIVE_TIMEZONE)
+
+    batches.map { batch =>
+      // Bounds and null counts are read from the input batch, which serializing then clears, so
+      // they have to be gathered first. The row is only assembled once the per-column sizes are
+      // known.
+      val (lower, upper, nulls) = gatherColumnStats(batch, attrs)
+      val numRows = batch.numRows()
+
+      val columns = if (Utils.isArrowBacked(batch)) {
+        Utils.serializeBatchColumns(batch)
+      } else {
+        val arrowBatch =
+          CometArrowConverters.columnarBatchToArrowBatch(batch, arrowSchema, CometArrowAllocator)
+        try Utils.serializeBatchColumns(arrowBatch)
+        finally arrowBatch.close()
+      }
+
+      val columnSizes = columns.map(_.size)
+      CometCachedBatch(
+        numRows = numRows,
+        sizeInBytes = columnSizes.sum,
+        stats = statsRow(lower, upper, nulls, numRows, columnSizes),
+        columns = columns)
+    }
+  }
+
+  // Resolve requested columns by exprId, not by name, because aliases may reuse names.
+  //
+  // An empty selection stays empty rather than expanding to every column. Spark asks for no
+  // columns when the query only needs the row count (SELECT count(*)), and since projection now
+  // decides what gets decoded, expanding it would turn the cheapest possible read into the most
+  // expensive one.
+  private def selectedIndices(
+      cacheAttributes: Seq[Attribute],
+      selectedAttributes: Seq[Attribute]): Array[Int] = {
+    val byExprId = cacheAttributes.zipWithIndex.map { case (attr, idx) =>
+      attr.exprId -> idx
+    }.toMap
+
+    selectedAttributes.map { attr =>
+      byExprId.getOrElse(
+        attr.exprId,
+        throw new IllegalStateException(
+          s"Could not resolve selected attribute ${attr.name} from cache attributes"))
+    }.toArray
+  }
+
+  // Spark's SimpleMetricsCachedBatchSerializer prunes a batch when the generated partition filter
+  // does not evaluate to true against the stats row. Bounds are only computed for the types
+  // tracksBounds accepts, and for every other column the lower and upper bounds stay null, which
+  // makes a comparison against them evaluate to null and therefore prune the batch. That would
+  // silently drop rows, so predicates over columns without bounds are not pushed down at all.
+  // Null counts and row counts are recorded for every column, so IsNull and IsNotNull stay safe.
+  override def buildFilter(
+      predicates: Seq[Expression],
+      cachedAttributes: Seq[Attribute]): (Int, Iterator[CachedBatch]) => Iterator[CachedBatch] = {
+    val prunable = cachedAttributes.collect {
+      case a if tracksBounds(a.dataType) => a.exprId
+    }.toSet
+
+    val prunablePredicates = predicates.filter {
+      case _: IsNull | _: IsNotNull => true
+      case p => p.references.forall(a => prunable.contains(a.exprId))
+    }
+
+    super.buildFilter(prunablePredicates, cachedAttributes)
+  }
+
+  // Comet's Arrow writer only handles the types listed in supportsSchema. Reporting false here
+  // sends the relation down the row path, where it is delegated to Spark's default serializer,
+  // instead of failing at cache materialization inside Utils.serializeBatches.
+  //
+  // This answer is schema-only, because attributes are all Spark gives us; it says nothing about
+  // the vectors. Returning true also makes InMemoryRelation strip the ColumnarToRow above the
+  // cached plan, so convertColumnarBatchToCachedBatch then receives whatever that plan produces:
+  // a Comet scan's CometVectors, but equally Spark's vectorized Parquet/ORC reader or a
+  // connector's own vectors. encodeBatches converts the non-Arrow ones; that conversion is load
+  // bearing, not defensive.
+  override def supportsColumnarInput(schema: Seq[Attribute]): Boolean = supportsSchema(schema)
+
+  // A relation Comet stores is always readable as columnar Arrow. Anything else holds
+  // DefaultCachedBatch, so defer to Spark, which only claims columnar output for the primitive
+  // types its ColumnAccessor.decompress path can actually decode.
+  override def supportsColumnarOutput(schema: StructType): Boolean = {
+    if (schema.fields.forall(f => ArrowCachedBatchSerializer.supportsType(f.dataType))) {
+      true
+    } else {
+      fallback.supportsColumnarOutput(schema)
+    }
+  }
+
+  // Columnar Comet output is stored as compressed Arrow stream bytes. Spark only calls this when
+  // supportsColumnarInput returned true, so the schema is known to be Comet-writable here.
+  override def convertColumnarBatchToCachedBatch(
+      input: RDD[ColumnarBatch],
+      schema: Seq[Attribute],
+      storageLevel: StorageLevel,
+      conf: SQLConf): RDD[CachedBatch] = {
+
+    input.mapPartitions { batches =>
+      encodeBatches(batches, schema)
+    }
+  }
+
+  override def convertCachedBatchToColumnarBatch(
+      input: RDD[CachedBatch],
+      cacheAttributes: Seq[Attribute],
+      selectedAttributes: Seq[Attribute],
+      conf: SQLConf): RDD[ColumnarBatch] = {
+    if (!supportsSchema(cacheAttributes)) {
+      return fallback.convertCachedBatchToColumnarBatch(
+        input,
+        cacheAttributes,
+        selectedAttributes,
+        conf)
+    }
+
+    val indices = selectedIndices(cacheAttributes, selectedAttributes)
+
+    input.mapPartitions { it =>
+      // A ColumnReaders closes its readers (releasing the vectors they are holding) only when the
+      // batch it produced has been consumed. A consumer that stops early -- LIMIT, take(), or a
+      // cancelled task -- leaves the readers for the batch in flight open, so close them on task
+      // completion. Spark's own ArrowCachedBatchSerializer registers a listener for the same
+      // reason.
+      //
+      // flatMap consumes each inner iterator fully before building the next, so at most one batch
+      // is open at a time and tracking the current one is enough. close() is idempotent, so
+      // closing one that already released itself is a no-op.
+      @volatile var current: ColumnReaders = null
+      Option(TaskContext.get()).foreach { tc =>
+        tc.addTaskCompletionListener[Unit] { _ =>
+          val readers = current
+          current = null
+          if (readers != null) {
+            readers.close()
+          }
+        }
+      }
+
+      it.flatMap {
+        case cb: CometCachedBatch =>
+          if (indices.isEmpty) {
+            // Nothing to decode: the row count is the whole answer, and it is already here.
+            Iterator.single(new ColumnarBatch(Array.empty[ColumnVector], cb.numRows))
+          } else {
+            val readers = new ColumnReaders(indices.map(i => cb.columns(i)), cb.numRows)
+            current = readers
+            readers.batches
+          }
+
+        case other =>
+          throw new IllegalStateException(
+            s"Unsupported cached batch type ${other.getClass.getName}")
+      }
+    }
+  }
+
+  // Decodes one selected column stream apiece and stitches the results back into a single batch.
+  //
+  // Each stream is self-contained, so the columns a scan did not select are never inflated. The
+  // decoded vectors stay owned by their readers: closing them releases the batch, which is why
+  // this yields a single-element iterator that closes on exhaustion, matching what
+  // ArrowReaderIterator did when the payload was one stream.
+  private class ColumnReaders(buffers: Array[ChunkedByteBuffer], numRows: Int) {
+    // decodeBatches opens a reader and eagerly decodes its first batch, so it allocates. If a
+    // later column throws, the readers already opened here are unreachable: the task-completion
+    // listener cannot release them because `current` is only assigned once this constructor
+    // returns, so they would leak off-heap for the life of the executor.
+    private val readers: Array[Iterator[ColumnarBatch]] = {
+      val opened = new Array[Iterator[ColumnarBatch]](buffers.length)
+      var i = 0
+      try {
+        while (i < buffers.length) {
+          opened(i) = Utils.decodeBatches(buffers(i), "CometCache")

Review Comment:
   **[P2] Close the reader whose own initialization fails**
   
   The new guard correctly closes previously returned readers, but `decodeBatches` can allocate a dictionary and then throw while loading that same column's first record batch. In that case it never returns a reader to `opened(i)`, and neither this catch nor the task-completion listener can close it.
   
   With one valid dictionary-encoded cached string column, I limited the allocator to allow its dictionary message but reject the following indices buffer. After three completed failed tasks, retained Arrow allocations grew from zero to 64, 128 and 192 bytes. The original two-plain-column reproduction now returns to zero. Please make reader initialization exception-safe inside `ArrowReaderIterator`/the reader factory too, so a failed first batch releases that reader's own allocations.



##########
spark/src/main/scala/org/apache/spark/sql/comet/CometInMemoryTableScanExec.scala:
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.columnar.CachedBatchSerializer
+import org.apache.spark.sql.execution.LeafExecNode
+import org.apache.spark.sql.execution.columnar.{CachedRDDBuilder, InMemoryTableScanExec}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.CometConf
+import org.apache.comet.serde.CometOperatorSerde
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.serializeDataType
+
+/**
+ * Reads Spark cached table data when the cache was written by Comet's cache serializer.
+ *
+ * Spark stores cached data through `CachedBatchSerializer`. This node keeps the scan inside Comet
+ * by asking the serializer to decode cached batches directly into `ColumnarBatch` output,
+ * avoiding the extra Spark columnar-to-Comet columnar conversion used by the default path.
+ *
+ * `relationOutput` is the full schema stored in the cache. `scanOutput` is the subset requested
+ * by this scan after pruning.
+ */
+case class CometInMemoryTableScanExec(
+    originalPlan: InMemoryTableScanExec,
+    serializer: CachedBatchSerializer,
+    cacheBuilder: CachedRDDBuilder,

Review Comment:
   **[P2] Canonicalize the embedded scan to preserve exchange reuse**
   
   This leaf extends `CometExec`, not `CometNativeExec`, so it does not get the latter's canonicalization of embedded plan fields. `originalPlan` is not a child and retains its expression IDs even after the Comet scan is canonicalized. Equivalent references to the same cache therefore fail `sameResult`.
   
   With AQE disabled, a UNION of two identical grouped aggregates over one cache produces one shuffle exchange plus `ReusedExchange` when native cache scanning is disabled, but two independent shuffle exchanges when enabled. Repeating a broadcast join has the same one-versus-two behavior. In the canonical scans, `cacheBuilder`, `relationOutput` and `scanOutput` compare equal; only `originalPlan` differs, and canonicalizing that field restores equality. Please canonicalize the wrapped scan while preserving its predicates and source semantics to avoid repeating scans, aggregation, shuffle and broadcast work.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3872507896


##########
spark/src/main/scala/org/apache/spark/sql/comet/CometInMemoryTableScanExec.scala:
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.columnar.CachedBatchSerializer
+import org.apache.spark.sql.execution.LeafExecNode
+import org.apache.spark.sql.execution.columnar.{CachedRDDBuilder, InMemoryTableScanExec}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.CometConf
+import org.apache.comet.serde.CometOperatorSerde
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.serializeDataType
+
+/**
+ * Reads Spark cached table data when the cache was written by Comet's cache serializer.
+ *
+ * Spark stores cached data through `CachedBatchSerializer`. This node keeps the scan inside Comet
+ * by asking the serializer to decode cached batches directly into `ColumnarBatch` output,
+ * avoiding the extra Spark columnar-to-Comet columnar conversion used by the default path.
+ *
+ * `relationOutput` is the full schema stored in the cache. `scanOutput` is the subset requested
+ * by this scan after pruning.
+ */
+case class CometInMemoryTableScanExec(
+    originalPlan: InMemoryTableScanExec,
+    serializer: CachedBatchSerializer,
+    cacheBuilder: CachedRDDBuilder,

Review Comment:
   Fixed in 70f046abf. `CometInMemoryTableScanExec` now overrides `doCanonicalize` and canonicalizes `originalPlan` by deferring to `InMemoryTableScanExec`, which normalizes its own attributes and predicates against the relation's output. `relationOutput` and `scanOutput` were already normalized by `QueryPlan`, and `cacheBuilder` and `serializer` are the same instances, so that one field was the whole difference.
   
   Regression test: `Comet in-memory cache scans of one cache canonicalize equal, so exchanges are reused`, using your UNION-of-two-aggregates shape. It asserts one exchange plus one `ReusedExchangeExec`; reverted, it finds two exchanges and no reuse.
   
   It also asserts that two scans differing only in their pushed predicates still canonicalize unequal. That is the reason for deferring rather than nulling the field the way `CometNativeExec.canonicalizePlans` does for embedded plans, which would have equated those as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "sandugood (via GitHub)" <gi...@apache.org>.
sandugood commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3677755637


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet.execution.arrow
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, GenericInternalRow, IsNotNull, IsNull, UnsafeProjection}
+import org.apache.spark.sql.columnar.{CachedBatch, SimpleMetricsCachedBatch, SimpleMetricsCachedBatchSerializer}
+import org.apache.spark.sql.comet.util.Utils
+import org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.unsafe.types.{ByteArray, UTF8String}
+import org.apache.spark.util.io.ChunkedByteBuffer
+
+import org.apache.comet.CometArrowAllocator
+
+/**
+ * Cached batch format used when Comet writes Spark in-memory cache data.
+ *
+ * `bytes` contains compressed Arrow stream data produced by `Utils.serializeBatches`. The cache
+ * manager still owns storage and eviction; this class only changes the cached payload.
+ */
+private case class CometCachedBatch(
+    override val numRows: Int,
+    override val sizeInBytes: Long,
+    override val stats: InternalRow,
+    bytes: ChunkedByteBuffer)
+    extends SimpleMetricsCachedBatch
+
+/**
+ * Cache serializer that stores Comet-compatible Arrow batches in Spark's in-memory cache.
+ *
+ * The cached payload format is decided by the schema alone. A relation whose schema Comet's Arrow
+ * writer supports is stored as `CometCachedBatch`, and every other relation is delegated in full
+ * to Spark's `DefaultCachedBatchSerializer`. The format deliberately does not depend on any
+ * runtime config: `spark.sql.cache.serializer` is a static conf, so installing this serializer is
+ * already a per-application decision, and a relation whose format could flip mid-session cannot
+ * be read back reliably. `spark.comet.exec.inMemoryCache.enabled` still governs whether a scan
+ * over the cache runs natively, and its value at startup is what makes `CometDriverPlugin`
+ * install this serializer in the first place.
+ *
+ * Reads of `CometCachedBatch` keep working when the native scan is disabled, because Spark then
+ * reads the same cached data through the SparkToColumnar fallback path.
+ */
+class ArrowCachedBatchSerializer extends SimpleMetricsCachedBatchSerializer {
+
+  import ArrowCachedBatchSerializer.supportsSchema
+
+  private val fallback = new DefaultCachedBatchSerializer()
+
+  // Row-to-Arrow conversion needs a StructType, while cache APIs pass attributes.
+  private def toStructType(schema: Seq[Attribute]): StructType = {
+    StructType(schema.map { attr =>
+      StructField(attr.name, attr.dataType, attr.nullable, attr.metadata)
+    })
+  }
+
+  // Build the statistics row expected by SimpleMetricsCachedBatchSerializer.
+  // For each cached column Spark expects five values in this order:
+  // lower bound, upper bound, null count, row count, and size in bytes.
+  private def computeStats(batch: ColumnarBatch, attrs: Seq[Attribute]): InternalRow = {
+    val numCols = attrs.length
+    val lower = new Array[Any](numCols)
+    val upper = new Array[Any](numCols)
+    val nulls = Array.fill[Int](numCols)(0)
+    val numRows = batch.numRows()
+
+    var c = 0
+    while (c < numCols) {
+      val dt = attrs(c).dataType
+      val col = batch.column(c)
+      var r = 0
+      while (r < numRows) {
+        if (col.isNullAt(r)) {
+          nulls(c) += 1
+        } else if (tracksBounds(dt)) {
+          val value = readValue(col, dt, r)
+          if (lower(c) == null || compare(dt, value, lower(c)) < 0) {
+            lower(c) = value
+          }
+          if (upper(c) == null || compare(dt, value, upper(c)) > 0) {
+            upper(c) = value
+          }
+        }
+        r += 1
+      }
+      c += 1
+    }
+
+    val values = new Array[Any](numCols * 5)
+    c = 0
+    while (c < numCols) {
+      val base = c * 5
+      values(base) = lower(c)
+      values(base + 1) = upper(c)
+      values(base + 2) = nulls(c)
+      values(base + 3) = numRows
+      // Spark reserves the fifth field for per-column size. Comet stores the whole
+      // Arrow stream as one compressed buffer, so per-column size is not tracked here.
+      // Cache pruning uses bounds/null-count/row-count, not this size field.
+      values(base + 4) = 0L
+      c += 1
+    }
+
+    new GenericInternalRow(values)
+  }
+
+  // Spark can prune cache batches only for types whose bounds can be compared.
+  // Other types still report null count and row count but leave bounds as null.
+  private def tracksBounds(dt: DataType): Boolean = dt match {
+    case BooleanType | ByteType | ShortType | IntegerType | LongType | FloatType | DoubleType |
+        _: DecimalType | StringType | DateType | TimestampType | TimestampNTZType =>
+      true
+    case _ => false
+  }
+
+  // Read a non-null value from a ColumnVector using Spark's internal value type
+  // for the corresponding DataType.
+  private def readValue(col: ColumnVector, dt: DataType, rowId: Int): Any = dt match {
+    case BooleanType => col.getBoolean(rowId)
+    case ByteType => col.getByte(rowId)
+    case ShortType => col.getShort(rowId)
+    case IntegerType | DateType => col.getInt(rowId)
+    case LongType | TimestampType | TimestampNTZType => col.getLong(rowId)
+    case FloatType => col.getFloat(rowId)
+    case DoubleType => col.getDouble(rowId)
+    case d: DecimalType => col.getDecimal(rowId, d.precision, d.scale)
+    case StringType => col.getUTF8String(rowId).copy()
+    case _ => null
+  }
+
+  // Compare values using the same physical representation used in the stats row.
+  private def compare(dt: DataType, left: Any, right: Any): Int = dt match {
+    case BooleanType =>
+      java.lang.Boolean.compare(left.asInstanceOf[Boolean], right.asInstanceOf[Boolean])
+    case ByteType =>
+      java.lang.Byte.compare(left.asInstanceOf[Byte], right.asInstanceOf[Byte])
+    case ShortType =>
+      java.lang.Short.compare(left.asInstanceOf[Short], right.asInstanceOf[Short])
+    case IntegerType | DateType =>
+      java.lang.Integer.compare(left.asInstanceOf[Int], right.asInstanceOf[Int])
+    case LongType | TimestampType | TimestampNTZType =>
+      java.lang.Long.compare(left.asInstanceOf[Long], right.asInstanceOf[Long])
+    case FloatType =>
+      java.lang.Float.compare(left.asInstanceOf[Float], right.asInstanceOf[Float])
+    case DoubleType =>
+      java.lang.Double.compare(left.asInstanceOf[Double], right.asInstanceOf[Double])
+    case _: DecimalType =>
+      left.asInstanceOf[Decimal].compare(right.asInstanceOf[Decimal])
+    case StringType =>
+      ByteArray.compareBinary(
+        left.asInstanceOf[UTF8String].getBytes,
+        right.asInstanceOf[UTF8String].getBytes)
+    case other =>
+      throw new IllegalStateException(s"compare called for unsupported type $other")
+  }
+
+  // Compute Spark-compatible cache stats before serializing each batch to Arrow.
+  // The stats are stored beside the Arrow bytes so Spark's cache filter can prune
+  // CometCachedBatch without decoding the batch first.
+  private def encodeBatches(
+      batches: Iterator[ColumnarBatch],
+      attrs: Seq[Attribute]): Iterator[CachedBatch] = {
+    batches.flatMap { batch =>
+      val stats = computeStats(batch, attrs)
+
+      Utils.serializeBatches(Iterator.single(batch)).map { case (rows, buffer) =>
+        CometCachedBatch(
+          numRows = rows.toInt,
+          sizeInBytes = buffer.size,
+          stats = stats,
+          bytes = buffer)
+      }
+    }
+  }
+
+  // Resolve requested columns by exprId, not by name, because aliases may reuse names.
+  private def selectedIndices(
+      cacheAttributes: Seq[Attribute],
+      selectedAttributes: Seq[Attribute]): Array[Int] = {
+    if (selectedAttributes.isEmpty) {
+      cacheAttributes.indices.toArray
+    } else {
+      val byExprId = cacheAttributes.zipWithIndex.map { case (attr, idx) =>
+        attr.exprId -> idx
+      }.toMap
+
+      selectedAttributes.map { attr =>
+        byExprId.getOrElse(
+          attr.exprId,
+          throw new IllegalStateException(
+            s"Could not resolve selected attribute ${attr.name} from cache attributes"))
+      }.toArray
+    }
+  }
+
+  // A full-width projection is only an identity projection if every selected index
+  // is already in column order. For example, [1, 0] must still be projected.
+  private def isIdentityProjection(indices: Array[Int], numCols: Int): Boolean =
+    indices.length == numCols && indices.indices.forall(i => indices(i) == i)
+
+  private def projectBatch(batch: ColumnarBatch, indices: Array[Int]): ColumnarBatch = {
+    if (isIdentityProjection(indices, batch.numCols())) {
+      batch
+    } else {
+      val cols = indices.map(i => batch.column(i).asInstanceOf[ColumnVector])
+      new ColumnarBatch(cols, batch.numRows())
+    }
+  }
+
+  // Spark's SimpleMetricsCachedBatchSerializer prunes a batch when the generated partition filter
+  // does not evaluate to true against the stats row. Bounds are only computed for the types
+  // tracksBounds accepts, and for every other column the lower and upper bounds stay null, which
+  // makes a comparison against them evaluate to null and therefore prune the batch. That would
+  // silently drop rows, so predicates over columns without bounds are not pushed down at all.
+  // Null counts and row counts are recorded for every column, so IsNull and IsNotNull stay safe.
+  override def buildFilter(
+      predicates: Seq[Expression],
+      cachedAttributes: Seq[Attribute]): (Int, Iterator[CachedBatch]) => Iterator[CachedBatch] = {
+    val prunable = cachedAttributes.collect {
+      case a if tracksBounds(a.dataType) => a.exprId
+    }.toSet
+
+    val prunablePredicates = predicates.filter {
+      case _: IsNull | _: IsNotNull => true
+      case p => p.references.forall(a => prunable.contains(a.exprId))
+    }
+
+    super.buildFilter(prunablePredicates, cachedAttributes)
+  }
+
+  // Comet's Arrow writer only handles the types listed in supportsSchema. Reporting false here
+  // sends the relation down the row path, where it is delegated to Spark's default serializer,
+  // instead of failing at cache materialization inside Utils.serializeBatches.
+  override def supportsColumnarInput(schema: Seq[Attribute]): Boolean = supportsSchema(schema)

Review Comment:
   > probably the cause of [#5051 (comment)](https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5122801192)
   
   looks like its directly related



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3677859989


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowConverters.scala:
##########
@@ -74,4 +74,45 @@ object CometArrowConverters extends Logging {
       }
     }
   }
+
+  /**
+   * Copy a Spark `ColumnarBatch` whose columns are not Arrow-backed (e.g.
+   * `On/OffHeapColumnVector` from Spark's vectorized Parquet reader, or a third-party connector's
+   * vectors) into a freshly allocated Arrow `ColumnarBatch` of `CometVector`s.
+   *
+   * The input batch is not consumed or closed; the caller owns the returned batch and must close
+   * it. Values are copied element-wise, since Spark's `ColumnVector` implementations do not
+   * expose Arrow buffers.
+   */
+  def columnarBatchToArrowBatch(

Review Comment:
   A `ColumnarBatch` is a container:
   ```scala
   class ColumnarBatch {
     ColumnVector[] columns;
   }
   ```
   
   Spark 4.1.2’s relevant `ColumnVector` implementations are:
   Producer | Vectors inside ColumnarBatch
   -- | --
   Vectorized Parquet | OnHeapColumnVector, OffHeapColumnVector, ConstantColumnVector
   Vectorized ORC | OrcAtomicColumnVector, OrcArrayColumnVector, OrcMapColumnVector, OrcStructColumnVector; sometimes heap/off-heap or constant vectors
   Spark Arrow/Python paths | ArrowColumnVector
   Comet | CometVector
   External columnar sources | Any implementation extending Spark’s ColumnVector
   
   [SparkColumnarArrowReader](https://github.com/apache/datafusion-comet/blob/63c5a916b2e5626b040dc1f097384ab1a443bf91/spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/SparkColumnarArrowReader.scala#L27-L101) does not inspect those concrete classes. It wraps each column in `ColumnarArray`, and `ArrowWriter` reads through Spark’s standard accessors such as `getLong`, `getUTF8String`, `getArray`, and `getStruct`. Therefore, it supports any implementation that correctly follows the [spark's ColumnVector](https://github.com/apache/spark/blob/v4.1.2/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnVector.java)
   
   `ColumnVector` contract for the declared data type.
   The cache writer could reuse it approximately like this:
   
   ```scala
   override def convertColumnarBatchToCachedBatch(
       input: RDD[ColumnarBatch],
       schema: Seq[Attribute],
       storageLevel: StorageLevel,
       conf: SQLConf): RDD[CachedBatch] = {
   
     val sparkSchema = toStructType(schema)
     val batchSize = conf.columnBatchSize
     val timeZoneId = conf.sessionLocalTimeZone
   
     input.mapPartitions { batches =&gt;
       // Arrow Schema is not serializable, so construct it inside the task.
       val arrowSchema = Utils.toArrowSchema(sparkSchema, timeZoneId)
   
       val arrowBatches = CometArrowStream.readerBatchIter(
         "CometCacheWrite",
         allocator =&gt;
           new SparkColumnarArrowReader(
             allocator,
             arrowSchema,
             batches,
             batchSize))
   
       encodeBatches(arrowBatches, schema)
     }
   }
   ```
   
   That gives this path:
   ```
   Spark ColumnarBatch
     → SparkColumnarArrowReader
     → Arrow-backed ColumnarBatch containing CometVector
     → existing encodeBatches/getBatchFieldVectors
     → CometCachedBatch</code>
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5122851449

   > Tried out on a pipeline that relies heavily on caching and got an error:
   > 
   > ```
   > org.apache.spark.SparkException: Comet execution only takes Arrow Arrays, but got class org.apache.spark.sql.execution.vectorized.OffHeapColumnVector. This typically happens when a Comet scan falls back to Spark due to unsupported data types (e.g., complex types like structs, arrays, or maps). To resolve this, you can: (1) enable spark.comet.scan.allowIncompatible=true to use a compatible native scan variant, or (2) enable spark.comet.convert.parquet.enabled=true to convert Spark Parquet data to Arrow format automatically.
   > 	at org.apache.spark.sql.comet.util.Utils$.$anonfun$getBatchFieldVectors$1(Utils.scala:417)
   > 	at org.apache.spark.sql.comet.util.Utils$.$anonfun$getBatchFieldVectors$1$adapted(Utils.scala:403)
   > 	at scala.collection.immutable.Range.map(Range.scala:60)
   > 	at org.apache.spark.sql.comet.util.Utils$.getBatchFieldVectors(Utils.scala:403)
   > 	at org.apache.spark.sql.comet.util.Utils$.$anonfun$serializeBatches$1(Utils.scala:248)
   > ```
   > 
   > Note that I've tried to enable the suggested config insertions, but it didn't work either. For context: pipeline has lots of fallbacks to default Spark operators. Also using Iceberg V2 as datasource with MOR tables.
   > 
   > cc @andygrove
   
   Thanks @sandugood. I'm looking into it now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3677859989


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowConverters.scala:
##########
@@ -74,4 +74,45 @@ object CometArrowConverters extends Logging {
       }
     }
   }
+
+  /**
+   * Copy a Spark `ColumnarBatch` whose columns are not Arrow-backed (e.g.
+   * `On/OffHeapColumnVector` from Spark's vectorized Parquet reader, or a third-party connector's
+   * vectors) into a freshly allocated Arrow `ColumnarBatch` of `CometVector`s.
+   *
+   * The input batch is not consumed or closed; the caller owns the returned batch and must close
+   * it. Values are copied element-wise, since Spark's `ColumnVector` implementations do not
+   * expose Arrow buffers.
+   */
+  def columnarBatchToArrowBatch(

Review Comment:
   A `ColumnarBatch` is a container:
   ```scala
   class ColumnarBatch {
     ColumnVector[] columns;
   }
   ```
   
   Spark 4.1.2’s relevant `ColumnVector` implementations are:
   Producer | Vectors inside ColumnarBatch
   -- | --
   Vectorized Parquet | OnHeapColumnVector, OffHeapColumnVector, ConstantColumnVector
   Vectorized ORC | OrcAtomicColumnVector, OrcArrayColumnVector, OrcMapColumnVector, OrcStructColumnVector; sometimes heap/off-heap or constant vectors
   Spark Arrow/Python paths | ArrowColumnVector
   Comet | CometVector
   External columnar sources | Any implementation extending Spark’s ColumnVector
   
   [SparkColumnarArrowReader](https://github.com/apache/datafusion-comet/blob/63c5a916b2e5626b040dc1f097384ab1a443bf91/spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/SparkColumnarArrowReader.scala#L27-L101) does not inspect those concrete classes. It wraps each column in `ColumnarArray`, and `ArrowWriter` reads through Spark’s standard accessors such as `getLong`, `getUTF8String`, `getArray`, and `getStruct`. Therefore, it supports any implementation that correctly follows the [spark's ColumnVector](https://github.com/apache/spark/blob/v4.1.2/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnVector.java)
   
   `ColumnVector` contract for the declared data type. The cache writer could reuse it approximately like this:
   
   ```scala
   override def convertColumnarBatchToCachedBatch(
       input: RDD[ColumnarBatch],
       schema: Seq[Attribute],
       storageLevel: StorageLevel,
       conf: SQLConf): RDD[CachedBatch] = {
   
     val sparkSchema = toStructType(schema)
     val batchSize = conf.columnBatchSize
     val timeZoneId = conf.sessionLocalTimeZone
   
     input.mapPartitions { batches =>;
       // Arrow Schema is not serializable, so construct it inside the task.
       val arrowSchema = Utils.toArrowSchema(sparkSchema, timeZoneId)
   
       val arrowBatches = CometArrowStream.readerBatchIter(
         "CometCacheWrite",
         allocator =&gt;
           new SparkColumnarArrowReader(
             allocator,
             arrowSchema,
             batches,
             batchSize))
   
       encodeBatches(arrowBatches, schema)
     }
   }
   ```
   
   That gives this path:
   ```
   Spark ColumnarBatch
     -> SparkColumnarArrowReader
     -> Arrow-backed ColumnarBatch containing CometVector
     -> existing encodeBatches/getBatchFieldVectors
     -> CometCachedBatch</code>
   ```
   
   _Assist with LLM_



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5093401967

   I found some issues with data type coverage and will push some fixes soon.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5448373191

   > @viirya @sunchao happy either way — say which you would rather review and I will restructure, or leave it as is if folding it in here is fine.
   
   Thanks @andygrove , that makes sense. I'm in favor of splitting the broadcast fix into its own PR under #5510, since it affects existing native broadcast joins without the cache feature. The shared registrations, registrator, startup warning, and a broadcast regression test can land there first, with #5051 keeping the cache-specific registrations and tests. That lets us fix the existing bug independently and keeps this PR focused.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3867601976


##########
spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala:
##########
@@ -268,6 +268,37 @@ object Utils extends CometTypeShim with Logging {
     }
   }
 
+  /**
+   * Serializes each column of `batch` into its own compressed Arrow IPC stream, in column order.
+   *
+   * [[serializeBatches]] writes one stream covering every column, so a reader has to inflate all
+   * of them before it can project. Comet's in-memory cache stores columns separately instead, so
+   * a scan decodes only the ones it selected. Each stream is self-contained, including its schema
+   * and any dictionaries the column needs.
+   *
+   * The row count is not recoverable from the result when `batch` has no columns, so callers keep
+   * it alongside. As with [[serializeBatches]], the batch's vectors are cleared once written.
+   */
+  def serializeBatchColumns(batch: ColumnarBatch): Array[ChunkedByteBuffer] = {
+    val (fieldVectors, batchProviderOpt) = getBatchFieldVectors(batch)
+    val provider = batchProviderOpt.getOrElse(new CDataDictionaryProvider)
+    val codec = CompressionCodec.createCodec(SparkEnv.get.conf)
+
+    fieldVectors.map { fieldVector =>
+      val cbbos = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)
+      val out = new DataOutputStream(codec.compressedOutputStream(cbbos))
+
+      val root = new VectorSchemaRoot(Seq(fieldVector).asJava)
+      val writer = new ArrowStreamWriter(root, provider, Channels.newChannel(out))
+      writer.start()
+      writer.writeBatch()

Review Comment:
   **[P2] Use each column's dictionary provider when serializing it**
   
   The new decoder opens one Arrow reader per column, so its dictionary-backed output columns have independent providers. `getBatchFieldVectors` returns only the first dictionary column's provider, which is then passed to every writer here. Re-encoding a decoded cache batch cannot resolve the later columns' dictionary IDs.
   
   I reproduced this through normal Spark operations: with Comet caching and `spark.comet.shuffle.mode=jvm`, cache a repartitioned DataFrame with two low-cardinality string columns. Then disable `spark.comet.exec.enabled` and `spark.comet.exec.inMemoryCache.enabled` and run `first.union(first).cache().count()`. Spark's columnar Union passes the decoded cached batches back to this serializer, and the second cache fails with `IllegalArgumentException: Could not find dictionary with ID 1`. The original cache reads correctly and its columns have separate provider namespaces. Please obtain the provider associated with each column rather than reusing the first one.



##########
spark/src/main/scala/org/apache/spark/sql/comet/CometInMemoryTableScanExec.scala:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.columnar.{CachedBatch, CachedBatchSerializer}
+import org.apache.spark.sql.execution.LeafExecNode
+import org.apache.spark.sql.execution.columnar.InMemoryTableScanExec
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.CometConf
+import org.apache.comet.serde.CometOperatorSerde
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.serializeDataType
+
+/**
+ * Reads Spark cached table data when the cache was written by Comet's cache serializer.
+ *
+ * Spark stores cached data through `CachedBatchSerializer`. This node keeps the scan inside Comet
+ * by asking the serializer to decode cached batches directly into `ColumnarBatch` output,
+ * avoiding the extra Spark columnar-to-Comet columnar conversion used by the default path.
+ *
+ * `relationOutput` is the full schema stored in the cache. `scanOutput` is the subset requested
+ * by this scan after pruning.
+ */
+case class CometInMemoryTableScanExec(
+    originalPlan: InMemoryTableScanExec,
+    serializer: CachedBatchSerializer,
+    cachedBuffers: RDD[CachedBatch],
+    relationOutput: Seq[Attribute],
+    scanOutput: Seq[Attribute])
+    extends CometExec
+    with LeafExecNode {
+
+  override lazy val metrics: Map[String, SQLMetric] = Map(
+    "numOutputRows" -> SQLMetrics.createMetric(sparkContext, "number of output rows"))
+
+  // For an empty-projection scan (`SELECT count(*)`) this is empty while `scanOutput` holds one
+  // placeholder column, so the emitted batches are wider than the declared output. That is safe
+  // because the only consumer of an empty-output scan is a count-style aggregate, which reads the
+  // row count rather than any column; see `scanOutputFor` for why the scan cannot simply be empty.
+  override def output: Seq[Attribute] = originalPlan.output
+
+  // Use the serializer's vector types because the cached batch layout is owned by the serializer.
+  override def vectorTypes: Option[Seq[String]] =
+    serializer.vectorTypes(scanOutput, conf)
+
+  // Apply Spark's cache batch filter before decoding. Spark's InMemoryTableScanExec does this in
+  // filteredCachedBatches(), but that method is private. Reusing the serializer's buildFilter here
+  // keeps Comet on the same stats-based pruning path instead of decoding every cached batch.
+  //
+  // Gated on conf.inMemoryPartitionPruning the same way Spark's filteredCachedBatches is, so
+  // spark.sql.inMemoryColumnarStorage.partitionPruning=false disables pruning here too. Pruning is
+  // normally a win, but the config exists to be able to turn it off -- for debugging a suspected
+  // stats bug, for instance -- and silently ignoring it would make Comet diverge from Spark on a
+  // knob a user reaching for it is specifically trying to control.
+  override def doExecuteColumnar(): RDD[ColumnarBatch] = {
+    val numOutputRows = longMetric("numOutputRows")
+
+    val filteredBuffers =
+      if (originalPlan.predicates.nonEmpty && conf.inMemoryPartitionPruning) {
+        val filter = serializer.buildFilter(originalPlan.predicates, relationOutput)
+        cachedBuffers.mapPartitionsWithIndex(filter)
+      } else {
+        cachedBuffers
+      }
+
+    serializer
+      .convertCachedBatchToColumnarBatch(filteredBuffers, relationOutput, scanOutput, conf)
+      .map { cb =>
+        numOutputRows += cb.numRows()
+        cb
+      }
+  }
+}
+
+object CometInMemoryTableScanExec extends CometOperatorSerde[InMemoryTableScanExec] {
+
+  override def enabledConfig: Option[org.apache.comet.ConfigEntry[Boolean]] =
+    Some(CometConf.COMET_EXEC_IN_MEMORY_CACHE_ENABLED)
+
+  override def convert(
+      op: InMemoryTableScanExec,
+      builder: OperatorOuterClass.Operator.Builder,
+      childOp: Operator*): Option[Operator] = {
+
+    val scanTypes = scanOutputFor(op).flatMap(attr => serializeDataType(attr.dataType))
+
+    val scanBuilder = OperatorOuterClass.Scan
+      .newBuilder()
+      .setSource(op.getClass.getSimpleName)
+      .addAllFields(scanTypes.asJava)
+
+    Some(builder.setScan(scanBuilder).build())
+  }
+
+  // Reuse Spark's InMemoryRelation metadata so cache materialization, pruning, and storage
+  // behavior remain controlled by Spark's cache manager.
+  override def createExec(nativeOp: Operator, op: InMemoryTableScanExec): CometNativeExec = {
+    val relation = op.relation
+
+    CometScanWrapper(
+      nativeOp,
+      CometInMemoryTableScanExec(
+        op,
+        relation.cacheBuilder.serializer,
+        relation.cacheBuilder.cachedColumnBuffers,
+        relation.output,
+        scanOutputFor(op)))
+  }
+
+  /**
+   * Columns the cache scan asks the serializer to decode.
+   *
+   * An empty-output scan (`SELECT count(*)`) still needs a non-empty schema for native planning,
+   * and the batches the node emits have to match that schema. Falling back to the whole cache
+   * schema satisfies both, but the serializer decodes exactly what it is asked for, so the
+   * cheapest query in the workload would decode every cached column. One column is enough: the
+   * aggregate above an empty-output scan reads the row count and never a value, so pick the
+   * cheapest to decode rather than all of them.
+   *
+   * `convert` and `createExec` must choose identically, or the native scan's declared schema and
+   * the batches fed to it disagree.
+   */
+  private def scanOutputFor(op: InMemoryTableScanExec): Seq[Attribute] = {
+    if (op.output.nonEmpty) {
+      op.output
+    } else if (op.relation.output.isEmpty) {
+      Nil
+    } else {
+      Seq(op.relation.output.minBy(a => decodeCostRank(a.dataType)))

Review Comment:
   **[P1] Preserve zero-column output for joins**
   
   An empty-output cache scan can feed a join, not only a count-style aggregate. Adding this column changes the native join's column positions while `output` still declares the original empty schema. With native caching enabled and AQE disabled, I reproduced:
   
   ```scala
   val left = spark.range(10L, 13L).cache()
   left.collect()
   left.createOrReplaceTempView("cached_left")
   spark.sql("""
     SELECT /*+ BROADCAST(r) */ sum(r.id)
     FROM cached_left l JOIN range(2) r ON true
   """).collect()
   ```
   
   The result is **66**, versus the correct **3** when native cache scanning is disabled. The native join receives the hidden left column before `r.id`, so the aggregate's bound ordinal 0 reads the wrong values. Selecting `r.id` directly also fails with `Output column count mismatch: expected 1, got 2`. Please preserve the zero-column scan schema/batches instead of introducing an unreported column. A genuinely zero-column cached relation already returns the correct count on this head.



##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala:
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet.execution.arrow
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.TaskContext
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, GenericInternalRow, IsNotNull, IsNull, UnsafeProjection}
+import org.apache.spark.sql.columnar.{CachedBatch, SimpleMetricsCachedBatch, SimpleMetricsCachedBatchSerializer}
+import org.apache.spark.sql.comet.util.Utils
+import org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.unsafe.types.{ByteArray, UTF8String}
+import org.apache.spark.util.io.ChunkedByteBuffer
+
+import org.apache.comet.CometArrowAllocator
+
+/**
+ * Cached batch format used when Comet writes Spark in-memory cache data.
+ *
+ * `columns` holds one compressed Arrow stream per cached column, in cache-schema order, produced
+ * by `Utils.serializeBatchColumns`. Storing columns separately is what lets a scan decode only
+ * the ones it projected; a single stream covering the whole batch would have to be inflated in
+ * full before any projection could be applied. The cache manager still owns storage and eviction;
+ * this class only changes the cached payload.
+ */
+private case class CometCachedBatch(
+    override val numRows: Int,
+    override val sizeInBytes: Long,
+    override val stats: InternalRow,
+    columns: Array[ChunkedByteBuffer])
+    extends SimpleMetricsCachedBatch
+
+/**
+ * Cache serializer that stores Comet-compatible Arrow batches in Spark's in-memory cache.
+ *
+ * The cached payload format is decided by the schema alone. A relation whose schema Comet's Arrow
+ * writer supports is stored as `CometCachedBatch`, and every other relation is delegated in full
+ * to Spark's `DefaultCachedBatchSerializer`. The format deliberately does not depend on any
+ * runtime config: `spark.sql.cache.serializer` is a static conf, so installing this serializer is
+ * already a per-application decision, and a relation whose format could flip mid-session cannot
+ * be read back reliably. `spark.comet.exec.inMemoryCache.enabled` still governs whether a scan
+ * over the cache runs natively, and its value at startup is what makes `CometDriverPlugin`
+ * install this serializer in the first place.
+ *
+ * Reads of `CometCachedBatch` keep working when the native scan is disabled, because Spark then
+ * reads the same cached data through the SparkToColumnar fallback path.
+ */
+class ArrowCachedBatchSerializer extends SimpleMetricsCachedBatchSerializer {
+
+  import ArrowCachedBatchSerializer.supportsSchema
+
+  private val fallback = new DefaultCachedBatchSerializer()
+
+  // Bounds and null counts per column, gathered before the batch is serialized: serializing
+  // clears the batch's vectors, and the per-column byte sizes that complete the statistics row
+  // are only known afterwards. See statsRow.
+  private def gatherColumnStats(
+      batch: ColumnarBatch,
+      attrs: Seq[Attribute]): (Array[Any], Array[Any], Array[Int]) = {
+    val numCols = attrs.length
+    val lower = new Array[Any](numCols)
+    val upper = new Array[Any](numCols)
+    val nulls = Array.fill[Int](numCols)(0)
+    val numRows = batch.numRows()
+
+    var c = 0
+    while (c < numCols) {
+      val dt = attrs(c).dataType
+      val col = batch.column(c)
+      var r = 0
+      while (r < numRows) {
+        if (col.isNullAt(r)) {
+          nulls(c) += 1
+        } else if (tracksBounds(dt)) {
+          val value = readValue(col, dt, r)
+          if (lower(c) == null || compare(dt, value, lower(c)) < 0) {
+            lower(c) = value
+          }
+          if (upper(c) == null || compare(dt, value, upper(c)) > 0) {
+            upper(c) = value
+          }
+        }
+        r += 1
+      }
+      c += 1
+    }
+
+    (lower, upper, nulls)
+  }
+
+  // Build the statistics row expected by SimpleMetricsCachedBatchSerializer.
+  // For each cached column Spark expects five values in this order:
+  // lower bound, upper bound, null count, row count, and size in bytes.
+  private def statsRow(
+      lower: Array[Any],
+      upper: Array[Any],
+      nulls: Array[Int],
+      numRows: Int,
+      columnSizes: Array[Long]): InternalRow = {
+    val numCols = lower.length
+    val values = new Array[Any](numCols * 5)
+    var c = 0
+    while (c < numCols) {
+      val base = c * 5
+      values(base) = lower(c)
+      values(base + 1) = upper(c)
+      values(base + 2) = nulls(c)
+      values(base + 3) = numRows
+      // Each column is its own compressed stream, so its size is known exactly. Cache pruning
+      // uses bounds/null-count/row-count rather than this field, but Spark reserves it and
+      // reports it, so record the real value.
+      values(base + 4) = columnSizes(c)
+      c += 1
+    }
+
+    new GenericInternalRow(values)
+  }
+
+  // Spark can prune cache batches only for types whose bounds can be compared.
+  // Other types still report null count and row count but leave bounds as null.
+  private def tracksBounds(dt: DataType): Boolean = dt match {
+    case BooleanType | ByteType | ShortType | IntegerType | LongType | FloatType | DoubleType |
+        _: DecimalType | StringType | DateType | TimestampType | TimestampNTZType =>
+      true
+    case _ => false
+  }
+
+  // Read a non-null value from a ColumnVector using Spark's internal value type
+  // for the corresponding DataType.
+  private def readValue(col: ColumnVector, dt: DataType, rowId: Int): Any = dt match {
+    case BooleanType => col.getBoolean(rowId)
+    case ByteType => col.getByte(rowId)
+    case ShortType => col.getShort(rowId)
+    case IntegerType | DateType => col.getInt(rowId)
+    case LongType | TimestampType | TimestampNTZType => col.getLong(rowId)
+    case FloatType => col.getFloat(rowId)
+    case DoubleType => col.getDouble(rowId)
+    case d: DecimalType => col.getDecimal(rowId, d.precision, d.scale)
+    case StringType => col.getUTF8String(rowId).copy()
+    case _ => null
+  }
+
+  // Compare values using the same physical representation used in the stats row.
+  private def compare(dt: DataType, left: Any, right: Any): Int = dt match {
+    case BooleanType =>
+      java.lang.Boolean.compare(left.asInstanceOf[Boolean], right.asInstanceOf[Boolean])
+    case ByteType =>
+      java.lang.Byte.compare(left.asInstanceOf[Byte], right.asInstanceOf[Byte])
+    case ShortType =>
+      java.lang.Short.compare(left.asInstanceOf[Short], right.asInstanceOf[Short])
+    case IntegerType | DateType =>
+      java.lang.Integer.compare(left.asInstanceOf[Int], right.asInstanceOf[Int])
+    case LongType | TimestampType | TimestampNTZType =>
+      java.lang.Long.compare(left.asInstanceOf[Long], right.asInstanceOf[Long])
+    case FloatType =>
+      java.lang.Float.compare(left.asInstanceOf[Float], right.asInstanceOf[Float])
+    case DoubleType =>
+      java.lang.Double.compare(left.asInstanceOf[Double], right.asInstanceOf[Double])
+    case _: DecimalType =>
+      left.asInstanceOf[Decimal].compare(right.asInstanceOf[Decimal])
+    case StringType =>
+      ByteArray.compareBinary(
+        left.asInstanceOf[UTF8String].getBytes,
+        right.asInstanceOf[UTF8String].getBytes)
+    case other =>
+      throw new IllegalStateException(s"compare called for unsupported type $other")
+  }
+
+  // Compute Spark-compatible cache stats before serializing each batch to Arrow.
+  // The stats are stored beside the Arrow bytes so Spark's cache filter can prune
+  // CometCachedBatch without decoding the batch first.
+  //
+  // A columnar input batch is not guaranteed to be Arrow-backed; see supportsColumnarInput for
+  // why. Batches that are not get copied into Arrow first, since Utils.serializeBatches only
+  // writes CometVector columns.
+  private def encodeBatches(
+      batches: Iterator[ColumnarBatch],
+      attrs: Seq[Attribute]): Iterator[CachedBatch] = {
+    val arrowSchema =
+      Utils.toArrowSchema(Utils.fromAttributes(attrs), CometArrowStream.NATIVE_TIMEZONE)
+
+    batches.map { batch =>
+      // Bounds and null counts are read from the input batch, which serializing then clears, so
+      // they have to be gathered first. The row is only assembled once the per-column sizes are
+      // known.
+      val (lower, upper, nulls) = gatherColumnStats(batch, attrs)
+      val numRows = batch.numRows()
+
+      val columns = if (Utils.isArrowBacked(batch)) {
+        Utils.serializeBatchColumns(batch)
+      } else {
+        val arrowBatch =
+          CometArrowConverters.columnarBatchToArrowBatch(batch, arrowSchema, CometArrowAllocator)
+        try Utils.serializeBatchColumns(arrowBatch)
+        finally arrowBatch.close()

Review Comment:
   **[P2] Handle large-offset Arrow vectors before taking the direct write path**
   
   `Utils.isArrowBacked` accepts any `CometVector`, including a `CometPlainVector` wrapping `LargeVarCharVector` or `LargeVarBinaryVector`. However, `serializeBatchColumns` calls `Utils.getFieldVector`, which rejects both representations. Thus `supportsColumnarInput` accepts the Spark `StringType`/`BinaryType` schema, but materializing the cache fails.
   
   At the serializer boundary, I reproduced both large-vector cases with just `hello` and `world`: Spark's default cache round-trips them, while this serializer throws `Unsupported Arrow Vector for serialize`. Ordinary `VarCharVector` succeeds on both paths. Accelerated `mapInArrow` is a relevant producer because its existing runner preserves returned Arrow vectors, including `pa.large_string()`/`pa.large_binary()`. That Python integration path was source-traced, not executed end to end; the serializer failure and default-cache comparison were run. Please normalize these physical representations through the conversion path or support them in the direct writer.



##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala:
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet.execution.arrow
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.TaskContext
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, GenericInternalRow, IsNotNull, IsNull, UnsafeProjection}
+import org.apache.spark.sql.columnar.{CachedBatch, SimpleMetricsCachedBatch, SimpleMetricsCachedBatchSerializer}
+import org.apache.spark.sql.comet.util.Utils
+import org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.unsafe.types.{ByteArray, UTF8String}
+import org.apache.spark.util.io.ChunkedByteBuffer
+
+import org.apache.comet.CometArrowAllocator
+
+/**
+ * Cached batch format used when Comet writes Spark in-memory cache data.
+ *
+ * `columns` holds one compressed Arrow stream per cached column, in cache-schema order, produced
+ * by `Utils.serializeBatchColumns`. Storing columns separately is what lets a scan decode only
+ * the ones it projected; a single stream covering the whole batch would have to be inflated in
+ * full before any projection could be applied. The cache manager still owns storage and eviction;
+ * this class only changes the cached payload.
+ */
+private case class CometCachedBatch(
+    override val numRows: Int,
+    override val sizeInBytes: Long,
+    override val stats: InternalRow,
+    columns: Array[ChunkedByteBuffer])
+    extends SimpleMetricsCachedBatch
+
+/**
+ * Cache serializer that stores Comet-compatible Arrow batches in Spark's in-memory cache.
+ *
+ * The cached payload format is decided by the schema alone. A relation whose schema Comet's Arrow
+ * writer supports is stored as `CometCachedBatch`, and every other relation is delegated in full
+ * to Spark's `DefaultCachedBatchSerializer`. The format deliberately does not depend on any
+ * runtime config: `spark.sql.cache.serializer` is a static conf, so installing this serializer is
+ * already a per-application decision, and a relation whose format could flip mid-session cannot
+ * be read back reliably. `spark.comet.exec.inMemoryCache.enabled` still governs whether a scan
+ * over the cache runs natively, and its value at startup is what makes `CometDriverPlugin`
+ * install this serializer in the first place.
+ *
+ * Reads of `CometCachedBatch` keep working when the native scan is disabled, because Spark then
+ * reads the same cached data through the SparkToColumnar fallback path.
+ */
+class ArrowCachedBatchSerializer extends SimpleMetricsCachedBatchSerializer {
+
+  import ArrowCachedBatchSerializer.supportsSchema
+
+  private val fallback = new DefaultCachedBatchSerializer()
+
+  // Bounds and null counts per column, gathered before the batch is serialized: serializing
+  // clears the batch's vectors, and the per-column byte sizes that complete the statistics row
+  // are only known afterwards. See statsRow.
+  private def gatherColumnStats(
+      batch: ColumnarBatch,
+      attrs: Seq[Attribute]): (Array[Any], Array[Any], Array[Int]) = {
+    val numCols = attrs.length
+    val lower = new Array[Any](numCols)
+    val upper = new Array[Any](numCols)
+    val nulls = Array.fill[Int](numCols)(0)
+    val numRows = batch.numRows()
+
+    var c = 0
+    while (c < numCols) {
+      val dt = attrs(c).dataType
+      val col = batch.column(c)
+      var r = 0
+      while (r < numRows) {
+        if (col.isNullAt(r)) {
+          nulls(c) += 1
+        } else if (tracksBounds(dt)) {
+          val value = readValue(col, dt, r)
+          if (lower(c) == null || compare(dt, value, lower(c)) < 0) {
+            lower(c) = value
+          }
+          if (upper(c) == null || compare(dt, value, upper(c)) > 0) {
+            upper(c) = value
+          }
+        }
+        r += 1
+      }
+      c += 1
+    }
+
+    (lower, upper, nulls)
+  }
+
+  // Build the statistics row expected by SimpleMetricsCachedBatchSerializer.
+  // For each cached column Spark expects five values in this order:
+  // lower bound, upper bound, null count, row count, and size in bytes.
+  private def statsRow(
+      lower: Array[Any],
+      upper: Array[Any],
+      nulls: Array[Int],
+      numRows: Int,
+      columnSizes: Array[Long]): InternalRow = {
+    val numCols = lower.length
+    val values = new Array[Any](numCols * 5)
+    var c = 0
+    while (c < numCols) {
+      val base = c * 5
+      values(base) = lower(c)
+      values(base + 1) = upper(c)
+      values(base + 2) = nulls(c)
+      values(base + 3) = numRows
+      // Each column is its own compressed stream, so its size is known exactly. Cache pruning
+      // uses bounds/null-count/row-count rather than this field, but Spark reserves it and
+      // reports it, so record the real value.
+      values(base + 4) = columnSizes(c)
+      c += 1
+    }
+
+    new GenericInternalRow(values)
+  }
+
+  // Spark can prune cache batches only for types whose bounds can be compared.
+  // Other types still report null count and row count but leave bounds as null.
+  private def tracksBounds(dt: DataType): Boolean = dt match {
+    case BooleanType | ByteType | ShortType | IntegerType | LongType | FloatType | DoubleType |
+        _: DecimalType | StringType | DateType | TimestampType | TimestampNTZType =>
+      true
+    case _ => false
+  }
+
+  // Read a non-null value from a ColumnVector using Spark's internal value type
+  // for the corresponding DataType.
+  private def readValue(col: ColumnVector, dt: DataType, rowId: Int): Any = dt match {
+    case BooleanType => col.getBoolean(rowId)
+    case ByteType => col.getByte(rowId)
+    case ShortType => col.getShort(rowId)
+    case IntegerType | DateType => col.getInt(rowId)
+    case LongType | TimestampType | TimestampNTZType => col.getLong(rowId)
+    case FloatType => col.getFloat(rowId)
+    case DoubleType => col.getDouble(rowId)
+    case d: DecimalType => col.getDecimal(rowId, d.precision, d.scale)
+    case StringType => col.getUTF8String(rowId).copy()
+    case _ => null
+  }
+
+  // Compare values using the same physical representation used in the stats row.
+  private def compare(dt: DataType, left: Any, right: Any): Int = dt match {
+    case BooleanType =>
+      java.lang.Boolean.compare(left.asInstanceOf[Boolean], right.asInstanceOf[Boolean])
+    case ByteType =>
+      java.lang.Byte.compare(left.asInstanceOf[Byte], right.asInstanceOf[Byte])
+    case ShortType =>
+      java.lang.Short.compare(left.asInstanceOf[Short], right.asInstanceOf[Short])
+    case IntegerType | DateType =>
+      java.lang.Integer.compare(left.asInstanceOf[Int], right.asInstanceOf[Int])
+    case LongType | TimestampType | TimestampNTZType =>
+      java.lang.Long.compare(left.asInstanceOf[Long], right.asInstanceOf[Long])
+    case FloatType =>
+      java.lang.Float.compare(left.asInstanceOf[Float], right.asInstanceOf[Float])
+    case DoubleType =>
+      java.lang.Double.compare(left.asInstanceOf[Double], right.asInstanceOf[Double])
+    case _: DecimalType =>
+      left.asInstanceOf[Decimal].compare(right.asInstanceOf[Decimal])
+    case StringType =>
+      ByteArray.compareBinary(
+        left.asInstanceOf[UTF8String].getBytes,
+        right.asInstanceOf[UTF8String].getBytes)
+    case other =>
+      throw new IllegalStateException(s"compare called for unsupported type $other")
+  }
+
+  // Compute Spark-compatible cache stats before serializing each batch to Arrow.
+  // The stats are stored beside the Arrow bytes so Spark's cache filter can prune
+  // CometCachedBatch without decoding the batch first.
+  //
+  // A columnar input batch is not guaranteed to be Arrow-backed; see supportsColumnarInput for
+  // why. Batches that are not get copied into Arrow first, since Utils.serializeBatches only
+  // writes CometVector columns.
+  private def encodeBatches(
+      batches: Iterator[ColumnarBatch],
+      attrs: Seq[Attribute]): Iterator[CachedBatch] = {
+    val arrowSchema =
+      Utils.toArrowSchema(Utils.fromAttributes(attrs), CometArrowStream.NATIVE_TIMEZONE)
+
+    batches.map { batch =>
+      // Bounds and null counts are read from the input batch, which serializing then clears, so
+      // they have to be gathered first. The row is only assembled once the per-column sizes are
+      // known.
+      val (lower, upper, nulls) = gatherColumnStats(batch, attrs)
+      val numRows = batch.numRows()
+
+      val columns = if (Utils.isArrowBacked(batch)) {
+        Utils.serializeBatchColumns(batch)
+      } else {
+        val arrowBatch =
+          CometArrowConverters.columnarBatchToArrowBatch(batch, arrowSchema, CometArrowAllocator)
+        try Utils.serializeBatchColumns(arrowBatch)
+        finally arrowBatch.close()
+      }
+
+      val columnSizes = columns.map(_.size)
+      CometCachedBatch(
+        numRows = numRows,
+        sizeInBytes = columnSizes.sum,
+        stats = statsRow(lower, upper, nulls, numRows, columnSizes),
+        columns = columns)
+    }
+  }
+
+  // Resolve requested columns by exprId, not by name, because aliases may reuse names.
+  //
+  // An empty selection stays empty rather than expanding to every column. Spark asks for no
+  // columns when the query only needs the row count (SELECT count(*)), and since projection now
+  // decides what gets decoded, expanding it would turn the cheapest possible read into the most
+  // expensive one.
+  private def selectedIndices(
+      cacheAttributes: Seq[Attribute],
+      selectedAttributes: Seq[Attribute]): Array[Int] = {
+    val byExprId = cacheAttributes.zipWithIndex.map { case (attr, idx) =>
+      attr.exprId -> idx
+    }.toMap
+
+    selectedAttributes.map { attr =>
+      byExprId.getOrElse(
+        attr.exprId,
+        throw new IllegalStateException(
+          s"Could not resolve selected attribute ${attr.name} from cache attributes"))
+    }.toArray
+  }
+
+  // Spark's SimpleMetricsCachedBatchSerializer prunes a batch when the generated partition filter
+  // does not evaluate to true against the stats row. Bounds are only computed for the types
+  // tracksBounds accepts, and for every other column the lower and upper bounds stay null, which
+  // makes a comparison against them evaluate to null and therefore prune the batch. That would
+  // silently drop rows, so predicates over columns without bounds are not pushed down at all.
+  // Null counts and row counts are recorded for every column, so IsNull and IsNotNull stay safe.
+  override def buildFilter(
+      predicates: Seq[Expression],
+      cachedAttributes: Seq[Attribute]): (Int, Iterator[CachedBatch]) => Iterator[CachedBatch] = {
+    val prunable = cachedAttributes.collect {
+      case a if tracksBounds(a.dataType) => a.exprId
+    }.toSet
+
+    val prunablePredicates = predicates.filter {
+      case _: IsNull | _: IsNotNull => true
+      case p => p.references.forall(a => prunable.contains(a.exprId))
+    }
+
+    super.buildFilter(prunablePredicates, cachedAttributes)
+  }
+
+  // Comet's Arrow writer only handles the types listed in supportsSchema. Reporting false here
+  // sends the relation down the row path, where it is delegated to Spark's default serializer,
+  // instead of failing at cache materialization inside Utils.serializeBatches.
+  //
+  // This answer is schema-only, because attributes are all Spark gives us; it says nothing about
+  // the vectors. Returning true also makes InMemoryRelation strip the ColumnarToRow above the
+  // cached plan, so convertColumnarBatchToCachedBatch then receives whatever that plan produces:
+  // a Comet scan's CometVectors, but equally Spark's vectorized Parquet/ORC reader or a
+  // connector's own vectors. encodeBatches converts the non-Arrow ones; that conversion is load
+  // bearing, not defensive.
+  override def supportsColumnarInput(schema: Seq[Attribute]): Boolean = supportsSchema(schema)
+
+  // A relation Comet stores is always readable as columnar Arrow. Anything else holds
+  // DefaultCachedBatch, so defer to Spark, which only claims columnar output for the primitive
+  // types its ColumnAccessor.decompress path can actually decode.
+  override def supportsColumnarOutput(schema: StructType): Boolean = {
+    if (schema.fields.forall(f => ArrowCachedBatchSerializer.supportsType(f.dataType))) {
+      true
+    } else {
+      fallback.supportsColumnarOutput(schema)
+    }
+  }
+
+  // Columnar Comet output is stored as compressed Arrow stream bytes. Spark only calls this when
+  // supportsColumnarInput returned true, so the schema is known to be Comet-writable here.
+  override def convertColumnarBatchToCachedBatch(
+      input: RDD[ColumnarBatch],
+      schema: Seq[Attribute],
+      storageLevel: StorageLevel,
+      conf: SQLConf): RDD[CachedBatch] = {
+
+    input.mapPartitions { batches =>
+      encodeBatches(batches, schema)
+    }
+  }
+
+  override def convertCachedBatchToColumnarBatch(
+      input: RDD[CachedBatch],
+      cacheAttributes: Seq[Attribute],
+      selectedAttributes: Seq[Attribute],
+      conf: SQLConf): RDD[ColumnarBatch] = {
+    if (!supportsSchema(cacheAttributes)) {
+      return fallback.convertCachedBatchToColumnarBatch(
+        input,
+        cacheAttributes,
+        selectedAttributes,
+        conf)
+    }
+
+    val indices = selectedIndices(cacheAttributes, selectedAttributes)
+
+    input.mapPartitions { it =>
+      // A ColumnReaders closes its readers (releasing the vectors they are holding) only when the
+      // batch it produced has been consumed. A consumer that stops early -- LIMIT, take(), or a
+      // cancelled task -- leaves the readers for the batch in flight open, so close them on task
+      // completion. Spark's own ArrowCachedBatchSerializer registers a listener for the same
+      // reason.
+      //
+      // flatMap consumes each inner iterator fully before building the next, so at most one batch
+      // is open at a time and tracking the current one is enough. close() is idempotent, so
+      // closing one that already released itself is a no-op.
+      @volatile var current: ColumnReaders = null
+      Option(TaskContext.get()).foreach { tc =>
+        tc.addTaskCompletionListener[Unit] { _ =>
+          val readers = current
+          current = null
+          if (readers != null) {
+            readers.close()
+          }
+        }
+      }
+
+      it.flatMap {
+        case cb: CometCachedBatch =>
+          if (indices.isEmpty) {
+            // Nothing to decode: the row count is the whole answer, and it is already here.
+            Iterator.single(new ColumnarBatch(Array.empty[ColumnVector], cb.numRows))
+          } else {
+            val readers = new ColumnReaders(indices.map(i => cb.columns(i)), cb.numRows)
+            current = readers
+            readers.batches
+          }
+
+        case other =>
+          throw new IllegalStateException(
+            s"Unsupported cached batch type ${other.getClass.getName}")
+      }
+    }
+  }
+
+  // Decodes one selected column stream apiece and stitches the results back into a single batch.
+  //
+  // Each stream is self-contained, so the columns a scan did not select are never inflated. The
+  // decoded vectors stay owned by their readers: closing them releases the batch, which is why
+  // this yields a single-element iterator that closes on exhaustion, matching what
+  // ArrowReaderIterator did when the payload was one stream.
+  private class ColumnReaders(buffers: Array[ChunkedByteBuffer], numRows: Int) {
+    private val readers: Array[Iterator[ColumnarBatch]] =
+      buffers.map(Utils.decodeBatches(_, "CometCache"))

Review Comment:
   **[P2] Close previously opened readers when initialization fails**
   
   `decodeBatches` opens and decodes a column eagerly. If opening a later column throws, the readers already created by this `map` are lost. The task-completion listener cannot release them because the new `ColumnReaders` instance is assigned to `current` only after this constructor returns.
   
   I reproduced this with valid cached data containing two integer columns and an allocator limit that permits the first column but rejects the second allocation. After task completion, the first column's 512 bytes remain allocated; three failed tasks leave **512, 1024, and 1536 bytes** respectively, starting from zero. This turns allocation failures into persistent off-heap leaks and worsens subsequent memory pressure. Please close every successfully opened reader if constructing the remaining readers fails.



##########
spark/src/main/scala/org/apache/spark/sql/comet/CometInMemoryTableScanExec.scala:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.columnar.{CachedBatch, CachedBatchSerializer}
+import org.apache.spark.sql.execution.LeafExecNode
+import org.apache.spark.sql.execution.columnar.InMemoryTableScanExec
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.CometConf
+import org.apache.comet.serde.CometOperatorSerde
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.serializeDataType
+
+/**
+ * Reads Spark cached table data when the cache was written by Comet's cache serializer.
+ *
+ * Spark stores cached data through `CachedBatchSerializer`. This node keeps the scan inside Comet
+ * by asking the serializer to decode cached batches directly into `ColumnarBatch` output,
+ * avoiding the extra Spark columnar-to-Comet columnar conversion used by the default path.
+ *
+ * `relationOutput` is the full schema stored in the cache. `scanOutput` is the subset requested
+ * by this scan after pruning.
+ */
+case class CometInMemoryTableScanExec(
+    originalPlan: InMemoryTableScanExec,
+    serializer: CachedBatchSerializer,
+    cachedBuffers: RDD[CachedBatch],
+    relationOutput: Seq[Attribute],
+    scanOutput: Seq[Attribute])
+    extends CometExec
+    with LeafExecNode {
+
+  override lazy val metrics: Map[String, SQLMetric] = Map(
+    "numOutputRows" -> SQLMetrics.createMetric(sparkContext, "number of output rows"))
+
+  // For an empty-projection scan (`SELECT count(*)`) this is empty while `scanOutput` holds one
+  // placeholder column, so the emitted batches are wider than the declared output. That is safe
+  // because the only consumer of an empty-output scan is a count-style aggregate, which reads the
+  // row count rather than any column; see `scanOutputFor` for why the scan cannot simply be empty.
+  override def output: Seq[Attribute] = originalPlan.output
+
+  // Use the serializer's vector types because the cached batch layout is owned by the serializer.
+  override def vectorTypes: Option[Seq[String]] =
+    serializer.vectorTypes(scanOutput, conf)
+
+  // Apply Spark's cache batch filter before decoding. Spark's InMemoryTableScanExec does this in
+  // filteredCachedBatches(), but that method is private. Reusing the serializer's buildFilter here
+  // keeps Comet on the same stats-based pruning path instead of decoding every cached batch.
+  //
+  // Gated on conf.inMemoryPartitionPruning the same way Spark's filteredCachedBatches is, so
+  // spark.sql.inMemoryColumnarStorage.partitionPruning=false disables pruning here too. Pruning is
+  // normally a win, but the config exists to be able to turn it off -- for debugging a suspected
+  // stats bug, for instance -- and silently ignoring it would make Comet diverge from Spark on a
+  // knob a user reaching for it is specifically trying to control.
+  override def doExecuteColumnar(): RDD[ColumnarBatch] = {
+    val numOutputRows = longMetric("numOutputRows")
+
+    val filteredBuffers =
+      if (originalPlan.predicates.nonEmpty && conf.inMemoryPartitionPruning) {
+        val filter = serializer.buildFilter(originalPlan.predicates, relationOutput)
+        cachedBuffers.mapPartitionsWithIndex(filter)
+      } else {
+        cachedBuffers
+      }
+
+    serializer
+      .convertCachedBatchToColumnarBatch(filteredBuffers, relationOutput, scanOutput, conf)
+      .map { cb =>
+        numOutputRows += cb.numRows()
+        cb
+      }
+  }
+}
+
+object CometInMemoryTableScanExec extends CometOperatorSerde[InMemoryTableScanExec] {
+
+  override def enabledConfig: Option[org.apache.comet.ConfigEntry[Boolean]] =
+    Some(CometConf.COMET_EXEC_IN_MEMORY_CACHE_ENABLED)
+
+  override def convert(
+      op: InMemoryTableScanExec,
+      builder: OperatorOuterClass.Operator.Builder,
+      childOp: Operator*): Option[Operator] = {
+
+    val scanTypes = scanOutputFor(op).flatMap(attr => serializeDataType(attr.dataType))
+
+    val scanBuilder = OperatorOuterClass.Scan
+      .newBuilder()
+      .setSource(op.getClass.getSimpleName)
+      .addAllFields(scanTypes.asJava)
+
+    Some(builder.setScan(scanBuilder).build())
+  }
+
+  // Reuse Spark's InMemoryRelation metadata so cache materialization, pruning, and storage
+  // behavior remain controlled by Spark's cache manager.
+  override def createExec(nativeOp: Operator, op: InMemoryTableScanExec): CometNativeExec = {
+    val relation = op.relation
+
+    CometScanWrapper(
+      nativeOp,
+      CometInMemoryTableScanExec(
+        op,
+        relation.cacheBuilder.serializer,
+        relation.cacheBuilder.cachedColumnBuffers,

Review Comment:
   **[P2] Defer cache RDD construction until execution**
   
   Accessing `cachedColumnBuffers` here is not a passive metadata lookup: Spark's cache builder constructs it by calling `cachedPlan.execute`/`executeColumnar`. If that plan is adaptive, this can execute shuffle stages while Comet is still planning the outer query.
   
   With AQE and native caching enabled, I reproduced:
   
   ```scala
   val cached = spark.range(100).repartition(2).cache()
   cached.createOrReplaceTempView("cached_adaptive")
   spark.sql("SELECT * FROM cached_adaptive").explain()
   ```
   
   `explain()` starts **one Spark job** and changes the cached adaptive plan to `isFinalPlan=true`; with native cache scanning disabled and the same cache serializer, it starts **zero jobs** and the cached plan stays unexecuted. Planning/EXPLAIN can therefore perform expensive computation or fail on execution errors. Please retain the relation/cache builder and obtain its RDD inside `doExecuteColumnar()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3677859989


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowConverters.scala:
##########
@@ -74,4 +74,45 @@ object CometArrowConverters extends Logging {
       }
     }
   }
+
+  /**
+   * Copy a Spark `ColumnarBatch` whose columns are not Arrow-backed (e.g.
+   * `On/OffHeapColumnVector` from Spark's vectorized Parquet reader, or a third-party connector's
+   * vectors) into a freshly allocated Arrow `ColumnarBatch` of `CometVector`s.
+   *
+   * The input batch is not consumed or closed; the caller owns the returned batch and must close
+   * it. Values are copied element-wise, since Spark's `ColumnVector` implementations do not
+   * expose Arrow buffers.
+   */
+  def columnarBatchToArrowBatch(

Review Comment:
   A `ColumnarBatch` is a container:
   ```scala
   class ColumnarBatch {
     ColumnVector[] columns;
   }
   ```
   
   Spark 4.1.2’s relevant `ColumnVector` implementations are:
   Producer | Vectors inside ColumnarBatch
   -- | --
   Vectorized Parquet | OnHeapColumnVector, OffHeapColumnVector, ConstantColumnVector
   Vectorized ORC | OrcAtomicColumnVector, OrcArrayColumnVector, OrcMapColumnVector, OrcStructColumnVector; sometimes heap/off-heap or constant vectors
   Spark Arrow/Python paths | ArrowColumnVector
   Comet | CometVector
   External columnar sources | Any implementation extending Spark’s ColumnVector
   
   [SparkColumnarArrowReader](https://github.com/apache/datafusion-comet/blob/63c5a916b2e5626b040dc1f097384ab1a443bf91/spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/SparkColumnarArrowReader.scala#L27-L101) does not inspect those concrete classes. It wraps each column in `ColumnarArray`, and `ArrowWriter` reads through Spark’s standard accessors such as `getLong`, `getUTF8String`, `getArray`, and `getStruct`. Therefore, it supports any implementation that correctly follows the [spark's ColumnVector](https://github.com/apache/spark/blob/v4.1.2/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnVector.java)
   
   `ColumnVector` contract for the declared data type. The cache writer could reuse it approximately like this:
   
   ```scala
   override def convertColumnarBatchToCachedBatch(
       input: RDD[ColumnarBatch],
       schema: Seq[Attribute],
       storageLevel: StorageLevel,
       conf: SQLConf): RDD[CachedBatch] = {
   
     val sparkSchema = toStructType(schema)
     val batchSize = conf.columnBatchSize
     val timeZoneId = conf.sessionLocalTimeZone
   
     input.mapPartitions { batches =>;
       // Arrow Schema is not serializable, so construct it inside the task.
       val arrowSchema = Utils.toArrowSchema(sparkSchema, timeZoneId)
   
       val arrowBatches = CometArrowStream.readerBatchIter(
         "CometCacheWrite",
         allocator =>;
           new SparkColumnarArrowReader(
             allocator,
             arrowSchema,
             batches,
             batchSize))
   
       encodeBatches(arrowBatches, schema)
     }
   }
   ```
   
   That gives this path:
   ```
   Spark ColumnarBatch
     -> SparkColumnarArrowReader
     -> Arrow-backed ColumnarBatch containing CometVector
     -> existing encodeBatches/getBatchFieldVectors
     -> CometCachedBatch</code>
   ```
   
   _Assist with LLM_



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5432172489

   @comphead @sunchao Could I get a review? This is an experimental feature, disabled by default, but I think this PR is already large and complex, and I think it would be helpful to merge it and then continue work in future PRs. WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5443392005

   Filed the broadcast half of the Kryo work as #5510 so it can be split out if reviewers prefer.
   
   To be clear about what is and is not in this PR's scope: `CometBroadcastExchangeExec` broadcasts an `Array[ChunkedByteBuffer]`, which Spark does not register, so a native broadcast join fails under `spark.kryo.registrationRequired=true` on `main` today with no cache involved. 7e19dff20 fixes it here because the registrator this PR had to add anyway covers the same type — but it is a pre-existing bug, not one this PR introduces, and it should not have to wait on an experimental default-off feature.
   
   The two are entangled only through the shared class list (`Utils.arrowBytesKryoClasses`), so splitting is straightforward: that list plus the registrator plus the warning land first, and this PR keeps only `ArrowCachedBatchSerializer.kryoClasses` and the cache tests. #5510 spells out what a standalone fix needs, including the reason Comet cannot install `spark.kryo.registrator` itself.
   
   @viirya @sunchao happy either way — say which you would rather review and I will restructure, or leave it as is if folding it in here is fine.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5441639876

   One small accuracy point on the benchmark, separate from the above.
   
   `CometInMemoryCacheBenchmark` compares "Comet cache disabled" against "Comet cache enabled", and the `withCachedTable` comment describes the disabled case as *"Spark execution over CometCachedBatch, NOT Spark's own cache format"*. The second half is the important clarification and it's right — both cases read a Comet-written `CometCachedBatch`, so this is not a comparison against Spark's cache format. But "Spark execution" reads as if the disabled case runs the query on Spark operators, and from `cacheConf` that doesn't look right: both cases set `COMET_ENABLED` / `COMET_EXEC_ENABLED = true` and `spark.comet.sparkToColumnar.enabled = true`, and the only flag that flips is `COMET_EXEC_IN_MEMORY_CACHE_ENABLED`.
   
   So unless I'm misreading the plan, the two cases differ only at the cache-scan boundary, not in the execution engine above it:
   
   - **enabled** → `CometInMemoryTableScan` feeds the Comet operators directly (and `verifyPlan` asserts no `CometSparkColumnarToColumnar`);
   - **disabled** → Spark's `InMemoryTableScanExec` feeds the same Comet operators through a `CometSparkColumnarToColumnar` bridge — which is a Spark-columnar→Arrow transition whose existence implies the operators above it are still Comet.
   
   The `sum(...)` aggregation runs on Comet in both cases; what the benchmark isolates is the cost of that scan-boundary conversion, which is exactly the overhead the PR sets out to remove. That makes the 1.5x / 1.1x a measure of "keep the cached scan native vs fall back to a Spark cache scan + convert", not "Comet vs Spark execution" — a narrower and, I think, more accurate framing than the comment's wording suggests.
   
   Worth tightening the `withCachedTable` comment (and the same "Spark execution over CometCachedBatch" phrasing in the PR description) so a reader doesn't take the disabled case as a Spark-execution baseline. If I've misread and the upper operators do fall back to Spark in the disabled case, I'd be glad to be corrected — the plan for the disabled run would settle it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3877987279


##########
spark/src/main/scala/org/apache/spark/sql/comet/CometInMemoryTableScanExec.scala:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.columnar.CachedBatchSerializer
+import org.apache.spark.sql.execution.{LeafExecNode, SparkPlan}
+import org.apache.spark.sql.execution.columnar.{CachedRDDBuilder, InMemoryTableScanExec}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.CometConf
+import org.apache.comet.serde.CometOperatorSerde
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.serializeDataType
+
+/**
+ * Reads Spark cached table data when the cache was written by Comet's cache serializer.
+ *
+ * Spark stores cached data through `CachedBatchSerializer`. This node keeps the scan inside Comet
+ * by asking the serializer to decode cached batches directly into `ColumnarBatch` output,
+ * avoiding the extra Spark columnar-to-Comet columnar conversion used by the default path.
+ *
+ * `relationOutput` is the full schema stored in the cache. `scanOutput` is the subset requested
+ * by this scan after pruning.
+ */
+case class CometInMemoryTableScanExec(
+    originalPlan: InMemoryTableScanExec,
+    serializer: CachedBatchSerializer,
+    cacheBuilder: CachedRDDBuilder,
+    relationOutput: Seq[Attribute],
+    scanOutput: Seq[Attribute])
+    extends CometExec
+    with LeafExecNode {
+
+  override lazy val metrics: Map[String, SQLMetric] = Map(
+    "numOutputRows" -> SQLMetrics.createMetric(sparkContext, "number of output rows"))
+
+  // `scanOutput` always equals this, including when it is empty. An empty-output scan
+  // (`SELECT count(*)`) emits genuinely zero-column batches carrying only a row count: widening it
+  // to a placeholder column, or to the whole cache schema, makes the emitted batches disagree with
+  // the declared output, and a consumer that reads by ordinal rather than by row count -- a join,
+  // for instance -- then reads the wrong column.
+  override def output: Seq[Attribute] = originalPlan.output
+
+  // `originalPlan` is a plan-typed field rather than a child, so QueryPlan's canonicalization
+  // walks straight past it: its attributes and predicates keep the expression IDs of whichever
+  // occurrence of the cached relation produced them. Two scans of one cache then compare unequal,
+  // and since sameResult is what exchange and broadcast reuse are keyed on, a UNION of two
+  // identical aggregates over a cached table runs two shuffles where Spark's own cache scan runs
+  // one and reuses it.
+  //
+  // Defer to `InMemoryTableScanExec`, which normalizes its own attributes and predicates against
+  // the relation's output. Dropping the field instead would also make the scans compare equal,
+  // but it would equate scans carrying different pruning predicates along with them.
+  override protected def doCanonicalize(): SparkPlan =
+    super
+      .doCanonicalize()
+      .asInstanceOf[CometInMemoryTableScanExec]
+      .copy(originalPlan = originalPlan.canonicalized.asInstanceOf[InMemoryTableScanExec])
+
+  // Use the serializer's vector types because the cached batch layout is owned by the serializer.
+  override def vectorTypes: Option[Seq[String]] =
+    serializer.vectorTypes(scanOutput, conf)
+
+  // Apply Spark's cache batch filter before decoding. Spark's InMemoryTableScanExec does this in
+  // filteredCachedBatches(), but that method is private. Reusing the serializer's buildFilter here
+  // keeps Comet on the same stats-based pruning path instead of decoding every cached batch.
+  //
+  // Gated on conf.inMemoryPartitionPruning the same way Spark's filteredCachedBatches is, so
+  // spark.sql.inMemoryColumnarStorage.partitionPruning=false disables pruning here too. Pruning is
+  // normally a win, but the config exists to be able to turn it off -- for debugging a suspected
+  // stats bug, for instance -- and silently ignoring it would make Comet diverge from Spark on a
+  // knob a user reaching for it is specifically trying to control.
+  override def doExecuteColumnar(): RDD[ColumnarBatch] = {
+    val numOutputRows = longMetric("numOutputRows")
+
+    // Resolved here rather than at planning time. CachedRDDBuilder.cachedColumnBuffers is not a
+    // metadata lookup: it builds the RDD by calling execute/executeColumnar on the cached plan,
+    // so touching it while Comet is still planning the outer query runs jobs during planning --
+    // visibly, an EXPLAIN of a query over an adaptively-cached relation would launch a job and
+    // finalize that plan.
+    val cachedBuffers = cacheBuilder.cachedColumnBuffers
+
+    val filteredBuffers =
+      if (originalPlan.predicates.nonEmpty && conf.inMemoryPartitionPruning) {
+        val filter = serializer.buildFilter(originalPlan.predicates, relationOutput)
+        cachedBuffers.mapPartitionsWithIndex(filter)
+      } else {
+        cachedBuffers
+      }
+
+    serializer
+      .convertCachedBatchToColumnarBatch(filteredBuffers, relationOutput, scanOutput, conf)
+      .map { cb =>
+        numOutputRows += cb.numRows()
+        cb
+      }
+  }
+}
+
+object CometInMemoryTableScanExec extends CometOperatorSerde[InMemoryTableScanExec] {
+
+  override def enabledConfig: Option[org.apache.comet.ConfigEntry[Boolean]] =
+    Some(CometConf.COMET_EXEC_IN_MEMORY_CACHE_ENABLED)
+
+  override def convert(
+      op: InMemoryTableScanExec,
+      builder: OperatorOuterClass.Operator.Builder,
+      childOp: Operator*): Option[Operator] = {
+
+    val scanTypes = op.output.flatMap(attr => serializeDataType(attr.dataType))

Review Comment:
   **[P2] Widen nested nullability in the cache scan schema**
   
   This serializes containsNull=false/valueContainsNull=false into the native scan schema, reintroducing the nested-type failures already fixed for CometLocalTableScanExec. On Spark 4.1.3, caching Seq(Seq(1, 2, 3), Seq(4, 5)) and evaluating slice(x, 2, 2) fails because spark_array_slice returns List(non-null Int32) while its declared result is List(Int32). map_entries over a cached Map[Int, Int] similarly panics on the value field's nullability. Both work uncached and with native cache scanning disabled over the same payload. An isolated change to serializeDataType(attr.dataType.asNullable) fixes both. Please preserve that normalization at this scan boundary and add cache regressions.



##########
spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala:
##########
@@ -398,33 +447,109 @@ object Utils extends CometTypeShim with Logging {
     }
   }
 
+  /**
+   * Whether every column in `batch` is an Arrow-backed `CometVector`, so [[getBatchFieldVectors]]
+   * can hand out its vectors directly. Callers that may receive batches from a plan they did not
+   * build (e.g. Comet's cache serializer, which Spark hands the cached plan's columnar output)
+   * use this to convert foreign vectors to Arrow instead of tripping the exception below.
+   *
+   * Stricter than what [[getBatchFieldVectors]] accepts: a `ConstantColumnVector` is rejected
+   * here even though that method materializes one, so such a batch takes the conversion path
+   * rather than being materialized column by column.
+   */
+  def isArrowBacked(batch: ColumnarBatch): Boolean =
+    (0 until batch.numCols()).forall { i =>
+      batch.column(i) match {
+        // Not every CometVector can be handed to getFieldVector: a CometPlainVector can wrap a
+        // LargeVarCharVector or LargeVarBinaryVector (an accelerated mapInArrow returning
+        // pa.large_string(), for instance), which it rejects. Answering true for those would
+        // send a batch down the direct write path that then fails, so check the vector itself
+        // and let the caller convert instead.
+        case v: CometVector => isSupportedFieldVector(v.getValueVector)
+        case _ => false
+      }
+    }
+
   def getBatchFieldVectors(
       batch: ColumnarBatch): (Seq[FieldVector], Option[DictionaryProvider]) = {
-    var provider: Option[DictionaryProvider] = None
+    val columns = getBatchFieldVectorsWithProviders(batch)
+    (columns.map(_._1), combineDictionaryProviders(columns))
+  }
+
+  /**
+   * The dictionaries every dictionary-encoded column of `columns` refers to, as one provider.
+   *
+   * Columns of a batch need not share a provider. Comet's cache decodes each column from its own
+   * Arrow stream, so a dictionary-backed column arrives carrying the provider its reader built,
+   * and a batch that reaches [[serializeBatches]] -- a native broadcast of a cache scan, say --
+   * can hold several. Writing the whole batch emits one schema covering every column and resolves
+   * each column's dictionary ID against the single provider the writer was given, so handing it
+   * any one column's provider fails with "Could not find dictionary with ID n" for the others.
+   */
+  private def combineDictionaryProviders(
+      columns: Seq[(FieldVector, Option[DictionaryProvider])]): Option[DictionaryProvider] = {
+    val dictionaries = scala.collection.mutable.LinkedHashMap.empty[Long, Dictionary]
+
+    columns.foreach { case (vector, providerOpt) =>
+      val encoding = vector.getField.getDictionary
+      if (encoding != null) {
+        val id = encoding.getId
+        val dictionary = providerOpt.map(_.lookup(id)).orNull
+        if (dictionary == null) {
+          throw new SparkException(
+            s"Column ${vector.getField.getName} is dictionary encoded with ID $id, but no " +
+              "dictionary with that ID was provided")
+        }
+        dictionaries.get(id) match {
+          // Every provider seen here descends from one upstream reader, which numbers the
+          // dictionaries it hands out, so two columns sharing an ID share the dictionary itself.
+          // A genuine clash would need renumbering, which means rewriting each vector's field,
+          // so refuse rather than silently decode one column against another's dictionary.
+          case Some(existing) if existing.getVector ne dictionary.getVector =>
+            throw new SparkException(
+              s"Columns of the same batch carry different dictionaries under ID $id")

Review Comment:
   **[P2] Handle equivalent dictionaries from repeated cached columns**
   
   Selecting one cached dictionary column twice opens two readers for the same IPC stream. Their dictionary IDs and values match, but their vector objects differ, so this identity check rejects valid data. With a dictionary-encoded cache produced through JVM shuffle and AQE disabled, spark.range(1).join(broadcast(cached.select(cached("s1"), cached("s1"))), lit(true), "inner").collect() fails with 'Columns of the same batch carry different dictionaries under ID 0'. Both plans use CometBroadcastExchange; disabling only native cache scanning returns all 2,000 rows correctly. Please support equivalent dictionaries or normalize IDs instead of requiring object identity.



##########
spark/src/test/scala/org/apache/spark/sql/benchmark/CometInMemoryCacheBenchmark.scala:
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.benchmark
+
+import org.apache.spark.SparkConf
+import org.apache.spark.benchmark.Benchmark
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.internal.SQLConf
+
+import org.apache.comet.{CometConf, CometSparkSessionExtensions}
+
+object CometInMemoryCacheBenchmark extends CometBenchmarkBase {
+  private val numRows = 5 * 1000 * 1000
+  private val cacheTable = "comet_cache_bench"
+  private val sourceTable = "comet_cache_bench_src"
+
+  override def getSparkSession: SparkSession = {
+    val conf = new SparkConf()
+      .setAppName("CometInMemoryCacheBenchmark")
+      .set("spark.master", "local[1]")
+      .setIfMissing("spark.driver.memory", "3g")
+      .setIfMissing("spark.executor.memory", "3g")
+      .set("spark.plugins", "org.apache.spark.CometPlugin")
+      .set(
+        "spark.shuffle.manager",
+        "org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager")
+      .set(
+        "spark.sql.cache.serializer",
+        "org.apache.spark.sql.comet.execution.arrow.ArrowCachedBatchSerializer")
+
+    val sparkSession = SparkSession
+      .builder()
+      .config(conf)
+      .withExtensions(new CometSparkSessionExtensions)
+      .getOrCreate()
+
+    sparkSession.conf.set(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key, "true")
+    sparkSession.conf.set(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key, "false")
+    sparkSession.conf.set(SQLConf.CACHE_VECTORIZED_READER_ENABLED.key, "true")
+    sparkSession.conf.set(SQLConf.ANSI_ENABLED.key, "false")
+    sparkSession.conf.set(CometConf.COMET_ENABLED.key, "false")
+    sparkSession.conf.set(CometConf.COMET_EXEC_ENABLED.key, "false")
+    sparkSession
+  }
+
+  override def runCometBenchmark(args: Array[String]): Unit = {
+    withTempTable(sourceTable, cacheTable) {
+      spark
+        .range(0, numRows, 1, 16)
+        .selectExpr(
+          "id",
+          "id % 1000 AS k",
+          "id + 1 AS v",
+          "concat('str_a_', cast(id % 100000 as string)) AS s1",
+          "concat('str_b_', cast(id % 7919 as string)) AS s2",
+          "concat('str_c_', cast(id as string)) AS s3")
+        .createOrReplaceTempView(sourceTable)
+
+      runCacheBenchmark(
+        "in-memory cache repeated scan",
+        s"SELECT sum(id), sum(k), sum(v) FROM $cacheTable")
+
+      runCacheBenchmark(
+        "in-memory cache selective filter",
+        s"""
+           |SELECT sum(id), sum(k), sum(v)
+           |FROM $cacheTable
+           |WHERE id >= 4500000 AND id < 4750000
+         """.stripMargin)
+
+      // A CometCachedBatch stores each column as its own stream, so a scan decodes only what it
+      // projected and cost tracks the width of the projection. These three cases span that range
+      // over one cached relation: no columns, one column, and all six.
+      runCacheBenchmark(
+        "in-memory cache row count only (0 of 6 columns)",
+        s"SELECT count(*) FROM $cacheTable")
+
+      runCacheBenchmark(
+        "in-memory cache narrow projection (1 of 6 columns)",
+        s"SELECT count(k) FROM $cacheTable")
+
+      runCacheBenchmark(
+        "in-memory cache full projection (6 of 6 columns)",
+        s"SELECT count(id), count(k), count(v), count(s1), count(s2), count(s3) FROM $cacheTable")

Review Comment:
   **[P3] Make the full projection benchmark consume all six columns**
   
   The Range-derived id, v, and s3 columns are non-nullable, so Catalyst rewrites count(id), count(v), and count(s3) to count(1). Both the native and fallback benchmark plans therefore scan only k, s1, and s2, despite this case being labeled 6 of 6 columns. This also omits the highest-cardinality string column s3. Use expressions that consume all six columns and assert the actual cache-scan projection, otherwise the benchmark does not measure the full-width per-column decoding cost it claims.



##########
spark/src/main/scala/org/apache/spark/sql/comet/CometInMemoryTableScanExec.scala:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.columnar.CachedBatchSerializer
+import org.apache.spark.sql.execution.{LeafExecNode, SparkPlan}
+import org.apache.spark.sql.execution.columnar.{CachedRDDBuilder, InMemoryTableScanExec}
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.CometConf
+import org.apache.comet.serde.CometOperatorSerde
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.serializeDataType
+
+/**
+ * Reads Spark cached table data when the cache was written by Comet's cache serializer.
+ *
+ * Spark stores cached data through `CachedBatchSerializer`. This node keeps the scan inside Comet
+ * by asking the serializer to decode cached batches directly into `ColumnarBatch` output,
+ * avoiding the extra Spark columnar-to-Comet columnar conversion used by the default path.
+ *
+ * `relationOutput` is the full schema stored in the cache. `scanOutput` is the subset requested
+ * by this scan after pruning.
+ */
+case class CometInMemoryTableScanExec(
+    originalPlan: InMemoryTableScanExec,
+    serializer: CachedBatchSerializer,
+    cacheBuilder: CachedRDDBuilder,
+    relationOutput: Seq[Attribute],
+    scanOutput: Seq[Attribute])
+    extends CometExec
+    with LeafExecNode {

Review Comment:
   **[P2] Preserve AQE cache materialization and reoptimization**
   
   Replacing Spark's InMemoryTableScanLike with an ordinary leaf during AQE preparation prevents TableCacheQueryStageExec creation. On Spark 4.1.3, first reading an adaptive cached join through a grouped aggregate executes an extra outer shuffle with native caching enabled: native disabled materializes the cache and removes that shuffle, while enabled retains it. A join against a cold cache already partitioned by its join key similarly runs two outer shuffles instead of one. Warm-cache queries avoid the difference. Results are correct, but the cold path unnecessarily reshuffles data. Please preserve materialization and reoptimization at the cache boundary. This is an executed failure of the scenarios requested in #5245, not just a coverage concern.



##########
spark/src/test/scala/org/apache/comet/exec/CometInMemoryCacheKryoSuite.scala:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.exec
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.CometTestBase
+import org.apache.spark.sql.execution.columnar.CometInMemoryRelationHelper
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.storage.StorageLevel
+
+import org.apache.comet.{CometConf, CometKryoRegistrator}
+
+/**
+ * Covers Comet's cached batch format under `spark.kryo.registrationRequired=true`.
+ *
+ * Kryo then rejects any class it has not been told about, and Spark serializes a `CachedBatch`
+ * whenever a cached block leaves the heap: the disk half of the default `MEMORY_AND_DISK`, the
+ * `_SER` levels, replication, and cross-executor fetches. So this is not a `DISK_ONLY`-only
+ * concern -- a plain `df.cache()` that spills is enough to reach it. Spark registers its own
+ * `ArrowCachedBatch` in `KryoSerializer.loadableSparkClasses`; Comet cannot add to that list, so
+ * [[CometKryoRegistrator]] has to be set explicitly, and this suite is what proves it is
+ * sufficient.
+ *
+ * This needs its own suite because `spark.serializer` and `spark.kryo.registrator` are read when
+ * `SparkEnv` builds the serializer, so they cannot be changed per test.
+ */
+class CometInMemoryCacheKryoSuite extends CometTestBase {

Review Comment:
   **[P2] Add the new Kryo suite to both CI test matrices**
   
   This new suite is absent from the suite lists in both .github/workflows/pr_build_linux.yml and .github/workflows/pr_build_macos.yml. python3 dev/ci/check-suites.py exits 255 with 'Suite not found in workflow ... CometInMemoryCacheKryoSuite'; the current GitHub Preflight job fails for exactly this reason and the platform/Spark test jobs are skipped. Add the suite to both matrices so CI can validate the PR and keep exercising the new Kryo regressions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5332658158

   I'll check it today
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5431888717

   @sandugood sorry for the long silence, and thanks for chasing it. Merged latest `apache/main` and have an answer on both halves of your report.
   
   **The crash.** #5138 merged on Aug 7, the day after you tested, so the build you hit did not have it. It normalizes nested field nullability, which is exactly the `expected Struct("col_1": Utf8, "col_2": Boolean, ...)` versus `found Struct("col_1": Utf8, "col_2": non-null Boolean, ...)` mismatch you saw. It is in this branch now. If you get a chance to retest, that is the one thing I still cannot verify locally, since Iceberg MOR is the input I have no reproduction for.
   
   **The performance regression.** You were right, and it is a real property of the cache format rather than anything specific to your pipeline. I reproduced it.
   
   `CometCachedBatch` stores each cached batch as one compressed Arrow IPC stream covering every cached column. `convertCachedBatchToColumnarBatch` decodes that whole stream and then projects, so read cost does not fall as the projection narrows. Spark's `DefaultCachedBatch` stores columns separately and only decodes the ones the scan selected.
   
   Measured on 5M rows, 6 columns (3 longs, 3 strings), cached and then read repeatedly by Spark operators, comparing the two serializers directly:
   
   | Read shape | Spark cache | Comet cache | Ratio |
   |------------|------------:|------------:|------:|
   | 1 of 6 columns | 353 ms | 1175 ms | 3.3x slower |
   | 3 of 6 columns | 1078 ms | 1859 ms | 1.7x slower |
   | 6 of 6 columns | 1227 ms | 1728 ms | 1.4x slower |
   | Materialize cache | 5050 ms | 2468 ms | 2.0x faster |
   
   Writing the cache got faster, reading it got slower, and a cache-once/read-many pipeline that projects a subset of a wide relation ends up behind. That matches your under 10 minutes going to over 15. Your later note that there is no regression without `.cache()` fits too: the cost is entirely on the cached read path.
   
   Two things I got wrong that are worth stating plainly, since they are why this was not caught here:
   
   1. **The benchmark in this PR never compared against Spark's cache format.** `spark.sql.cache.serializer` is a static config, so one session cannot materialize both formats. Both benchmark cases read a `CometCachedBatch`, and "Comet cache disabled" meant Spark execution over Comet's payload. The 1.5x I quoted was not a speedup over Spark's cache. I have corrected the PR description and relabelled the cases.
   2. **`InMemoryRelation` memoizes the resolved serializer in a JVM-static field.** My first attempt at a cross-serializer measurement read `DefaultCachedBatch` in every phase and showed no regression at all, vacuously. It needs `InMemoryRelation.clearSerializer()` between sessions.
   
   **What is in the branch now**, on top of the `apache/main` merge:
   
   * The benchmark relation is widened with string columns and has narrow (1 of 6) and full (6 of 6) projection cases, so the effect is visible rather than invisible.
   * The missing baseline is stated at the materialization site, so nobody reads those numbers as a Spark comparison again.
   * The read cost is documented on `spark.comet.exec.inMemoryCache.enabled`.
   
   Fixing it properly means making decode proportional to the projection, most likely one Arrow stream per column so only the selected ones are inflated. That is a change to the cache format itself and too large to fold in here, so it is filed as #5484 with the numbers and the options.
   
   The feature stays off by default, which I think is the right call given the above. @peterxcli @comphead this changes what the benchmark numbers in the description mean, so worth a look if you have already reviewed them.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "sandugood (via GitHub)" <gi...@apache.org>.
sandugood commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5183245585

   > @sandugood could you test this PR again when you have time? Thanks.
   
   Sure, going to test it, thanks @andygrove 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3709230076


##########
spark/src/main/scala/org/apache/spark/sql/comet/CometInMemoryTableScanExec.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.columnar.{CachedBatch, CachedBatchSerializer}
+import org.apache.spark.sql.execution.LeafExecNode
+import org.apache.spark.sql.execution.columnar.InMemoryTableScanExec
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.CometConf
+import org.apache.comet.serde.CometOperatorSerde
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.serializeDataType
+
+/**
+ * Reads Spark cached table data when the cache was written by Comet's cache serializer.
+ *
+ * Spark stores cached data through `CachedBatchSerializer`. This node keeps the scan inside Comet
+ * by asking the serializer to decode cached batches directly into `ColumnarBatch` output,
+ * avoiding the extra Spark columnar-to-Comet columnar conversion used by the default path.
+ *
+ * `relationOutput` is the full schema stored in the cache. `scanOutput` is the subset requested
+ * by this scan after pruning.
+ */
+case class CometInMemoryTableScanExec(

Review Comment:
   Filed as #5245, scoped to the three scenarios you linked, run with `spark.comet.exec.inMemoryCache.enabled=true` and asserting the cached scan is `CometInMemoryTableScanExec`.
   
   I called out SPARK-37742 as the one to weight most: the Comet serializer reports its own sizes for `CometCachedBatch`, so the runtime statistics AQE reads when picking a join strategy come from Comet code rather than Spark's `DefaultCachedBatch` accounting. That is the scenario where a divergence would be silent and would change plans rather than results.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "sandugood (via GitHub)" <gi...@apache.org>.
sandugood commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5119035862

   lgtm👍


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3677718311


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet.execution.arrow
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, GenericInternalRow, IsNotNull, IsNull, UnsafeProjection}
+import org.apache.spark.sql.columnar.{CachedBatch, SimpleMetricsCachedBatch, SimpleMetricsCachedBatchSerializer}
+import org.apache.spark.sql.comet.util.Utils
+import org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.unsafe.types.{ByteArray, UTF8String}
+import org.apache.spark.util.io.ChunkedByteBuffer
+
+import org.apache.comet.CometArrowAllocator
+
+/**
+ * Cached batch format used when Comet writes Spark in-memory cache data.
+ *
+ * `bytes` contains compressed Arrow stream data produced by `Utils.serializeBatches`. The cache
+ * manager still owns storage and eviction; this class only changes the cached payload.
+ */
+private case class CometCachedBatch(
+    override val numRows: Int,
+    override val sizeInBytes: Long,
+    override val stats: InternalRow,
+    bytes: ChunkedByteBuffer)
+    extends SimpleMetricsCachedBatch
+
+/**
+ * Cache serializer that stores Comet-compatible Arrow batches in Spark's in-memory cache.
+ *
+ * The cached payload format is decided by the schema alone. A relation whose schema Comet's Arrow
+ * writer supports is stored as `CometCachedBatch`, and every other relation is delegated in full
+ * to Spark's `DefaultCachedBatchSerializer`. The format deliberately does not depend on any
+ * runtime config: `spark.sql.cache.serializer` is a static conf, so installing this serializer is
+ * already a per-application decision, and a relation whose format could flip mid-session cannot
+ * be read back reliably. `spark.comet.exec.inMemoryCache.enabled` still governs whether a scan
+ * over the cache runs natively, and its value at startup is what makes `CometDriverPlugin`
+ * install this serializer in the first place.
+ *
+ * Reads of `CometCachedBatch` keep working when the native scan is disabled, because Spark then
+ * reads the same cached data through the SparkToColumnar fallback path.
+ */
+class ArrowCachedBatchSerializer extends SimpleMetricsCachedBatchSerializer {
+
+  import ArrowCachedBatchSerializer.supportsSchema
+
+  private val fallback = new DefaultCachedBatchSerializer()
+
+  // Row-to-Arrow conversion needs a StructType, while cache APIs pass attributes.
+  private def toStructType(schema: Seq[Attribute]): StructType = {
+    StructType(schema.map { attr =>
+      StructField(attr.name, attr.dataType, attr.nullable, attr.metadata)
+    })
+  }
+
+  // Build the statistics row expected by SimpleMetricsCachedBatchSerializer.
+  // For each cached column Spark expects five values in this order:
+  // lower bound, upper bound, null count, row count, and size in bytes.
+  private def computeStats(batch: ColumnarBatch, attrs: Seq[Attribute]): InternalRow = {
+    val numCols = attrs.length
+    val lower = new Array[Any](numCols)
+    val upper = new Array[Any](numCols)
+    val nulls = Array.fill[Int](numCols)(0)
+    val numRows = batch.numRows()
+
+    var c = 0
+    while (c < numCols) {
+      val dt = attrs(c).dataType
+      val col = batch.column(c)
+      var r = 0
+      while (r < numRows) {
+        if (col.isNullAt(r)) {
+          nulls(c) += 1
+        } else if (tracksBounds(dt)) {
+          val value = readValue(col, dt, r)
+          if (lower(c) == null || compare(dt, value, lower(c)) < 0) {
+            lower(c) = value
+          }
+          if (upper(c) == null || compare(dt, value, upper(c)) > 0) {
+            upper(c) = value
+          }
+        }
+        r += 1
+      }
+      c += 1
+    }
+
+    val values = new Array[Any](numCols * 5)
+    c = 0
+    while (c < numCols) {
+      val base = c * 5
+      values(base) = lower(c)
+      values(base + 1) = upper(c)
+      values(base + 2) = nulls(c)
+      values(base + 3) = numRows
+      // Spark reserves the fifth field for per-column size. Comet stores the whole
+      // Arrow stream as one compressed buffer, so per-column size is not tracked here.
+      // Cache pruning uses bounds/null-count/row-count, not this size field.
+      values(base + 4) = 0L
+      c += 1
+    }
+
+    new GenericInternalRow(values)
+  }
+
+  // Spark can prune cache batches only for types whose bounds can be compared.
+  // Other types still report null count and row count but leave bounds as null.
+  private def tracksBounds(dt: DataType): Boolean = dt match {
+    case BooleanType | ByteType | ShortType | IntegerType | LongType | FloatType | DoubleType |
+        _: DecimalType | StringType | DateType | TimestampType | TimestampNTZType =>
+      true
+    case _ => false
+  }
+
+  // Read a non-null value from a ColumnVector using Spark's internal value type
+  // for the corresponding DataType.
+  private def readValue(col: ColumnVector, dt: DataType, rowId: Int): Any = dt match {
+    case BooleanType => col.getBoolean(rowId)
+    case ByteType => col.getByte(rowId)
+    case ShortType => col.getShort(rowId)
+    case IntegerType | DateType => col.getInt(rowId)
+    case LongType | TimestampType | TimestampNTZType => col.getLong(rowId)
+    case FloatType => col.getFloat(rowId)
+    case DoubleType => col.getDouble(rowId)
+    case d: DecimalType => col.getDecimal(rowId, d.precision, d.scale)
+    case StringType => col.getUTF8String(rowId).copy()
+    case _ => null
+  }
+
+  // Compare values using the same physical representation used in the stats row.
+  private def compare(dt: DataType, left: Any, right: Any): Int = dt match {
+    case BooleanType =>
+      java.lang.Boolean.compare(left.asInstanceOf[Boolean], right.asInstanceOf[Boolean])
+    case ByteType =>
+      java.lang.Byte.compare(left.asInstanceOf[Byte], right.asInstanceOf[Byte])
+    case ShortType =>
+      java.lang.Short.compare(left.asInstanceOf[Short], right.asInstanceOf[Short])
+    case IntegerType | DateType =>
+      java.lang.Integer.compare(left.asInstanceOf[Int], right.asInstanceOf[Int])
+    case LongType | TimestampType | TimestampNTZType =>
+      java.lang.Long.compare(left.asInstanceOf[Long], right.asInstanceOf[Long])
+    case FloatType =>
+      java.lang.Float.compare(left.asInstanceOf[Float], right.asInstanceOf[Float])
+    case DoubleType =>
+      java.lang.Double.compare(left.asInstanceOf[Double], right.asInstanceOf[Double])
+    case _: DecimalType =>
+      left.asInstanceOf[Decimal].compare(right.asInstanceOf[Decimal])
+    case StringType =>
+      ByteArray.compareBinary(
+        left.asInstanceOf[UTF8String].getBytes,
+        right.asInstanceOf[UTF8String].getBytes)
+    case other =>
+      throw new IllegalStateException(s"compare called for unsupported type $other")
+  }
+
+  // Compute Spark-compatible cache stats before serializing each batch to Arrow.
+  // The stats are stored beside the Arrow bytes so Spark's cache filter can prune
+  // CometCachedBatch without decoding the batch first.
+  private def encodeBatches(
+      batches: Iterator[ColumnarBatch],
+      attrs: Seq[Attribute]): Iterator[CachedBatch] = {
+    batches.flatMap { batch =>
+      val stats = computeStats(batch, attrs)
+
+      Utils.serializeBatches(Iterator.single(batch)).map { case (rows, buffer) =>
+        CometCachedBatch(
+          numRows = rows.toInt,
+          sizeInBytes = buffer.size,
+          stats = stats,
+          bytes = buffer)
+      }
+    }
+  }
+
+  // Resolve requested columns by exprId, not by name, because aliases may reuse names.
+  private def selectedIndices(
+      cacheAttributes: Seq[Attribute],
+      selectedAttributes: Seq[Attribute]): Array[Int] = {
+    if (selectedAttributes.isEmpty) {
+      cacheAttributes.indices.toArray
+    } else {
+      val byExprId = cacheAttributes.zipWithIndex.map { case (attr, idx) =>
+        attr.exprId -> idx
+      }.toMap
+
+      selectedAttributes.map { attr =>
+        byExprId.getOrElse(
+          attr.exprId,
+          throw new IllegalStateException(
+            s"Could not resolve selected attribute ${attr.name} from cache attributes"))
+      }.toArray
+    }
+  }
+
+  // A full-width projection is only an identity projection if every selected index
+  // is already in column order. For example, [1, 0] must still be projected.
+  private def isIdentityProjection(indices: Array[Int], numCols: Int): Boolean =
+    indices.length == numCols && indices.indices.forall(i => indices(i) == i)
+
+  private def projectBatch(batch: ColumnarBatch, indices: Array[Int]): ColumnarBatch = {
+    if (isIdentityProjection(indices, batch.numCols())) {
+      batch
+    } else {
+      val cols = indices.map(i => batch.column(i).asInstanceOf[ColumnVector])
+      new ColumnarBatch(cols, batch.numRows())
+    }
+  }
+
+  // Spark's SimpleMetricsCachedBatchSerializer prunes a batch when the generated partition filter
+  // does not evaluate to true against the stats row. Bounds are only computed for the types
+  // tracksBounds accepts, and for every other column the lower and upper bounds stay null, which
+  // makes a comparison against them evaluate to null and therefore prune the batch. That would
+  // silently drop rows, so predicates over columns without bounds are not pushed down at all.
+  // Null counts and row counts are recorded for every column, so IsNull and IsNotNull stay safe.
+  override def buildFilter(
+      predicates: Seq[Expression],
+      cachedAttributes: Seq[Attribute]): (Int, Iterator[CachedBatch]) => Iterator[CachedBatch] = {
+    val prunable = cachedAttributes.collect {
+      case a if tracksBounds(a.dataType) => a.exprId
+    }.toSet
+
+    val prunablePredicates = predicates.filter {
+      case _: IsNull | _: IsNotNull => true
+      case p => p.references.forall(a => prunable.contains(a.exprId))
+    }
+
+    super.buildFilter(prunablePredicates, cachedAttributes)
+  }
+
+  // Comet's Arrow writer only handles the types listed in supportsSchema. Reporting false here
+  // sends the relation down the row path, where it is delegated to Spark's default serializer,
+  // instead of failing at cache materialization inside Utils.serializeBatches.
+  override def supportsColumnarInput(schema: Seq[Attribute]): Boolean = supportsSchema(schema)

Review Comment:
   `supportsColumnarInput` promises support for any columnar producer whose schema is supported, but `convertColumnarBatchToCachedBatch` eventually calls `Utils.getBatchFieldVectors`, which only accepts `CometVector`. Spark can expose vectorized Parquet/ORC plans here, causing otherwise supported caches to receive `OnHeapColumnVector`/`OffHeapColumnVector` and throw. Please either return `false` here and use the already-working row-to-Arrow path—the smallest safe fix—or add generic Spark-vector conversion. A vectorized Parquet/ORC cache-materialization regression with Comet execution disabled would cover this.
   
   _Assist with LLM_



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3677808670


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowConverters.scala:
##########
@@ -74,4 +74,45 @@ object CometArrowConverters extends Logging {
       }
     }
   }
+
+  /**
+   * Copy a Spark `ColumnarBatch` whose columns are not Arrow-backed (e.g.
+   * `On/OffHeapColumnVector` from Spark's vectorized Parquet reader, or a third-party connector's
+   * vectors) into a freshly allocated Arrow `ColumnarBatch` of `CometVector`s.
+   *
+   * The input batch is not consumed or closed; the caller owns the returned batch and must close
+   * it. Values are copied element-wise, since Spark's `ColumnVector` implementations do not
+   * expose Arrow buffers.
+   */
+  def columnarBatchToArrowBatch(

Review Comment:
   checking which type of spark column vector does the `SparkColumnarArrowReader` support...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5094118532

   @0lai0 @manuzhang @sandugood @peterxcli could you help with reviews on this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3693198326


##########
spark/src/main/scala/org/apache/spark/sql/comet/CometInMemoryTableScanExec.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.columnar.{CachedBatch, CachedBatchSerializer}
+import org.apache.spark.sql.execution.LeafExecNode
+import org.apache.spark.sql.execution.columnar.InMemoryTableScanExec
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.CometConf
+import org.apache.comet.serde.CometOperatorSerde
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.serializeDataType
+
+/**
+ * Reads Spark cached table data when the cache was written by Comet's cache serializer.
+ *
+ * Spark stores cached data through `CachedBatchSerializer`. This node keeps the scan inside Comet
+ * by asking the serializer to decode cached batches directly into `ColumnarBatch` output,
+ * avoiding the extra Spark columnar-to-Comet columnar conversion used by the default path.
+ *
+ * `relationOutput` is the full schema stored in the cache. `scanOutput` is the subset requested
+ * by this scan after pruning.
+ */
+case class CometInMemoryTableScanExec(
+    originalPlan: InMemoryTableScanExec,
+    serializer: CachedBatchSerializer,
+    cachedBuffers: RDD[CachedBatch],
+    relationOutput: Seq[Attribute],
+    scanOutput: Seq[Attribute])
+    extends CometExec
+    with LeafExecNode {
+
+  override lazy val metrics: Map[String, SQLMetric] = Map(
+    "numOutputRows" -> SQLMetrics.createMetric(sparkContext, "number of output rows"))
+
+  // For an empty-projection scan (`SELECT count(*)`) this is empty while `scanOutput` holds the
+  // full cache schema, so the emitted batches are wider than the declared output. That is safe
+  // because the only consumer of an empty-output scan is a count-style aggregate, which reads
+  // the row count rather than any column; `convert` and `createExec` deliberately fall back to
+  // the cache schema in that case because the native plan still needs a non-empty scan schema.
+  override def output: Seq[Attribute] = originalPlan.output
+
+  // Use the serializer's vector types because the cached batch layout is owned by the serializer.
+  override def vectorTypes: Option[Seq[String]] =
+    serializer.vectorTypes(scanOutput, conf)
+
+  // Apply Spark's cache batch filter before decoding. Spark's InMemoryTableScanExec does this in
+  // filteredCachedBatches(), but that method is private. Reusing the serializer's buildFilter here
+  // keeps Comet on the same stats-based pruning path instead of decoding every cached batch.
+  override def doExecuteColumnar(): RDD[ColumnarBatch] = {

Review Comment:
   Real bug, fixed in 1367665bd — thanks. Spark's `InMemoryTableScanExec.filteredCachedBatches` gates on `conf.inMemoryPartitionPruning` and Comet was applying the filter unconditionally, so the config was silently ignored.
   
   To your "not sure enable partitionPruning is always better than disable": pruning is normally the win, but that is not really the point — the config exists so a user *can* turn it off, typically to rule out a stats bug when results look wrong. Ignoring it means the one knob they are reaching for does nothing, and Comet diverges from Spark precisely when someone is debugging. So honoring it is right regardless of which default is faster.
   
   Covered by a new test that observes the scan's `numOutputRows`, which counts rows in the batches actually decoded: **100 with pruning on** (a single 100-row batch) and **1000 with it off** (every batch decoded). One wrinkle worth noting for anyone writing a similar test: the metric has to be read after forcing that exact `df` to run, because `checkSparkAnswer` executes its own copies and leaves the collected plan instance at zero — my first version passed vacuously with 0 on both sides.



##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet.execution.arrow
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, GenericInternalRow, IsNotNull, IsNull, UnsafeProjection}
+import org.apache.spark.sql.columnar.{CachedBatch, SimpleMetricsCachedBatch, SimpleMetricsCachedBatchSerializer}
+import org.apache.spark.sql.comet.util.Utils
+import org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.unsafe.types.{ByteArray, UTF8String}
+import org.apache.spark.util.io.ChunkedByteBuffer
+
+import org.apache.comet.CometArrowAllocator
+
+/**
+ * Cached batch format used when Comet writes Spark in-memory cache data.
+ *
+ * `bytes` contains compressed Arrow stream data produced by `Utils.serializeBatches`. The cache
+ * manager still owns storage and eviction; this class only changes the cached payload.
+ */
+private case class CometCachedBatch(
+    override val numRows: Int,
+    override val sizeInBytes: Long,
+    override val stats: InternalRow,
+    bytes: ChunkedByteBuffer)
+    extends SimpleMetricsCachedBatch
+
+/**
+ * Cache serializer that stores Comet-compatible Arrow batches in Spark's in-memory cache.
+ *
+ * The cached payload format is decided by the schema alone. A relation whose schema Comet's Arrow
+ * writer supports is stored as `CometCachedBatch`, and every other relation is delegated in full
+ * to Spark's `DefaultCachedBatchSerializer`. The format deliberately does not depend on any
+ * runtime config: `spark.sql.cache.serializer` is a static conf, so installing this serializer is
+ * already a per-application decision, and a relation whose format could flip mid-session cannot
+ * be read back reliably. `spark.comet.exec.inMemoryCache.enabled` still governs whether a scan
+ * over the cache runs natively, and its value at startup is what makes `CometDriverPlugin`
+ * install this serializer in the first place.
+ *
+ * Reads of `CometCachedBatch` keep working when the native scan is disabled, because Spark then
+ * reads the same cached data through the SparkToColumnar fallback path.
+ */
+class ArrowCachedBatchSerializer extends SimpleMetricsCachedBatchSerializer {

Review Comment:
   Real leak, fixed in 1367665bd. Confirmed the mechanism: `ArrowReaderIterator.close()` is only reached from `hasNext` when the stream is exhausted, so LIMIT, `take()`, or a cancelled task leaves the reader it was part-way through open.
   
   `convertCachedBatchToColumnarBatch` now registers a `TaskCompletionListener`, following the Spark implementation you linked. One simplification: `flatMap` consumes each inner iterator fully before constructing the next, so at most one reader is open at a time and tracking the current one is enough rather than a collection. `close()` is already idempotent and `synchronized`, so closing one that exhausted itself is a no-op and the listener is safe to run from another thread.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3693199806


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet.execution.arrow
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, GenericInternalRow, IsNotNull, IsNull, UnsafeProjection}
+import org.apache.spark.sql.columnar.{CachedBatch, SimpleMetricsCachedBatch, SimpleMetricsCachedBatchSerializer}
+import org.apache.spark.sql.comet.util.Utils
+import org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.unsafe.types.{ByteArray, UTF8String}
+import org.apache.spark.util.io.ChunkedByteBuffer
+
+import org.apache.comet.CometArrowAllocator
+
+/**
+ * Cached batch format used when Comet writes Spark in-memory cache data.
+ *
+ * `bytes` contains compressed Arrow stream data produced by `Utils.serializeBatches`. The cache
+ * manager still owns storage and eviction; this class only changes the cached payload.
+ */
+private case class CometCachedBatch(
+    override val numRows: Int,
+    override val sizeInBytes: Long,
+    override val stats: InternalRow,
+    bytes: ChunkedByteBuffer)

Review Comment:
   Added in 1367665bd, following your spec: "supports DISK_ONLY storage level" materializes the cache at `StorageLevel.DISK_ONLY` and asserts `memSize == 0`, `diskSize > 0`, `numCachedPartitions == numPartitions`, that the payload is still `CometCachedBatch`, and that a second query returns the right answer through `CometInMemoryTableScan`.
   
   Your reasoning was right — `ChunkedByteBuffer` being `Externalizable` means `BlockManager` spills it like any other block, and it worked first time with no production change needed. Worth having the regression test precisely because nothing in Comet's code makes that true, so it could break without notice.



##########
spark/src/main/scala/org/apache/spark/sql/comet/CometInMemoryTableScanExec.scala:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.columnar.{CachedBatch, CachedBatchSerializer}
+import org.apache.spark.sql.execution.LeafExecNode
+import org.apache.spark.sql.execution.columnar.InMemoryTableScanExec
+import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import org.apache.comet.CometConf
+import org.apache.comet.serde.CometOperatorSerde
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.QueryPlanSerde.serializeDataType
+
+/**
+ * Reads Spark cached table data when the cache was written by Comet's cache serializer.
+ *
+ * Spark stores cached data through `CachedBatchSerializer`. This node keeps the scan inside Comet
+ * by asking the serializer to decode cached batches directly into `ColumnarBatch` output,
+ * avoiding the extra Spark columnar-to-Comet columnar conversion used by the default path.
+ *
+ * `relationOutput` is the full schema stored in the cache. `scanOutput` is the subset requested
+ * by this scan after pruning.
+ */
+case class CometInMemoryTableScanExec(

Review Comment:
   Not done, and I would rather flag that than half-do it. Porting the three AQE scenarios you linked (SPARK-42101 cold/warm materialization, the `TableCacheQueryStageExec` join, SPARK-37742) is a meaningful chunk of work and touches AQE plan-shape assertions rather than the cache format this PR changes, so I have left it out of this round rather than rushing it alongside the four correctness fixes.
   
   You said your comments could be follow-ups, so unless you would rather block on it, my suggestion is a tracking issue for AQE coverage of `CometInMemoryTableScanExec`. I have not filed one yet — say the word and I will, or feel free to open it yourself if you have a preference for how it should be scoped.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3693196569


##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowConverters.scala:
##########
@@ -74,4 +76,52 @@ object CometArrowConverters extends Logging {
       }
     }
   }
+
+  /**
+   * Copy `numRows` rows starting at `startRow` from a Spark `ColumnarBatch` into `root`.
+   *
+   * Spark's `ColumnVector` implementations do not expose Arrow buffers, so values are necessarily
+   * copied element-wise. Shared by [[SparkColumnarArrowReader]], which slices into a stable root,
+   * and [[columnarBatchToArrowBatch]], which fills a fresh one.
+   */
+  private[arrow] def writeColumns(
+      root: VectorSchemaRoot,
+      batch: ColumnarBatch,
+      startRow: Int,
+      numRows: Int): Unit = {
+    val writer = ArrowWriter.create(root)
+    var col = 0
+    while (col < batch.numCols()) {
+      val column = batch.column(col)
+      val columnArray = new ColumnarArray(column, startRow, numRows)
+      if (column.hasNull) {
+        writer.writeCol(columnArray, col)
+      } else {
+        writer.writeColNoNull(columnArray, col)
+      }
+      col += 1
+    }
+    writer.finish()
+    // ArrowWriter derives the root row count from its per-column writes, so a zero-column input
+    // batch (Spark's count-from-metadata scan: numRows > 0, numCols == 0) would otherwise produce
+    // a root with rowCount == 0 and silently drop the rows.
+    root.setRowCount(numRows)
+  }
+
+  /**
+   * Copy a Spark `ColumnarBatch` whose columns are not Arrow-backed (e.g.
+   * `On/OffHeapColumnVector` from Spark's vectorized Parquet reader, or a third-party connector's
+   * vectors) into a freshly allocated Arrow `ColumnarBatch` of `CometVector`s.
+   *
+   * The input batch is not consumed or closed; the caller owns the returned batch and must close
+   * it.
+   */
+  def columnarBatchToArrowBatch(
+      batch: ColumnarBatch,
+      arrowSchema: Schema,
+      allocator: BufferAllocator): ColumnarBatch = {
+    val root = VectorSchemaRoot.create(arrowSchema, allocator)

Review Comment:
   Good catch, fixed in 1367665bd. Guarded with `try`/`catch NonFatal` releasing the root before rethrowing, as you suggested.
   
   You are right that the normal path cannot hit it, but `writeColumns` walks arbitrary `ColumnVector` implementations, so a badly-behaved external vector is exactly the case that would throw here — and that is the path this PR added.
   
   `rowToArrowBatchIter` just above has the same shape (root allocated, then `writer.write(rowIter.next())` before `rootAsBatch` takes ownership), so I gave it the same guard rather than leaving one of the two fixed.



##########
spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala:
##########
@@ -397,6 +397,15 @@ object Utils extends CometTypeShim with Logging {
     }
   }
 
+  /**
+   * Whether every column in `batch` satisfies [[getBatchFieldVectors]]'s precondition, i.e. is an
+   * Arrow-backed `CometVector`. Callers that may receive batches from a plan they did not build
+   * (e.g. Comet's cache serializer, which Spark hands the cached plan's columnar output) use this
+   * to convert foreign vectors to Arrow instead of tripping the exception below.
+   */
+  def isArrowBacked(batch: ColumnarBatch): Boolean =

Review Comment:
   Yes — and this is the case the PR exists to handle.
   
   `isArrowBacked` is false whenever Comet is handed a columnar batch from a plan it did not build. The concrete reproduction is the one from the report on this PR: with `spark.comet.scan.enabled=false`, Spark's own vectorized Parquet reader produces `OnHeapColumnVector`s, `supportsColumnarInput` returns true so `InMemoryRelation.apply` strips the `ColumnarToRow` above that scan, and the cache serializer receives non-Comet vectors. Before the converter path those batches hit the exception in `getBatchFieldVectors`.
   
   It is also false for any external columnar source — an Iceberg or other connector's `ColumnVector` implementation — which is why the conversion is written against the `ColumnVector` contract rather than against specific classes.
   
   The two tests at the end of `CometInMemoryCacheSuite` ("cache a Spark columnar plan whose vectors are not Arrow-backed" and the complex-types variant) cover exactly this, so the false branch is exercised.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5432138279

   Fixed #5484 here rather than leaving it as a follow-up. A cache format that loses to Spark's on the reads people actually do is not worth shipping, even off by default, so it seemed better to fix it than to document it.
   
   **What changed.** A `CometCachedBatch` now stores one compressed Arrow IPC stream per column instead of one stream covering all of them, and `convertCachedBatchToColumnarBatch` decodes only the columns it was asked for. Two related cases fell out of the same change: an empty selection now stays empty rather than expanding to every column, and the scan asks for a single cheap column rather than the whole cache schema when a query needs only the row count. That last one is why `count(*)` was the worst case rather than the best: `CometInMemoryTableScanExec` widened an empty projection to the full schema, because the native plan needs a non-empty scan schema, and once the serializer decodes exactly what it is asked for that widening meant decoding everything. One column satisfies the native planner just as well.
   
   Per-column sizes are also known now, so the fifth statistics field Spark reserves for them holds the real value instead of zero.
   
   **Results**, 5M rows and 6 columns, min of 5 runs after a discarded warm-up, against a real `DefaultCachedBatch` baseline in a separate session:
   
   | Read shape | Spark cache | Comet before | Comet after |
   |------------|------------:|-------------:|------------:|
   | `count(*)` | 62 ms | 241 ms | **77 ms** |
   | 1 of 6 columns | 105 ms | 592 ms | **57 ms** |
   | 3 of 6 columns | 339 ms | 655 ms | **204 ms** |
   | 6 of 6 columns | 375 ms | 448 ms | 448 ms |
   | Materialize | 3497 ms | 1563 ms | 1563 ms |
   | Footprint | 309 MiB | 60 MiB | 61 MiB |
   
   Narrow and medium projections now beat Spark's cache by 1.7x to 1.8x rather than losing to it. `count(*)` is at parity. A full-width projection is still 1.2x behind, which is decode and conversion cost rather than anything to do with projection.
   
   **The cost**, since I said I would measure it rather than assume it. Per-column framing adds a schema block and compression framing per column per batch, and gives up cross-column compression. Footprint grows 2.5% at 6 columns and 32% at 60 columns. At 60 columns the cached relation is still 107 MiB against Spark's 2369 MiB, so I took the tradeoff, but it is a real one and it scales with width, so it is worth knowing about.
   
   **What is still slower.** Reads that feed Spark operators rather than Comet ones remain 1.7x to 2.5x behind Spark's cache. Those pay a row conversion that Spark's format avoids with a codegen'd path over its own layout. That is a different cost from the projection problem and I have not touched it here.
   
   @sandugood this should help your pipeline on the read side, but that fallback gap is exactly the shape you were running, so I would not promise it closes the whole 10-to-15-minute difference. If you do retest, the two things worth separating are cache materialization and the repeated reads, since they now move in opposite directions.
   
   **Testing.** Five new tests in `CometInMemoryCacheSuite`. The two that matter corrupt the column streams a read must not touch, so they assert on what was actually decoded rather than on timings; I checked both fail against the previous single-stream format rather than passing for free. All 26 tests in the suite pass, along with `UtilsSuite`, `CometNativeColumnarToRowSuite` and `NativeUtilSuite`.
   
   @peterxcli this changes the cached payload and the read path, so your approval no longer covers it. Sorry to move the target after you had signed off. The lifetime handling around the readers is the part I would most want a second pair of eyes on: a cached batch is now several independent Arrow streams, so the `TaskCompletionListener` you asked for has to release all the readers for the batch in flight rather than one.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#discussion_r3872504786


##########
spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala:
##########
@@ -398,33 +430,69 @@ object Utils extends CometTypeShim with Logging {
     }
   }
 
+  /**
+   * Whether every column in `batch` is an Arrow-backed `CometVector`, so [[getBatchFieldVectors]]
+   * can hand out its vectors directly. Callers that may receive batches from a plan they did not
+   * build (e.g. Comet's cache serializer, which Spark hands the cached plan's columnar output)
+   * use this to convert foreign vectors to Arrow instead of tripping the exception below.
+   *
+   * Stricter than what [[getBatchFieldVectors]] accepts: a `ConstantColumnVector` is rejected
+   * here even though that method materializes one, so such a batch takes the conversion path
+   * rather than being materialized column by column.
+   */
+  def isArrowBacked(batch: ColumnarBatch): Boolean =
+    (0 until batch.numCols()).forall { i =>
+      batch.column(i) match {
+        // Not every CometVector can be handed to getFieldVector: a CometPlainVector can wrap a
+        // LargeVarCharVector or LargeVarBinaryVector (an accelerated mapInArrow returning
+        // pa.large_string(), for instance), which it rejects. Answering true for those would
+        // send a batch down the direct write path that then fails, so check the vector itself
+        // and let the caller convert instead.
+        case v: CometVector => isSupportedFieldVector(v.getValueVector)
+        case _ => false
+      }
+    }
+
   def getBatchFieldVectors(
       batch: ColumnarBatch): (Seq[FieldVector], Option[DictionaryProvider]) = {
-    var provider: Option[DictionaryProvider] = None
+    val columns = getBatchFieldVectorsWithProviders(batch)
+    (columns.map(_._1), columns.flatMap(_._2).headOption)

Review Comment:
   Fixed in 70f046abf. `getBatchFieldVectors` now combines the providers instead of taking the first: each dictionary-encoded column's dictionary is looked up under the provider that column was decoded with, and the result is one `MapDictionaryProvider` covering the whole batch.
   
   Your repro is the regression test, `Comet in-memory cache broadcasts a batch whose columns have separate dictionaries`. With the fix reverted it fails with exactly `IllegalArgumentException: Could not find dictionary with ID 1`.
   
   On normalize versus combine: combine. Renumbering would mean rewriting each vector's `Field` dictionary ID, which is not settable without copying the vector, so a same-ID/different-dictionary clash now raises rather than resolving one column against another's dictionary. That case is unreachable today, since every provider reaching this path descends either from a single `NativeUtil` importer or from a stream whose IDs that importer wrote, but silently decoding one column with the wrong dictionary was the worse of the two failure modes to leave available.



##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowCachedBatchSerializer.scala:
##########
@@ -0,0 +1,539 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet.execution.arrow
+
+import scala.collection.JavaConverters._
+import scala.util.control.NonFatal
+
+import org.apache.spark.TaskContext
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, GenericInternalRow, IsNotNull, IsNull, UnsafeProjection}
+import org.apache.spark.sql.columnar.{CachedBatch, SimpleMetricsCachedBatch, SimpleMetricsCachedBatchSerializer}
+import org.apache.spark.sql.comet.util.Utils
+import org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.unsafe.types.{ByteArray, UTF8String}
+import org.apache.spark.util.io.ChunkedByteBuffer
+
+import org.apache.comet.CometArrowAllocator
+
+/**
+ * Cached batch format used when Comet writes Spark in-memory cache data.
+ *
+ * `columns` holds one compressed Arrow stream per cached column, in cache-schema order, produced
+ * by `Utils.serializeBatchColumns`. Storing columns separately is what lets a scan decode only
+ * the ones it projected; a single stream covering the whole batch would have to be inflated in
+ * full before any projection could be applied. The cache manager still owns storage and eviction;
+ * this class only changes the cached payload.
+ */
+private case class CometCachedBatch(
+    override val numRows: Int,
+    override val sizeInBytes: Long,
+    override val stats: InternalRow,
+    columns: Array[ChunkedByteBuffer])
+    extends SimpleMetricsCachedBatch
+
+/**
+ * Cache serializer that stores Comet-compatible Arrow batches in Spark's in-memory cache.
+ *
+ * The cached payload format is decided by the schema alone. A relation whose schema Comet's Arrow
+ * writer supports is stored as `CometCachedBatch`, and every other relation is delegated in full
+ * to Spark's `DefaultCachedBatchSerializer`. The format deliberately does not depend on any
+ * runtime config: `spark.sql.cache.serializer` is a static conf, so installing this serializer is
+ * already a per-application decision, and a relation whose format could flip mid-session cannot
+ * be read back reliably. `spark.comet.exec.inMemoryCache.enabled` still governs whether a scan
+ * over the cache runs natively, and its value at startup is what makes `CometDriverPlugin`
+ * install this serializer in the first place.
+ *
+ * Reads of `CometCachedBatch` keep working when the native scan is disabled, because Spark then
+ * reads the same cached data through the SparkToColumnar fallback path.
+ */
+class ArrowCachedBatchSerializer extends SimpleMetricsCachedBatchSerializer {
+
+  import ArrowCachedBatchSerializer.supportsSchema
+
+  private val fallback = new DefaultCachedBatchSerializer()
+
+  // Bounds and null counts per column, gathered before the batch is serialized: serializing
+  // clears the batch's vectors, and the per-column byte sizes that complete the statistics row
+  // are only known afterwards. See statsRow.
+  private def gatherColumnStats(
+      batch: ColumnarBatch,
+      attrs: Seq[Attribute]): (Array[Any], Array[Any], Array[Int]) = {
+    val numCols = attrs.length
+    val lower = new Array[Any](numCols)
+    val upper = new Array[Any](numCols)
+    val nulls = Array.fill[Int](numCols)(0)
+    val numRows = batch.numRows()
+
+    var c = 0
+    while (c < numCols) {
+      val dt = attrs(c).dataType
+      val col = batch.column(c)
+      var r = 0
+      while (r < numRows) {
+        if (col.isNullAt(r)) {
+          nulls(c) += 1
+        } else if (tracksBounds(dt)) {
+          val value = readValue(col, dt, r)
+          if (lower(c) == null || compare(dt, value, lower(c)) < 0) {
+            lower(c) = value
+          }
+          if (upper(c) == null || compare(dt, value, upper(c)) > 0) {
+            upper(c) = value
+          }
+        }
+        r += 1
+      }
+      c += 1
+    }
+
+    (lower, upper, nulls)
+  }
+
+  // Build the statistics row expected by SimpleMetricsCachedBatchSerializer.
+  // For each cached column Spark expects five values in this order:
+  // lower bound, upper bound, null count, row count, and size in bytes.
+  private def statsRow(
+      lower: Array[Any],
+      upper: Array[Any],
+      nulls: Array[Int],
+      numRows: Int,
+      columnSizes: Array[Long]): InternalRow = {
+    val numCols = lower.length
+    val values = new Array[Any](numCols * 5)
+    var c = 0
+    while (c < numCols) {
+      val base = c * 5
+      values(base) = lower(c)
+      values(base + 1) = upper(c)
+      values(base + 2) = nulls(c)
+      values(base + 3) = numRows
+      // Each column is its own compressed stream, so its size is known exactly. Cache pruning
+      // uses bounds/null-count/row-count rather than this field, but Spark reserves it and
+      // reports it, so record the real value.
+      values(base + 4) = columnSizes(c)
+      c += 1
+    }
+
+    new GenericInternalRow(values)
+  }
+
+  // Spark can prune cache batches only for types whose bounds can be compared.
+  // Other types still report null count and row count but leave bounds as null.
+  private def tracksBounds(dt: DataType): Boolean = dt match {
+    case BooleanType | ByteType | ShortType | IntegerType | LongType | FloatType | DoubleType |
+        _: DecimalType | StringType | DateType | TimestampType | TimestampNTZType =>
+      true
+    case _ => false
+  }
+
+  // Read a non-null value from a ColumnVector using Spark's internal value type
+  // for the corresponding DataType.
+  private def readValue(col: ColumnVector, dt: DataType, rowId: Int): Any = dt match {
+    case BooleanType => col.getBoolean(rowId)
+    case ByteType => col.getByte(rowId)
+    case ShortType => col.getShort(rowId)
+    case IntegerType | DateType => col.getInt(rowId)
+    case LongType | TimestampType | TimestampNTZType => col.getLong(rowId)
+    case FloatType => col.getFloat(rowId)
+    case DoubleType => col.getDouble(rowId)
+    case d: DecimalType => col.getDecimal(rowId, d.precision, d.scale)
+    case StringType => col.getUTF8String(rowId).copy()
+    case _ => null
+  }
+
+  // Compare values using the same physical representation used in the stats row.
+  private def compare(dt: DataType, left: Any, right: Any): Int = dt match {
+    case BooleanType =>
+      java.lang.Boolean.compare(left.asInstanceOf[Boolean], right.asInstanceOf[Boolean])
+    case ByteType =>
+      java.lang.Byte.compare(left.asInstanceOf[Byte], right.asInstanceOf[Byte])
+    case ShortType =>
+      java.lang.Short.compare(left.asInstanceOf[Short], right.asInstanceOf[Short])
+    case IntegerType | DateType =>
+      java.lang.Integer.compare(left.asInstanceOf[Int], right.asInstanceOf[Int])
+    case LongType | TimestampType | TimestampNTZType =>
+      java.lang.Long.compare(left.asInstanceOf[Long], right.asInstanceOf[Long])
+    case FloatType =>
+      java.lang.Float.compare(left.asInstanceOf[Float], right.asInstanceOf[Float])
+    case DoubleType =>
+      java.lang.Double.compare(left.asInstanceOf[Double], right.asInstanceOf[Double])
+    case _: DecimalType =>
+      left.asInstanceOf[Decimal].compare(right.asInstanceOf[Decimal])
+    case StringType =>
+      ByteArray.compareBinary(
+        left.asInstanceOf[UTF8String].getBytes,
+        right.asInstanceOf[UTF8String].getBytes)
+    case other =>
+      throw new IllegalStateException(s"compare called for unsupported type $other")
+  }
+
+  // Compute Spark-compatible cache stats before serializing each batch to Arrow.
+  // The stats are stored beside the Arrow bytes so Spark's cache filter can prune
+  // CometCachedBatch without decoding the batch first.
+  //
+  // A columnar input batch is not guaranteed to be Arrow-backed; see supportsColumnarInput for
+  // why. Batches that are not get copied into Arrow first, since Utils.serializeBatches only
+  // writes CometVector columns.
+  private def encodeBatches(
+      batches: Iterator[ColumnarBatch],
+      attrs: Seq[Attribute]): Iterator[CachedBatch] = {
+    val arrowSchema =
+      Utils.toArrowSchema(Utils.fromAttributes(attrs), CometArrowStream.NATIVE_TIMEZONE)
+
+    batches.map { batch =>
+      // Bounds and null counts are read from the input batch, which serializing then clears, so
+      // they have to be gathered first. The row is only assembled once the per-column sizes are
+      // known.
+      val (lower, upper, nulls) = gatherColumnStats(batch, attrs)
+      val numRows = batch.numRows()
+
+      val columns = if (Utils.isArrowBacked(batch)) {
+        Utils.serializeBatchColumns(batch)
+      } else {
+        val arrowBatch =
+          CometArrowConverters.columnarBatchToArrowBatch(batch, arrowSchema, CometArrowAllocator)
+        try Utils.serializeBatchColumns(arrowBatch)
+        finally arrowBatch.close()
+      }
+
+      val columnSizes = columns.map(_.size)
+      CometCachedBatch(
+        numRows = numRows,
+        sizeInBytes = columnSizes.sum,
+        stats = statsRow(lower, upper, nulls, numRows, columnSizes),
+        columns = columns)
+    }
+  }
+
+  // Resolve requested columns by exprId, not by name, because aliases may reuse names.
+  //
+  // An empty selection stays empty rather than expanding to every column. Spark asks for no
+  // columns when the query only needs the row count (SELECT count(*)), and since projection now
+  // decides what gets decoded, expanding it would turn the cheapest possible read into the most
+  // expensive one.
+  private def selectedIndices(
+      cacheAttributes: Seq[Attribute],
+      selectedAttributes: Seq[Attribute]): Array[Int] = {
+    val byExprId = cacheAttributes.zipWithIndex.map { case (attr, idx) =>
+      attr.exprId -> idx
+    }.toMap
+
+    selectedAttributes.map { attr =>
+      byExprId.getOrElse(
+        attr.exprId,
+        throw new IllegalStateException(
+          s"Could not resolve selected attribute ${attr.name} from cache attributes"))
+    }.toArray
+  }
+
+  // Spark's SimpleMetricsCachedBatchSerializer prunes a batch when the generated partition filter
+  // does not evaluate to true against the stats row. Bounds are only computed for the types
+  // tracksBounds accepts, and for every other column the lower and upper bounds stay null, which
+  // makes a comparison against them evaluate to null and therefore prune the batch. That would
+  // silently drop rows, so predicates over columns without bounds are not pushed down at all.
+  // Null counts and row counts are recorded for every column, so IsNull and IsNotNull stay safe.
+  override def buildFilter(
+      predicates: Seq[Expression],
+      cachedAttributes: Seq[Attribute]): (Int, Iterator[CachedBatch]) => Iterator[CachedBatch] = {
+    val prunable = cachedAttributes.collect {
+      case a if tracksBounds(a.dataType) => a.exprId
+    }.toSet
+
+    val prunablePredicates = predicates.filter {
+      case _: IsNull | _: IsNotNull => true
+      case p => p.references.forall(a => prunable.contains(a.exprId))
+    }
+
+    super.buildFilter(prunablePredicates, cachedAttributes)
+  }
+
+  // Comet's Arrow writer only handles the types listed in supportsSchema. Reporting false here
+  // sends the relation down the row path, where it is delegated to Spark's default serializer,
+  // instead of failing at cache materialization inside Utils.serializeBatches.
+  //
+  // This answer is schema-only, because attributes are all Spark gives us; it says nothing about
+  // the vectors. Returning true also makes InMemoryRelation strip the ColumnarToRow above the
+  // cached plan, so convertColumnarBatchToCachedBatch then receives whatever that plan produces:
+  // a Comet scan's CometVectors, but equally Spark's vectorized Parquet/ORC reader or a
+  // connector's own vectors. encodeBatches converts the non-Arrow ones; that conversion is load
+  // bearing, not defensive.
+  override def supportsColumnarInput(schema: Seq[Attribute]): Boolean = supportsSchema(schema)
+
+  // A relation Comet stores is always readable as columnar Arrow. Anything else holds
+  // DefaultCachedBatch, so defer to Spark, which only claims columnar output for the primitive
+  // types its ColumnAccessor.decompress path can actually decode.
+  override def supportsColumnarOutput(schema: StructType): Boolean = {
+    if (schema.fields.forall(f => ArrowCachedBatchSerializer.supportsType(f.dataType))) {
+      true
+    } else {
+      fallback.supportsColumnarOutput(schema)
+    }
+  }
+
+  // Columnar Comet output is stored as compressed Arrow stream bytes. Spark only calls this when
+  // supportsColumnarInput returned true, so the schema is known to be Comet-writable here.
+  override def convertColumnarBatchToCachedBatch(
+      input: RDD[ColumnarBatch],
+      schema: Seq[Attribute],
+      storageLevel: StorageLevel,
+      conf: SQLConf): RDD[CachedBatch] = {
+
+    input.mapPartitions { batches =>
+      encodeBatches(batches, schema)
+    }
+  }
+
+  override def convertCachedBatchToColumnarBatch(
+      input: RDD[CachedBatch],
+      cacheAttributes: Seq[Attribute],
+      selectedAttributes: Seq[Attribute],
+      conf: SQLConf): RDD[ColumnarBatch] = {
+    if (!supportsSchema(cacheAttributes)) {
+      return fallback.convertCachedBatchToColumnarBatch(
+        input,
+        cacheAttributes,
+        selectedAttributes,
+        conf)
+    }
+
+    val indices = selectedIndices(cacheAttributes, selectedAttributes)
+
+    input.mapPartitions { it =>
+      // A ColumnReaders closes its readers (releasing the vectors they are holding) only when the
+      // batch it produced has been consumed. A consumer that stops early -- LIMIT, take(), or a
+      // cancelled task -- leaves the readers for the batch in flight open, so close them on task
+      // completion. Spark's own ArrowCachedBatchSerializer registers a listener for the same
+      // reason.
+      //
+      // flatMap consumes each inner iterator fully before building the next, so at most one batch
+      // is open at a time and tracking the current one is enough. close() is idempotent, so
+      // closing one that already released itself is a no-op.
+      @volatile var current: ColumnReaders = null
+      Option(TaskContext.get()).foreach { tc =>
+        tc.addTaskCompletionListener[Unit] { _ =>
+          val readers = current
+          current = null
+          if (readers != null) {
+            readers.close()
+          }
+        }
+      }
+
+      it.flatMap {
+        case cb: CometCachedBatch =>
+          if (indices.isEmpty) {
+            // Nothing to decode: the row count is the whole answer, and it is already here.
+            Iterator.single(new ColumnarBatch(Array.empty[ColumnVector], cb.numRows))
+          } else {
+            val readers = new ColumnReaders(indices.map(i => cb.columns(i)), cb.numRows)
+            current = readers
+            readers.batches
+          }
+
+        case other =>
+          throw new IllegalStateException(
+            s"Unsupported cached batch type ${other.getClass.getName}")
+      }
+    }
+  }
+
+  // Decodes one selected column stream apiece and stitches the results back into a single batch.
+  //
+  // Each stream is self-contained, so the columns a scan did not select are never inflated. The
+  // decoded vectors stay owned by their readers: closing them releases the batch, which is why
+  // this yields a single-element iterator that closes on exhaustion, matching what
+  // ArrowReaderIterator did when the payload was one stream.
+  private class ColumnReaders(buffers: Array[ChunkedByteBuffer], numRows: Int) {
+    // decodeBatches opens a reader and eagerly decodes its first batch, so it allocates. If a
+    // later column throws, the readers already opened here are unreachable: the task-completion
+    // listener cannot release them because `current` is only assigned once this constructor
+    // returns, so they would leak off-heap for the life of the executor.
+    private val readers: Array[Iterator[ColumnarBatch]] = {
+      val opened = new Array[Iterator[ColumnarBatch]](buffers.length)
+      var i = 0
+      try {
+        while (i < buffers.length) {
+          opened(i) = Utils.decodeBatches(buffers(i), "CometCache")

Review Comment:
   Fixed in 70f046abf. `ArrowReaderIterator` now closes the reader if its eager first decode throws, so a column that loads its dictionary and then fails on the record batch releases it. I also guarded `StreamReader`'s own `getVectorSchemaRoot`, which allocates the root's vectors under the same condition: nothing holds the reader until the constructor returns, so nothing else can close it.
   
   Regression test: `Comet in-memory cache releases a reader whose own first batch fails to decode`. Instead of an allocator limit it truncates the decoded Arrow stream, dropping the end-of-stream marker plus part of the record batch body, so the dictionary loads and the read that follows runs out of input. Reverted, it leaks 64 bytes per attempt, matching the growth you measured; with the fix the allocator returns to baseline.
   
   The existing `corruptColumnStream` helper was not usable for this. A small column compresses to a single LZ4 block, so truncating the compressed bytes fails the decompressor before Arrow reads anything at all, which is the case that was already covered. The new helper cuts the decoded bytes and re-compresses.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5443347184

   @viirya you read it correctly and I had it wrong. Fixed in 7e19dff20.
   
   Here is the plan for the disabled run, which settles it:
   
   ```
   *(1) CometColumnarToRow
   +- CometHashAggregate [sum#265L, sum#266L, sum#267L], [Final], [sum(id#0L), sum(k#1L), sum(v#2L)]
      +- CometExchange SinglePartition, ENSURE_REQUIREMENTS, CometNativeShuffle
         +- CometHashAggregate [id#0L, k#1L, v#2L], [Partial], [partial_sum(id#0L), ...]
            +- CometSparkColumnarToColumnar
               +- Scan In-memory table bench_cache [id#0L, k#1L, v#2L]
                     +- InMemoryRelation [...], StorageLevel(disk, memory, deserialized, 1 replicas)
   ```
   
   The aggregation is `CometHashAggregate` on both sides, over `CometExchange`, and the only difference is that `CometSparkColumnarToColumnar` + `Scan In-memory table` becomes `CometInMemoryTableScan`. So your reading of `cacheConf` was right and the "Spark execution" half of that comment was simply wrong — the correct framing is the narrower one you gave: keep the cached scan native versus fall back to a Spark cache scan and convert.
   
   What changed:
   
   * The case labels are now `Spark cache scan + CometSparkColumnarToColumnar` and `CometInMemoryTableScan`, so the comparison is legible from the benchmark output alone rather than depending on a comment.
   * The `withCachedTable` comment states which flag moves and which does not, and keeps the separate point that neither case is a baseline for Spark's own cache format.
   * The PR description is corrected the same way. I had already fixed one wrong framing there — presenting the numbers as a speedup over Spark's cache — and replaced it with a second wrong one, so the description now says both explicitly.
   
   One addition beyond what you asked for: `verifyPlan` now asserts `CometSparkColumnarToColumnar` is present in the disabled case, not just that `CometInMemoryTableScan` is absent. The labels claim a plan shape, so the benchmark should fail rather than mislabel itself if that shape ever changes. I verified the assertion holds for all five of the benchmark's queries, including the count-only and full-width ones where the scan output differs.
   
   I could not re-run the benchmark itself in my current environment — `exec:java` cannot bind a socket there, unrelated to this change — so the numbers in the description are the previous run's. Only labels and comments moved, so they still describe the same measurement, but flagging it rather than implying I re-measured.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "sandugood (via GitHub)" <gi...@apache.org>.
sandugood commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5254380631

   > > Tested on Spark4.0 against current branch. Using PySpark Got this error on the same pipeline that was testing earlier. And I can clearly see a regression in performance. Same job with `.cache()` on default Spark took <10 min, Comet ran >15min.
   > > Here is error:
   > > ```
   > > Py4JJavaError: An error occurred while calling o12641.createOrReplace.
   > > : org.apache.spark.SparkException: Job aborted due to stage failure: Task 92 in stage 27.0 failed 4 times, most recent failure: Lost task 92.3 in stage 27.0 (TID 15624) (1.1.1.1 executor 76): org.apache.comet.CometNativeException: Invalid argument error: column types must match schema types, expected List(Struct("col_1": Utf8, "col_2": Boolean, "col_3": Boolean)) but found List(Struct("col_1": Utf8, "col_2": non-null Boolean, "col_3": Boolean)) at column index 5
   > > 	at org.apache.comet.Native.executePlan(Native Method)
   > > 	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2(CometExecIterator.scala:155)
   > > 	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2$adapted(CometExecIterator.scala:154)
   > > 	at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:189)
   > > 	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$1(CometExecIterator.scala:154)
   > > 	at org.apache.comet.Tracing$.withTrace(Tracing.scala:31)
   > > 	at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:152)
   > > 	at org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:196)
   > > ```
   > > 
   > > 
   > >     
   > >       
   > >     
   > > 
   > >       
   > >     
   > > 
   > >     
   > >   
   > > (obfuscated column names)
   > 
   > I think the crash may get fixed once #5138 is merged. Looking into perf regression...
   
   Thanks, @andygrove 
   Would love to try it out once ready


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add experimental native support for in-memory cache, disabled by default [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5051:
URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5457339387

   Merged. Thanks for the reviews @sandugood @peterxcli @0lai0 @sunchao @viirya. I'll start work on the follow on items now as a fresh PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org