You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "wangyum (via GitHub)" <gi...@apache.org> on 2026/10/21 15:54:32 UTC
[PR] [SPARK-60425][SQL] Fix InternalRow.copyValue to clone Array[Byte] values [spark]
wangyum opened a new pull request, #59675:
URL: https://github.com/apache/spark/pull/59675
### What changes were proposed in this pull request?
`InternalRow.copyValue` copies `UTF8String`, `BinaryView`, `InternalRow`, `ArrayData`, and `MapData` values to detach them from any mutable buffer they may be backed by, but fell through to the default case (returning the same reference) for `Array[Byte]`. This PR adds a case that clones the byte array.
### Why are the changes needed?
Several code paths rely on `copyValue` to defensively copy values before retaining them past the lifetime of the row they came from: aggregate buffers (`collect_list`/`collect_set`, `max_by`/`min_by`), higher-order function closures (`transform`/`aggregate`/`reduce`), `GenericArrayData.copy()`, and `FrequentItems`. Some `InternalRow` implementations return a `byte[]` that aliases a buffer which is mutated or reused afterward (e.g. buffers backed by vectorized/columnar readers), so without cloning, values retained through `copyValue` could silently change or get corrupted after the fact.
### Does this PR introduce _any_ user-facing change?
No. This only fixes a latent data-correctness bug for `BinaryType` values retained via `InternalRow.copyValue`.
### How was this patch tested?
Added `InternalRowSuite`, which verifies `copyValue` detaches binary, string, array, and row values from a buffer the caller mutates afterward. Also ran the existing `DataFrameAggregateSuite."SPARK-31500: collect_set() of BinaryType returns duplicate elements"` to confirm no regression.
### Was this patch authored or co-authored using generative AI tooling?
Yes, Claude Code (Claude Sonnet 5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60425][SQL] Fix InternalRow.copyValue to clone Array[Byte] values [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #59675:
URL: https://github.com/apache/spark/pull/59675#discussion_r4345288096
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala:
##########
@@ -127,11 +127,12 @@ object InternalRow {
val empty = apply()
/**
- * Copies the given value if it's string/struct/array/map type.
+ * Copies the given value if it's string/binary/struct/array/map type.
*/
def copyValue(value: Any): Any = value match {
case v: UTF8String => v.copy()
case v: BinaryView => v.copy()
+ case v: Array[Byte] => v.clone()
Review Comment:
**If the premise holds, this fixes only one of several paths that keep a `byte[]`.** These still store it by reference:
- `CodeGenerator.setColumn` ([L2733-L2739](https://github.com/apache/spark/blob/cd06735c008285bddb9bd976b3e931f37eb4990f/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala#L2733-L2739)) and `getWriterDefault` in this object ([L216](https://github.com/apache/spark/blob/c2ea7f7550a0af1413b24f5669c32a772ebc9102/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala#L216)), which mutable projections use for `max`/`min`/`first`/`last` buffers and window aggregates. The two-argument `max_by`/`min_by` listed in the description is a `DeclarativeAggregate` on this path ([MaxByAndMinBy.scala L30](https://github.com/apache/spark/blob/cd06735c008285bddb9bd976b3e931f37eb4990f/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/MaxByAndMinBy.scala#L30)). Only the `k` variant uses `copyValue`.
- `ColumnarRow.copy()`, `ColumnarBatchRow.copy()` and `MutableColumnarRow.copy()` do `row.update(i, getBinary(i))`, while they copy `UTF8String` and `BinaryView` ([ColumnarRow.java L79-L83](https://github.com/apache/spark/blob/cd06735c008285bddb9bd976b3e931f37eb4990f/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java#L79-L83)).
- `VariantVal` still falls through `case _ => value`, though it wraps two arrays from the same `getBinary` calls ([ColumnVector.java L360](https://github.com/apache/spark/blob/cd06735c008285bddb9bd976b3e931f37eb4990f/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnVector.java#L360)).
After this PR, `first(b)` keeps the reference while `first(array(b))` clones it in the same projection. Could we either make these paths consistent in one change, or keep `byte[]` as a pass-through?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala:
##########
@@ -127,11 +127,12 @@ object InternalRow {
val empty = apply()
/**
- * Copies the given value if it's string/struct/array/map type.
+ * Copies the given value if it's string/binary/struct/array/map type.
*/
def copyValue(value: Any): Any = value match {
case v: UTF8String => v.copy()
case v: BinaryView => v.copy()
+ case v: Array[Byte] => v.clone()
Review Comment:
**With this change, `freqItems` can return a wrong result for a binary column whose rows share one array instance.** `CollectFrequentItems.update` passes `InternalRow.copyValue(key)` to a `mutable.Map[Any, Long]`, which compares `byte[]` keys by reference ([FrequentItems.scala L88-L119](https://github.com/apache/spark/blob/cd06735c008285bddb9bd976b3e931f37eb4990f/sql/core/src/main/scala/org/apache/spark/sql/execution/stat/FrequentItems.scala#L88-L119)). Before this PR, a repeated instance hit the same entry. After it, every row inserts a new clone as a new key.
For example, `spark.range(0, 3, 1, 1).select(lit(Array[Byte](1)).as("b")).stat.freqItems(Seq("b"), 0.5)`:
- `FoldablePropagation` replaces `b` in the aggregate with the literal, and `Literal.eval` returns the same array for every row.
- Before: the map ends with one key whose count is 3, so the result is `[X'01']`.
- After: the three clones are three keys. The third insert reaches `size = 2`, and `filterInPlace((k, v) => v > minCount)` evicts all of them, so the result is `[]`. With the default support, the result has three copies of `X'01'` instead.
Keying binary values by reference is already broken for ordinary input (SPARK-59898), but this makes the constant case worse, while the PR description lists `FrequentItems` as a beneficiary. I traced this statically and did not run it. Could we land the content-based key from SPARK-59898 first, or leave `FrequentItems` out of this change?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala:
##########
@@ -127,11 +127,12 @@ object InternalRow {
val empty = apply()
/**
- * Copies the given value if it's string/struct/array/map type.
+ * Copies the given value if it's string/binary/struct/array/map type.
*/
def copyValue(value: Any): Any = value match {
case v: UTF8String => v.copy()
case v: BinaryView => v.copy()
+ case v: Array[Byte] => v.clone()
Review Comment:
**This adds a second allocation and copy for almost every binary value, since the input is usually a fresh array already.** Some places where it adds up (estimated from the code, not measured):
- A running window such as `collect_list(bin) OVER (PARTITION BY p ORDER BY t ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)`: `AggregateProcessor.evaluate` writes each result through the mutable projection, whose `.copy()` on the array now clones every element in the frame for each output row. With 10,000 rows of 1 KiB, that is about 48 GiB of copying per partition, instead of copying references.
- `first(struct(id, payload))` or `max_by(struct(id, payload), ts)` in `SortAggregateExec` re-copies the struct buffer on every input row, so the payload is now cloned even when the value doesn't change.
- `Dataset.observe` snapshots its buffer with `buffer.copy()` in `LastAttemptAggregatingAccumulator` on the driver's DAGScheduler thread, which now clones the serialized aggregate buffers of every task update.
- `aggregate(arr, X'', (acc, x) -> concat(acc, x))` clones the growing accumulator on each step, and `ColumnarArray.copy()` copies each binary element twice.
- `mode`, `collect_set(struct(...))` and `collect_top_k` clone before the lookup or admission, so the clone is often thrown away right after.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala:
##########
@@ -127,11 +127,12 @@ object InternalRow {
val empty = apply()
/**
- * Copies the given value if it's string/struct/array/map type.
+ * Copies the given value if it's string/binary/struct/array/map type.
*/
def copyValue(value: Any): Any = value match {
case v: UTF8String => v.copy()
case v: BinaryView => v.copy()
+ case v: Array[Byte] => v.clone()
Review Comment:
**I couldn't find any in-tree producer that returns a `byte[]` and mutates it afterwards, so I'm not sure which bug this fixes.** The description mentions "buffers backed by vectorized/columnar readers", but:
- `UnsafeRow.getBinary`, `UnsafeArrayData.getBinary`, `OnHeapColumnVector.getBytes`, `OffHeapColumnVector.getBytes`, `OrcAtomicColumnVector.getBinary` and the Arrow accessors allocate a new array on every call.
- `ConstantColumnVector.getBinary` and the Parquet dictionary path (`getBytesUnsafe()`) share one array across rows, but nothing writes to it afterwards.
- `InternalRow.update` says "the caller side should guarantee that this value won't be changed afterwards". `ColumnVector.getUTF8String` documents that its result may point into the vector, while `getBinary` doesn't. SPARK-17528, which added `copyValue`, and SPARK-57058, which added `BinaryView`, both rely on `getBinary` returning a fresh array.
The new test shows the problem only by mutating the array after passing it in, which is what that contract rules out. If there is a real producer, e.g. a UDF or a third-party DSv2 reader that reuses its output buffer, could you name it in the description with a reproducer? Then we can decide whether to fix it at that boundary, document the ownership rule on `SpecializedGetters.getBinary`, or keep this change. If this is a correctness fix, the JIRA would also need the `correctness` label.
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/InternalRowSuite.scala:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayData, GenericArrayData}
+import org.apache.spark.unsafe.types.UTF8String
+
+class InternalRowSuite extends SparkFunSuite {
Review Comment:
**The existing test cited in the description, "SPARK-31500: collect_set() of BinaryType returns duplicate elements", doesn't reach the new branch.** `CollectSet` converts a top-level binary with `UnsafeArrayData.fromPrimitiveArray` before any `copyValue` ([collect.scala L271](https://github.com/apache/spark/blob/cd06735c008285bddb9bd976b3e931f37eb4990f/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala#L271)), and its struct part copies an `UnsafeRow`, so it passes with or without this change. "SPARK-58201: MIN/MAX on DECIMAL and BINARY types with spill" in `MonotonicDequeWindowFunctionSuite` ([L379](https://github.com/apache/spark/blob/cd06735c008285bddb9bd976b3e931f37eb4990f/sql/core/src/test/scala/org/apache/spark/sql/execution/window/MonotonicDequeWindowFunctionSuite.scala#L379)) does go through it. Could we update "How was this patch tested?", since the description becomes the commit message?
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/InternalRowSuite.scala:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayData, GenericArrayData}
+import org.apache.spark.unsafe.types.UTF8String
+
+class InternalRowSuite extends SparkFunSuite {
+
+ test("copyValue detaches a binary value from a buffer the caller mutates afterward") {
+ val buffer = Array[Byte](1, 2, 3, 4)
+ val copy = InternalRow.copyValue(buffer).asInstanceOf[Array[Byte]]
+ assert(copy sameElements Array[Byte](1, 2, 3, 4))
+ java.util.Arrays.fill(buffer, 0.toByte)
+ assert(copy sameElements Array[Byte](1, 2, 3, 4),
+ "copyValue must clone a byte[] instead of aliasing the caller's buffer")
+ }
+
+ test("copyValue detaches a UTF8String from a buffer the caller mutates afterward") {
Review Comment:
**The UTF8String, row/array and primitive tests (L36-L60) pass without this PR, because they only exercise branches it doesn't change.** `ComplexDataSuite` ([L57-L104](https://github.com/apache/spark/blob/cd06735c008285bddb9bd976b3e931f37eb4990f/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/ComplexDataSuite.scala#L57-L104), added with `copyValue` in SPARK-17528) already covers the same copies, and more realistically, by overwriting a reused `UnsafeProjection` buffer. Also, `InternalRow.copyValue(null) == null` can't fail. Could we drop these three tests?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala:
##########
@@ -127,11 +127,12 @@ object InternalRow {
val empty = apply()
/**
- * Copies the given value if it's string/struct/array/map type.
+ * Copies the given value if it's string/binary/struct/array/map type.
Review Comment:
**This isn't from this PR, so it may be better as a separate JIRA.** In `getWriterDefault` of the same object, `case StringType =>` ([L210](https://github.com/apache/spark/blob/c2ea7f7550a0af1413b24f5669c32a772ebc9102/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala#L210)) matches only the default `UTF8_BINARY` string type, so collated `STRING`, `CHAR(n)` and `VARCHAR(n)` values fall through to `case _` and are stored without `.copy()`, while `CodeGenerator.setColumn` matches `_: StringType`. With interpreted projections, e.g. `spark.sql.codegen.factoryMode=NO_CODEGEN`, a collated string kept in a buffer may still point into a reused `UnsafeRow`.
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/InternalRowSuite.scala:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayData, GenericArrayData}
+import org.apache.spark.unsafe.types.UTF8String
+
+class InternalRowSuite extends SparkFunSuite {
+
+ test("copyValue detaches a binary value from a buffer the caller mutates afterward") {
+ val buffer = Array[Byte](1, 2, 3, 4)
+ val copy = InternalRow.copyValue(buffer).asInstanceOf[Array[Byte]]
+ assert(copy sameElements Array[Byte](1, 2, 3, 4))
+ java.util.Arrays.fill(buffer, 0.toByte)
+ assert(copy sameElements Array[Byte](1, 2, 3, 4),
+ "copyValue must clone a byte[] instead of aliasing the caller's buffer")
+ }
+
+ test("copyValue detaches a UTF8String from a buffer the caller mutates afterward") {
+ val buffer = "hello".getBytes("UTF-8")
+ val str = UTF8String.fromBytes(buffer)
+ val copy = InternalRow.copyValue(str).asInstanceOf[UTF8String]
+ java.util.Arrays.fill(buffer, 'X'.toByte)
+ assert(copy.toString == "hello")
+ }
+
+ test("copyValue detaches array/row values from a buffer the caller mutates afterward") {
Review Comment:
**This test doesn't check what its name says.** The array half (L50-L53) mutates nothing. The row half replaces a slot with `update`, which never affected a copy, and it stores a `java.lang.String` instead of a `UTF8String`. If `GenericInternalRow.copy()` or `GenericArrayData.copy()` stopped copying their elements, it would still pass. Also, no test puts a `byte[]` inside a row or an array, which is where this PR changes the behavior of `GenericInternalRow`, `SpecificInternalRow` and `GenericArrayData.copy()`.
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/InternalRowSuite.scala:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.expressions.GenericInternalRow
+import org.apache.spark.sql.catalyst.util.{ArrayData, GenericArrayData}
+import org.apache.spark.unsafe.types.UTF8String
+
+class InternalRowSuite extends SparkFunSuite {
+
+ test("copyValue detaches a binary value from a buffer the caller mutates afterward") {
Review Comment:
**Since this is the only test that fails without the fix, could we move it into `ComplexDataSuite` next to the SPARK-17528 copy tests, add the `SPARK-60425: ` prefix, and go through the copies the description names?** The check before the mutation (L30) is already covered by L32. For example:
```scala
test("SPARK-60425: GenericInternalRow/GenericArrayData.copy clone binary values") {
val bytes = Array[Byte](1, 2)
val copiedRow = new GenericInternalRow(Array[Any](bytes)).copy()
val copiedArray = new GenericArrayData(Array[Any](bytes)).copy()
bytes(0) = 9
assert(copiedRow.getBinary(0) === Array[Byte](1, 2))
assert(copiedArray.getBinary(0) === Array[Byte](1, 2))
}
```
`ComplexDataSuite` already imports `GenericInternalRow`, so this new file wouldn't be needed.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala:
##########
@@ -127,11 +127,12 @@ object InternalRow {
val empty = apply()
/**
- * Copies the given value if it's string/struct/array/map type.
+ * Copies the given value if it's string/binary/struct/array/map type.
Review Comment:
**Could we also update the other comments that describe which types `copyValue` copies?**
- `SlidingWindowMinMaxFunctionFrame` ([L282-L286](https://github.com/apache/spark/blob/cd06735c008285bddb9bd976b3e931f37eb4990f/sql/core/src/main/scala/org/apache/spark/sql/execution/window/SlidingWindowMinMaxFunctionFrame.scala#L282-L286)) says "Binary (byte[]) and Decimal fall through to the identity case but are still safe", which is no longer true. The reason it gives, that `UnsafeRow.getBinary` allocates a fresh array, also means the new clone is redundant on that path.
- `MonotonicDequeWindowFunctionSuite` ([L375-L377](https://github.com/apache/spark/blob/cd06735c008285bddb9bd976b3e931f37eb4990f/sql/core/src/test/scala/org/apache/spark/sql/execution/window/MonotonicDequeWindowFunctionSuite.scala#L375-L377)) has a comment with the same premise.
- This line still leaves out geometry and geography, which the `BinaryView` case covers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60425][SQL] Fix InternalRow.copyValue to clone Array[Byte] values [spark]
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum closed pull request #59675: [SPARK-60425][SQL] Fix InternalRow.copyValue to clone Array[Byte] values
URL: https://github.com/apache/spark/pull/59675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60425][SQL] Fix InternalRow.copyValue to clone Array[Byte] values [spark]
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #59675:
URL: https://github.com/apache/spark/pull/59675#issuecomment-6340347315
Thanks for the thorough review, @dongjoon-hyun.
I followed up on point 2 by checking the one lead we hadn't ruled out: the new in-process Arrow UDF path (SPARK-59718), since its description mentions exchanging Arrow buffers without copying across the JVM/Python boundary. It isn't a counter-example either: that "zero-copy" is specifically about avoiding the socket-serialization hop between the JVM and a separate Python process, not about the `byte[]` eventually handed to `InternalRow`. The actual binary accessor on that path, `ArrowColumnVector.BinaryAccessor.getBinary()`, goes through Arrow's own `VarBinaryVector.getObject()` (the same shared accessor class already used elsewhere in Spark, not new code for this feature), which allocates a fresh array and copies out of the `ArrowBuf` on every call. The result is then copied a second time into an `UnsafeRow` via `UnsafeProjection` before it leaves the evaluator.
So between your audit and this one, I couldn't find an in-tree path - old or newly added - that hands `copyValue` a `byte[]` which is later mutated in violation of `InternalRow.update`'s documented contract ("the caller side should guarantee that this value won't be changed afterwards"). The bug this PR describes would require a producer to violate that contract itself, which isn't something `copyValue` should need to defend against.
On the `FrequentItems` regression: #59162 (SPARK-59898) already fixes the underlying issue correctly, by keying binary values on content instead of reference, so once that lands this PR's interaction with `FrequentItems` is moot either way.
Given there's no reproducer-backed correctness case for the `Array[Byte]` branch, I'm closing this rather than continuing to chase one or patching around the regression. Appreciate the detailed audit - it's a more thorough treatment of this than the PR had going in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org