[spark] branch branch-3.0 updated: [SPARK-35224][SQL][TESTS][3.1][3.0] Fix buffer overflow in `MutableProjectionSuite`

[ma...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

maxgekk pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.0 by this push:
     new d433457  [SPARK-35224][SQL][TESTS][3.1][3.0] Fix buffer overflow in `MutableProjectionSuite`
d433457 is described below

commit d433457d5dbbd6790b1cdb63b7a44b0a7da6f392
Author: Max Gekk <ma...@gmail.com>
AuthorDate: Mon Apr 26 12:41:15 2021 +0300

    [SPARK-35224][SQL][TESTS][3.1][3.0] Fix buffer overflow in `MutableProjectionSuite`
    
    ### What changes were proposed in this pull request?
    In the test `"unsafe buffer with NO_CODEGEN"` of `MutableProjectionSuite`, fix unsafe buffer size calculation to be able to place all input fields without buffer overflow + meta-data.
    
    ### Why are the changes needed?
    To make the test suite `MutableProjectionSuite` more stable.
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    By running the affected test suite:
    ```
    $ build/sbt "test:testOnly *MutableProjectionSuite"
    ```
    
    Authored-by: Max Gekk <max.gekkgmail.com>
    Signed-off-by: Max Gekk <max.gekkgmail.com>
    (cherry picked from commit d572a859891547f73b57ae8c9a3b800c48029678)
    
    Closes #32347 from MaxGekk/fix-buffer-overflow-MutableProjectionSuite-3.1.
    
    Authored-by: Max Gekk <ma...@gmail.com>
    Signed-off-by: Max Gekk <ma...@gmail.com>
    (cherry picked from commit c59db3d294cadb53d4a6729649e86b69058262af)
    Signed-off-by: Max Gekk <ma...@gmail.com>
---
 .../spark/sql/catalyst/expressions/MutableProjectionSuite.scala    | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MutableProjectionSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MutableProjectionSuite.scala
index c31310b..d0e3990 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MutableProjectionSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MutableProjectionSuite.scala
@@ -23,6 +23,7 @@ import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
 import org.apache.spark.sql.catalyst.util.IntervalUtils
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.Platform
 import org.apache.spark.unsafe.types.UTF8String
 
 class MutableProjectionSuite extends SparkFunSuite with ExpressionEvalHelper {
@@ -48,8 +49,10 @@ class MutableProjectionSuite extends SparkFunSuite with ExpressionEvalHelper {
 
   testBothCodegenAndInterpreted("unsafe buffer") {
     val inputRow = InternalRow.fromSeq(Seq(false, 1.toByte, 9.toShort, -18, 53L, 3.2f, 7.8, 4, 9L))
-    val numBytes = UnsafeRow.calculateBitSetWidthInBytes(fixedLengthTypes.length)
-    val unsafeBuffer = UnsafeRow.createFromByteArray(numBytes, fixedLengthTypes.length)
+    val numFields = fixedLengthTypes.length
+    val numBytes = Platform.BYTE_ARRAY_OFFSET + UnsafeRow.calculateBitSetWidthInBytes(numFields) +
+      UnsafeRow.WORD_SIZE * numFields
+    val unsafeBuffer = UnsafeRow.createFromByteArray(numBytes, numFields)
     val proj = createMutableProjection(fixedLengthTypes)
     val projUnsafeRow = proj.target(unsafeBuffer)(inputRow)
     assert(SafeProjection.create(fixedLengthTypes)(projUnsafeRow) === inputRow)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org