[I] NegativeArraySizeException reading many variant paths on MOR and bootstrap tables (Spark 4.1+ pushVariantIntoScan) [hudi]

["voonhous (via GitHub)" <gi...@apache.org>]

voonhous opened a new issue, #20040:
URL: https://github.com/apache/hudi/issues/20040

   **Describe the problem you faced**
   
   On Spark 4.1+ with `spark.sql.variant.pushVariantIntoScan=true` (the default), a query that extracts several paths from a variant column fails with `java.lang.NegativeArraySizeException` once a row lacks enough of those paths. Reproduced on master with eight `try_variant_get(v, '$.x', 'bigint')` paths over a row that holds only `$.a`:
   
   - MOR table with a log file in the latest slice: any such query.
   - Bootstrapped table (METADATA_ONLY), COW or MOR: any such query that also needs a meta column, and every query on MOR.
   
   Snapshot reads of a COW table without bootstrap are unaffected. Fewer extracted paths (or rows that hold most of them) pass, which is why the existing pushVariantIntoScan legs never hit it.
   
   **Cause**
   
   PushVariantIntoScan rewrites the variant column into a projection struct in the scan schema. `SparkFileFormatInternalRowReaderContext` reads base-file rows in that shape and rewrites log rows into it (#19783), but two row writers are still built from the engine `HoodieSchema`, whose variant field converts to `VariantType`:
   
   1. the output converter that projects the reader's required schema down to the requested one (`FileGroupReaderSchemaHandler.getOutputConverter` via `BaseSparkInternalRecordContext.projectRecord`), present whenever merge columns or `_hoodie_commit_time` widen the required schema;
   2. the bootstrap skeleton/data join (`BaseSparkInternalRowReaderContext.getBootstrapProjection`).
   
   A `VariantType`-typed writer copies the struct through `UnsafeRow.getVariant`, which reads the struct's null bitset as the variant value length. That is byte-identical while the low bitset word stays below the struct's byte size, and throws once enough pushed fields are null (seven null fields of eight: `NegativeArraySizeException: -186`).
   
   **To Reproduce**
   
   ```sql
   create table t (id int, v variant, ts long) using hudi tblproperties (primaryKey = 'id', preCombineField = 'ts', type = 'mor');
   insert into t select id, parse_json(concat('{"a":', id, '}')), 1000 from range(0, 10);
   update t set v = parse_json(concat('{"a":', 100 + id, '}')), ts = 1001 where id >= 5;
   select try_variant_get(v, '$.a', 'bigint'), try_variant_get(v, '$.b', 'bigint'), try_variant_get(v, '$.c', 'bigint'),
          try_variant_get(v, '$.d', 'bigint'), try_variant_get(v, '$.e', 'bigint'), try_variant_get(v, '$.f', 'bigint'),
          try_variant_get(v, '$.g', 'bigint'), try_variant_get(v, '$.h', 'bigint') from t where id = 2;
   ```
   
   **Expected behavior**
   
   `2, null, null, null, null, null, null, null`.
   
   **Environment Description**
   
   * Hudi version : master (7430134280ae)
   * Spark version : 4.1.1 (pushVariantIntoScan on by default)
   * Storage : local
   
   **Stacktrace**
   
   <details><summary>MOR read</summary>
   
   ```
   java.lang.NegativeArraySizeException: -186
   	at org.apache.spark.unsafe.types.VariantVal.readFromUnsafeRow(VariantVal.java:79)
   	...
   	at org.apache.spark.sql.HoodieInternalRowUtils$.$anonfun$genUnsafeStructWriter$3(HoodieInternalRowUtils.scala:210)
   	at org.apache.spark.sql.HoodieInternalRowUtils$.$anonfun$modifiableRowWriter$2(HoodieInternalRowUtils.scala:239)
   	at org.apache.spark.sql.HoodieInternalRowUtils$.$anonfun$genUnsafeRowWriter$2(HoodieInternalRowUtils.scala:165)
   	at org.apache.hudi.BaseSparkInternalRecordContext.lambda$projectRecord$0(BaseSparkInternalRecordContext.java:221)
   	at org.apache.hudi.common.table.read.BufferedRecord.project(BufferedRecord.java:90)
   	at org.apache.hudi.common.table.read.HoodieFileGroupReader.next(HoodieFileGroupReader.java:351)
   ```
   </details>
   
   <details><summary>Bootstrap read</summary>
   
   ```
   java.lang.NegativeArraySizeException: -186
   	at org.apache.spark.unsafe.types.VariantVal.readFromUnsafeRow(VariantVal.java:79)
   	...
   	at org.apache.hudi.BaseSparkInternalRowReaderContext.lambda$getBootstrapProjection$2(BaseSparkInternalRowReaderContext.java:96)
   	at org.apache.hudi.SparkFileFormatInternalRowReaderContext$$anon$2.doHasNext(SparkFileFormatInternalRowReaderContext.scala:405)
   	at org.apache.hudi.common.util.collection.CachingIterator.hasNext(CachingIterator.java:30)
   	at org.apache.hudi.common.table.read.HoodieFileGroupReader.hasNext(HoodieFileGroupReader.java:339)
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] NegativeArraySizeException reading many variant paths on MOR and bootstrap tables (Spark 4.1+ pushVariantIntoScan) [hudi]

["voonhous (via GitHub)" <gi...@apache.org>]

voonhous closed issue #20040: NegativeArraySizeException reading many variant paths on MOR and bootstrap tables (Spark 4.1+ pushVariantIntoScan)
URL: https://github.com/apache/hudi/issues/20040


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org