You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "comphead (via GitHub)" <gi...@apache.org> on 2026/08/29 16:02:32 UTC

[I] Fix remaining reported issues for arrays of map [datafusion-comet]

comphead opened a new issue, #5544:
URL: https://github.com/apache/datafusion-comet/issues/5544

   @comphead, smaller follow-ups are reasonable, provided this PR preserves the previous safe behavior for the affected cases. The remaining **[P2]** concerns come from the actual BASE-to-HEAD changes:
   
   - The TRY/bracket and collation examples contain folded map literals. [BASE declines those non-null complex literals](https://github.com/apache/datafusion-comet/blob/e0ab0a6fe60c05bd679654f0201ebe88319cc3a7/spark/src/main/scala/org/apache/comet/serde/literals.scala#L43-L64); [HEAD newly admits and rebuilds them](https://github.com/apache/datafusion-comet/blob/67fbae59727d108dda1a875e5f00c33a545855e4/spark/src/main/scala/org/apache/comet/serde/literals.scala#L43-L86), exposing the consumer limitations. That admission is not restricted to an outer array-of-map literal.
   - The stateful example exercises the [new ANSI guard](https://github.com/apache/datafusion-comet/blob/67fbae59727d108dda1a875e5f00c33a545855e4/spark/src/main/scala/org/apache/comet/serde/arrays.scala#L623-L696), which serializes the left operand twice. [BASE has no second serialization](https://github.com/apache/datafusion-comet/blob/e0ab0a6fe60c05bd679654f0201ebe88319cc3a7/spark/src/main/scala/org/apache/comet/serde/arrays.scala#L601-L639). This change affects nullable array lookups beyond literal expansion.
   - The Slice example does construct an array of maps. The [new `deepNullable` casts](https://github.com/apache/datafusion-comet/blob/67fbae59727d108dda1a875e5f00c33a545855e4/spark/src/main/scala/org/apache/comet/serde/arrays.scala#L520-L534) change its nested map type, while [Slice still declares the original element type](https://github.com/apache/datafusion-comet/blob/67fbae59727d108dda1a875e5f00c33a545855e4/spark/src/main/scala/org/apache/comet/serde/arrays.scala#L465-L484). BASE passes that single map child through without the new cast.
   
   So a narrow solution here is fine: restrict the newly enabled paths or retain safe fallback until the corresponding consumer fix lands; broader collation/collection improvements can remain separate PRs. The scope distinction above is from pinned source comparison, not an additional executed reproduction.
   
   _Originally posted by @sunchao in https://github.com/apache/datafusion-comet/pull/5452#pullrequestreview-5058500152_
               


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Fix remaining reported issues for arrays of map [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead closed issue #5544: Fix remaining reported issues for arrays of map
URL: https://github.com/apache/datafusion-comet/issues/5544


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Fix remaining reported issues for arrays of map [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on issue #5544:
URL: https://github.com/apache/datafusion-comet/issues/5544#issuecomment-5464226832

   Two additional **P2 follow-up cases** from the re-review of #5452 at [`ec9832366`](https://github.com/apache/datafusion-comet/commit/ec9832366bf05b2c645f72c8fabeacfb7af48925), recorded here under the agreed follow-up scope.
   
   Both were reproduced on Spark 4.1.3 / JDK 17 with normal constant folding and a Parquet table `t(id INT)` containing `1, 2, 3` in one file.
   
   **1. Duplicate struct field names inside folded map values**
   
   ```sql
   SELECT id, array(map(1, named_struct('x', 10, 'x', 20))) AS a
   FROM t;
   ```
   
   Spark preserves both struct values, `10` and `20`. Comet selects `CometProject` and fails with `NullPointerException` when the generated writer writes the second field. The [literal reconstruction](https://github.com/apache/datafusion-comet/blob/ec9832366bf05b2c645f72c8fabeacfb7af48925/spark/src/main/scala/org/apache/comet/serde/literals.scala#L273-L281) sends the struct value through whole-map JVM codegen, bypassing the duplicate-name guard on `CreateNamedStruct`. Arrow initializes struct children by name, collapsing the two `x` fields into one child; the writer's ordinal-1 vector is null.
   
   The direct `map(...)` variant also fails. Distinct `x`/`y` field-name controls pass. A narrow follow-up can retain Spark fallback for these nested shapes until the output path preserves duplicate field positions.
   
   **2. Calendar interval map values outside Arrow's nanosecond range**
   
   ```sql
   SELECT id, array(map(1, make_interval(0, 0, 0, 0, 3000000, 0, 0))) AS a
   FROM t;
   ```
   
   Spark accepts the `3000000 hours` value, while Comet throws `ArithmeticException: long overflow`. The [new admission check](https://github.com/apache/datafusion-comet/blob/ec9832366bf05b2c645f72c8fabeacfb7af48925/spark/src/main/scala/org/apache/comet/serde/literals.scala#L295-L302) allows this map value type; the calendar-interval restriction applies to map keys. The [generated output writer](https://github.com/apache/datafusion-comet/blob/ec9832366bf05b2c645f72c8fabeacfb7af48925/spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegenOutput.scala#L222-L229) multiplies the interval's microseconds by `1000L` using `Math.multiplyExact`, exceeding the nanosecond field's range.
   
   The direct `map(...)` variant also fails; a 24-hour control passes natively. A narrow follow-up can retain Spark fallback for out-of-range calendar interval values.
   
   **Validation:** all four failing variants (direct map and array of maps for each case) were reproduced with a freshly built native library, current JVM classes, and `CometProject` asserted. Ordinary Spark succeeds. All eight reproduction/control tests pass after substituting the exact-base [`215ab706a` literal serializer](https://github.com/apache/datafusion-comet/blob/215ab706aa9af074e589f400cd6843dee9938030/spark/src/main/scala/org/apache/comet/serde/literals.scala), restoring Spark projection for these folded literals. That control replaces only the literal serializer on the current native/JVM runtime; it is not a full base build. Local execution was limited to Spark 4.1.3.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Fix remaining reported issues for arrays of map [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on issue #5544:
URL: https://github.com/apache/datafusion-comet/issues/5544#issuecomment-5464398708

   Two **P1 updates** from the final pass of #5452 at [`ec9832366`](https://github.com/apache/datafusion-comet/commit/ec9832366bf05b2c645f72c8fabeacfb7af48925), against base [`215ab706a`](https://github.com/apache/datafusion-comet/commit/215ab706aa9af074e589f400cd6843dee9938030). The UTF-8 case is new; the stateful `element_at` case below strengthens and raises the priority of the previously reported P2.
   
   **1. [P1] Newly admitted malformed UTF-8 map literals can abort the JVM**
   
   With normal constant folding, ANSI disabled, and `t(id INT)` read from one Parquet file containing `1, 2, 3`:
   
   ```sql
   SET spark.sql.ansi.enabled = false;
   
   SELECT id,
          levenshtein(element_at(map(1, CAST(X'FF' AS STRING)), id), 'a') AS v
   FROM t;
   ```
   
   Spark returns `(1, 1), (2, NULL), (3, NULL)`. The current head selects `CometProject` and aborts the entire JVM with **SIGABRT**, rather than returning a query exception:
   
   ```text
   unsafe precondition(s) violated: hint::unreachable_unchecked must never be reached
   thread caused non-unwinding panic. aborting.
   ```
   
   The [new literal admission](https://github.com/apache/datafusion-comet/blob/ec9832366bf05b2c645f72c8fabeacfb7af48925/spark/src/main/scala/org/apache/comet/serde/literals.scala#L295-L302) rebuilds the folded map without validating its string values. The [generated writer copies Spark's raw string bytes into Arrow](https://github.com/apache/datafusion-comet/blob/ec9832366bf05b2c645f72c8fabeacfb7af48925/spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegenOutput.scala#L243-L264), and the [native bridge imports them through unchecked Arrow FFI](https://github.com/apache/datafusion-comet/blob/ec9832366bf05b2c645f72c8fabeacfb7af48925/native/spark-expr/src/jvm_udf/mod.rs#L245-L252). Native character iteration then receives bytes that violate its valid-UTF-8 requirement.
   
   This also has a silent-result manifestation: `length(element_at(map(1, CAST(X'C080' AS STRING)), id))` returns `1` for `id = 1`, where Spark returns `2`; a binary round-trip control preserves `C080` on both paths.
   
   The underlying bridge hazard predates this PR: disabling ConstantFolding also exposes it with the base serializers. The regression here is admitting these folded queries under normal optimizer settings, where the base retains safe Spark evaluation. A narrow guard can decline expansion of literals containing malformed UTF-8 until the broader bridge/string handling is fixed. Replacing the bytes with normalized text would not preserve Spark's raw-byte semantics.
   
   **2. [P1] The ANSI `element_at` guard silently changes values and NULL placement for stateful operands**
   
   This is the existing duplicated-left-operand family, now reproduced with a simpler primitive-array query. For one Parquet file written with IDs `1` through `16` in order:
   
   ```sql
   SET spark.sql.ansi.enabled = true;
   
   SELECT id,
          element_at(
            IF(monotonically_increasing_id() % 2 = 0,
               array(1), CAST(NULL AS ARRAY<INT>)),
            1) AS v
   FROM t;
   ```
   
   Spark returns non-NULL values for all eight odd IDs. Comet returns non-NULL values only for IDs `1, 5, 9, 13`: **four expected values silently become NULL**. The simpler random variant, `element_at(IF(rand(7L) < 0.5, array(1), CAST(NULL AS ARRAY<INT>)), 1)`, also changes NULL placement, including producing a value on a row where Spark returns NULL.
   
   The left operand is serialized [for the lookup](https://github.com/apache/datafusion-comet/blob/ec9832366bf05b2c645f72c8fabeacfb7af48925/spark/src/main/scala/org/apache/comet/serde/arrays.scala#L623-L628) and [again for the new null guard](https://github.com/apache/datafusion-comet/blob/ec9832366bf05b2c645f72c8fabeacfb7af48925/spark/src/main/scala/org/apache/comet/serde/arrays.scala#L663-L686). Reevaluating a stateful expression over different row selections changes its state and results. Both examples pass with ANSI disabled, and both pass with the base serializers while still executing a native projection.
   
   I would raise this from P2 to P1 because it regresses an already-supported primitive-array path, requires no malformed data or incompatible-expression flag, and silently changes downstream counts, filters, or persisted results. ANSI is enabled by default in the tested Spark 4.1.3 session. The demonstrated trigger is an **inline nullable, stateful left operand**; these tests do not show corruption of ordinary deterministic array columns. Retaining Spark fallback for affected nondeterministic operands is a narrow safeguard while single-evaluation handling is implemented.
   
   **Validation and limits:** both findings were independently reproduced on Spark 4.1.3 / JDK 17 with native and JVM artifacts built from the pinned head and `CometProject` verified. Exact-base controls substitute only `arrays.scala`, `maps.scala`, and `literals.scala` serializers on the current native/JVM runtime; they are not a full base build. The UTF-8 process abort was reproduced with the debug native build and active unsafe-precondition checks. Release-build failure behavior, heap corruption, and exploitability were not tested or established.
   
   I recommend retaining safe fallback for these cases before merging #5452; the broader fixes can remain follow-ups here.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Fix remaining reported issues for arrays of map [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #5544:
URL: https://github.com/apache/datafusion-comet/issues/5544#issuecomment-5546992022

   Another edge case is 
   ```
     import org.apache.spark.sql.functions._                                                                                                                                                                        
                                                                                                                                                                                                                    
     Seq(Option(1)).toDF("k").write.mode("overwrite").parquet("/tmp/comet_repro/empty_map")                                                                                                                         
     val df = spark.read.parquet("/tmp/comet_repro/empty_map")                                                                                                                                                      
                                                                                                                                                                                                                    
     val emptyMapping = typedLit(Map.empty[Int, List[Int]])                                                                                                                                                         
     println(emptyMapping.expr.dataType)   // MapType(IntegerType,ArrayType(IntegerType,false),true)                                                                                                                
                                                                                                                                                                                                                    
     val out = df.select(array(emptyMapping($"k")).as("a"))                                                                                                                                                         
     out.show()                                                                                                                                                                                                     
     println(new org.apache.comet.ExtendedExplainInfo().getFallbackReasons(out.queryExecution.executedPlan))
   ```
   
   I still see a fallback for it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org