You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/01/15 00:58:54 UTC

[I] [Feature] Support Spark expression: length_of_json_array [datafusion-comet]

andygrove opened a new issue, #3153:
URL: https://github.com/apache/datafusion-comet/issues/3153

   ## What is the problem the feature request solves?
   
   > **Note:** This issue was generated with AI assistance. The specification details have been extracted from Spark documentation and may need verification.
   
   Comet does not currently support the Spark `length_of_json_array` function, causing queries using this function to fall back to Spark's JVM execution instead of running natively on DataFusion.
   
   The `LengthOfJsonArray` expression returns the number of elements in a JSON array string. It is implemented as a runtime-replaceable expression that delegates to the JsonExpressionUtils utility class for actual JSON parsing and length calculation.
   
   Supporting this expression would allow more Spark workloads to benefit from Comet's native acceleration.
   
   ## Describe the potential solution
   
   ### Spark Specification
   
   **Syntax:**
   ```sql
   json_array_length(json_string)
   ```
   
   ```scala
   // DataFrame API
   import org.apache.spark.sql.functions.expr
   df.select(expr("json_array_length(json_column)"))
   ```
   
   **Arguments:**
   | Argument | Type | Description |
   |----------|------|-------------|
   | json_string | String (with collation support) | A JSON string representing an array whose length is to be calculated |
   
   **Return Type:** Returns `IntegerType` representing the number of elements in the JSON array.
   
   **Supported Data Types:**
   - String types with collation support (specifically supports trim collation)
   
   - Input must be a valid JSON array string format
   
   **Edge Cases:**
   - Returns `NULL` when input is `NULL` (expression is nullable)
   
   - Behavior with malformed JSON depends on JsonExpressionUtils implementation
   
   - Empty JSON array `[]` should return `0`
   
   - Non-array JSON values (objects, primitives) handling depends on underlying utility
   
   - Invalid JSON strings may return `NULL` or throw exceptions based on implementation
   
   **Examples:**
   ```sql
   -- Basic usage
   SELECT json_array_length('[1, 2, 3, 4]') AS length;
   -- Returns: 4
   
   -- Empty array
   SELECT json_array_length('[]') AS length;  
   -- Returns: 0
   
   -- NULL input
   SELECT json_array_length(NULL) AS length;
   -- Returns: NULL
   
   -- With table data
   SELECT id, json_array_length(json_data) AS array_size 
   FROM table_with_json 
   WHERE json_data IS NOT NULL;
   ```
   
   ```scala
   // DataFrame API usage
   import org.apache.spark.sql.functions._
   
   val df = spark.createDataFrame(Seq(
     ("1", "[1, 2, 3]"),
     ("2", "[]"),
     ("3", null)
   )).toDF("id", "json_array")
   
   df.select(
     col("id"),
     expr("json_array_length(json_array)").as("length")
   ).show()
   ```
   
   ### Implementation Approach
   
   See the [Comet guide on adding new expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html) for detailed instructions.
   
   1. **Scala Serde**: Add expression handler in `spark/src/main/scala/org/apache/comet/serde/`
   2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
   3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if needed
   4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has built-in support first)
   
   
   ## Additional context
   
   **Difficulty:** Medium
   **Spark Expression Class:** `org.apache.spark.sql.catalyst.expressions.LengthOfJsonArray`
   
   **Related:**
   - Other JSON functions in the `json_funcs` group
   - `JsonExpressionUtils` utility class
   - `StaticInvoke` expression for runtime delegation
   - JSON parsing and extraction functions
   
   ---
   *This issue was auto-generated from Spark reference documentation.*
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] [Feature] Support Spark expression: length_of_json_array [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #3153: [Feature] Support Spark expression: length_of_json_array
URL: https://github.com/apache/datafusion-comet/issues/3153


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org