[I] array_has_any/all drop input NULLs in the empty-needle fast path [datafusion]

["yashrb24 (via GitHub)" <gi...@apache.org>]

yashrb24 opened a new issue, #24978:
URL: https://github.com/apache/datafusion/issues/24978

   ### Describe the bug
   
   `array_has_any` and `array_has_all` drop top-level input NULLs when every row in the needle array is NULL or empty. In that batch shape, nullable rows return `false` / `true` instead of NULL.
   
   The result can depend on another row in the same batch: adding one non-empty needle row disables the fast path and restores the expected NULL result.
   
   This reproduces on DataFusion 52.1.0 and current `main` at `722cbf2e7ad9a43bc6723f7795fe87e0552eb81f`.
   
   ### To Reproduce
   
   Run with DataFusion CLI:
   
   ```sql
   WITH t(id, haystack, needle) AS (
     VALUES
       (1, [1, 2], arrow_cast(NULL, 'List(Int64)')),
       (2, arrow_cast(NULL, 'List(Int64)'), arrow_cast([], 'List(Int64)')),
       (3, [3, 4], arrow_cast([], 'List(Int64)'))
   )
   SELECT array_has_any(haystack, needle),
          array_has_all(haystack, needle)
   FROM t
   ORDER BY id;
   ```
   
   Actual output:
   
   ```text
   false true
   false true
   false true
   ```
   
   As a control, put a non-empty needle in another row:
   
   ```sql
   WITH t(haystack, needle) AS (
     VALUES
       ([1, 2], arrow_cast(NULL, 'List(Int64)')),
       ([3, 4], [9])
   )
   SELECT array_has_any(haystack, needle) FROM t;
   ```
   
   This returns `NULL`, `false`, showing that the nullable row changes with the batch contents.
   
   ### Expected behavior
   
   Rows where either input list is NULL should return NULL. Non-null rows with an empty needle should keep the existing empty-set results: `false` for `array_has_any` and `true` for `array_has_all`.
   
   For the first query, the expected output is:
   
   ```text
   NULL  NULL
   NULL  NULL
   false true
   ```
   
   ### Additional context
   
   The empty-needle fast path constructs a `BooleanArray` without the input null mask, while the general and string paths preserve the union of the input null masks. The affected code is unchanged from upstream 52.1.0 in the downstream fork.
   
   This was found while wiring `ARRAYS_OVERLAP` in e6data/e6-native-executor#818. A downstream fix is in e6data/e6-datafusion#223.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] array_has_any/all drop input NULLs in the empty-needle fast path [datafusion]

["yashrb24 (via GitHub)" <gi...@apache.org>]

yashrb24 closed issue #24978: array_has_any/all drop input NULLs in the empty-needle fast path
URL: https://github.com/apache/datafusion/issues/24978


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org