You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "bhanreddy1973 (via GitHub)" <gi...@apache.org> on 2026/01/02 10:56:58 UTC

Re: [PR] Add preserve_empty_as_null option to unnest for Spark explode_outer compatibility [datafusion]

bhanreddy1973 commented on PR #19574:
URL: https://github.com/apache/datafusion/pull/19574#issuecomment-3705085824

   Good question @Jefffrey!
   
   If `preserve_nulls = false` and `preserve_empty_as_null = true`:
   - Empty arrays `[]` are first treated as NULL
   - Then, since `preserve_nulls = false`, NULLs produce 0 output rows
   - So empty arrays would produce 0 rows
   
   The two flags work independently:
   1. `preserve_empty_as_null` converts `[]` → NULL
   2. `preserve_nulls` determines if NULLs produce output
   
   | preserve_nulls | preserve_empty_as_null | Empty [] produces |
   |----------------|------------------------|-------------------|
   | true | true | 1 row (NULL) |
   | true | false | 0 rows |
   | false | true | 0 rows |
   | false | false | 0 rows |
   
   Should I add a test case for this combination, or update the documentation to clarify?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org