You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "neilconway (via GitHub)" <gi...@apache.org> on 2026/05/10 13:07:04 UTC
[I] Substrait literal conversion should preserve nested nullability [datafusion]
neilconway opened a new issue, #22095:
URL: https://github.com/apache/datafusion/issues/22095
### Is your feature request related to a problem or challenge?
DataFusion’s Substrait consumer currently normalizes some nested literal fields to nullable when converting Substrait literals into ScalarValues.
This is intentional today because `VirtualTable` / `Values` consumption converts the schema and literal rows separately. DataFusion later materializes the values into Arrow arrays and builds a `RecordBatch` using the schema. Arrow requires nested field types to match exactly, including nullability, so a mismatch like this can fail even
when the value itself is valid:
```
schema: List(Field { name: "item", data_type: Int32, nullable: false })
literal: List(Field { name: "item", data_type: Int32, nullable: true })
```
### Describe the solution you'd like
Add an expected-type-aware Substrait literal conversion path.
For `VirtualTable` / `Values`, the consumer has access to the `NamedStruct` schema. When converting each literal row, pass the expected top-level Field into literal conversion, and recursively use the expected nested fields for Struct, List, and Map literals.
The existing schema-less literal conversion path should remain available for ordinary expression literals where no expected field/type is available.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Substrait literal conversion should preserve nested nullability [datafusion]
Posted by "nathanb9 (via GitHub)" <gi...@apache.org>.
nathanb9 commented on issue #22095:
URL: https://github.com/apache/datafusion/issues/22095#issuecomment-4416836584
if its okay, I can take this @neilconway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Substrait literal conversion should preserve nested nullability [datafusion]
Posted by "huymq1710 (via GitHub)" <gi...@apache.org>.
huymq1710 commented on issue #22095:
URL: https://github.com/apache/datafusion/issues/22095#issuecomment-5213373303
@neilconway , @nathanb9
Since this issue has been open without action for a while, I picked it up and opened https://github.com/apache/datafusion/issues/22095
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Substrait literal conversion should preserve nested nullability [datafusion]
Posted by "nathanb9 (via GitHub)" <gi...@apache.org>.
nathanb9 commented on issue #22095:
URL: https://github.com/apache/datafusion/issues/22095#issuecomment-5216338914
untake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Substrait literal conversion should preserve nested nullability [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on issue #22095:
URL: https://github.com/apache/datafusion/issues/22095#issuecomment-4421226538
@nathanb9 Absolutely! It make make sense to wait until #12727 is landed to avoid merge conflicts, but that's up to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org