You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jacksonrnewhouse (via GitHub)" <gi...@apache.org> on 2023/12/12 02:12:23 UTC
[I] non-null sub-field on nullable struct-field has wrong nullity. [arrow-datafusion]
jacksonrnewhouse opened a new issue, #8507:
URL: https://github.com/apache/arrow-datafusion/issues/8507
### Describe the bug
If you have table schema like
```
DFSchema {
fields: [
DFField {
qualifier: Some(
Bare {
table: "nexmark",
},
),
field: Field {
name: "bid",
data_type: Struct(
[
Field {
name: "auction",
data_type: Int64,
nullable: false,
},
nullable: true,
}
```
And run a query like `SELECT bid.auction FROM nexmark` you'll get an error when bid is null. Error looks like
```
ArrowError(InvalidArgumentError("Column 'nexmark.bid[datetime]' is declared as non-nullable but contains null values"))
```
The problem is that the nullable() method only looks at the sub-fields nullability, ignoring that the parent field may be null.
https://github.com/apache/arrow-datafusion/blob/main/datafusion/expr/src/expr_schema.rs#L280.
### To Reproduce
_No response_
### Expected behavior
The expression should be nullable, as its parent may not be present.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] non-null sub-field on nullable struct-field has wrong nullity. [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8507:
URL: https://github.com/apache/arrow-datafusion/issues/8507#issuecomment-1863392087
> Hi @alamb, what I understand after reading the issue is that I should check condition whether the parent is null || sub-fields are null. if I am wrong please correct me.
That sounds about right @ravikiran232 -- what I would suggest doing first is creating a reproducer / test that shows the behavior described in this ticket.
Then the fix should likely be straightforward
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] non-null sub-field on nullable struct-field has wrong nullity. [arrow-datafusion]
Posted by "ravikiran232 (via GitHub)" <gi...@apache.org>.
ravikiran232 commented on issue #8507:
URL: https://github.com/apache/arrow-datafusion/issues/8507#issuecomment-1867312825
Hi @marvinlanhenke, continue with the issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] non-null sub-field on nullable struct-field has wrong nullity. [arrow-datafusion]
Posted by "marvinlanhenke (via GitHub)" <gi...@apache.org>.
marvinlanhenke commented on issue #8507:
URL: https://github.com/apache/arrow-datafusion/issues/8507#issuecomment-1867271239
Hi @ravikiran232 are you still working on this?
Since I'm a `first-timer` (as well) I took the liberty to look into this Issue as well, in order to learn and get a better feel for the codebase. I think I already found a solution. However, I don't want to steal your opportunity to contribute.
Hi @alamb just to confirm the approach.
I wrote a test creating a nested schema and an expr = col("bid).field("auction") on which I executed nullable(). Then I included a guard close in the relevant match arm to check, if the parent col is nullable and return early if it is? Sounds about right?
If @ravikiran232 is not currently working on this anymore, I'd be more than happy to contribute.
Thank you for you feedback and your efforts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] non-null sub-field on nullable struct-field has wrong nullity. [arrow-datafusion]
Posted by "ravikiran232 (via GitHub)" <gi...@apache.org>.
ravikiran232 commented on issue #8507:
URL: https://github.com/apache/arrow-datafusion/issues/8507#issuecomment-1859331359
@alamb I will try to solve the issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] non-null sub-field on nullable struct-field has wrong nullity. [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #8507: non-null sub-field on nullable struct-field has wrong nullity.
URL: https://github.com/apache/arrow-datafusion/issues/8507
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] non-null sub-field on nullable struct-field has wrong nullity. [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8507:
URL: https://github.com/apache/arrow-datafusion/issues/8507#issuecomment-1852826150
Thank you for the report @jacksonrnewhouse -- this looks like a good first issue to me so marking it as such
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] non-null sub-field on nullable struct-field has wrong nullity. [arrow-datafusion]
Posted by "ravikiran232 (via GitHub)" <gi...@apache.org>.
ravikiran232 commented on issue #8507:
URL: https://github.com/apache/arrow-datafusion/issues/8507#issuecomment-1862114562
Hi @alamb, what I understand after reading the issue is that I should check condition whether the parent is null || sub-fields are null. if I am wrong please correct me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org