You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jonahgao (via GitHub)" <gi...@apache.org> on 2024/01/07 13:14:26 UTC
[PR] fix: failed to create ValuesExec with non-nullable schema [arrow-datafusion]
jonahgao opened a new pull request, #8776:
URL: https://github.com/apache/arrow-datafusion/pull/8776
## Which issue does this PR close?
Closes #8763.
## Rationale for this change
In `ValuesExec::try_new`, we use the schema of `ValuesExec` and **null arrays** to construct a placeholder batch.
https://github.com/apache/arrow-datafusion/blob/dd4263f843e093c807d63edf73a571b1ba2669b5/datafusion/physical-plan/src/values.rs#L56-L64
This batch placeholder is used to evaluate the physical expressions in Values.
But users can define a schema containing non-nullable fields just like issue #8763, which is in conflict with null arrays.
```rust
let field = Field::new("a", DataType::Int32, false);
let schema = Schema::new(vec![field]);
let df_schema = DFSchema::try_from(schema.clone()).unwrap();
let values = vec![vec![Expr::Literal(ScalarValue::Int32(Some(1)))]];
let values_plan = LogicalPlan::Values(Values {
schema: df_schema.clone().into(),
values: values.clone(),
})
```
In this case, an ArrowError ‘InvalidArgumentError("Column 'a' is declared as non-nullable but contains null values"))' will be raised.
Since `ValuesExec` has no input, I think the schema for this batch placeholder can be empty, rather than using the schema from `ValuesExec`.
## What changes are included in this PR?
Use correct schema to build the placeholder batch in `ValuesExec::try_new`.
## Are these changes tested?
Yes
## Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: failed to create ValuesExec with non-nullable schema [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8776:
URL: https://github.com/apache/arrow-datafusion/pull/8776#issuecomment-1881583107
Also thank you for:
1. The very clear PR description
2. The drive by cleanups to comments
3. The fix of the other bug you found (and the clear comments and tests) 🙏
Very much appreciated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: failed to create ValuesExec with non-nullable schema [arrow-datafusion]
Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on PR #8776:
URL: https://github.com/apache/arrow-datafusion/pull/8776#issuecomment-1882165687
Thank you for reviewing @alamb .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: failed to create ValuesExec with non-nullable schema [arrow-datafusion]
Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on code in PR #8776:
URL: https://github.com/apache/arrow-datafusion/pull/8776#discussion_r1444004985
##########
datafusion/physical-plan/src/values.rs:
##########
@@ -71,7 +69,7 @@ impl ValuesExec {
match r {
Ok(ColumnarValue::Scalar(scalar)) => Ok(scalar),
Ok(ColumnarValue::Array(a)) if a.len() == 1 => {
- Ok(ScalarValue::List(a))
+ ScalarValue::try_from_array(&a, 0)
Review Comment:
Returning `ScalarValue::List(a)` makes the following query fail.
```sh
DataFusion CLI v34.0.0
❯ select * from (values(random()));
Internal error: Inconsistent types in ScalarValue::iter_to_array. Expected Float64, got List(0.5174512019067328).
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
```
Related to pr #7629. I think it's a bug, so I fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: failed to create ValuesExec with non-nullable schema [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #8776:
URL: https://github.com/apache/arrow-datafusion/pull/8776
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: failed to create ValuesExec with non-nullable schema [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8776:
URL: https://github.com/apache/arrow-datafusion/pull/8776#discussion_r1445091772
##########
datafusion/sqllogictest/test_files/select.slt:
##########
@@ -114,6 +114,14 @@ VALUES (1,2,3,4,5,6,7,8,9,10,11,12,13,NULL,'F',3.5)
----
1 2 3 4 5 6 7 8 9 10 11 12 13 NULL F 3.5
+# Test non-literal expressions in VALUES
Review Comment:
👍
##########
datafusion/physical-plan/src/values.rs:
##########
@@ -53,15 +52,14 @@ impl ValuesExec {
}
let n_row = data.len();
let n_col = schema.fields().len();
- // we have this single row, null, typed batch as a placeholder to satisfy evaluation argument
- let batch = RecordBatch::try_new(
- schema.clone(),
- schema
- .fields()
- .iter()
- .map(|field| new_null_array(field.data_type(), 1))
- .collect::<Vec<_>>(),
+ // we have this single row batch as a placeholder to satisfy evaluation argument
Review Comment:
👍 this code to use a null array probably pre-dates the ability to create zero column record batches
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org