You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "rebasedming (via GitHub)" <gi...@apache.org> on 2024/01/05 14:57:14 UTC
[I] Cannot insert with DMLStatement into a table with non-nullable fields [arrow-datafusion]
rebasedming opened a new issue, #8763:
URL: https://github.com/apache/arrow-datafusion/issues/8763
### Describe the bug
When a table is created with a non-nullable `Field`, executing a `DMLStatement` logical plan to write to that table always fails with the error `Column [name] is declared as non-nullable but contains null values`. This happens no matter what values I try to insert.
### To Reproduce
The following code will reproduce the error. It creates a table called `t1` with a non-nullable int32 `Field` called `a`. When we try to add an integer into this table, it errors.
```rust
use async_std::task;
use datafusion::arrow::datatypes::{DataType, Field, Schema};
use datafusion::arrow::record_batch::RecordBatch;
use datafusion::common::{DFSchema, ScalarValue};
use datafusion::datasource::MemTable;
use datafusion::logical_expr::{DmlStatement, Expr, LogicalPlan, Values};
use datafusion::logical_expr::WriteOp::InsertInto;
use datafusion::prelude::SessionContext;
use std::sync::Arc;
fn main() {
let table_name = "t1";
let field = Field::new("a", DataType::Int32, false);
let ctx = SessionContext::new();
let schema = Schema::new(vec![field]);
let df_schema = DFSchema::try_from(schema.clone()).unwrap();
match MemTable::try_new(schema.clone().into(), vec![Vec::<RecordBatch>::new()]).ok() {
Some(mem_table) => {
let _ = ctx.register_table(table_name.clone(), Arc::new(mem_table));
}
None => panic!("An unexpected error occured creating the table"),
};
let values = vec![vec![Expr::Literal(ScalarValue::Int32(Some(1)))]];
let values_plan = LogicalPlan::Values(Values {
schema: df_schema.clone().into(),
values: values.clone(),
});
let dml_plan = LogicalPlan::Dml(DmlStatement {
table_name: table_name.into(),
table_schema: df_schema.clone().into(),
op: InsertInto,
input: values_plan.into(),
});
// This step always errors out
// Error:
// ArrowError(InvalidArgumentError("Column 'a' is declared as non-nullable but
// contains null values"))'
let dataframe = task::block_on(ctx.execute_logical_plan(dml_plan.clone()))
.expect("Could not execute plan");
let _ = task::block_on(dataframe.collect()).expect("Could not collect dataframe");
println!("Success!");
}
```
### Expected behavior
We would expect this insert to succeed, as the values we are inserting are not null.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot insert with DMLStatement into a table with non-nullable fields [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8763:
URL: https://github.com/apache/arrow-datafusion/issues/8763#issuecomment-1879245883
https://github.com/apache/arrow-datafusion/issues/7693 and https://github.com/apache/arrow-datafusion/issues/7636 may be related
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot insert with DMLStatement into a table with non-nullable fields [arrow-datafusion]
Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on issue #8763:
URL: https://github.com/apache/arrow-datafusion/issues/8763#issuecomment-1879484539
I will try to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot insert with DMLStatement into a table with non-nullable fields [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8763:
URL: https://github.com/apache/arrow-datafusion/issues/8763#issuecomment-1879248006
This definitely sounds like a bug -- thank you for filing this @rebasedming
I think @jonahgao has worked on something similar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot insert with DMLStatement into a table with non-nullable fields [arrow-datafusion]
Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on issue #8763:
URL: https://github.com/apache/arrow-datafusion/issues/8763#issuecomment-1880065725
I have verified that the snippet in the issue can be executed successfully on [this branch](https://github.com/jonahgao/arrow-datafusion/blob/fix_values_exec_with_tests/datafusion/core/tests/sql/mod.rs#L799).
The corresponding PR is #8776.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot insert with DMLStatement into a table with non-nullable fields [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #8763: Cannot insert with DMLStatement into a table with non-nullable fields
URL: https://github.com/apache/arrow-datafusion/issues/8763
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org