[jira] [Created] (ARROW-9778) [Rust] [DataFusion] Logical and physical schemas' nullability does not match in 8 out of 20 end-to-end tests

["Jorge (Jira)" <ji...@apache.org>]

Jorge created ARROW-9778:
----------------------------

             Summary: [Rust] [DataFusion] Logical and physical schemas' nullability does not match in 8 out of 20 end-to-end tests
                 Key: ARROW-9778
                 URL: https://issues.apache.org/jira/browse/ARROW-9778
             Project: Apache Arrow
          Issue Type: Bug
          Components: Rust, Rust - DataFusion
            Reporter: Jorge


In `tests/sql.rs`, if we re-write the ```execute``` function to test the end schemas, as

```
/// Execute query and return result set as tab delimited string
fn execute(ctx: &mut ExecutionContext, sql: &str) -> Vec<String> {
    let plan = ctx.create_logical_plan(&sql).unwrap();
    let plan = ctx.optimize(&plan).unwrap();
    let physical_plan = ctx.create_physical_plan(&plan).unwrap();
    let results = ctx.collect(physical_plan.as_ref()).unwrap();
    if results.len() > 0 {
        // results must match the logical schema
        assert_eq!(plan.schema().as_ref(), results[0].schema().as_ref());
    }
    result_str(&results)
}
```

we end up with 8 tests failing, which indicates that our physical and logical plans are not aligned. In all cases, the issue is nullability: our logical plan assumes nullability = true, while our physical plan may change the nullability field.

If we do not plan to track nullability on the logical level, we could consider replacing Schema by a type that does not track nullability.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)