You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "brancz (via GitHub)" <gi...@apache.org> on 2025/11/13 22:33:45 UTC
[I] Unexpected schema merging nullability behavior [arrow-rs]
brancz opened a new issue, #8840:
URL: https://github.com/apache/arrow-rs/issues/8840
**Describe the bug**
When merging two schemas using `Schema::try_merge` where one side doesn't have a column but the other does, then it keep the nullability setting of the preexisting column. However, this semantically doesn't make sense, the merged version of that would be the field with nullable being true, since that is the implicit property of the schema that doesn't have the field.
Consequently what this means is that it's impossible to merge schemas and therefore record batches where one side has a field that is nullable false, and the other doesn't have it at all.
**To Reproduce**
```
#[test]
fn test_schema_merge_nullability() {
let merged = Schema::try_merge(vec![
Schema::new(vec![
Field::new("first_name", DataType::Utf8, false),
]),
Schema::new(vec![
Field::new("last_name", DataType::Utf8, false),
]),
])
.unwrap();
assert_eq!(
merged,
Schema::new(
vec![
Field::new("first_name", DataType::Utf8, true),
Field::new("last_name", DataType::Utf8, true),
],
)
);
}
```
**Expected behavior**
The above test passes.
**Additional context**
This of course assumes that the whole intention of schema merging is to merge record batches of merged schemas, so if that is a wrong assumption this can be solved other ways for us by creating our own schema merge functionality, however, I see no useful reason to merge schemas and not also merge record batches. And if that's the intention then I think this is a legitimate bug.
@alamb @vegarsti
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unexpected schema merging nullability behavior [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8840:
URL: https://github.com/apache/arrow-rs/issues/8840#issuecomment-3536450296
> When merging two schemas using Schema::try_merge where one side doesn't have a column but the other does, then it keep the nullability setting of the preexisting column. However, this semantically doesn't make sense, the merged version of that would be the field with nullable being true, since that is the implicit property of the schema that doesn't have the field.
I agree this doesn't make sense. I will look at the PR now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unexpected schema merging nullability behavior [arrow-rs]
Posted by "vegarsti (via GitHub)" <gi...@apache.org>.
vegarsti commented on issue #8840:
URL: https://github.com/apache/arrow-rs/issues/8840#issuecomment-3531212384
Agree that this seems like a legitimate bug. The nullability going from false to true when the field exists on both schemas works as expected
https://github.com/apache/arrow-rs/blob/2bc269c3eec23f6794fcd793b641ea4c08325d54/arrow-schema/src/schema.rs#L1330
In `try_merge` we simply iterate over all the schemas and merge each field:
https://github.com/apache/arrow-rs/blob/2bc269c3eec23f6794fcd793b641ea4c08325d54/arrow-schema/src/schema.rs#L295-L319
Might there be existing use cases where this behavior is a feature and not a bug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unexpected schema merging nullability behavior [arrow-rs]
Posted by "vegarsti (via GitHub)" <gi...@apache.org>.
vegarsti commented on issue #8840:
URL: https://github.com/apache/arrow-rs/issues/8840#issuecomment-3541317116
> > To clarify, are you [@alamb](https://github.com/alamb) saying you're in favour of changing the existing semantics with something like the linked PR, in parallel to building this API in DataFusion?
>
> I suggest starting with an API in DataFusion where we can work out whatever the most useful semantics are. Then when we port it back upstream to arrow we can figure out what the best defaults, etc
Yep, sounds good! Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unexpected schema merging nullability behavior [arrow-rs]
Posted by "vegarsti (via GitHub)" <gi...@apache.org>.
vegarsti commented on issue #8840:
URL: https://github.com/apache/arrow-rs/issues/8840#issuecomment-3538995331
To clarify, are you @alamb saying you're in favour of changing the existing semantics with something like the linked PR in parallel?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unexpected schema merging nullability behavior [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8840:
URL: https://github.com/apache/arrow-rs/issues/8840#issuecomment-3541311933
> To clarify, are you [@alamb](https://github.com/alamb) saying you're in favour of changing the existing semantics with something like the linked PR, in parallel to building this API in DataFusion?
I suggest starting with an API in DataFusion where we can work out whatever the most useful semantics are. Then when we port it back upstream to arrow we can figure out what the best defaults, etc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unexpected schema merging nullability behavior [arrow-rs]
Posted by "vegarsti (via GitHub)" <gi...@apache.org>.
vegarsti commented on issue #8840:
URL: https://github.com/apache/arrow-rs/issues/8840#issuecomment-3538169429
Yeah, this sounds like a good idea! And the API seems nice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unexpected schema merging nullability behavior [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8840:
URL: https://github.com/apache/arrow-rs/issues/8840#issuecomment-3538560251
> Only suggestion I would have is to not even have the option to preserve nullability at first until there is a clear use case (I still struggle to come up with one).
Make sense to me -- the only potential reason I can see is if we need to preserve the existing (strange) semantics for some reason. Defaulting to the nullability behavior makes sense to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unexpected schema merging nullability behavior [arrow-rs]
Posted by "brancz (via GitHub)" <gi...@apache.org>.
brancz commented on issue #8840:
URL: https://github.com/apache/arrow-rs/issues/8840#issuecomment-3536477673
I like that idea of creating a new API in datafusion first. Only suggestion I would have is to not even have the option to preserve nullability at first until there is a clear use case (I still struggle to come up with one). Having the schema merger like that though makes sense to me so we can extend it without breaking APIs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unexpected schema merging nullability behavior [arrow-rs]
Posted by "vegarsti (via GitHub)" <gi...@apache.org>.
vegarsti commented on issue #8840:
URL: https://github.com/apache/arrow-rs/issues/8840#issuecomment-3536190994
Pushed a PR here: #8851
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unexpected schema merging nullability behavior [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8840:
URL: https://github.com/apache/arrow-rs/issues/8840#issuecomment-3536458954
So the topic (and pain) of schema merging comes up a bunch in DataFusion (and perhaps elsewhere)
For example, here is the code in DataFusion that handles schema merging for Parquet
https://github.com/apache/datafusion/blob/6ab4d216b768c9327982e59376a62a29c69ca436/datafusion/datasource-parquet/src/file_format.rs#L406-L421
We also have a similar challenge when comparing schemas (in some cases a field that is less nullable than another should be compatibility).
I am not quite sure what merging logic belongs in what crate (e.g. I don't have a sense for if there is a broadly agreed upon definition of what schema merging means for schema outsides the context of the schema evolution context of DataFusion)
Thus what I suggest is:
1. We start by moving schema merging logic into DataFusion, and iterate there until we get it right
2. the consider if the logic belongs upstream in arrow-rs or not where we can commit to an API
Given all the various potential options, the API I would suggest is some sort of Merger structure. Something like
```rust
let mut schema_merger = SchemaMerger::new()
.with_preserve_nulls(true); // set various options builder style;
// try to merge the schemas
schema_merger.try_merge(schemas)?;
// get the built schema
let merged_schema = schema_merger.build()?
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unexpected schema merging nullability behavior [arrow-rs]
Posted by "brancz (via GitHub)" <gi...@apache.org>.
brancz commented on issue #8840:
URL: https://github.com/apache/arrow-rs/issues/8840#issuecomment-3533005185
I can't imagine a use case where merging two schemas isn't to also then merge/concat the record batches of the unified schema. But perhaps I'm only seeing my own use case, so if someone can come up with one, I'm all ears.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unexpected schema merging nullability behavior [arrow-rs]
Posted by "vegarsti (via GitHub)" <gi...@apache.org>.
vegarsti commented on issue #8840:
URL: https://github.com/apache/arrow-rs/issues/8840#issuecomment-3531198346
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org