You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2025/07/02 04:20:26 UTC
Re: [I] Support for "Schema evolution" / Schema Adapters [arrow-rs]
adriangb commented on issue #6735:
URL: https://github.com/apache/arrow-rs/issues/6735#issuecomment-3026363292
> Often computing the transformation may be non trivial (e.g. matching columns by name) so it would be nice to do the mapping calculation once per schema rather than once per batch / StructArrayschema. For example DF's SchemaAdapter computes the mapping once and can then apply that to multiple batches.
I'm tot sure how this would happen in practice: there's no state in `UDFs` in DataFusion. So if we e.g. wanted to implement `cast(...)` in terms of a SchemaAdapter we have nowhere to store the pre-computed value. I think we'd have to introduce some sort of build step that goes around the expression tree and optimizes expressions for the given input / output schemas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org