You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "paleolimbot (via GitHub)" <gi...@apache.org> on 2025/04/09 03:24:13 UTC
[PR] Type extension maybe [arrow-rs]
paleolimbot opened a new pull request, #7398:
URL: https://github.com/apache/arrow-rs/pull/7398
# Which issue does this PR close?
Experiments in pursuit of #7063 and https://github.com/apache/datafusion/issues/12644 .
# Rationale for this change
Customizing behaviour (e.g., sorting, signatures for user defined functions, import/export from/to formats like Parquet, Arrow IPC, and FFI) is increasingly requested. arrow-rs is also increasingly the implementation of choice for compute frameworks and many APIs have been built around the `DataType` and `ArrayRef` (e.g., GeoArrow, parquet, many parts of DataFusion). To support additional types, those frameworks have to either invent a new array type and rewrite substantial wrappers (e.g., GeoArrow), or use something other than a `DataType` (e.g., a Field or MyOwnDataType) and forego having that type automatically pass through APIs from other arrow-rs-based libraries (or arrow itself).
Incorporating an ExtensionType as a first-class data type provides a potentially less disruptive route for libraries like parquet and DataFusion to implement new Parquet types and other types other databases implement like JSON and UUID. I also have an experiment going for swapping out a `Field` for a `DataType` in DataFusion ( https://github.com/apache/datafusion/pull/15036 ).
# What changes are included in this PR?
A new `DataType::Extension()` enum member was added. I had hoped to have this be `DataType::Extension(Arc<dyn ExtensionType>)` but the current design of the ExtensionType is not dyn-compatible. The `DynExtensionType` trait is really a placeholder to collect the requirements of the rest of arrow-rs for this experiment.
This is currently just an experiment/minimum change that actually builds! I'm happy to continue the experiment if there is interest.
# Are there any user-facing changes?
Yes, a new data type enum member at least (which is a breaking change).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Experiment with Extension as a DataType [arrow-rs]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot closed pull request #7398: Experiment with Extension as a DataType
URL: https://github.com/apache/arrow-rs/pull/7398
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Experiment with Extension as a DataType [arrow-rs]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #7398:
URL: https://github.com/apache/arrow-rs/pull/7398#discussion_r2036110803
##########
arrow-data/src/transform/mod.rs:
##########
@@ -276,6 +276,7 @@ fn build_extend(array: &ArrayData) -> Extend {
UnionMode::Dense => union::build_extend_dense(array),
},
DataType::RunEndEncoded(_, _) => todo!(),
+ DataType::Extension(_) => unimplemented!("Extension not implemented"),
Review Comment:
Yes, the tradeoff is taking on the burden of extensibility (so that other projects don't have to implement it separately, for example, by redefining wrapper DataTypes or Arrays). This extra burden hasn't caused issues in Arrow C++/pyarrow that I'm aware of but absolutely fair that it might here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Experiment with Extension as a DataType [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #7398:
URL: https://github.com/apache/arrow-rs/pull/7398#discussion_r2034858541
##########
arrow-data/src/transform/mod.rs:
##########
@@ -276,6 +276,7 @@ fn build_extend(array: &ArrayData) -> Extend {
UnionMode::Dense => union::build_extend_dense(array),
},
DataType::RunEndEncoded(_, _) => todo!(),
+ DataType::Extension(_) => unimplemented!("Extension not implemented"),
Review Comment:
This highlights the challenge with this approach, every kernel/function must now be updated to accommodate this new extension type. Not only does this stand a very high chance of regressing functionality, but also is IMO not in the spirit of extension types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Experiment with Extension as a DataType [arrow-rs]
Posted by "sdf-jkl (via GitHub) via github" <gi...@arrow.apache.org>.
sdf-jkl commented on PR #7398:
URL: https://github.com/apache/arrow-rs/pull/7398#issuecomment-6580841643
Something like this would be so nice to have! It's very inconvenient that we need a `Field` every time we want to pass an ext type :weary:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org