You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "mbutrovich (via GitHub)" <gi...@apache.org> on 2025/02/14 17:48:54 UTC
[I] Utf8View and BinaryView (i.e., StringView in Arrow, colloquially German-style strings) support [datafusion-comet]
mbutrovich opened a new issue, #1403:
URL: https://github.com/apache/datafusion-comet/issues/1403
### What is the problem the feature request solves?
With the experimental native scans built on DataFusion's ParquetExec and our update to DataFusion 45, we have the opportunity to start adding support for StringView. I have started scoping out this work and would like to start aggregating findings here.
### Describe the potential solution
Project-level:
- Bump arrow-java version. We're currently on 16.0.0. I believe the view types were added in 17.0.0. I tested bumping to 18.2.0 and so far it doesn't seem too painful.
Java-side:
- Add support for decoding `Utf8View` and `BinaryView` to `CometVector`. I prototyped this [here](https://github.com/mbutrovich/datafusion-comet/blob/7c4eede25ba6672befa9beeb6f8c3e95dba7cc75/common/src/main/java/org/apache/comet/vector/CometPlainVector.java#L161) and [here](https://github.com/mbutrovich/datafusion-comet/blob/7c4eede25ba6672befa9beeb6f8c3e95dba7cc75/common/src/main/java/org/apache/comet/vector/CometPlainVector.java#L201) for Utf8View and BinaryView, respectively.
Native-side:
- Enable StringViewArray by default in query execution and Parquet reader. [We're a recent enough DataFusion version that this is done already](https://github.com/apache/datafusion/pull/13101).
- planner.rs and serde.rs should generate Utf8View and BinaryView types when possible.
- Shuffle:
- Add support to hash_util.
- Add support to shuffle_writer (slot_size, etc.)
I'm sure there's more than this, and will continue adding as I find stuff broken in my proof-of-concept branch.
### Additional context
Related DataFusion blogs:
https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1/
https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-2/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Utf8View and BinaryView (i.e., StringView in Arrow, colloquially German-style strings) support [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #1403:
URL: https://github.com/apache/datafusion-comet/issues/1403#issuecomment-2967992880
I'll be picking this up again now that we dropped Java 8 support and bumped our Arrow Java version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org