You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/06/14 18:51:32 UTC
[I] [Epic] Implement support for `StringView` in DataFusion [datafusion]
alamb opened a new issue, #10918:
URL: https://github.com/apache/datafusion/issues/10918
### Is your feature request related to a problem or challenge?
_No response_
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] Implement support for `StringView` in DataFusion [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10918:
URL: https://github.com/apache/datafusion/issues/10918#issuecomment-2293359212
In case anyone wants an overview of adding StringView to DataFusion, here is a presentation and slides from @XiangpengHao
Video: https://www.youtube.com/watch?v=RVLshX6fbds
Slides: https://drive.google.com/file/d/1Qqd8V6cfS9rSQ_-JrinasQJwI79qlUEV/view?usp=drive_link
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] Implement support for `StringView` in DataFusion [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10918:
URL: https://github.com/apache/datafusion/issues/10918#issuecomment-2231728026
Update: we merged the basic support for StringView in https://github.com/apache/datafusion/pull/11402
I have created a branch to collect any changes that rely on the pre-release parquet/arrow `52.2.0` version here: https://github.com/apache/datafusion/tree/string-view2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] Implement support for `StringView` in DataFusion [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10918:
URL: https://github.com/apache/datafusion/issues/10918#issuecomment-2243212906
Update here is that we hope to have enough code merged / implemented to report benchmark results by the end of the week. Once arrow-rs 52.2.0 is released https://github.com/apache/arrow-rs/issues/5998 then we'll merge the contents of https://github.com/apache/datafusion/tree/string-view2 to main
Once we do that we'll evaluate what, if anything, is left before we can close this ticket
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] Implement support for `StringView` in DataFusion [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10918:
URL: https://github.com/apache/datafusion/issues/10918#issuecomment-2168625482
I think we should aim for a first "milestone" of showing improvements for some clickbench queries
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] Implement support for `StringView` in DataFusion [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10918:
URL: https://github.com/apache/datafusion/issues/10918#issuecomment-2261502279
Update here is that we have all the basic support in now but it is behind a feature flag. Let's declare this project a success for now and track the remaining work to enable this feature by default in a separate issue: https://github.com/apache/datafusion/issues/11752
Thanks to everyone who has helped so far. We are close ™️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] Implement support for `StringView` in DataFusion [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10918:
URL: https://github.com/apache/datafusion/issues/10918#issuecomment-2255827552
Update: the next major PR in this sequence is ready to merge: https://github.com/apache/datafusion/pull/11667
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] Implement support for `StringView` in DataFusion [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10918:
URL: https://github.com/apache/datafusion/issues/10918#issuecomment-2218678888
Now that we have upgraded to arrow 52.1.0, I think we could merge the `string-view` branch to main. I'll try and make a PR if no one beats me to it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] Implement support for `StringView` in DataFusion [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10918:
URL: https://github.com/apache/datafusion/issues/10918#issuecomment-2228800548
An update here on the plan from @XiangpengHao:
* He has local changes (that also require some additional features that will be released in arrow `52.2.0`) that show significant performance improvements for TPCH and ClickBench queries
* The hope is that these proposals are all up for review by the end of this week
* So by the time arrow `52.2.0` is released (early August 2024) we'll be able to add an option that makes DataFusion use StringView when reading from Parquet / filtering
* We also plan to write a blog post about this work / adventure
I for one am very excited
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] Implement support for `StringView` in DataFusion [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #10918: [Epic] Implement support for `StringView` in DataFusion
URL: https://github.com/apache/datafusion/issues/10918
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org