You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/07/26 10:44:19 UTC
[PR] Merge string-view2 branch to main [datafusion]
alamb opened a new pull request, #11667:
URL: https://github.com/apache/datafusion/pull/11667
Draft until arrow `52.2.0` is released to crates.io (expected Sat July 27)
## Which issue does this PR close?
Part of https://github.com/apache/datafusion/issues/10918
## Rationale for this change
We have been integrating a set of StringView changes on the [`string-view2`](https://github.com/apache/datafusion/tree/string-view2) branch as they rely on un-released code in arrow-rs . Once those changes are released and DataFusion uses them we can bring this code directly to main.
## What changes are included in this PR?
* Add a config to force using string view in benchmark (#11514) - @XiangpengHao
* Add String view helper functions (#11517) - @XiangpengHao
* Add ArrowBytesViewMap and ArrowBytesViewSet (#11515) - @XiangpengHao
* Enable GroupValueBytesView for aggregation with StringView types (#… - @XiangpengHao
* Initial support for regex_replace on StringViewArray (#11556) - @XiangpengHao
* Add support for Utf8View for date/temporal codepaths (#11518) - @a10y
* GC StringViewArray in CoalesceBatchesStream (#11587) - @XiangpengHao
## Are these changes tested?
CI
## Are there any user-facing changes?
When StringView is enabled, benchmarks run significantly faster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Merge `string-view2` branch: reading from parquet up to 2x faster for some ClickBench queries (not on by default) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #11667:
URL: https://github.com/apache/datafusion/pull/11667
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Merge string-view2 branch to main [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11667:
URL: https://github.com/apache/datafusion/pull/11667#issuecomment-2252864547
CI appears to be failing due to https://github.com/apache/datafusion/issues/11671
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Merge string-view2 branch to main [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11667:
URL: https://github.com/apache/datafusion/pull/11667#issuecomment-2254118604
Update here is that we are on track to release arrow `52.2.0` to crates.io tomorrow Saturday July 28. (thank you to @waynexia @viirya @wjones127 for verifying / voting 🙏 ).
Then we'll need a PR to update datafusion to arrow 52.2.0 (which should be straightforward)
Then I will change this PR to ready to review and we should be able to merge it into DataFusion main
🤞
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Merge `string-view2` branch: reading from parquet up to 2x faster for some ClickBench queries (not on by default) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11667:
URL: https://github.com/apache/datafusion/pull/11667#issuecomment-2256934236
🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Merge `string-view2` branch: reading from parquet up to 2x faster for some ClickBench queries (not on by default) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11667:
URL: https://github.com/apache/datafusion/pull/11667#issuecomment-2256933192
Thanks @Dandandan -- so exciting!
I plan to work with @XiangpengHao and figure out what we need to do to get this feature on by default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Merge `string-view` branch: reading from parquet up to 2x faster for some ClickBench queries (not on by default) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11667:
URL: https://github.com/apache/datafusion/pull/11667#issuecomment-2255826417
This PR is now ready for review (mostly I am hoping another committer will approve/merge it as I can't approve my own PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org