You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "pepijnve (via GitHub)" <gi...@apache.org> on 2026/06/20 16:21:19 UTC
[PR] Improve performance of binary and string concatenation operator [datafusion]
pepijnve opened a new pull request, #23057:
URL: https://github.com/apache/datafusion/pull/23057
## Which issue does this PR close?
- None yet
## Rationale for this change
During recent profiling work string concatenation proved to be a hotspot. Investigation of the current kernel implementation for string views showed that there was still some room for improvement.
- Preallocating the exact size of the required output buffers can avoid reallocations.
- By copying data directly to the final data buffer a memcpy from the temp buffer can be avoided.
Together this can result in ~30% improvement per the string concat benchmark
Note that this work is a port of https://github.com/apache/arrow-rs/pull/10161. Ideally the implementation from Arrow is used by DataFusion once the PR in that project is merged and released. Since DataFusion currently uses a custom kernel it seemed to make sense to temporarily port the proposed PR from Arrow.
## What changes are included in this PR?
- Rewrite the byte view concatenation kernels
## Are these changes tested?
Covered by existing tests
## Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve performance of binary and string concatenation operator [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #23057:
URL: https://github.com/apache/datafusion/pull/23057#issuecomment-4812540451
> And just as I pressed the button, I thought it might be useful to keep track of the fact that we should replace the custom DataFusion kernel with the arrow-rw one. I'll make an issue to track that.
Bonus points if you make the issue and then make a PR to add the refernce to the code as comment (so fugure coders can easily find the todo)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve performance of binary and string concatenation operator [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #23057:
URL: https://github.com/apache/datafusion/pull/23057#issuecomment-4811971474
> Marking this for consideration again since the arrow-rs PR has been merged. @alamb since you reviewed the arrow-rs work, wdyt? Is it worth temporarily porting this to DataFusion or do we wait for the next arrow-rs release?
I think the next arrow-release will come out before the next Datafusion release, so I don't think we need to add the same optimization in both places
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve performance of binary and string concatenation operator [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on PR #23057:
URL: https://github.com/apache/datafusion/pull/23057#issuecomment-4812564875
https://github.com/apache/arrow-rs/pull/10161
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve performance of binary and string concatenation operator [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on PR #23057:
URL: https://github.com/apache/datafusion/pull/23057#issuecomment-4812507276
And just as I pressed the button, I thought it might be useful to keep track of the fact that we should replace the custom DataFusion kernel with the arrow-rw one. I'll make an issue to track that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve performance of binary and string concatenation operator [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on PR #23057:
URL: https://github.com/apache/datafusion/pull/23057#issuecomment-4759674746
I'll mark this one as draft for now and add the TODO already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve performance of binary and string concatenation operator [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve closed pull request #23057: Improve performance of binary and string concatenation operator
URL: https://github.com/apache/datafusion/pull/23057
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve performance of binary and string concatenation operator [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23057:
URL: https://github.com/apache/datafusion/pull/23057#issuecomment-4759685747
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
error: `cargo metadata` exited with an error: Updating crates.io index
error: failed to get `whoami` as a dependency of package `tokio-postgres v0.7.18`
... which satisfies dependency `tokio-postgres = "^0.7.18"` (locked to 0.7.18) of package `datafusion-sqllogictest v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/sqllogictest)`
Caused by:
failed to load source for dependency `whoami`
Caused by:
unable to update registry `crates-io`
Caused by:
download of wh/oa/whoami failed
Caused by:
curl failed
Caused by:
[56] Failure when receiving data from the peer (Recv failure: Connection reset by peer)
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve performance of binary and string concatenation operator [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on PR #23057:
URL: https://github.com/apache/datafusion/pull/23057#issuecomment-4809963605
Marking this for consideration again since the arrow-rs PR has been merged. @alamb since you reviewed the arrow-rs work, wdyt? Is it worth temporarily porting this to DataFusion or do we wait for the next arrow-rs release?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve performance of binary and string concatenation operator [datafusion]
Posted by "pepijnve (via GitHub)" <gi...@apache.org>.
pepijnve commented on PR #23057:
URL: https://github.com/apache/datafusion/pull/23057#issuecomment-4812495190
Closing this one then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org