You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "simonvandel (via GitHub)" <gi...@apache.org> on 2025/02/16 21:59:49 UTC
[PR] Speed up `chr` UDF (~4x faster) [datafusion]
simonvandel opened a new pull request, #14700:
URL: https://github.com/apache/datafusion/pull/14700
## Which issue does this PR close?
N/A
## Rationale for this change
It is wasteful to allocate temporary memory with intermediate `to_string` calls.
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
```
chr time: [3.7600 µs 3.7730 µs 3.7892 µs]
change: [-74.908% -74.553% -74.265%] (p = 0.00 < 0.05)
Performance has improved.
Found 12 outliers among 100 measurements (12.00%)
2 (2.00%) low mild
2 (2.00%) high mild
8 (8.00%) high severe
```
## What changes are included in this PR?
- Add a bench for `chr`
- Avoid intermediate String allocations by encoding directly to the string array
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
Relying on existing tests
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
Faster execution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Speed up `chr` UDF (~4x faster) [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #14700:
URL: https://github.com/apache/datafusion/pull/14700#discussion_r1957866816
##########
datafusion/functions/src/string/chr.rs:
##########
@@ -36,26 +36,40 @@ use datafusion_macros::user_doc;
pub fn chr(args: &[ArrayRef]) -> Result<ArrayRef> {
let integer_array = as_int64_array(&args[0])?;
- // first map is the iterator, second is for the `Option<_>`
- let result = integer_array
- .iter()
- .map(|integer: Option<i64>| {
- integer
- .map(|integer| {
- if integer == 0 {
- exec_err!("null character not permitted.")
- } else {
- match core::char::from_u32(integer as u32) {
- Some(integer) => Ok(integer.to_string()),
- None => {
- exec_err!("requested character too large for encoding.")
- }
+ let mut builder = GenericStringBuilder::<i32>::with_capacity(
+ integer_array.len(),
+ // This assumes the maximum length of a single character is 4 bytes,
+ // which might be overly pessimistic
+ integer_array.len() * 4,
Review Comment:
Maybe it can assume 1 or 2 bytes instead? This means max 4 (or 2) allocations per array and in common cases 1-2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Speed up `chr` UDF (~4x faster) [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #14700:
URL: https://github.com/apache/datafusion/pull/14700#discussion_r1957866816
##########
datafusion/functions/src/string/chr.rs:
##########
@@ -36,26 +36,40 @@ use datafusion_macros::user_doc;
pub fn chr(args: &[ArrayRef]) -> Result<ArrayRef> {
let integer_array = as_int64_array(&args[0])?;
- // first map is the iterator, second is for the `Option<_>`
- let result = integer_array
- .iter()
- .map(|integer: Option<i64>| {
- integer
- .map(|integer| {
- if integer == 0 {
- exec_err!("null character not permitted.")
- } else {
- match core::char::from_u32(integer as u32) {
- Some(integer) => Ok(integer.to_string()),
- None => {
- exec_err!("requested character too large for encoding.")
- }
+ let mut builder = GenericStringBuilder::<i32>::with_capacity(
+ integer_array.len(),
+ // This assumes the maximum length of a single character is 4 bytes,
+ // which might be overly pessimistic
+ integer_array.len() * 4,
Review Comment:
Maybe it can assume 1 or 2 bytes instead? This means max 4 allocations per array and in common cases 1-2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Speed up `chr` UDF (~4x faster) [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #14700:
URL: https://github.com/apache/datafusion/pull/14700#discussion_r1957866816
##########
datafusion/functions/src/string/chr.rs:
##########
@@ -36,26 +36,40 @@ use datafusion_macros::user_doc;
pub fn chr(args: &[ArrayRef]) -> Result<ArrayRef> {
let integer_array = as_int64_array(&args[0])?;
- // first map is the iterator, second is for the `Option<_>`
- let result = integer_array
- .iter()
- .map(|integer: Option<i64>| {
- integer
- .map(|integer| {
- if integer == 0 {
- exec_err!("null character not permitted.")
- } else {
- match core::char::from_u32(integer as u32) {
- Some(integer) => Ok(integer.to_string()),
- None => {
- exec_err!("requested character too large for encoding.")
- }
+ let mut builder = GenericStringBuilder::<i32>::with_capacity(
+ integer_array.len(),
+ // This assumes the maximum length of a single character is 4 bytes,
+ // which might be overly pessimistic
+ integer_array.len() * 4,
Review Comment:
Maybe it can assume 1 or 2 bytes instead? This means max 4 (or 2 when assuming 2 bytes) allocations per array and in common cases 1-2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Speed up `chr` UDF (~4x faster) [datafusion]
Posted by "simonvandel (via GitHub)" <gi...@apache.org>.
simonvandel commented on code in PR #14700:
URL: https://github.com/apache/datafusion/pull/14700#discussion_r1960597244
##########
datafusion/functions/src/string/chr.rs:
##########
@@ -36,26 +36,40 @@ use datafusion_macros::user_doc;
pub fn chr(args: &[ArrayRef]) -> Result<ArrayRef> {
let integer_array = as_int64_array(&args[0])?;
- // first map is the iterator, second is for the `Option<_>`
- let result = integer_array
- .iter()
- .map(|integer: Option<i64>| {
- integer
- .map(|integer| {
- if integer == 0 {
- exec_err!("null character not permitted.")
- } else {
- match core::char::from_u32(integer as u32) {
- Some(integer) => Ok(integer.to_string()),
- None => {
- exec_err!("requested character too large for encoding.")
- }
+ let mut builder = GenericStringBuilder::<i32>::with_capacity(
+ integer_array.len(),
+ // This assumes the maximum length of a single character is 4 bytes,
+ // which might be overly pessimistic
+ integer_array.len() * 4,
Review Comment:
Makes sense, changed in https://github.com/apache/datafusion/pull/14700/commits/9d158d4364d53321a80f4604cc24b9896dc42c5b
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Speed up `chr` UDF (~4x faster) [datafusion]
Posted by "ajita-asthana (via GitHub)" <gi...@apache.org>.
ajita-asthana commented on PR #14700:
URL: https://github.com/apache/datafusion/pull/14700#issuecomment-3033951891
how long does it take to compile the benchs library on running `cargo bench`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Speed up `chr` UDF (~4x faster) [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #14700:
URL: https://github.com/apache/datafusion/pull/14700#issuecomment-2666587347
LGTM, thx!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Speed up `chr` UDF (~4x faster) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead merged PR #14700:
URL: https://github.com/apache/datafusion/pull/14700
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Speed up `chr` UDF (~4x faster) [datafusion]
Posted by "waynexia (via GitHub)" <gi...@apache.org>.
waynexia commented on PR #14700:
URL: https://github.com/apache/datafusion/pull/14700#issuecomment-3104237037
> how long does it take to compile the benches library on running `cargo bench`
for me it takes ~1min to compile `cargo bench --bench chr`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org