You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "coderfender (via GitHub)" <gi...@apache.org> on 2025/12/31 06:11:21 UTC
[PR] perf: Improve string to date perf [datafusion-comet]
coderfender opened a new pull request, #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #2981
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## How are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3712347520
Sure @andygrove . If my understanding is right, we just want to be _inspired_ by and implement a spark-like parser right ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3706870168
Results :
```
================================================================================================
Running benchmark cast operation from : StringType to : IntegerType
================================================================================================
OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Cast function to : IntegerType , ansi mode enabled : false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------
Spark 668 683 14 15.7 63.8 1.0X
Comet (Scan) 708 739 30 14.8 67.5 0.9X
Comet (Scan + Exec) 816 829 18 12.9 77.8 0.8X
================================================================================================
Running benchmark cast operation from : StringType to : ShortType
================================================================================================
OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Cast function to : ShortType , ansi mode enabled : false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
----------------------------------------------------------------------------------------------------------------------------------------
Spark 689 692 2 15.2 65.7 1.0X
Comet (Scan) 639 658 21 16.4 60.9 1.1X
Comet (Scan + Exec) 793 798 4 13.2 75.6 0.9X
================================================================================================
Running benchmark cast operation from : StringType to : ByteType
================================================================================================
OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 16.0
Apple M2 Max
Cast function to : ByteType , ansi mode enabled : false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------------
Spark 659 694 39 15.9 62.9 1.0X
Comet (Scan) 639 655 12 16.4 60.9 1.0X
Comet (Scan + Exec) 795 810 20 13.2 75.8 0.8X
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3712953799
```
group feature feature_separate_eval_mode main
----- ------- -------------------------- ----
cast_string_to_int/cast_string_to_i16 1.14 20.0±0.17µs ? ?/sec 1.00 17.6±0.21µs ? ?/sec 1.55 27.3±0.21µs ? ?/sec
cast_string_to_int/cast_string_to_i32 1.14 19.9±0.34µs ? ?/sec 1.00 17.4±0.51µs ? ?/sec 1.58 27.5±0.81µs ? ?/sec
cast_string_to_int/cast_string_to_i64 1.07 26.3±2.55µs ? ?/sec 1.00 24.5±1.44µs ? ?/sec 1.34 32.8±0.26µs ? ?/sec
cast_string_to_int/cast_string_to_i8 1.15 20.3±1.23µs ? ?/sec 1.00 17.6±0.28µs ? ?/sec 1.55 27.4±1.00µs ? ?/sec
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3707363706
## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/3017?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 9.99%. Comparing base ([`f09f8af`](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`2b331f0`](https://app.codecov.io/gh/apache/datafusion-comet/commit/2b331f01c023f45e0577c59b9dbb56fbfdb34583?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 814 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #3017 +/- ##
============================================
- Coverage 56.12% 9.99% -46.14%
+ Complexity 976 234 -742
============================================
Files 119 167 +48
Lines 11743 15482 +3739
Branches 2251 2543 +292
============================================
- Hits 6591 1547 -5044
- Misses 4012 13758 +9746
+ Partials 1140 177 -963
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/3017?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3711785788
In other notes, I was also experimenting in implementing a two pass fast algorithm using a switch fallthrough (similar to what my friend wrote here) but the implementation became super complicated with diminishing returns :
https://github.com/CameronHosking/fast-atoi/blob/master/fast_atoi.h
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to date perf [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3701531698
Initial benchmarks showed improvement to match with Spark's performance. I will continue to profile the code to see if we can squeeze in additional optimization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3706874748
Proceeding with some rather unsafe options to see if we can squeeze in further optimizations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3711333041
Sure @andygrove . Let me get the benchmark file from stash and push a commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3712412258
Removed branching below are the latest bench numbers :
main - main branch
feature - feature branch (before separate implementations for each eval mode)
feature_remove_branch - after separate eval modes
```
group feature feature_remove_branching main
----- ------- ------------------------ ----
cast_string_to_int/cast_string_to_i16 1.12 20.0±0.17µs ? ?/sec 1.00 17.8±1.92µs ? ?/sec 1.53 27.3±0.21µs ? ?/sec
cast_string_to_int/cast_string_to_i32 1.00 19.9±0.34µs ? ?/sec 1.54 30.6±46.63µs ? ?/sec 1.38 27.5±0.81µs ? ?/sec
cast_string_to_int/cast_string_to_i64 1.08 26.3±2.55µs ? ?/sec 1.00 24.2±0.35µs ? ?/sec 1.35 32.8±0.26µs ? ?/sec
cast_string_to_int/cast_string_to_i8 1.16 20.3±1.23µs ? ?/sec 1.00 17.5±0.21µs ? ?/sec 1.57 27.4±1.00µs ? ?/sec
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3711389280
@andygrove , sure
Here are the benchmarks compared through `critcmp`
```
group feature main
----- ------- ----
cast_string_to_int/cast_string_to_i16 1.00 21.3±1.92µs ? ?/sec 1.27 27.2±2.95µs ? ?/sec
cast_string_to_int/cast_string_to_i32 1.00 19.9±0.40µs ? ?/sec 1.34 26.8±1.08µs ? ?/sec
cast_string_to_int/cast_string_to_i64 1.00 25.6±0.85µs ? ?/sec 1.24 31.8±0.35µs ? ?/sec
cast_string_to_int/cast_string_to_i8 1.00 19.8±0.36µs ? ?/sec 1.34 26.5±0.43µs ? ?/sec
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3711326648
Thanks @coderfender. I think it would be useful to add a criterion benchmark as well, so we can more easily measure the improvement compared to the main branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3709013471
```
| Type | Before (main) | After (feature) | Improvement |
|------|---------------|-----------------|-------------|
| i8 | 26.5 µs | 19.8 µs | 1.34x (34%) |
| i16 | 27.2 µs | 21.3 µs | 1.27x (27%) |
| i32 | 26.8 µs | 19.9 µs | 1.34x (34%) |
| i64 | 31.8 µs | 25.6 µs | 1.24x (24%) |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve performance of CAST from string to int [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3715931380
Sure thank you very much @andygrove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3711279788
Tried a bunch of unsafe , low level (SIMD) ops but the gains were diminishing and perhaps would maintenance difficult.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3712955271
@andygrove , I removed redundant processing of decimals (in try and ansi eval modes) and that improved the benchmarks as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#discussion_r2658673305
##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -1965,33 +1975,41 @@ fn do_cast_string_to_int<
type_name: &str,
min_value: T,
) -> SparkResult<Option<T>> {
- let trimmed_str = str.trim();
- if trimmed_str.is_empty() {
+ let bytes = str.as_bytes();
+ let mut start = 0;
+ let mut end = bytes.len();
+
+ while start < end && bytes[start].is_ascii_whitespace() {
+ start += 1;
+ }
+ while end > start && bytes[end - 1].is_ascii_whitespace() {
+ end -= 1;
+ }
+
+ if start == end {
return none_or_err(eval_mode, type_name, str);
}
+ let trimmed_str = &str[start..end];
let len = trimmed_str.len();
+ let trimmed_bytes = trimmed_str.as_bytes();
let mut result: T = T::zero();
- let mut negative = false;
+ let mut idx = 0;
+ let first_char = trimmed_bytes[0];
+ let negative = first_char == b'-';
+ if negative || first_char == b'+' {
+ idx = 1;
+ if len == 1 {
+ return none_or_err(eval_mode, type_name, str);
+ }
+ }
Review Comment:
Same (removed unwanted `If` branching)
##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -389,13 +389,23 @@ macro_rules! cast_utf8_to_int {
($array:expr, $eval_mode:expr, $array_type:ty, $cast_method:ident) => {{
let len = $array.len();
let mut cast_array = PrimitiveArray::<$array_type>::builder(len);
- for i in 0..len {
- if $array.is_null(i) {
- cast_array.append_null()
- } else if let Some(cast_value) = $cast_method($array.value(i), $eval_mode)? {
- cast_array.append_value(cast_value);
- } else {
- cast_array.append_null()
+ if $array.null_count() == 0 {
+ for i in 0..len {
+ if let Some(cast_value) = $cast_method($array.value(i), $eval_mode)? {
+ cast_array.append_value(cast_value);
+ } else {
+ cast_array.append_null()
+ }
+ }
+ } else {
+ for i in 0..len {
+ if $array.is_null(i) {
+ cast_array.append_null()
+ } else if let Some(cast_value) = $cast_method($array.value(i), $eval_mode)? {
+ cast_array.append_value(cast_value);
+ } else {
+ cast_array.append_null()
+ }
Review Comment:
made null check conditional to remove unwanted branching
##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -1965,33 +1975,41 @@ fn do_cast_string_to_int<
type_name: &str,
min_value: T,
) -> SparkResult<Option<T>> {
- let trimmed_str = str.trim();
- if trimmed_str.is_empty() {
+ let bytes = str.as_bytes();
+ let mut start = 0;
+ let mut end = bytes.len();
+
+ while start < end && bytes[start].is_ascii_whitespace() {
+ start += 1;
+ }
+ while end > start && bytes[end - 1].is_ascii_whitespace() {
+ end -= 1;
+ }
+
+ if start == end {
return none_or_err(eval_mode, type_name, str);
}
+ let trimmed_str = &str[start..end];
let len = trimmed_str.len();
+ let trimmed_bytes = trimmed_str.as_bytes();
let mut result: T = T::zero();
- let mut negative = false;
+ let mut idx = 0;
+ let first_char = trimmed_bytes[0];
+ let negative = first_char == b'-';
+ if negative || first_char == b'+' {
+ idx = 1;
+ if len == 1 {
+ return none_or_err(eval_mode, type_name, str);
+ }
+ }
+
let radix = T::from(10);
let stop_value = min_value / radix;
let mut parse_sign_and_digits = true;
- for (i, ch) in trimmed_str.char_indices() {
+ for &ch in &trimmed_bytes[idx..] {
Review Comment:
Cleaner and faster approach to access the chars directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#discussion_r2662539689
##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -1957,41 +1967,46 @@ fn cast_string_to_int_with_range_check(
/// Equivalent to
/// - org.apache.spark.unsafe.types.UTF8String.toInt(IntWrapper intWrapper, boolean allowDecimal)
/// - org.apache.spark.unsafe.types.UTF8String.toLong(LongWrapper longWrapper, boolean allowDecimal)
-fn do_cast_string_to_int<
- T: Num + PartialOrd + Integer + CheckedSub + CheckedNeg + From<i32> + Copy,
->(
+fn do_cast_string_to_int<T: Integer + CheckedSub + CheckedNeg + From<u8> + Copy>(
str: &str,
eval_mode: EvalMode,
type_name: &str,
min_value: T,
) -> SparkResult<Option<T>> {
- let trimmed_str = str.trim();
- if trimmed_str.is_empty() {
+ let bytes = str.as_bytes();
+ let mut start = 0;
+ let mut end = bytes.len();
+
+ while start < end && bytes[start].is_ascii_whitespace() {
+ start += 1;
+ }
+ while end > start && bytes[end - 1].is_ascii_whitespace() {
+ end -= 1;
+ }
+
+ if start == end {
return none_or_err(eval_mode, type_name, str);
}
- let len = trimmed_str.len();
+ let trimmed_bytes = &bytes[start..end];
+ let len = trimmed_bytes.len();
let mut result: T = T::zero();
- let mut negative = false;
- let radix = T::from(10);
+ let mut idx = 0;
+ let first_char = trimmed_bytes[0];
+ let negative = first_char == b'-';
+ if negative || first_char == b'+' {
+ idx = 1;
+ if len == 1 {
+ return none_or_err(eval_mode, type_name, str);
+ }
+ }
+
+ let radix = T::from(10_u8);
let stop_value = min_value / radix;
let mut parse_sign_and_digits = true;
- for (i, ch) in trimmed_str.char_indices() {
+ for &ch in &trimmed_bytes[idx..] {
if parse_sign_and_digits {
- if i == 0 {
- negative = ch == '-';
- let positive = ch == '+';
- if negative || positive {
- if i + 1 == len {
- // input string is just "+" or "-"
- return none_or_err(eval_mode, type_name, str);
- }
- // consume this char
- continue;
- }
- }
-
- if ch == '.' {
+ if ch == b'.' {
if eval_mode == EvalMode::Legacy {
// truncate decimal in legacy mode
parse_sign_and_digits = false;
Review Comment:
Great suggestion . Let me go ahead and make separate paths for each eval mode to prevent hot loops and update this thread with benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3712413622
TODO : remove common methods to a utility function to follow DRY principle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#discussion_r2665950351
##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -1954,100 +1964,247 @@ fn cast_string_to_int_with_range_check(
}
}
+// Returns (start, end) indices after trimming whitespace
+fn trim_whitespace(bytes: &[u8]) -> (usize, usize) {
+ let mut start = 0;
+ let mut end = bytes.len();
+
+ while start < end && bytes[start].is_ascii_whitespace() {
+ start += 1;
+ }
+ while end > start && bytes[end - 1].is_ascii_whitespace() {
+ end -= 1;
+ }
+
+ (start, end)
+}
+
+// Parses sign and returns (is_negative, start_idx after sign)
+// Returns None if invalid (e.g., just "+" or "-")
+fn parse_sign(trimmed_bytes: &[u8]) -> Option<(bool, usize)> {
+ let len = trimmed_bytes.len();
+ if len == 0 {
+ return None;
+ }
+
+ let first_char = trimmed_bytes[0];
+ let negative = first_char == b'-';
+
+ if negative || first_char == b'+' {
+ if len == 1 {
+ return None;
+ }
+ Some((negative, 1))
+ } else {
+ Some((false, 0))
+ }
+}
+
/// Equivalent to
/// - org.apache.spark.unsafe.types.UTF8String.toInt(IntWrapper intWrapper, boolean allowDecimal)
/// - org.apache.spark.unsafe.types.UTF8String.toLong(LongWrapper longWrapper, boolean allowDecimal)
-fn do_cast_string_to_int<
- T: Num + PartialOrd + Integer + CheckedSub + CheckedNeg + From<i32> + Copy,
->(
+fn do_parse_string_to_int_legacy<T: Integer + CheckedSub + CheckedNeg + From<u8> + Copy>(
str: &str,
- eval_mode: EvalMode,
- type_name: &str,
min_value: T,
) -> SparkResult<Option<T>> {
- let trimmed_str = str.trim();
- if trimmed_str.is_empty() {
- return none_or_err(eval_mode, type_name, str);
+ let bytes = str.as_bytes();
+ let (start, end) = trim_whitespace(bytes);
+
+ if start == end {
+ return Ok(None);
}
- let len = trimmed_str.len();
+ let trimmed_bytes = &bytes[start..end];
+
+ let (negative, idx) = match parse_sign(trimmed_bytes) {
+ Some(result) => result,
+ None => return Ok(None),
+ };
+
let mut result: T = T::zero();
- let mut negative = false;
- let radix = T::from(10);
+
+ let radix = T::from(10_u8);
let stop_value = min_value / radix;
let mut parse_sign_and_digits = true;
- for (i, ch) in trimmed_str.char_indices() {
+ for &ch in &trimmed_bytes[idx..] {
if parse_sign_and_digits {
- if i == 0 {
- negative = ch == '-';
- let positive = ch == '+';
- if negative || positive {
- if i + 1 == len {
- // input string is just "+" or "-"
- return none_or_err(eval_mode, type_name, str);
- }
- // consume this char
- continue;
- }
+ if ch == b'.' {
+ // truncate decimal in legacy mode
+ parse_sign_and_digits = false;
+ continue;
}
- if ch == '.' {
- if eval_mode == EvalMode::Legacy {
- // truncate decimal in legacy mode
- parse_sign_and_digits = false;
- continue;
- } else {
- return none_or_err(eval_mode, type_name, str);
- }
+ if !ch.is_ascii_digit() {
+ return Ok(None);
}
- let digit = if ch.is_ascii_digit() {
- (ch as u32) - ('0' as u32)
- } else {
- return none_or_err(eval_mode, type_name, str);
- };
+ let digit: T = T::from(ch - b'0');
- // We are going to process the new digit and accumulate the result. However, before
- // doing this, if the result is already smaller than the
- // stopValue(Integer.MIN_VALUE / radix), then result * 10 will definitely be
- // smaller than minValue, and we can stop
if result < stop_value {
- return none_or_err(eval_mode, type_name, str);
+ return Ok(None);
}
-
- // Since the previous result is greater than or equal to stopValue(Integer.MIN_VALUE /
- // radix), we can just use `result > 0` to check overflow. If result
- // overflows, we should stop
let v = result * radix;
- let digit = (digit as i32).into();
match v.checked_sub(&digit) {
Some(x) if x <= T::zero() => result = x,
_ => {
- return none_or_err(eval_mode, type_name, str);
+ return Ok(None);
}
}
} else {
- // make sure fractional digits are valid digits but ignore them
+ // in legacy mode we still process chars after the dot and make sure the chars are digits
if !ch.is_ascii_digit() {
- return none_or_err(eval_mode, type_name, str);
+ return Ok(None);
+ }
+ }
+ }
+
+ if !negative {
+ if let Some(neg) = result.checked_neg() {
+ if neg < T::zero() {
+ return Ok(None);
+ }
+ result = neg;
+ } else {
+ return Ok(None);
+ }
+ }
+
+ Ok(Some(result))
+}
+
+fn do_parse_string_to_int_ansi<T: Integer + CheckedSub + CheckedNeg + From<u8> + Copy>(
+ str: &str,
+ type_name: &str,
+ min_value: T,
+) -> SparkResult<Option<T>> {
+ let bytes = str.as_bytes();
+ let (start, end) = trim_whitespace(bytes);
+
+ if start == end {
+ return Err(invalid_value(str, "STRING", type_name));
+ }
+ let trimmed_bytes = &bytes[start..end];
+
+ let (negative, idx) = match parse_sign(trimmed_bytes) {
+ Some(result) => result,
+ None => return Err(invalid_value(str, "STRING", type_name)),
+ };
+
+ let mut result: T = T::zero();
+
+ let radix = T::from(10_u8);
+ let stop_value = min_value / radix;
+
+ for &ch in &trimmed_bytes[idx..] {
+ if ch == b'.' {
+ return Err(invalid_value(str, "STRING", type_name));
+ }
+
+ if !ch.is_ascii_digit() {
+ return Err(invalid_value(str, "STRING", type_name));
+ }
+
+ let digit: T = T::from(ch - b'0');
+
+ if result < stop_value {
+ return Err(invalid_value(str, "STRING", type_name));
+ }
+ let v = result * radix;
+ match v.checked_sub(&digit) {
+ Some(x) if x <= T::zero() => result = x,
+ _ => {
+ return Err(invalid_value(str, "STRING", type_name));
+ }
+ }
+ }
+
+ if !negative {
+ if let Some(neg) = result.checked_neg() {
+ if neg < T::zero() {
+ return Err(invalid_value(str, "STRING", type_name));
+ }
+ result = neg;
+ } else {
+ return Err(invalid_value(str, "STRING", type_name));
+ }
+ }
+
+ Ok(Some(result))
+}
+
+fn do_parse_string_to_int_try<T: Integer + CheckedSub + CheckedNeg + From<u8> + Copy>(
+ str: &str,
+ min_value: T,
+) -> SparkResult<Option<T>> {
+ let bytes = str.as_bytes();
+ let (start, end) = trim_whitespace(bytes);
+
+ if start == end {
+ return Ok(None);
+ }
+ let trimmed_bytes = &bytes[start..end];
+
+ let (negative, idx) = match parse_sign(trimmed_bytes) {
+ Some(result) => result,
+ None => return Ok(None),
+ };
+
+ let mut result: T = T::zero();
+
+ let radix = T::from(10_u8);
+ let stop_value = min_value / radix;
+
+ // we don't have to go beyond decimal point in try eval mode - early return NULL
+ for &ch in &trimmed_bytes[idx..] {
+ if ch == b'.' {
+ return Ok(None);
+ }
+
+ if !ch.is_ascii_digit() {
+ return Ok(None);
+ }
+
+ let digit: T = T::from(ch - b'0');
+
+ if result < stop_value {
+ return Ok(None);
+ }
+ let v = result * radix;
+ match v.checked_sub(&digit) {
+ Some(x) if x <= T::zero() => result = x,
+ _ => {
+ return Ok(None);
}
}
}
if !negative {
if let Some(neg) = result.checked_neg() {
if neg < T::zero() {
- return none_or_err(eval_mode, type_name, str);
+ return Ok(None);
}
result = neg;
} else {
- return none_or_err(eval_mode, type_name, str);
+ return Ok(None);
}
}
Ok(Some(result))
}
+fn do_cast_string_to_int<T: Integer + CheckedSub + CheckedNeg + From<u8> + Copy>(
+ str: &str,
+ eval_mode: EvalMode,
+ type_name: &str,
+ min_value: T,
+) -> SparkResult<Option<T>> {
+ match eval_mode {
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve performance of CAST from string to int [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve performance of CAST from string to int [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#discussion_r2665972048
##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -1957,41 +1967,46 @@ fn cast_string_to_int_with_range_check(
/// Equivalent to
/// - org.apache.spark.unsafe.types.UTF8String.toInt(IntWrapper intWrapper, boolean allowDecimal)
/// - org.apache.spark.unsafe.types.UTF8String.toLong(LongWrapper longWrapper, boolean allowDecimal)
-fn do_cast_string_to_int<
- T: Num + PartialOrd + Integer + CheckedSub + CheckedNeg + From<i32> + Copy,
->(
+fn do_cast_string_to_int<T: Integer + CheckedSub + CheckedNeg + From<u8> + Copy>(
str: &str,
eval_mode: EvalMode,
type_name: &str,
min_value: T,
) -> SparkResult<Option<T>> {
- let trimmed_str = str.trim();
- if trimmed_str.is_empty() {
+ let bytes = str.as_bytes();
+ let mut start = 0;
+ let mut end = bytes.len();
+
+ while start < end && bytes[start].is_ascii_whitespace() {
+ start += 1;
+ }
+ while end > start && bytes[end - 1].is_ascii_whitespace() {
+ end -= 1;
+ }
Review Comment:
I will look at using `trim_ascii` in a separate PR along with some other minor changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#discussion_r2662535704
##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -1957,41 +1967,46 @@ fn cast_string_to_int_with_range_check(
/// Equivalent to
/// - org.apache.spark.unsafe.types.UTF8String.toInt(IntWrapper intWrapper, boolean allowDecimal)
/// - org.apache.spark.unsafe.types.UTF8String.toLong(LongWrapper longWrapper, boolean allowDecimal)
-fn do_cast_string_to_int<
- T: Num + PartialOrd + Integer + CheckedSub + CheckedNeg + From<i32> + Copy,
->(
+fn do_cast_string_to_int<T: Integer + CheckedSub + CheckedNeg + From<u8> + Copy>(
str: &str,
eval_mode: EvalMode,
type_name: &str,
min_value: T,
) -> SparkResult<Option<T>> {
- let trimmed_str = str.trim();
- if trimmed_str.is_empty() {
+ let bytes = str.as_bytes();
+ let mut start = 0;
+ let mut end = bytes.len();
+
+ while start < end && bytes[start].is_ascii_whitespace() {
+ start += 1;
+ }
+ while end > start && bytes[end - 1].is_ascii_whitespace() {
+ end -= 1;
+ }
+
+ if start == end {
return none_or_err(eval_mode, type_name, str);
}
- let len = trimmed_str.len();
+ let trimmed_bytes = &bytes[start..end];
+ let len = trimmed_bytes.len();
let mut result: T = T::zero();
- let mut negative = false;
- let radix = T::from(10);
+ let mut idx = 0;
+ let first_char = trimmed_bytes[0];
+ let negative = first_char == b'-';
+ if negative || first_char == b'+' {
+ idx = 1;
+ if len == 1 {
+ return none_or_err(eval_mode, type_name, str);
+ }
+ }
+
+ let radix = T::from(10_u8);
let stop_value = min_value / radix;
let mut parse_sign_and_digits = true;
- for (i, ch) in trimmed_str.char_indices() {
+ for &ch in &trimmed_bytes[idx..] {
if parse_sign_and_digits {
- if i == 0 {
- negative = ch == '-';
- let positive = ch == '+';
- if negative || positive {
- if i + 1 == len {
- // input string is just "+" or "-"
- return none_or_err(eval_mode, type_name, str);
- }
- // consume this char
- continue;
- }
- }
-
- if ch == '.' {
+ if ch == b'.' {
if eval_mode == EvalMode::Legacy {
// truncate decimal in legacy mode
parse_sign_and_digits = false;
Review Comment:
The `eval_mode` does not change for different rows. It would likely be more performant to have separate implementations for legacy vs other modes to avoid the conditional in the hot loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3715630175
Thank you @andygrove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#discussion_r2662544764
##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -1957,41 +1967,46 @@ fn cast_string_to_int_with_range_check(
/// Equivalent to
/// - org.apache.spark.unsafe.types.UTF8String.toInt(IntWrapper intWrapper, boolean allowDecimal)
/// - org.apache.spark.unsafe.types.UTF8String.toLong(LongWrapper longWrapper, boolean allowDecimal)
-fn do_cast_string_to_int<
- T: Num + PartialOrd + Integer + CheckedSub + CheckedNeg + From<i32> + Copy,
->(
+fn do_cast_string_to_int<T: Integer + CheckedSub + CheckedNeg + From<u8> + Copy>(
str: &str,
eval_mode: EvalMode,
type_name: &str,
min_value: T,
) -> SparkResult<Option<T>> {
- let trimmed_str = str.trim();
- if trimmed_str.is_empty() {
+ let bytes = str.as_bytes();
+ let mut start = 0;
+ let mut end = bytes.len();
+
+ while start < end && bytes[start].is_ascii_whitespace() {
+ start += 1;
+ }
+ while end > start && bytes[end - 1].is_ascii_whitespace() {
+ end -= 1;
+ }
Review Comment:
I was actually wrong in an earlier review when I suggested stopping using `trim`. `trim` does just return a slice on a `&str`, not a new `String`.
It may be worth considering using `trim_ascii` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3711805196
> In other notes, I was also experimenting in implementing a two pass fast algorithm using a switch fallthrough but the implementation became super complicated with diminishing returns largely due to rust's inability to not have a fallthrough switch statement : https://github.com/CameronHosking/fast-atoi/blob/master/fast_atoi.h
We could also look at https://crates.io/crates/atoi_simd (as a separate PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#discussion_r2662319466
##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -1957,41 +1967,46 @@ fn cast_string_to_int_with_range_check(
/// Equivalent to
/// - org.apache.spark.unsafe.types.UTF8String.toInt(IntWrapper intWrapper, boolean allowDecimal)
/// - org.apache.spark.unsafe.types.UTF8String.toLong(LongWrapper longWrapper, boolean allowDecimal)
-fn do_cast_string_to_int<
- T: Num + PartialOrd + Integer + CheckedSub + CheckedNeg + From<i32> + Copy,
->(
+fn do_cast_string_to_int<T: Integer + CheckedSub + CheckedNeg + From<u8> + Copy>(
str: &str,
eval_mode: EvalMode,
type_name: &str,
min_value: T,
) -> SparkResult<Option<T>> {
- let trimmed_str = str.trim();
- if trimmed_str.is_empty() {
+ let bytes = str.as_bytes();
+ let mut start = 0;
+ let mut end = bytes.len();
+
+ while start < end && bytes[start].is_ascii_whitespace() {
+ start += 1;
+ }
+ while end > start && bytes[end - 1].is_ascii_whitespace() {
+ end -= 1;
+ }
Review Comment:
not creating a new string through `trim` function and just looping through whitespaces
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Improve string to int perf [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #3017:
URL: https://github.com/apache/datafusion-comet/pull/3017#issuecomment-3715626677
These are nice speedups. Thanks @coderfender. I will review again later today.
```
$ cargo bench --bench cast_from_string -- --baseline main
Finished `bench` profile [optimized + debuginfo] target(s) in 0.17s
Running benches/cast_from_string.rs (target/release/deps/cast_from_string-9c9caa2565779a11)
Gnuplot not found, using plotters backend
cast_string_to_int/cast_string_to_i8
time: [10.269 µs 10.308 µs 10.356 µs]
change: [−44.807% −44.528% −44.214%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) high mild
1 (1.00%) high severe
cast_string_to_int/cast_string_to_i16
time: [10.260 µs 10.295 µs 10.335 µs]
change: [−44.683% −44.444% −44.202%] (p = 0.00 < 0.05)
Performance has improved.
Found 13 outliers among 100 measurements (13.00%)
9 (9.00%) high mild
4 (4.00%) high severe
cast_string_to_int/cast_string_to_i32
time: [10.348 µs 10.380 µs 10.417 µs]
change: [−43.119% −42.940% −42.761%] (p = 0.00 < 0.05)
Performance has improved.
Found 14 outliers among 100 measurements (14.00%)
11 (11.00%) high mild
3 (3.00%) high severe
cast_string_to_int/cast_string_to_i64
time: [14.733 µs 14.773 µs 14.824 µs]
change: [−39.389% −38.514% −37.613%] (p = 0.00 < 0.05)
Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
4 (4.00%) high mild
2 (2.00%) high severe
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org