You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "coderfender (via GitHub)" <gi...@apache.org> on 2025/08/09 18:32:47 UTC
[PR] implement_comet_native_lpad_expr [datafusion-comet]
coderfender opened a new pull request, #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102
## Which issue does this PR close?
<!--
Implements #2101
-->
Closes #.
## Rationale for this change
Moving towards goal to support more native operations and not fallback to spark .
## What changes are included in this PR?
## How are these changes tested?
Unit tests in comet expression suite
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#issuecomment-3358605422
I will raise a doc change PR shortly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#issuecomment-3300627643
## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2102?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `20.00000%` with `8 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 33.02%. Comparing base ([`f09f8af`](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`d76d530`](https://app.codecov.io/gh/apache/datafusion-comet/commit/d76d53063e20467982d17330143eddb14bb4c75f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 511 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion-comet/pull/2102?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...rc/main/scala/org/apache/comet/serde/strings.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/2102?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2Fserde%2Fstrings.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9zZXJkZS9zdHJpbmdzLnNjYWxh) | 11.11% | [8 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/2102?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #2102 +/- ##
=============================================
- Coverage 56.12% 33.02% -23.10%
+ Complexity 976 736 -240
=============================================
Files 119 147 +28
Lines 11743 13470 +1727
Branches 2251 2336 +85
=============================================
- Hits 6591 4449 -2142
- Misses 4012 8227 +4215
+ Partials 1140 794 -346
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/2102?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#issuecomment-3330455711
Thank you @comphead , It seems like the tests mentioned are adding non space chars while we still only support spaces (which will be fixed with #2436 ). Do you think it is a good idea to hold on to this PR until #2436 is closed ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#issuecomment-3300589495
@comphead , @mbutrovich I leveraged https://github.com/apache/datafusion-comet/pull/2099 to implement `lpad` expression in the same class given the high similarity among the expressions. Please take a look whenever you get a chance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#issuecomment-3358597113
@comphead thank you for kicking off the CI . Please review the changes whenever you get a chance
Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#issuecomment-3358604860
Thank you very much for the swift approval @comphead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#discussion_r2395667394
##########
native/spark-expr/src/static_invoke/char_varchar_utils/read_side_padding.rs:
##########
@@ -28,17 +28,22 @@ use std::sync::Arc;
const SPACE: &str = " ";
/// Similar to DataFusion `rpad`, but not to truncate when the string is already longer than length
pub fn spark_read_side_padding(args: &[ColumnarValue]) -> Result<ColumnarValue, DataFusionError> {
- spark_read_side_padding2(args, false)
+ spark_read_side_padding2(args, false, false)
}
/// Custom `rpad` because DataFusion's `rpad` has differences in unicode handling
pub fn spark_rpad(args: &[ColumnarValue]) -> Result<ColumnarValue, DataFusionError> {
- spark_read_side_padding2(args, true)
+ spark_read_side_padding2(args, true, false)
+}
+
+pub fn spark_lpad(args: &[ColumnarValue]) -> Result<ColumnarValue, DataFusionError> {
Review Comment:
Done
##########
native/spark-expr/src/static_invoke/char_varchar_utils/read_side_padding.rs:
##########
@@ -250,6 +271,10 @@ fn add_padding_string(
} else {
let pad_needed = length - char_len;
let pad: String = pad_string.chars().cycle().take(pad_needed).collect();
- Ok(string + &pad)
+ if is_left_pad {
+ Ok(format!("{}{}", pad, string))
+ } else {
+ Ok(string + &pad)
+ }
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#issuecomment-3185188248
@mbutrovich , Once the PR #2099 is merged, I plan to change code in this branch to leverage the same class since most of the code for rpad could be leveraged to support lpad operation as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#issuecomment-3336076553
@comphead Changes are ready for review (please rerun the failed test caused due to a transient error)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#discussion_r2395607453
##########
native/spark-expr/src/static_invoke/char_varchar_utils/read_side_padding.rs:
##########
@@ -250,6 +271,10 @@ fn add_padding_string(
} else {
let pad_needed = length - char_len;
let pad: String = pad_string.chars().cycle().take(pad_needed).collect();
- Ok(string + &pad)
+ if is_left_pad {
+ Ok(format!("{}{}", pad, string))
+ } else {
+ Ok(string + &pad)
+ }
Review Comment:
To avoid extra allocations we can try
```
let mut result = String::with_capacity(string.len() + pad.len());
if is_left_pad {
result.push_str(&pad);
result.push_str(&string);
} else {
result.push_str(&string);
result.push_str(&pad);
}
Ok(result)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#issuecomment-3357778834
Thank you @comphead . Let me push fixes right away and re run CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#discussion_r2353874675
##########
native/spark-expr/src/static_invoke/char_varchar_utils/read_side_padding.rs:
##########
@@ -175,7 +190,11 @@ fn add_padding_string(
} else {
Ok(string)
}
+ } else if is_left_pad {
+ println!("is lpad so flipping padding and the string");
Review Comment:
Unwanted print statements (I used it for testing purposes) and have removed in the latest commit. I will push these changes along with review comments I receive to save time on CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#discussion_r2395599836
##########
native/spark-expr/src/static_invoke/char_varchar_utils/read_side_padding.rs:
##########
@@ -28,17 +28,22 @@ use std::sync::Arc;
const SPACE: &str = " ";
/// Similar to DataFusion `rpad`, but not to truncate when the string is already longer than length
pub fn spark_read_side_padding(args: &[ColumnarValue]) -> Result<ColumnarValue, DataFusionError> {
- spark_read_side_padding2(args, false)
+ spark_read_side_padding2(args, false, false)
}
/// Custom `rpad` because DataFusion's `rpad` has differences in unicode handling
pub fn spark_rpad(args: &[ColumnarValue]) -> Result<ColumnarValue, DataFusionError> {
- spark_read_side_padding2(args, true)
+ spark_read_side_padding2(args, true, false)
+}
+
+pub fn spark_lpad(args: &[ColumnarValue]) -> Result<ColumnarValue, DataFusionError> {
Review Comment:
please add comments for public method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead merged PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org