You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "coderfender (via GitHub)" <gi...@apache.org> on 2025/08/09 18:32:47 UTC

[PR] implement_comet_native_lpad_expr [datafusion-comet]

coderfender opened a new pull request, #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102

   ## Which issue does this PR close?
   
   <!--
   Implements  #2101 
   -->
   
   Closes #.
   
   ## Rationale for this change
   
   Moving towards goal to support more native operations and not fallback to spark . 
   
   ## What changes are included in this PR?
   
   
   ## How are these changes tested?
   
   Unit tests in comet expression suite 
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#issuecomment-3358605422

   I will raise a doc change PR shortly


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#issuecomment-3300627643

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2102?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `20.00000%` with `8 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 33.02%. Comparing base ([`f09f8af`](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`d76d530`](https://app.codecov.io/gh/apache/datafusion-comet/commit/d76d53063e20467982d17330143eddb14bb4c75f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 511 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion-comet/pull/2102?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...rc/main/scala/org/apache/comet/serde/strings.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/2102?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2Fserde%2Fstrings.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9zZXJkZS9zdHJpbmdzLnNjYWxh) | 11.11% | [8 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/2102?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@              Coverage Diff              @@
   ##               main    #2102       +/-   ##
   =============================================
   - Coverage     56.12%   33.02%   -23.10%     
   + Complexity      976      736      -240     
   =============================================
     Files           119      147       +28     
     Lines         11743    13470     +1727     
     Branches       2251     2336       +85     
   =============================================
   - Hits           6591     4449     -2142     
   - Misses         4012     8227     +4215     
   + Partials       1140      794      -346     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/2102?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#issuecomment-3330455711

   Thank you @comphead , It seems like the tests mentioned are adding non space chars while we still only support spaces (which will be fixed with #2436 ). Do you think it is a good idea to hold on to this PR until #2436 is closed ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#issuecomment-3300589495

   @comphead , @mbutrovich  I leveraged https://github.com/apache/datafusion-comet/pull/2099 to implement `lpad` expression in the same class given the high similarity among the expressions. Please take a look whenever you get a chance  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#issuecomment-3358597113

   @comphead thank you for kicking off the CI . Please review the changes whenever you get a chance 
   Thank you 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#issuecomment-3358604860

   Thank you very much for the swift approval @comphead 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#discussion_r2395667394


##########
native/spark-expr/src/static_invoke/char_varchar_utils/read_side_padding.rs:
##########
@@ -28,17 +28,22 @@ use std::sync::Arc;
 const SPACE: &str = " ";
 /// Similar to DataFusion `rpad`, but not to truncate when the string is already longer than length
 pub fn spark_read_side_padding(args: &[ColumnarValue]) -> Result<ColumnarValue, DataFusionError> {
-    spark_read_side_padding2(args, false)
+    spark_read_side_padding2(args, false, false)
 }
 
 /// Custom `rpad` because DataFusion's `rpad` has differences in unicode handling
 pub fn spark_rpad(args: &[ColumnarValue]) -> Result<ColumnarValue, DataFusionError> {
-    spark_read_side_padding2(args, true)
+    spark_read_side_padding2(args, true, false)
+}
+
+pub fn spark_lpad(args: &[ColumnarValue]) -> Result<ColumnarValue, DataFusionError> {

Review Comment:
   Done



##########
native/spark-expr/src/static_invoke/char_varchar_utils/read_side_padding.rs:
##########
@@ -250,6 +271,10 @@ fn add_padding_string(
     } else {
         let pad_needed = length - char_len;
         let pad: String = pad_string.chars().cycle().take(pad_needed).collect();
-        Ok(string + &pad)
+        if is_left_pad {
+            Ok(format!("{}{}", pad, string))
+        } else {
+            Ok(string + &pad)
+        }

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#issuecomment-3185188248

   @mbutrovich  , Once the PR #2099  is merged, I plan to change code in this branch to leverage the same class since most of the code for rpad could be leveraged to support lpad operation as well 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#issuecomment-3336076553

   @comphead Changes are ready for review (please rerun the failed test caused due to a transient error) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#discussion_r2395607453


##########
native/spark-expr/src/static_invoke/char_varchar_utils/read_side_padding.rs:
##########
@@ -250,6 +271,10 @@ fn add_padding_string(
     } else {
         let pad_needed = length - char_len;
         let pad: String = pad_string.chars().cycle().take(pad_needed).collect();
-        Ok(string + &pad)
+        if is_left_pad {
+            Ok(format!("{}{}", pad, string))
+        } else {
+            Ok(string + &pad)
+        }

Review Comment:
   To avoid extra allocations we can try
   
   ```
   let mut result = String::with_capacity(string.len() + pad.len());
   if is_left_pad {
       result.push_str(&pad);
       result.push_str(&string);
   } else {
       result.push_str(&string);
       result.push_str(&pad);
   }
   Ok(result)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#issuecomment-3357778834

   Thank you @comphead  . Let me push fixes right away and re run CI 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]

Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#discussion_r2353874675


##########
native/spark-expr/src/static_invoke/char_varchar_utils/read_side_padding.rs:
##########
@@ -175,7 +190,11 @@ fn add_padding_string(
         } else {
             Ok(string)
         }
+    } else if is_left_pad {
+        println!("is lpad so flipping padding and the string");

Review Comment:
   Unwanted print statements (I used it for testing purposes) and have removed in the latest commit. I will push these changes along with review comments I receive to save time on CI 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102#discussion_r2395599836


##########
native/spark-expr/src/static_invoke/char_varchar_utils/read_side_padding.rs:
##########
@@ -28,17 +28,22 @@ use std::sync::Arc;
 const SPACE: &str = " ";
 /// Similar to DataFusion `rpad`, but not to truncate when the string is already longer than length
 pub fn spark_read_side_padding(args: &[ColumnarValue]) -> Result<ColumnarValue, DataFusionError> {
-    spark_read_side_padding2(args, false)
+    spark_read_side_padding2(args, false, false)
 }
 
 /// Custom `rpad` because DataFusion's `rpad` has differences in unicode handling
 pub fn spark_rpad(args: &[ColumnarValue]) -> Result<ColumnarValue, DataFusionError> {
-    spark_read_side_padding2(args, true)
+    spark_read_side_padding2(args, true, false)
+}
+
+pub fn spark_lpad(args: &[ColumnarValue]) -> Result<ColumnarValue, DataFusionError> {

Review Comment:
   please add comments for public method



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: implement_comet_native_lpad_expr [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead merged PR #2102:
URL: https://github.com/apache/datafusion-comet/pull/2102


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org