You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kazuyukitanimura (via GitHub)" <gi...@apache.org> on 2024/08/03 21:27:40 UTC
[PR] fix: Optimize rpad [datafusion-comet]
kazuyukitanimura opened a new pull request, #772:
URL: https://github.com/apache/datafusion-comet/pull/772
## Which issue does this PR close?
## Rationale for this change
This PR improves spark_rpad that is used for CHAR() schema
## What changes are included in this PR?
Optimized spark_rpad
## How are these changes tested?
Existing tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Optimize read_side_padding" [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on PR #772:
URL: https://github.com/apache/datafusion-comet/pull/772#issuecomment-2276255096
Merged, Thanks @comphead @viirya @andygrove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Optimize rpad [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #772:
URL: https://github.com/apache/datafusion-comet/pull/772#discussion_r1702943347
##########
native/spark-expr/src/scalar_funcs.rs:
##########
@@ -410,29 +410,31 @@ fn spark_rpad_internal<T: OffsetSizeTrait>(
length: i32,
) -> Result<ColumnarValue, DataFusionError> {
let string_array = as_generic_string_array::<T>(array)?;
+ let length = 0.max(length) as usize;
+ let space_string = " ".repeat(length);
+
+ let mut builder =
+ GenericStringBuilder::<T>::with_capacity(string_array.len(), string_array.len() * length);
- let result = string_array
- .iter()
- .map(|string| match string {
+ for string in string_array.iter() {
+ match string {
Some(string) => {
- let length = if length < 0 { 0 } else { length as usize };
- if length == 0 {
- Ok(Some("".to_string()))
+ // It looks Spark's UTF8String is closer to chars rather than graphemes
+ // https://stackoverflow.com/a/46290728
+ let char_len = string.chars().count();
+ if length <= char_len {
+ builder.append_value(string);
Review Comment:
If the required len is less than string's length, don't we need to take substring of it? Spark RPad does it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Optimize rpad [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #772:
URL: https://github.com/apache/datafusion-comet/pull/772#discussion_r1702945120
##########
native/spark-expr/src/scalar_funcs.rs:
##########
@@ -410,29 +410,31 @@ fn spark_rpad_internal<T: OffsetSizeTrait>(
length: i32,
) -> Result<ColumnarValue, DataFusionError> {
let string_array = as_generic_string_array::<T>(array)?;
+ let length = 0.max(length) as usize;
+ let space_string = " ".repeat(length);
+
+ let mut builder =
+ GenericStringBuilder::<T>::with_capacity(string_array.len(), string_array.len() * length);
- let result = string_array
- .iter()
- .map(|string| match string {
+ for string in string_array.iter() {
+ match string {
Some(string) => {
- let length = if length < 0 { 0 } else { length as usize };
- if length == 0 {
- Ok(Some("".to_string()))
+ // It looks Spark's UTF8String is closer to chars rather than graphemes
+ // https://stackoverflow.com/a/46290728
Review Comment:
Can you add an unit test for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Optimize rpad [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #772:
URL: https://github.com/apache/datafusion-comet/pull/772#discussion_r1707491294
##########
spark/src/test/resources/tpcds-micro-benchmarks/char_type.sql:
##########
@@ -0,0 +1,7 @@
+SELECT
Review Comment:
They are related as their schema types are CHAR()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Optimize read_side_padding" [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura merged PR #772:
URL: https://github.com/apache/datafusion-comet/pull/772
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Optimize rpad [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on PR #772:
URL: https://github.com/apache/datafusion-comet/pull/772#issuecomment-2267165391
### Before

### After

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Optimize rpad [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #772:
URL: https://github.com/apache/datafusion-comet/pull/772#discussion_r1702943485
##########
native/spark-expr/src/scalar_funcs.rs:
##########
@@ -410,29 +410,31 @@ fn spark_rpad_internal<T: OffsetSizeTrait>(
length: i32,
) -> Result<ColumnarValue, DataFusionError> {
let string_array = as_generic_string_array::<T>(array)?;
+ let length = 0.max(length) as usize;
+ let space_string = " ".repeat(length);
+
+ let mut builder =
+ GenericStringBuilder::<T>::with_capacity(string_array.len(), string_array.len() * length);
- let result = string_array
- .iter()
- .map(|string| match string {
+ for string in string_array.iter() {
+ match string {
Some(string) => {
- let length = if length < 0 { 0 } else { length as usize };
- if length == 0 {
- Ok(Some("".to_string()))
+ // It looks Spark's UTF8String is closer to chars rather than graphemes
+ // https://stackoverflow.com/a/46290728
+ let char_len = string.chars().count();
+ if length <= char_len {
+ builder.append_value(string);
Review Comment:
Current implementation already has this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Optimize rpad [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #772:
URL: https://github.com/apache/datafusion-comet/pull/772#discussion_r1707930950
##########
native/spark-expr/src/scalar_funcs.rs:
##########
@@ -410,29 +410,31 @@ fn spark_rpad_internal<T: OffsetSizeTrait>(
length: i32,
) -> Result<ColumnarValue, DataFusionError> {
let string_array = as_generic_string_array::<T>(array)?;
+ let length = 0.max(length) as usize;
+ let space_string = " ".repeat(length);
+
+ let mut builder =
+ GenericStringBuilder::<T>::with_capacity(string_array.len(), string_array.len() * length);
- let result = string_array
- .iter()
- .map(|string| match string {
+ for string in string_array.iter() {
+ match string {
Some(string) => {
- let length = if length < 0 { 0 } else { length as usize };
- if length == 0 {
- Ok(Some("".to_string()))
+ // It looks Spark's UTF8String is closer to chars rather than graphemes
+ // https://stackoverflow.com/a/46290728
Review Comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Optimize rpad [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #772:
URL: https://github.com/apache/datafusion-comet/pull/772#discussion_r1708070265
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -1911,6 +1911,18 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
+ test("readSidePadding") {
+ // https://stackoverflow.com/a/46290728
+ val table = "test"
+ withTable(table) {
+ sql(s"create table $table(col1 CHAR(2)) using parquet")
+ sql(s"insert into $table values('é')") // unicode 'e\\u{301}'
+ sql(s"insert into $table values('é')") // unicode '\\u{e9}'
Review Comment:
for completeness, could you also test for empty string and 2 char string?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Optimize read_side_padding" [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #772:
URL: https://github.com/apache/datafusion-comet/pull/772#discussion_r1708167370
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -1911,6 +1911,18 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
+ test("readSidePadding") {
+ // https://stackoverflow.com/a/46290728
+ val table = "test"
+ withTable(table) {
+ sql(s"create table $table(col1 CHAR(2)) using parquet")
+ sql(s"insert into $table values('é')") // unicode 'e\\u{301}'
+ sql(s"insert into $table values('é')") // unicode '\\u{e9}'
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Optimize rpad [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #772:
URL: https://github.com/apache/datafusion-comet/pull/772#discussion_r1702926845
##########
spark/src/test/resources/tpcds-micro-benchmarks/char_type.sql:
##########
@@ -0,0 +1,7 @@
+SELECT
Review Comment:
how this test related to rpad? 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Optimize rpad [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #772:
URL: https://github.com/apache/datafusion-comet/pull/772#discussion_r1707499456
##########
native/spark-expr/src/scalar_funcs.rs:
##########
@@ -410,29 +410,31 @@ fn spark_rpad_internal<T: OffsetSizeTrait>(
length: i32,
) -> Result<ColumnarValue, DataFusionError> {
let string_array = as_generic_string_array::<T>(array)?;
+ let length = 0.max(length) as usize;
+ let space_string = " ".repeat(length);
+
+ let mut builder =
+ GenericStringBuilder::<T>::with_capacity(string_array.len(), string_array.len() * length);
- let result = string_array
- .iter()
- .map(|string| match string {
+ for string in string_array.iter() {
+ match string {
Some(string) => {
- let length = if length < 0 { 0 } else { length as usize };
- if length == 0 {
- Ok(Some("".to_string()))
+ // It looks Spark's UTF8String is closer to chars rather than graphemes
+ // https://stackoverflow.com/a/46290728
+ let char_len = string.chars().count();
+ if length <= char_len {
+ builder.append_value(string);
Review Comment:
In the line 389 there is an existing comment
```
/// Similar to DataFusion `rpad`, but not to truncate when the string is already longer than length
```
Perhaps I should change the name of this method, this is not used for rpad
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org