You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/20 02:12:46 UTC

[GitHub] [spark] yaooqinn opened a new pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

yaooqinn opened a new pull request #31253:
URL: https://github.com/apache/spark/pull/31253


   
   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   For varchar(N), we currently trim all spaces first to check whether the remained length exceeds, it not necessary to visit them all but at most to those after N.
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   improve varchar performance for write side
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   no
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   
   benchmark and existing ut


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] yaooqinn commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r560975231



##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
##########
@@ -47,19 +47,23 @@ public static UTF8String charTypeReadSideCheck(UTF8String inputStr, int limit) {
   }
 
   public static UTF8String varcharTypeWriteSideCheck(UTF8String inputStr, int limit) {
-    if (inputStr != null && inputStr.numChars() <= limit) {
-      return inputStr;
-    } else if (inputStr != null) {
-      // Trailing spaces do not count in the length check. We need to retain the trailing spaces
-      // (truncate to length N), as there is no read-time padding for varchar type.
-      // TODO: create a special TrimRight function that can trim to a certain length.
-      UTF8String trimmed = inputStr.trimRight();
-      if (trimmed.numChars() > limit) {
-        throw new RuntimeException("Exceeds varchar type length limitation: " + limit);
-      }
-      return inputStr.substring(0, limit);
-    } else {
+    if (inputStr == null) {
       return null;
+    } else {
+      int numChars = inputStr.numChars();
+      if (numChars <= limit) {
+        return inputStr;
+      } else {
+        // Trailing spaces do not count in the length check. We need to retain the trailing spaces
+        // (truncate to length N), as there is no read-time padding for varchar type.
+        int tailSpacesMayNeedTrim = numChars - limit;
+        int lengthAfterTrim = inputStr.effectiveLength(tailSpacesMayNeedTrim);
+        if (lengthAfterTrim == limit) {

Review comment:
       oh...




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r561049706



##########
File path: sql/core/benchmarks/CharVarcharBenchmark-results.txt
##########
@@ -6,85 +6,130 @@ Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
 Read with length 20, hasSpaces: false:    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 ------------------------------------------------------------------------------------------------------------------------
-read string with length 20                         1504           1508           4         66.5          15.0       1.0X
-read char with length 20                           1680           1684           3         59.5          16.8       0.9X
-read varchar with length 20                        1659           1682          26         60.3          16.6       0.9X
+read string with length 20                         1605           1636          39         62.3          16.1       1.0X
+read char with length 20                           1731           1740          12         57.8          17.3       0.9X
+read varchar with length 20                        1716           1729          15         58.3          17.2       0.9X
 
 Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
 Read with length 40, hasSpaces: false:    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 ------------------------------------------------------------------------------------------------------------------------
-read string with length 40                         1662           1678          15         60.2          16.6       1.0X
-read char with length 40                           1721           1731           9         58.1          17.2       1.0X
-read varchar with length 40                        1694           1706          12         59.0          16.9       1.0X
+read string with length 40                         1712           1716           5         58.4          17.1       1.0X
+read char with length 40                           1660           1686          41         60.2          16.6       1.0X
+read varchar with length 40                        1670           1721          46         59.9          16.7       1.0X
 
 Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
 Read with length 60, hasSpaces: false:    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 ------------------------------------------------------------------------------------------------------------------------
-read string with length 60                         1623           1643          23         61.6          16.2       1.0X
-read char with length 60                           1644           1685          66         60.8          16.4       1.0X
-read varchar with length 60                        1660           1680          18         60.2          16.6       1.0X
+read string with length 60                         1708           1720          15         58.6          17.1       1.0X
+read char with length 60                           1652           1677          29         60.5          16.5       1.0X
+read varchar with length 60                        1649           1675          22         60.6          16.5       1.0X
 
 Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
 Read with length 80, hasSpaces: true:     Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 ------------------------------------------------------------------------------------------------------------------------
-read string with length 80                         1629           1678          57         61.4          16.3       1.0X
-read char with length 80                           1630           1667          65         61.3          16.3       1.0X
-read varchar with length 80                        1664           1684          34         60.1          16.6       1.0X
+read string with length 80                         1720           1726           6         58.1          17.2       1.0X
+read char with length 80                           1650           1670          33         60.6          16.5       1.0X
+read varchar with length 80                        1653           1667          21         60.5          16.5       1.0X
 
 Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
 Read with length 100, hasSpaces: true:    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 ------------------------------------------------------------------------------------------------------------------------
-read string with length 100                        1594           1612          17         62.7          15.9       1.0X
-read char with length 100                          1631           1642          11         61.3          16.3       1.0X
-read varchar with length 100                       1635           1644          13         61.1          16.4       1.0X
+read string with length 100                        1699           1703           5         58.8          17.0       1.0X
+read char with length 100                          1660           1677          15         60.2          16.6       1.0X
+read varchar with length 100                       1654           1658           3         60.4          16.5       1.0X
 
 
 ================================================================================================
-Char Varchar Write Side Perf
+Char Varchar Write Side Perf w/o Tailing Spaces
 ================================================================================================
 
 Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
 Write with length 20, hasSpaces: false:   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 ------------------------------------------------------------------------------------------------------------------------
-write string with length 20                        2760           2784          21          3.6         276.0       1.0X
-write char with length 20                          2898           2917          22          3.5         289.8       1.0X
-write varchar with length 20                       2876           2892          14          3.5         287.6       1.0X
+write string with length 20                        2322           2334          12          4.3         232.2       1.0X
+write char with length 20                          2490           2498           7          4.0         249.0       0.9X
+write varchar with length 20                       2481           2494          16          4.0         248.1       0.9X
 
 Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
 Write with length 40, hasSpaces: false:   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 ------------------------------------------------------------------------------------------------------------------------
-write string with length 40                        2726           2734           9          3.7         272.6       1.0X
-write char with length 40                          2885           2898          16          3.5         288.5       0.9X
-write varchar with length 40                       2844           2860          15          3.5         284.4       1.0X
+write string with length 40                        2302           2316          13          4.3         230.2       1.0X
+write char with length 40                          2464           2488          24          4.1         246.4       0.9X
+write varchar with length 40                       2471           2481          14          4.0         247.1       0.9X
 
 Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
 Write with length 60, hasSpaces: false:   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 ------------------------------------------------------------------------------------------------------------------------
-write string with length 60                        2724           2739          21          3.7         272.4       1.0X
-write char with length 60                          2868           2912          44          3.5         286.8       0.9X
-write varchar with length 60                       2870           2896          23          3.5         287.0       0.9X
+write string with length 60                        2329           2341          11          4.3         232.9       1.0X
+write char with length 60                          2483           2510          26          4.0         248.3       0.9X
+write varchar with length 60                       2449           2470          19          4.1         244.9       1.0X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
+Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
+Write with length 80, hasSpaces: false:   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
+------------------------------------------------------------------------------------------------------------------------
+write string with length 80                        2293           2310          24          4.4         229.3       1.0X
+write char with length 80                          2483           2504          37          4.0         248.3       0.9X
+write varchar with length 80                       2469           2480          12          4.1         246.9       0.9X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
+Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
+Write with length 100, hasSpaces: false:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
+------------------------------------------------------------------------------------------------------------------------
+write string with length 100                       2310           2327          16          4.3         231.0       1.0X
+write char with length 100                         2476           2515          35          4.0         247.6       0.9X
+write varchar with length 100                      2466           2486          23          4.1         246.6       0.9X
+
+
+================================================================================================
+Char Varchar Write Side Perf w Tailing Spaces
+================================================================================================
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
+Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
+Write with length 20, hasSpaces: true:    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative

Review comment:
       I think we don't need to print `hasSpaces: true` now.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r560967583



##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
##########
@@ -47,19 +47,23 @@ public static UTF8String charTypeReadSideCheck(UTF8String inputStr, int limit) {
   }
 
   public static UTF8String varcharTypeWriteSideCheck(UTF8String inputStr, int limit) {
-    if (inputStr != null && inputStr.numChars() <= limit) {
-      return inputStr;
-    } else if (inputStr != null) {
-      // Trailing spaces do not count in the length check. We need to retain the trailing spaces
-      // (truncate to length N), as there is no read-time padding for varchar type.
-      // TODO: create a special TrimRight function that can trim to a certain length.
-      UTF8String trimmed = inputStr.trimRight();
-      if (trimmed.numChars() > limit) {
-        throw new RuntimeException("Exceeds varchar type length limitation: " + limit);
-      }
-      return inputStr.substring(0, limit);
-    } else {
+    if (inputStr == null) {
       return null;
+    } else {
+      int numChars = inputStr.numChars();
+      if (numChars <= limit) {
+        return inputStr;
+      } else {
+        // Trailing spaces do not count in the length check. We need to retain the trailing spaces
+        // (truncate to length N), as there is no read-time padding for varchar type.
+        int tailSpacesMayNeedTrim = numChars - limit;
+        int lengthAfterTrim = inputStr.effectiveLength(tailSpacesMayNeedTrim);
+        if (lengthAfterTrim == limit) {

Review comment:
       This isn't right. `lengthAfterTrim` is the length of bytes, while `limit` is the length of chars.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] yaooqinn commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763279649


   cc @cloud-fan @maropu @HyukjinKwon thanks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763898642


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134288/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763830906


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38874/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763898642


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134288/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763357280


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134246/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-764383580


   thanks, merging to master!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763346696


   **[Test build #134246 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134246/testReport)** for PR 31253 at commit [`73e8979`](https://github.com/apache/spark/commit/73e89792474e9d72018c0ecf85f438b6723dcabe).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763357280


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134246/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763413924


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38841/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763363142


   **[Test build #134255 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134255/testReport)** for PR 31253 at commit [`e4d29fe`](https://github.com/apache/spark/commit/e4d29feedc757e03eb6ea2035cf4fb6603541b8d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763333116


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38831/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan closed pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31253:
URL: https://github.com/apache/spark/pull/31253


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763727063


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38865/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763363142


   **[Test build #134255 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134255/testReport)** for PR 31253 at commit [`e4d29fe`](https://github.com/apache/spark/commit/e4d29feedc757e03eb6ea2035cf4fb6603541b8d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763381962


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38841/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-764383580


   thanks, merging to master!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] yaooqinn commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r560976452



##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##########
@@ -683,6 +683,16 @@ public UTF8String trimRight() {
     return copyUTF8String(0, e);
   }
 
+  /**
+   * Return the length of string stripped at most `numOfSpaces` tailing spaces
+   */
+  public int effectiveLength(int numOfSpaces) {

Review comment:
       ah good point!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r561048188



##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##########
@@ -683,6 +683,16 @@ public UTF8String trimRight() {
     return copyUTF8String(0, e);
   }
 
+  /**
+   * Return the string stripped at most `numOfSpaces` tailing spaces

Review comment:
       `Returns the string that is trimmed at most ``numSpaces`` tailing spaces`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763865208


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38874/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763318715


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38831/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r561049938



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/CharVarcharBenchmark.scala
##########
@@ -112,9 +112,16 @@ object CharVarcharBenchmark extends SqlBasedBenchmark {
       }
     }
 
-    runBenchmark("Char Varchar Write Side Perf") {
+    runBenchmark("Char Varchar Write Side Perf w/o Tailing Spaces") {
       for (len <- range) {
-        writeBenchmark(N / 10, len, len >= 80)
+        writeBenchmark(N / 10, len, hasSpaces = false)
+      }
+    }
+
+    runBenchmark("Char Varchar Write Side Perf w Tailing Spaces") {

Review comment:
       can we do the same for the read benchmark?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r560965664



##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##########
@@ -683,6 +683,16 @@ public UTF8String trimRight() {
     return copyUTF8String(0, e);
   }
 
+  /**
+   * Return the length of string stripped at most `numOfSpaces` tailing spaces
+   */
+  public int effectiveLength(int numOfSpaces) {

Review comment:
       ah good point!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763296450


   **[Test build #134246 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134246/testReport)** for PR 31253 at commit [`73e8979`](https://github.com/apache/spark/commit/73e89792474e9d72018c0ecf85f438b6723dcabe).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763815420


   **[Test build #134278 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134278/testReport)** for PR 31253 at commit [`0a2682f`](https://github.com/apache/spark/commit/0a2682fe1845f15dad1cced76fc23556d310a501).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] yaooqinn commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r560699454



##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##########
@@ -683,6 +683,16 @@ public UTF8String trimRight() {
     return copyUTF8String(0, e);
   }
 
+  /**
+   * Return the length of string stripped the tailing space after `limit`.
+   */
+  public int canTrimRightTo(int numOfSpaces) {

Review comment:
       renamed to `effectiveLength`m WDYT?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763449569


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134255/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763804954


   **[Test build #134288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134288/testReport)** for PR 31253 at commit [`3fa67b3`](https://github.com/apache/spark/commit/3fa67b360ee381afda78e0bf42d71a8d6a14c28f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] yaooqinn commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r560815133



##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##########
@@ -683,6 +683,16 @@ public UTF8String trimRight() {
     return copyUTF8String(0, e);
   }
 
+  /**
+   * Return the length of string stripped at most `numOfSpaces` tailing spaces
+   */
+  public int effectiveLength(int numOfSpaces) {

Review comment:
       Considering that we need to calculate the new string's numChars again for error raising.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r561049176



##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
##########
@@ -47,19 +47,23 @@ public static UTF8String charTypeReadSideCheck(UTF8String inputStr, int limit) {
   }
 
   public static UTF8String varcharTypeWriteSideCheck(UTF8String inputStr, int limit) {
-    if (inputStr != null && inputStr.numChars() <= limit) {
-      return inputStr;
-    } else if (inputStr != null) {
-      // Trailing spaces do not count in the length check. We need to retain the trailing spaces
-      // (truncate to length N), as there is no read-time padding for varchar type.
-      // TODO: create a special TrimRight function that can trim to a certain length.
-      UTF8String trimmed = inputStr.trimRight();
-      if (trimmed.numChars() > limit) {
-        throw new RuntimeException("Exceeds varchar type length limitation: " + limit);
-      }
-      return inputStr.substring(0, limit);
-    } else {
+    if (inputStr == null) {
       return null;
+    } else {
+      int numChars = inputStr.numChars();
+      if (numChars <= limit) {
+        return inputStr;
+      } else {
+        // Trailing spaces do not count in the length check. We need to retain the trailing spaces
+        // (truncate to length N), as there is no read-time padding for varchar type.
+        int tailSpacesMayNeedTrim = numChars - limit;

Review comment:
       `maxAllowedNumTailSpaces`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763898642


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134288/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763865208


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38874/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763395526


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38841/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763700776


   **[Test build #134278 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134278/testReport)** for PR 31253 at commit [`0a2682f`](https://github.com/apache/spark/commit/0a2682fe1845f15dad1cced76fc23556d310a501).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763789718


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38865/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763804954


   **[Test build #134288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134288/testReport)** for PR 31253 at commit [`3fa67b3`](https://github.com/apache/spark/commit/3fa67b360ee381afda78e0bf42d71a8d6a14c28f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763898642


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134288/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r560788460



##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##########
@@ -683,6 +683,16 @@ public UTF8String trimRight() {
     return copyUTF8String(0, e);
   }
 
+  /**
+   * Return the length of string stripped at most `numOfSpaces` tailing spaces
+   */
+  public int effectiveLength(int numOfSpaces) {

Review comment:
       how about `public UTF8String trimTrailingSpaces(int numSpaces)`? I don't think we need to save the string creating for the error case, as perf doesn't matter much for that case.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] yaooqinn commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-764317190


   @cloud-fan I’ve addressed the comments,can you check again


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r561048188



##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##########
@@ -683,6 +683,16 @@ public UTF8String trimRight() {
     return copyUTF8String(0, e);
   }
 
+  /**
+   * Return the string stripped at most `numOfSpaces` tailing spaces

Review comment:
       `Trims at most ``numSpaces`` space characters (ASCII 32) from the end of this string.`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r560966639



##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##########
@@ -683,6 +683,16 @@ public UTF8String trimRight() {
     return copyUTF8String(0, e);
   }
 
+  /**
+   * Return the length of string stripped at most `numOfSpaces` tailing spaces
+   */
+  public int effectiveLength(int numOfSpaces) {

Review comment:
       `inputStr.substring(0, limit)` counts the chars as well.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan closed pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31253:
URL: https://github.com/apache/spark/pull/31253


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763858954


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38874/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] yaooqinn commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-764317190


   @cloud-fan I’ve addressed the comments,can you check again


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763449569


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134255/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763700776


   **[Test build #134278 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134278/testReport)** for PR 31253 at commit [`0a2682f`](https://github.com/apache/spark/commit/0a2682fe1845f15dad1cced76fc23556d310a501).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r560634035



##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##########
@@ -683,6 +683,16 @@ public UTF8String trimRight() {
     return copyUTF8String(0, e);
   }
 
+  /**
+   * Return the length of string stripped the tailing space after `limit`.
+   */
+  public int canTrimRightTo(int numOfSpaces) {

Review comment:
       it's a bit weird that a method with the name `canTrimRightTo` returns int not boolean.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763829865


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134278/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763829865


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134278/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763789718


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38865/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763752513


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38865/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763884274


   **[Test build #134288 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134288/testReport)** for PR 31253 at commit [`3fa67b3`](https://github.com/apache/spark/commit/3fa67b360ee381afda78e0bf42d71a8d6a14c28f).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763296450


   **[Test build #134246 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134246/testReport)** for PR 31253 at commit [`73e8979`](https://github.com/apache/spark/commit/73e89792474e9d72018c0ecf85f438b6723dcabe).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763429153


   **[Test build #134255 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134255/testReport)** for PR 31253 at commit [`e4d29fe`](https://github.com/apache/spark/commit/e4d29feedc757e03eb6ea2035cf4fb6603541b8d).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763309265


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38831/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763333116


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38831/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763413924


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38841/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org