You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/20 02:12:46 UTC
[GitHub] [spark] yaooqinn opened a new pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
yaooqinn opened a new pull request #31253:
URL: https://github.com/apache/spark/pull/31253
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
For varchar(N), we currently trim all spaces first to check whether the remained length exceeds, it not necessary to visit them all but at most to those after N.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
improve varchar performance for write side
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
no
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
benchmark and existing ut
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r560975231
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
##########
@@ -47,19 +47,23 @@ public static UTF8String charTypeReadSideCheck(UTF8String inputStr, int limit) {
}
public static UTF8String varcharTypeWriteSideCheck(UTF8String inputStr, int limit) {
- if (inputStr != null && inputStr.numChars() <= limit) {
- return inputStr;
- } else if (inputStr != null) {
- // Trailing spaces do not count in the length check. We need to retain the trailing spaces
- // (truncate to length N), as there is no read-time padding for varchar type.
- // TODO: create a special TrimRight function that can trim to a certain length.
- UTF8String trimmed = inputStr.trimRight();
- if (trimmed.numChars() > limit) {
- throw new RuntimeException("Exceeds varchar type length limitation: " + limit);
- }
- return inputStr.substring(0, limit);
- } else {
+ if (inputStr == null) {
return null;
+ } else {
+ int numChars = inputStr.numChars();
+ if (numChars <= limit) {
+ return inputStr;
+ } else {
+ // Trailing spaces do not count in the length check. We need to retain the trailing spaces
+ // (truncate to length N), as there is no read-time padding for varchar type.
+ int tailSpacesMayNeedTrim = numChars - limit;
+ int lengthAfterTrim = inputStr.effectiveLength(tailSpacesMayNeedTrim);
+ if (lengthAfterTrim == limit) {
Review comment:
oh...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r561049706
##########
File path: sql/core/benchmarks/CharVarcharBenchmark-results.txt
##########
@@ -6,85 +6,130 @@ Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Read with length 20, hasSpaces: false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-read string with length 20 1504 1508 4 66.5 15.0 1.0X
-read char with length 20 1680 1684 3 59.5 16.8 0.9X
-read varchar with length 20 1659 1682 26 60.3 16.6 0.9X
+read string with length 20 1605 1636 39 62.3 16.1 1.0X
+read char with length 20 1731 1740 12 57.8 17.3 0.9X
+read varchar with length 20 1716 1729 15 58.3 17.2 0.9X
Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Read with length 40, hasSpaces: false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-read string with length 40 1662 1678 15 60.2 16.6 1.0X
-read char with length 40 1721 1731 9 58.1 17.2 1.0X
-read varchar with length 40 1694 1706 12 59.0 16.9 1.0X
+read string with length 40 1712 1716 5 58.4 17.1 1.0X
+read char with length 40 1660 1686 41 60.2 16.6 1.0X
+read varchar with length 40 1670 1721 46 59.9 16.7 1.0X
Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Read with length 60, hasSpaces: false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-read string with length 60 1623 1643 23 61.6 16.2 1.0X
-read char with length 60 1644 1685 66 60.8 16.4 1.0X
-read varchar with length 60 1660 1680 18 60.2 16.6 1.0X
+read string with length 60 1708 1720 15 58.6 17.1 1.0X
+read char with length 60 1652 1677 29 60.5 16.5 1.0X
+read varchar with length 60 1649 1675 22 60.6 16.5 1.0X
Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Read with length 80, hasSpaces: true: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-read string with length 80 1629 1678 57 61.4 16.3 1.0X
-read char with length 80 1630 1667 65 61.3 16.3 1.0X
-read varchar with length 80 1664 1684 34 60.1 16.6 1.0X
+read string with length 80 1720 1726 6 58.1 17.2 1.0X
+read char with length 80 1650 1670 33 60.6 16.5 1.0X
+read varchar with length 80 1653 1667 21 60.5 16.5 1.0X
Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Read with length 100, hasSpaces: true: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-read string with length 100 1594 1612 17 62.7 15.9 1.0X
-read char with length 100 1631 1642 11 61.3 16.3 1.0X
-read varchar with length 100 1635 1644 13 61.1 16.4 1.0X
+read string with length 100 1699 1703 5 58.8 17.0 1.0X
+read char with length 100 1660 1677 15 60.2 16.6 1.0X
+read varchar with length 100 1654 1658 3 60.4 16.5 1.0X
================================================================================================
-Char Varchar Write Side Perf
+Char Varchar Write Side Perf w/o Tailing Spaces
================================================================================================
Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Write with length 20, hasSpaces: false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-write string with length 20 2760 2784 21 3.6 276.0 1.0X
-write char with length 20 2898 2917 22 3.5 289.8 1.0X
-write varchar with length 20 2876 2892 14 3.5 287.6 1.0X
+write string with length 20 2322 2334 12 4.3 232.2 1.0X
+write char with length 20 2490 2498 7 4.0 249.0 0.9X
+write varchar with length 20 2481 2494 16 4.0 248.1 0.9X
Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Write with length 40, hasSpaces: false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-write string with length 40 2726 2734 9 3.7 272.6 1.0X
-write char with length 40 2885 2898 16 3.5 288.5 0.9X
-write varchar with length 40 2844 2860 15 3.5 284.4 1.0X
+write string with length 40 2302 2316 13 4.3 230.2 1.0X
+write char with length 40 2464 2488 24 4.1 246.4 0.9X
+write varchar with length 40 2471 2481 14 4.0 247.1 0.9X
Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Write with length 60, hasSpaces: false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-write string with length 60 2724 2739 21 3.7 272.4 1.0X
-write char with length 60 2868 2912 44 3.5 286.8 0.9X
-write varchar with length 60 2870 2896 23 3.5 287.0 0.9X
+write string with length 60 2329 2341 11 4.3 232.9 1.0X
+write char with length 60 2483 2510 26 4.0 248.3 0.9X
+write varchar with length 60 2449 2470 19 4.1 244.9 1.0X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
+Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
+Write with length 80, hasSpaces: false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+write string with length 80 2293 2310 24 4.4 229.3 1.0X
+write char with length 80 2483 2504 37 4.0 248.3 0.9X
+write varchar with length 80 2469 2480 12 4.1 246.9 0.9X
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
+Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
+Write with length 100, hasSpaces: false: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+write string with length 100 2310 2327 16 4.3 231.0 1.0X
+write char with length 100 2476 2515 35 4.0 247.6 0.9X
+write varchar with length 100 2466 2486 23 4.1 246.6 0.9X
+
+
+================================================================================================
+Char Varchar Write Side Perf w Tailing Spaces
+================================================================================================
+
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.16
+Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
+Write with length 20, hasSpaces: true: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
Review comment:
I think we don't need to print `hasSpaces: true` now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r560967583
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
##########
@@ -47,19 +47,23 @@ public static UTF8String charTypeReadSideCheck(UTF8String inputStr, int limit) {
}
public static UTF8String varcharTypeWriteSideCheck(UTF8String inputStr, int limit) {
- if (inputStr != null && inputStr.numChars() <= limit) {
- return inputStr;
- } else if (inputStr != null) {
- // Trailing spaces do not count in the length check. We need to retain the trailing spaces
- // (truncate to length N), as there is no read-time padding for varchar type.
- // TODO: create a special TrimRight function that can trim to a certain length.
- UTF8String trimmed = inputStr.trimRight();
- if (trimmed.numChars() > limit) {
- throw new RuntimeException("Exceeds varchar type length limitation: " + limit);
- }
- return inputStr.substring(0, limit);
- } else {
+ if (inputStr == null) {
return null;
+ } else {
+ int numChars = inputStr.numChars();
+ if (numChars <= limit) {
+ return inputStr;
+ } else {
+ // Trailing spaces do not count in the length check. We need to retain the trailing spaces
+ // (truncate to length N), as there is no read-time padding for varchar type.
+ int tailSpacesMayNeedTrim = numChars - limit;
+ int lengthAfterTrim = inputStr.effectiveLength(tailSpacesMayNeedTrim);
+ if (lengthAfterTrim == limit) {
Review comment:
This isn't right. `lengthAfterTrim` is the length of bytes, while `limit` is the length of chars.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763279649
cc @cloud-fan @maropu @HyukjinKwon thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763898642
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134288/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763830906
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38874/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763898642
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134288/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763357280
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134246/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-764383580
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763346696
**[Test build #134246 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134246/testReport)** for PR 31253 at commit [`73e8979`](https://github.com/apache/spark/commit/73e89792474e9d72018c0ecf85f438b6723dcabe).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763357280
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134246/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763413924
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38841/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763363142
**[Test build #134255 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134255/testReport)** for PR 31253 at commit [`e4d29fe`](https://github.com/apache/spark/commit/e4d29feedc757e03eb6ea2035cf4fb6603541b8d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763333116
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38831/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31253:
URL: https://github.com/apache/spark/pull/31253
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763727063
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38865/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763363142
**[Test build #134255 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134255/testReport)** for PR 31253 at commit [`e4d29fe`](https://github.com/apache/spark/commit/e4d29feedc757e03eb6ea2035cf4fb6603541b8d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763381962
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38841/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-764383580
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r560976452
##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##########
@@ -683,6 +683,16 @@ public UTF8String trimRight() {
return copyUTF8String(0, e);
}
+ /**
+ * Return the length of string stripped at most `numOfSpaces` tailing spaces
+ */
+ public int effectiveLength(int numOfSpaces) {
Review comment:
ah good point!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r561048188
##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##########
@@ -683,6 +683,16 @@ public UTF8String trimRight() {
return copyUTF8String(0, e);
}
+ /**
+ * Return the string stripped at most `numOfSpaces` tailing spaces
Review comment:
`Returns the string that is trimmed at most ``numSpaces`` tailing spaces`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763865208
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38874/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763318715
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38831/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r561049938
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/CharVarcharBenchmark.scala
##########
@@ -112,9 +112,16 @@ object CharVarcharBenchmark extends SqlBasedBenchmark {
}
}
- runBenchmark("Char Varchar Write Side Perf") {
+ runBenchmark("Char Varchar Write Side Perf w/o Tailing Spaces") {
for (len <- range) {
- writeBenchmark(N / 10, len, len >= 80)
+ writeBenchmark(N / 10, len, hasSpaces = false)
+ }
+ }
+
+ runBenchmark("Char Varchar Write Side Perf w Tailing Spaces") {
Review comment:
can we do the same for the read benchmark?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r560965664
##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##########
@@ -683,6 +683,16 @@ public UTF8String trimRight() {
return copyUTF8String(0, e);
}
+ /**
+ * Return the length of string stripped at most `numOfSpaces` tailing spaces
+ */
+ public int effectiveLength(int numOfSpaces) {
Review comment:
ah good point!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763296450
**[Test build #134246 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134246/testReport)** for PR 31253 at commit [`73e8979`](https://github.com/apache/spark/commit/73e89792474e9d72018c0ecf85f438b6723dcabe).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763815420
**[Test build #134278 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134278/testReport)** for PR 31253 at commit [`0a2682f`](https://github.com/apache/spark/commit/0a2682fe1845f15dad1cced76fc23556d310a501).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r560699454
##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##########
@@ -683,6 +683,16 @@ public UTF8String trimRight() {
return copyUTF8String(0, e);
}
+ /**
+ * Return the length of string stripped the tailing space after `limit`.
+ */
+ public int canTrimRightTo(int numOfSpaces) {
Review comment:
renamed to `effectiveLength`m WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763449569
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134255/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763804954
**[Test build #134288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134288/testReport)** for PR 31253 at commit [`3fa67b3`](https://github.com/apache/spark/commit/3fa67b360ee381afda78e0bf42d71a8d6a14c28f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r560815133
##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##########
@@ -683,6 +683,16 @@ public UTF8String trimRight() {
return copyUTF8String(0, e);
}
+ /**
+ * Return the length of string stripped at most `numOfSpaces` tailing spaces
+ */
+ public int effectiveLength(int numOfSpaces) {
Review comment:
Considering that we need to calculate the new string's numChars again for error raising.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r561049176
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
##########
@@ -47,19 +47,23 @@ public static UTF8String charTypeReadSideCheck(UTF8String inputStr, int limit) {
}
public static UTF8String varcharTypeWriteSideCheck(UTF8String inputStr, int limit) {
- if (inputStr != null && inputStr.numChars() <= limit) {
- return inputStr;
- } else if (inputStr != null) {
- // Trailing spaces do not count in the length check. We need to retain the trailing spaces
- // (truncate to length N), as there is no read-time padding for varchar type.
- // TODO: create a special TrimRight function that can trim to a certain length.
- UTF8String trimmed = inputStr.trimRight();
- if (trimmed.numChars() > limit) {
- throw new RuntimeException("Exceeds varchar type length limitation: " + limit);
- }
- return inputStr.substring(0, limit);
- } else {
+ if (inputStr == null) {
return null;
+ } else {
+ int numChars = inputStr.numChars();
+ if (numChars <= limit) {
+ return inputStr;
+ } else {
+ // Trailing spaces do not count in the length check. We need to retain the trailing spaces
+ // (truncate to length N), as there is no read-time padding for varchar type.
+ int tailSpacesMayNeedTrim = numChars - limit;
Review comment:
`maxAllowedNumTailSpaces`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763898642
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134288/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763865208
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38874/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763395526
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38841/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763700776
**[Test build #134278 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134278/testReport)** for PR 31253 at commit [`0a2682f`](https://github.com/apache/spark/commit/0a2682fe1845f15dad1cced76fc23556d310a501).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763789718
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38865/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763804954
**[Test build #134288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134288/testReport)** for PR 31253 at commit [`3fa67b3`](https://github.com/apache/spark/commit/3fa67b360ee381afda78e0bf42d71a8d6a14c28f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763898642
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134288/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r560788460
##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##########
@@ -683,6 +683,16 @@ public UTF8String trimRight() {
return copyUTF8String(0, e);
}
+ /**
+ * Return the length of string stripped at most `numOfSpaces` tailing spaces
+ */
+ public int effectiveLength(int numOfSpaces) {
Review comment:
how about `public UTF8String trimTrailingSpaces(int numSpaces)`? I don't think we need to save the string creating for the error case, as perf doesn't matter much for that case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-764317190
@cloud-fan I’ve addressed the comments,can you check again
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r561048188
##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##########
@@ -683,6 +683,16 @@ public UTF8String trimRight() {
return copyUTF8String(0, e);
}
+ /**
+ * Return the string stripped at most `numOfSpaces` tailing spaces
Review comment:
`Trims at most ``numSpaces`` space characters (ASCII 32) from the end of this string.`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r560966639
##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##########
@@ -683,6 +683,16 @@ public UTF8String trimRight() {
return copyUTF8String(0, e);
}
+ /**
+ * Return the length of string stripped at most `numOfSpaces` tailing spaces
+ */
+ public int effectiveLength(int numOfSpaces) {
Review comment:
`inputStr.substring(0, limit)` counts the chars as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31253:
URL: https://github.com/apache/spark/pull/31253
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763858954
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38874/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-764317190
@cloud-fan I’ve addressed the comments,can you check again
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763449569
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134255/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763700776
**[Test build #134278 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134278/testReport)** for PR 31253 at commit [`0a2682f`](https://github.com/apache/spark/commit/0a2682fe1845f15dad1cced76fc23556d310a501).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31253:
URL: https://github.com/apache/spark/pull/31253#discussion_r560634035
##########
File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##########
@@ -683,6 +683,16 @@ public UTF8String trimRight() {
return copyUTF8String(0, e);
}
+ /**
+ * Return the length of string stripped the tailing space after `limit`.
+ */
+ public int canTrimRightTo(int numOfSpaces) {
Review comment:
it's a bit weird that a method with the name `canTrimRightTo` returns int not boolean.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763829865
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134278/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763829865
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134278/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763789718
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38865/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763752513
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38865/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763884274
**[Test build #134288 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134288/testReport)** for PR 31253 at commit [`3fa67b3`](https://github.com/apache/spark/commit/3fa67b360ee381afda78e0bf42d71a8d6a14c28f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763296450
**[Test build #134246 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134246/testReport)** for PR 31253 at commit [`73e8979`](https://github.com/apache/spark/commit/73e89792474e9d72018c0ecf85f438b6723dcabe).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763429153
**[Test build #134255 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134255/testReport)** for PR 31253 at commit [`e4d29fe`](https://github.com/apache/spark/commit/e4d29feedc757e03eb6ea2035cf4fb6603541b8d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763309265
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38831/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763333116
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38831/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31253: [SPARK-34164][SQL] Improve write side varchar check to visit only last few tailing spaces
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31253:
URL: https://github.com/apache/spark/pull/31253#issuecomment-763413924
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38841/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org