You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/21 14:25:21 UTC
[GitHub] [spark] yaooqinn opened a new pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check in read side
yaooqinn opened a new pull request #31281:
URL: https://github.com/apache/spark/pull/31281
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
On the read side, the char length check and padding bring issues to CBO and PPD and other issues to the catalyst.
It's more reasonable to do the padding on the write side and not necessary to raise an error for data that may be dirty, as Spark doesn't take full control of the storage layer.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
improvement
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
yes, spark will not raise error for oversized char/varchar values in read side
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
modified ut
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765248555
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38943/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765243810
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38941/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766778959
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766675102
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134421/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563490566
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/PartitioningUtils.scala
##########
@@ -30,14 +35,33 @@ object PartitioningUtils {
*/
def normalizePartitionSpec[T](
partitionSpec: Map[String, T],
- partColNames: Seq[String],
+ partCols: StructType,
tblName: String,
resolver: Resolver): Map[String, T] = {
+ val rawSchema = CharVarcharUtils.getRawSchema(partCols)
val normalizedPartSpec = partitionSpec.toSeq.map { case (key, value) =>
- val normalizedKey = partColNames.find(resolver(_, key)).getOrElse {
+ val normalizedFiled = rawSchema.find(f => resolver(f.name, key)).getOrElse {
throw new AnalysisException(s"$key is not a valid partition column in table $tblName.")
}
- normalizedKey -> value
+
+ val normalizedVal = normalizedFiled.dataType match {
+ case CharType(len) if value != null && value != DEFAULT_PARTITION_NAME =>
+ val v = value match {
+ case Some(v) => Some(charTypeWriteSideCheck(v.toString, len))
+ case None => None
+ case other => charTypeWriteSideCheck(other.toString, len)
Review comment:
I got match errors after removing these 2;
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766663335
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39011/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766497990
ping @cloud-fan @maropu @HyukjinKwon thanks for reviewing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765355543
**[Test build #134357 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134357/testReport)** for PR 31281 at commit [`afa3e3a`](https://github.com/apache/spark/commit/afa3e3a1113e0edf470286f35c9dfe1c9a142ff8).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765254346
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765319591
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38945/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562458348
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/PartitioningUtils.scala
##########
@@ -30,14 +33,20 @@ object PartitioningUtils {
*/
def normalizePartitionSpec[T](
partitionSpec: Map[String, T],
- partColNames: Seq[String],
+ partCols: StructType,
tblName: String,
resolver: Resolver): Map[String, T] = {
val normalizedPartSpec = partitionSpec.toSeq.map { case (key, value) =>
- val normalizedKey = partColNames.find(resolver(_, key)).getOrElse {
+ val normalizedFiled = partCols.find(f => resolver(f.name, key)).getOrElse {
throw new AnalysisException(s"$key is not a valid partition column in table $tblName.")
}
- normalizedKey -> value
+
+ val normalizedVal = normalizedFiled.dataType match {
+ case CharType(len) if value != null =>
+ StringUtils.rightPad(value.toString, len).asInstanceOf[T]
Review comment:
Here for the session catalog, we will have null and `__HIVE_DEFAULT_PARTITION__` here, what's the behavior for v2 tables?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766638415
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39007/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766646392
**[Test build #134425 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134425/testReport)** for PR 31281 at commit [`e357cf1`](https://github.com/apache/spark/commit/e357cf1d1a9409d2e34c025b550d78d8f912ea25).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562099377
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
##########
@@ -22,28 +22,25 @@
public class CharVarcharCodegenUtils {
private static final UTF8String SPACE = UTF8String.fromString(" ");
- /**
- * Trailing spaces do not count in the length check. We don't need to retain the trailing
- * spaces, as we will pad char type columns/fields at read time.
- */
public static UTF8String charTypeWriteSideCheck(UTF8String inputStr, int limit) {
if (inputStr == null) {
return null;
} else {
- UTF8String trimmed = inputStr.trimRight();
- if (trimmed.numChars() > limit) {
- throw new RuntimeException("Exceeds char type length limitation: " + limit);
+ int numChars = inputStr.numChars();
+ if (numChars == limit) {
+ return inputStr;
+ } else if (numChars <= limit) {
+ return inputStr.rpad(limit, SPACE);
+ } else {
+ int maxAllowedNumTailSpaces = numChars - limit;
Review comment:
nit: `numTailSpacesToTrim` seems better
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562463977
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/PartitioningUtils.scala
##########
@@ -30,14 +33,20 @@ object PartitioningUtils {
*/
def normalizePartitionSpec[T](
partitionSpec: Map[String, T],
- partColNames: Seq[String],
+ partCols: StructType,
tblName: String,
resolver: Resolver): Map[String, T] = {
val normalizedPartSpec = partitionSpec.toSeq.map { case (key, value) =>
- val normalizedKey = partColNames.find(resolver(_, key)).getOrElse {
+ val normalizedFiled = partCols.find(f => resolver(f.name, key)).getOrElse {
throw new AnalysisException(s"$key is not a valid partition column in table $tblName.")
}
- normalizedKey -> value
+
+ val normalizedVal = normalizedFiled.dataType match {
+ case CharType(len) if value != null =>
+ StringUtils.rightPad(value.toString, len).asInstanceOf[T]
Review comment:
`__HIVE_DEFAULT_PARTITION__ ` is the same as `null`, and we pass them to v2 catalog as it is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766976169
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134451/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765424993
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38954/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563667073
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,134 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
+
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
}
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ sql(s"ALTER TABLE t DROP PARTITION(c='a')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: static partitioned columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t PARTITION (c ='$v') VALUES ('1')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
Review comment:
According to SQL standard, for varchar type, the tailing spaces within the length limit is valid, so this case should remain as it is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-764714864
**[Test build #134336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134336/testReport)** for PR 31281 at commit [`10d03d0`](https://github.com/apache/spark/commit/10d03d0ebb15f3b8a1a402058c78d4a161cb8452).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765424993
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38954/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562450773
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/PartitioningUtils.scala
##########
@@ -30,14 +33,20 @@ object PartitioningUtils {
*/
def normalizePartitionSpec[T](
partitionSpec: Map[String, T],
- partColNames: Seq[String],
+ partCols: StructType,
tblName: String,
resolver: Resolver): Map[String, T] = {
val normalizedPartSpec = partitionSpec.toSeq.map { case (key, value) =>
- val normalizedKey = partColNames.find(resolver(_, key)).getOrElse {
+ val normalizedFiled = partCols.find(f => resolver(f.name, key)).getOrElse {
throw new AnalysisException(s"$key is not a valid partition column in table $tblName.")
}
- normalizedKey -> value
+
+ val normalizedVal = normalizedFiled.dataType match {
+ case CharType(len) if value != null =>
+ StringUtils.rightPad(value.toString, len).asInstanceOf[T]
Review comment:
T can be `Option[String]`, and calling `toString` blindly seems risky.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765292310
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38945/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766579761
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562448792
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
##########
@@ -22,55 +22,34 @@
public class CharVarcharCodegenUtils {
private static final UTF8String SPACE = UTF8String.fromString(" ");
- /**
- * Trailing spaces do not count in the length check. We don't need to retain the trailing
- * spaces, as we will pad char type columns/fields at read time.
- */
- public static UTF8String charTypeWriteSideCheck(UTF8String inputStr, int limit) {
- if (inputStr == null) {
- return null;
+ private static UTF8String trimTrailingSpaces(
+ UTF8String inputStr, int numChars, int limit) {
+ int numTailSpacesToTrim = numChars - limit;
Review comment:
nit: to be safe we should add `assert(numChars > limit)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766172855
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134401/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766416397
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134412/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766411961
**[Test build #134412 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134412/testReport)** for PR 31281 at commit [`4a0f6f1`](https://github.com/apache/spark/commit/4a0f6f129281e6b906c35b7335c9e6c43d614705).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765356228
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134357/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562099552
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
##########
@@ -54,8 +51,6 @@ public static UTF8String varcharTypeWriteSideCheck(UTF8String inputStr, int limi
if (numChars <= limit) {
return inputStr;
} else {
- // Trailing spaces do not count in the length check. We need to retain the trailing spaces
- // (truncate to length N), as there is no read-time padding for varchar type.
int maxAllowedNumTailSpaces = numChars - limit;
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563491086
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,109 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
+
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
}
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ val e1 = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e1.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+
+ val e2 = intercept[RuntimeException](sql("ALTER TABLE t DROP PARTITION(c='abcdef')"))
+ assert(e2.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+
+ val e3 = intercept[RuntimeException](sql("ALTER TABLE t ADD PARTITION(c='abcdef')"))
+ assert(e3.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
- sql("ALTER TABLE t DROP PARTITION(c='a')")
+ test("varchar type values length check: partitioned columns dynamic") {
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ checkColType(spark.table("t").schema(1), VarcharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val e = intercept[RuntimeException](sql("ALTER TABLE t DROP PARTITION(c='abcdef')"))
+ assert(e.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
}
}
+ test("varchar type values length check: partitioned columns of other types") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ Seq(1, 10, 100, 1000, 10000).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', $v)")
+ sql(s"ALTER TABLE t DROP PARTITION(c=$v)")
+ checkAnswer(spark.table("t"), Nil)
+ }
+
+ val e1 = intercept[SparkException](sql(s"INSERT OVERWRITE t VALUES ('1', 100000)"))
+ assert(e1.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+
+ val e2 = intercept[RuntimeException](sql("ALTER TABLE t DROP PARTITION(c='100000')"))
Review comment:
this is not "other types"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766153603
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38987/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766197480
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766778961
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765478480
**[Test build #134373 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134373/testReport)** for PR 31281 at commit [`6b239d1`](https://github.com/apache/spark/commit/6b239d129f544f38e3ff3146a475affef5efa8e3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766859159
**[Test build #134451 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134451/testReport)** for PR 31281 at commit [`a1a0aa5`](https://github.com/apache/spark/commit/a1a0aa53ead80c6396cf93d9e08b566661d23e5d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31281:
URL: https://github.com/apache/spark/pull/31281
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765332849
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134359/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765332849
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134359/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562101505
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -56,7 +56,7 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
checkColType(spark.table("t").schema(1), CharType(5))
- sql("ALTER TABLE t DROP PARTITION(c='a')")
+ sql("ALTER TABLE t DROP PARTITION(c='a ')")
Review comment:
is it the same with hive? or we should also add padding for char type column in partition spec?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-764761422
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38923/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766193904
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38990/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766179587
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38990/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765396176
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38954/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-767007988
This is a much simpler solution and makes sense to me: almost all databases enforce the data constraints at the write side, Spark should expect the data sources or vendors to enforce it at the write side, instead of doing read side checks.
This changes the storage of char type (because of the write side padding), which makes more sense to better preserve the char type semantic. Usually, we can't change storage afterward, but fortunately, Spark 3.1 is not released yet. It's a good chance to get this merged and fix the perf regression regarding CBO and predicate pushdown.
Thanks, merging to master/3.1!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765356500
**[Test build #134367 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134367/testReport)** for PR 31281 at commit [`4614bef`](https://github.com/apache/spark/commit/4614bef84a17d73b4992e25004b68d5e674ca459).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766169311
**[Test build #134401 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134401/testReport)** for PR 31281 at commit [`a0924d8`](https://github.com/apache/spark/commit/a0924d8cd0d3de4e5c31b74e14688c31112ad6e4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562455777
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
##########
@@ -22,55 +22,34 @@
public class CharVarcharCodegenUtils {
private static final UTF8String SPACE = UTF8String.fromString(" ");
- /**
- * Trailing spaces do not count in the length check. We don't need to retain the trailing
- * spaces, as we will pad char type columns/fields at read time.
- */
- public static UTF8String charTypeWriteSideCheck(UTF8String inputStr, int limit) {
- if (inputStr == null) {
- return null;
+ private static UTF8String trimTrailingSpaces(
+ UTF8String inputStr, int numChars, int limit) {
+ int numTailSpacesToTrim = numChars - limit;
Review comment:
to be safer we can add this assert to UTF8String
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766406906
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38998/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765356228
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134357/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563529545
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,134 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
+
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
}
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ sql(s"ALTER TABLE t DROP PARTITION(c='a')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: static partitioned columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t PARTITION (c ='$v') VALUES ('1')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
+ }
+
+ test("oversize char/varchar values for alter table partition operations") {
+ Seq("CHAR(5)", "VARCHAR(5)").foreach { typ =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c $typ) USING $format PARTITIONED BY (c)")
+ Seq("ADD", "DROP").foreach { op =>
+ val e = intercept[RuntimeException](sql(s"ALTER TABLE t $op PARTITION(c='abcdef')"))
+ assert(e.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+ val e1 = intercept[RuntimeException] {
+ sql(s"ALTER TABLE t PARTITION (c='abcdef') RENAME TO PARTITION (c='2')")
+ }
+ assert(e1.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ val e2 = intercept[RuntimeException] {
+ sql(s"ALTER TABLE t PARTITION (c='1') RENAME TO PARTITION (c='abcdef')")
+ }
+ assert(e2.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+ }
+ }
+
+ test("varchar type values length check: partitioned columns dynamic") {
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ checkColType(spark.table("t").schema(1), VarcharType(5))
+ }
+ }
- sql("ALTER TABLE t DROP PARTITION(c='a')")
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val e = intercept[RuntimeException](sql("ALTER TABLE t DROP PARTITION(c='abcdef')"))
+ assert(e.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
}
}
+ test("varchar type values length check: partitioned columns of other types") {
Review comment:
can we test both char and varchar?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766156637
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38987/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563523677
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,134 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
+
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
}
+
+ withTable("t") {
Review comment:
If we change https://github.com/apache/spark/pull/31281/files#r563523300, we can remove this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765268203
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38943/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765594773
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766675102
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134421/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-764761422
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765295929
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38943/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766579761
**[Test build #134421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134421/testReport)** for PR 31281 at commit [`6f19b5e`](https://github.com/apache/spark/commit/6f19b5ec539c503fe1e05627cd8d82d14359d3df).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562464324
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
##########
@@ -22,55 +22,34 @@
public class CharVarcharCodegenUtils {
private static final UTF8String SPACE = UTF8String.fromString(" ");
- /**
- * Trailing spaces do not count in the length check. We don't need to retain the trailing
- * spaces, as we will pad char type columns/fields at read time.
- */
- public static UTF8String charTypeWriteSideCheck(UTF8String inputStr, int limit) {
- if (inputStr == null) {
- return null;
+ private static UTF8String trimTrailingSpaces(
+ UTF8String inputStr, int numChars, int limit) {
+ int numTailSpacesToTrim = numChars - limit;
Review comment:
yea we can add it in `trimTrailingSpaces`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766889125
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39037/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766663697
**[Test build #134421 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134421/testReport)** for PR 31281 at commit [`6f19b5e`](https://github.com/apache/spark/commit/6f19b5ec539c503fe1e05627cd8d82d14359d3df).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766389446
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765471112
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563493183
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,109 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
+
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
Review comment:
yes checked that, I will remove all related
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563732389
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
##########
@@ -31,6 +31,8 @@ object CharVarcharUtils extends Logging {
private val CHAR_VARCHAR_TYPE_STRING_METADATA_KEY = "__CHAR_VARCHAR_TYPE_STRING"
+ def getRawTypeString: String = CHAR_VARCHAR_TYPE_STRING_METADATA_KEY
Review comment:
and `getRawType` can call this method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563489429
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,109 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
+
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
}
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ val e1 = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e1.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
Review comment:
nit: we should test that we can always drop partition with `c='a'`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563491756
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -988,7 +989,15 @@ private[hive] class HiveClientImpl(
private[hive] object HiveClientImpl extends Logging {
/** Converts the native StructField to Hive's FieldSchema. */
def toHiveColumn(c: StructField): FieldSchema = {
- val typeString = HiveVoidType.replaceVoidType(c.dataType).catalogString
+ val typeString = if (c.metadata.contains(CHAR_VARCHAR_TYPE_STRING_METADATA_KEY)) {
Review comment:
where do we hide the char/varchar from schema before reaching here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766891687
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39037/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562453102
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/PartitioningUtils.scala
##########
@@ -30,14 +33,20 @@ object PartitioningUtils {
*/
def normalizePartitionSpec[T](
partitionSpec: Map[String, T],
- partColNames: Seq[String],
+ partCols: StructType,
tblName: String,
resolver: Resolver): Map[String, T] = {
val normalizedPartSpec = partitionSpec.toSeq.map { case (key, value) =>
- val normalizedKey = partColNames.find(resolver(_, key)).getOrElse {
+ val normalizedFiled = partCols.find(f => resolver(f.name, key)).getOrElse {
throw new AnalysisException(s"$key is not a valid partition column in table $tblName.")
}
- normalizedKey -> value
+
+ val normalizedVal = normalizedFiled.dataType match {
+ case CharType(len) if value != null =>
+ StringUtils.rightPad(value.toString, len).asInstanceOf[T]
Review comment:
shall we call `charTypeWriteSideCheck` here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766719450
**[Test build #134431 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134431/testReport)** for PR 31281 at commit [`df41014`](https://github.com/apache/spark/commit/df41014361e34134fef515d54b9c68e2787f9b7c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563503267
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -988,7 +989,15 @@ private[hive] class HiveClientImpl(
private[hive] object HiveClientImpl extends Logging {
/** Converts the native StructField to Hive's FieldSchema. */
def toHiveColumn(c: StructField): FieldSchema = {
- val typeString = HiveVoidType.replaceVoidType(c.dataType).catalogString
+ val typeString = if (c.metadata.contains(CHAR_VARCHAR_TYPE_STRING_METADATA_KEY)) {
Review comment:
HiveTableScanExec -> HiveTableRelation -> CatalogTable, we need this chage to covert it back to `HiveTable` properly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563513856
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/HiveCharVarcharTestSuite.scala
##########
@@ -50,6 +51,35 @@ class HiveCharVarcharTestSuite extends CharVarcharTestSuite with TestHiveSinglet
assert(rest.contains("CHAR(5)"))
}
}
+
+ // TODO(SPARK-34203): Move this too super class when the ticket gets fixed
Review comment:
I rechecked this, this is a valid case. moved to super. The null case seems an unrelated issue and need not to add a TODO here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562367529
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -56,7 +56,7 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
checkColType(spark.table("t").schema(1), CharType(5))
- sql("ALTER TABLE t DROP PARTITION(c='a')")
+ sql("ALTER TABLE t DROP PARTITION(c='a ')")
Review comment:
Not same now, we should add padding to this, as for char(4), 'a', 'a ', 'a ', 'a ' are all same
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563496408
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,109 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
+
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
}
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ val e1 = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e1.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
Review comment:
good point, I will add one
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562367529
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -56,7 +56,7 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
checkColType(spark.table("t").schema(1), CharType(5))
- sql("ALTER TABLE t DROP PARTITION(c='a')")
+ sql("ALTER TABLE t DROP PARTITION(c='a ')")
Review comment:
Not same now, we should add padding to this, as for char(4), 'a', 'a ', 'a ', 'a ' are all same
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766738634
**[Test build #134425 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134425/testReport)** for PR 31281 at commit [`e357cf1`](https://github.com/apache/spark/commit/e357cf1d1a9409d2e34c025b550d78d8f912ea25).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766746908
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134425/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563609350
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,134 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
+
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
}
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ sql(s"ALTER TABLE t DROP PARTITION(c='a')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: static partitioned columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t PARTITION (c ='$v') VALUES ('1')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
+ }
+
+ test("oversize char/varchar values for alter table partition operations") {
+ Seq("CHAR(5)", "VARCHAR(5)").foreach { typ =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c $typ) USING $format PARTITIONED BY (c)")
+ Seq("ADD", "DROP").foreach { op =>
+ val e = intercept[RuntimeException](sql(s"ALTER TABLE t $op PARTITION(c='abcdef')"))
+ assert(e.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+ val e1 = intercept[RuntimeException] {
+ sql(s"ALTER TABLE t PARTITION (c='abcdef') RENAME TO PARTITION (c='2')")
+ }
+ assert(e1.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ val e2 = intercept[RuntimeException] {
+ sql(s"ALTER TABLE t PARTITION (c='1') RENAME TO PARTITION (c='abcdef')")
+ }
+ assert(e2.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+ }
+ }
+
+ test("varchar type values length check: partitioned columns dynamic") {
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ checkColType(spark.table("t").schema(1), VarcharType(5))
+ }
+ }
- sql("ALTER TABLE t DROP PARTITION(c='a')")
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val e = intercept[RuntimeException](sql("ALTER TABLE t DROP PARTITION(c='abcdef')"))
+ assert(e.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
}
}
+ test("varchar type values length check: partitioned columns of other types") {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766638415
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39007/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563490346
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,109 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
+
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
}
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ val e1 = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e1.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+
+ val e2 = intercept[RuntimeException](sql("ALTER TABLE t DROP PARTITION(c='abcdef')"))
+ assert(e2.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+
+ val e3 = intercept[RuntimeException](sql("ALTER TABLE t ADD PARTITION(c='abcdef')"))
+ assert(e3.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
- sql("ALTER TABLE t DROP PARTITION(c='a')")
+ test("varchar type values length check: partitioned columns dynamic") {
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ checkColType(spark.table("t").schema(1), VarcharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766974584
**[Test build #134451 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134451/testReport)** for PR 31281 at commit [`a1a0aa5`](https://github.com/apache/spark/commit/a1a0aa53ead80c6396cf93d9e08b566661d23e5d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765427247
**[Test build #134367 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134367/testReport)** for PR 31281 at commit [`4614bef`](https://github.com/apache/spark/commit/4614bef84a17d73b4992e25004b68d5e674ca459).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765383041
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38954/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766416397
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134412/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562451393
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CharTypePadding.scala
##########
@@ -17,63 +17,23 @@
package org.apache.spark.sql.execution.datasources
-import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
import org.apache.spark.sql.catalyst.expressions.{Attribute, BinaryComparison, Expression, In, Literal, StringRPad}
-import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.util.CharVarcharUtils
-import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
import org.apache.spark.sql.types.{CharType, StringType}
import org.apache.spark.unsafe.types.UTF8String
/**
- * This rule performs char type padding and length check for both char and varchar.
- *
- * When reading values from column/field of type CHAR(N) or VARCHAR(N), the underlying string value
- * might be over length (e.g. tables w/ external locations), it will fail in this case.
- * Otherwise, right-pad the values to length N for CHAR(N) and remain the same for VARCHAR(N).
+ * This rule performs char type padding both char.
Review comment:
`This rule performs string padding for char type comparison.`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765189914
**[Test build #134355 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134355/testReport)** for PR 31281 at commit [`4235420`](https://github.com/apache/spark/commit/42354200aa4c6898588bbd2233ad28cfff7554a1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765478480
**[Test build #134373 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134373/testReport)** for PR 31281 at commit [`6b239d1`](https://github.com/apache/spark/commit/6b239d129f544f38e3ff3146a475affef5efa8e3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562449366
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
##########
@@ -222,16 +155,14 @@ object CharVarcharUtils extends Logging {
classOf[CharVarcharCodegenUtils],
StringType,
"charTypeWriteSideCheck",
- expr :: Literal(length) :: Nil,
- propagateNull = false)
+ expr :: Literal(length) :: Nil)
Review comment:
we should set `returnNullable` to `false`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766602770
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39007/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766712019
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39011/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766885780
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39037/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-764714864
**[Test build #134336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134336/testReport)** for PR 31281 at commit [`10d03d0`](https://github.com/apache/spark/commit/10d03d0ebb15f3b8a1a402058c78d4a161cb8452).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766164386
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38987/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-764836343
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134336/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-764836343
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134336/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765425761
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38955/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766145652
**[Test build #134401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134401/testReport)** for PR 31281 at commit [`a0924d8`](https://github.com/apache/spark/commit/a0924d8cd0d3de4e5c31b74e14688c31112ad6e4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766174321
**[Test build #134404 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134404/testReport)** for PR 31281 at commit [`04c6b52`](https://github.com/apache/spark/commit/04c6b52581da032d4ad242875da5cba262f1d747).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-764711672
cc @cloud-fan @maropu thanks for reviewing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765356500
**[Test build #134367 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134367/testReport)** for PR 31281 at commit [`4614bef`](https://github.com/apache/spark/commit/4614bef84a17d73b4992e25004b68d5e674ca459).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563523300
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,134 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
+
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
Review comment:
To be complete, we need to make sure we can drop the partition with `a`, `a `, ..., or `a `. That's too many tests, and most users would just write `a` as they know the padding semantic. I think we only need to test `ALTER TABLE t DROP PARTITION(c='a')` here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766194925
**[Test build #134404 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134404/testReport)** for PR 31281 at commit [`04c6b52`](https://github.com/apache/spark/commit/04c6b52581da032d4ad242875da5cba262f1d747).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766164386
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766891687
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39037/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766859159
**[Test build #134451 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134451/testReport)** for PR 31281 at commit [`a1a0aa5`](https://github.com/apache/spark/commit/a1a0aa53ead80c6396cf93d9e08b566661d23e5d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-764736936
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38923/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563488581
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -465,13 +465,13 @@ case class AlterTableAddPartitionCommand(
override def run(sparkSession: SparkSession): Seq[Row] = {
val catalog = sparkSession.sessionState.catalog
- val table = catalog.getTableMetadata(tableName)
+ val table = catalog.getTableRawMetadata(tableName)
Review comment:
do we need to make this change? `PartitioningUtils.normalizePartitionSpec` will restore the raw schema of partition columns.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765583622
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38960/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563523908
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,134 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
+
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
}
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ sql(s"ALTER TABLE t DROP PARTITION(c='a')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: static partitioned columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t PARTITION (c ='$v') VALUES ('1')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
Review comment:
ditto, we only need to test `PARTITION(c='a')`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-764714864
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562452667
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/PartitioningUtils.scala
##########
@@ -30,14 +33,20 @@ object PartitioningUtils {
*/
def normalizePartitionSpec[T](
partitionSpec: Map[String, T],
- partColNames: Seq[String],
+ partCols: StructType,
tblName: String,
resolver: Resolver): Map[String, T] = {
val normalizedPartSpec = partitionSpec.toSeq.map { case (key, value) =>
- val normalizedKey = partColNames.find(resolver(_, key)).getOrElse {
+ val normalizedFiled = partCols.find(f => resolver(f.name, key)).getOrElse {
throw new AnalysisException(s"$key is not a valid partition column in table $tblName.")
}
- normalizedKey -> value
+
+ val normalizedVal = normalizedFiled.dataType match {
+ case CharType(len) if value != null =>
+ StringUtils.rightPad(value.toString, len).asInstanceOf[T]
Review comment:
we probably should do a pattern match and handle `String` `Option[String]` differently.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766579761
**[Test build #134421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134421/testReport)** for PR 31281 at commit [`6f19b5e`](https://github.com/apache/spark/commit/6f19b5ec539c503fe1e05627cd8d82d14359d3df).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765295929
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766145652
**[Test build #134401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134401/testReport)** for PR 31281 at commit [`a0924d8`](https://github.com/apache/spark/commit/a0924d8cd0d3de4e5c31b74e14688c31112ad6e4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-764761422
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563478821
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/PartitioningUtils.scala
##########
@@ -30,14 +35,34 @@ object PartitioningUtils {
*/
def normalizePartitionSpec[T](
partitionSpec: Map[String, T],
- partColNames: Seq[String],
+ partCols: StructType,
tblName: String,
resolver: Resolver): Map[String, T] = {
val normalizedPartSpec = partitionSpec.toSeq.map { case (key, value) =>
- val normalizedKey = partColNames.find(resolver(_, key)).getOrElse {
- throw new AnalysisException(s"$key is not a valid partition column in table $tblName.")
+ val normalizedFiled = CharVarcharUtils.getRawSchema(partCols)
Review comment:
nit: we don't need to call `getRawSchema` in the loop body. We can do `val partSchema = ...` at the beginning.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765391923
**[Test build #134368 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134368/testReport)** for PR 31281 at commit [`ca37499`](https://github.com/apache/spark/commit/ca3749951927e1a2e08148c85ca52cbf0b4a0655).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563530341
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -988,7 +989,15 @@ private[hive] class HiveClientImpl(
private[hive] object HiveClientImpl extends Logging {
/** Converts the native StructField to Hive's FieldSchema. */
def toHiveColumn(c: StructField): FieldSchema = {
- val typeString = HiveVoidType.replaceVoidType(c.dataType).catalogString
+ val typeString = if (c.metadata.contains(CHAR_VARCHAR_TYPE_STRING_METADATA_KEY)) {
+ // For Hive Serde, we still need to to restore the raw type for char and varchar type.
+ // When reading data in parquet, orc, or avro file format with string type for char,
+ // the tailing spaces may lost if we are not going to pad it.
+ c.metadata.getString(CHAR_VARCHAR_TYPE_STRING_METADATA_KEY)
Review comment:
we can add a `getRawTypeString` in `CharVarcharUtils`, instead of exposing `CHAR_VARCHAR_TYPE_STRING_METADATA_KEY`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766399117
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38998/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765216645
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38941/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563488313
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/PartitioningUtils.scala
##########
@@ -30,14 +35,33 @@ object PartitioningUtils {
*/
def normalizePartitionSpec[T](
partitionSpec: Map[String, T],
- partColNames: Seq[String],
+ partCols: StructType,
tblName: String,
resolver: Resolver): Map[String, T] = {
+ val rawSchema = CharVarcharUtils.getRawSchema(partCols)
val normalizedPartSpec = partitionSpec.toSeq.map { case (key, value) =>
- val normalizedKey = partColNames.find(resolver(_, key)).getOrElse {
+ val normalizedFiled = rawSchema.find(f => resolver(f.name, key)).getOrElse {
throw new AnalysisException(s"$key is not a valid partition column in table $tblName.")
}
- normalizedKey -> value
+
+ val normalizedVal = normalizedFiled.dataType match {
+ case CharType(len) if value != null && value != DEFAULT_PARTITION_NAME =>
+ val v = value match {
+ case Some(v) => Some(charTypeWriteSideCheck(v.toString, len))
+ case None => None
+ case other => charTypeWriteSideCheck(other.toString, len)
+ }
+ v.asInstanceOf[T]
+ case VarcharType(len) if value != null && value != DEFAULT_PARTITION_NAME =>
+ val v = value match {
+ case Some(v) => Some(varcharTypeWriteSideCheck(v.toString, len))
+ case None => None
+ case other => varcharTypeWriteSideCheck(other.toString, len)
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766976169
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134451/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765332158
**[Test build #134359 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134359/testReport)** for PR 31281 at commit [`6957941`](https://github.com/apache/spark/commit/69579415e752f1b647ab1782f98b33c5c2607219).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563488635
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -521,19 +521,19 @@ case class AlterTableRenamePartitionCommand(
override def run(sparkSession: SparkSession): Seq[Row] = {
val catalog = sparkSession.sessionState.catalog
- val table = catalog.getTableMetadata(tableName)
+ val table = catalog.getTableRawMetadata(tableName)
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563492926
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/HiveCharVarcharTestSuite.scala
##########
@@ -50,6 +51,35 @@ class HiveCharVarcharTestSuite extends CharVarcharTestSuite with TestHiveSinglet
assert(rest.contains("CHAR(5)"))
}
}
+
+ // TODO(SPARK-34203): Move this too super class when the ticket gets fixed
Review comment:
BTW the jira is about null partition value, while this test doesn't cover null partition at all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766197480
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563490099
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,109 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
+
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
Review comment:
I think we already tested it in the length check write tests.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,109 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
+
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
}
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ val e1 = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-764817307
**[Test build #134336 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134336/testReport)** for PR 31281 at commit [`10d03d0`](https://github.com/apache/spark/commit/10d03d0ebb15f3b8a1a402058c78d4a161cb8452).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765577267
**[Test build #134373 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134373/testReport)** for PR 31281 at commit [`6b239d1`](https://github.com/apache/spark/commit/6b239d129f544f38e3ff3146a475affef5efa8e3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766683027
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39011/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765471113
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563483203
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/PartitioningUtils.scala
##########
@@ -30,14 +35,34 @@ object PartitioningUtils {
*/
def normalizePartitionSpec[T](
partitionSpec: Map[String, T],
- partColNames: Seq[String],
+ partCols: StructType,
tblName: String,
resolver: Resolver): Map[String, T] = {
val normalizedPartSpec = partitionSpec.toSeq.map { case (key, value) =>
- val normalizedKey = partColNames.find(resolver(_, key)).getOrElse {
- throw new AnalysisException(s"$key is not a valid partition column in table $tblName.")
+ val normalizedFiled = CharVarcharUtils.getRawSchema(partCols)
Review comment:
fixed
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/PartitioningUtils.scala
##########
@@ -30,14 +35,33 @@ object PartitioningUtils {
*/
def normalizePartitionSpec[T](
partitionSpec: Map[String, T],
- partColNames: Seq[String],
+ partCols: StructType,
tblName: String,
resolver: Resolver): Map[String, T] = {
+ val rawSchema = CharVarcharUtils.getRawSchema(partCols)
val normalizedPartSpec = partitionSpec.toSeq.map { case (key, value) =>
- val normalizedKey = partColNames.find(resolver(_, key)).getOrElse {
+ val normalizedFiled = rawSchema.find(f => resolver(f.name, key)).getOrElse {
throw new AnalysisException(s"$key is not a valid partition column in table $tblName.")
}
- normalizedKey -> value
+
+ val normalizedVal = normalizedFiled.dataType match {
+ case CharType(len) if value != null && value != DEFAULT_PARTITION_NAME =>
+ val v = value match {
+ case Some(v) => Some(charTypeWriteSideCheck(v.toString, len))
+ case None => None
+ case other => charTypeWriteSideCheck(other.toString, len)
Review comment:
I got match errors after removing these 2;
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,109 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
+
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
Review comment:
yes checked that, I will remove all related
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,109 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
+
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
}
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ val e1 = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e1.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
Review comment:
good point, I will add one
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -988,7 +989,15 @@ private[hive] class HiveClientImpl(
private[hive] object HiveClientImpl extends Logging {
/** Converts the native StructField to Hive's FieldSchema. */
def toHiveColumn(c: StructField): FieldSchema = {
- val typeString = HiveVoidType.replaceVoidType(c.dataType).catalogString
+ val typeString = if (c.metadata.contains(CHAR_VARCHAR_TYPE_STRING_METADATA_KEY)) {
Review comment:
HiveTableScanExec -> HiveTableRelation -> CatalogTable, we need this chage to covert it back to `HiveTable` properly
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/HiveCharVarcharTestSuite.scala
##########
@@ -50,6 +51,35 @@ class HiveCharVarcharTestSuite extends CharVarcharTestSuite with TestHiveSinglet
assert(rest.contains("CHAR(5)"))
}
}
+
+ // TODO(SPARK-34203): Move this too super class when the ticket gets fixed
Review comment:
I rechecked this, this is a valid case. moved to super. The null case seems an unrelated issue and need not to add a TODO here
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,134 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
+
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
}
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ sql(s"ALTER TABLE t DROP PARTITION(c='a')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: static partitioned columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t PARTITION (c ='$v') VALUES ('1')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
+ }
+
+ test("oversize char/varchar values for alter table partition operations") {
+ Seq("CHAR(5)", "VARCHAR(5)").foreach { typ =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c $typ) USING $format PARTITIONED BY (c)")
+ Seq("ADD", "DROP").foreach { op =>
+ val e = intercept[RuntimeException](sql(s"ALTER TABLE t $op PARTITION(c='abcdef')"))
+ assert(e.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+ val e1 = intercept[RuntimeException] {
+ sql(s"ALTER TABLE t PARTITION (c='abcdef') RENAME TO PARTITION (c='2')")
+ }
+ assert(e1.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ val e2 = intercept[RuntimeException] {
+ sql(s"ALTER TABLE t PARTITION (c='1') RENAME TO PARTITION (c='abcdef')")
+ }
+ assert(e2.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+ }
+ }
+
+ test("varchar type values length check: partitioned columns dynamic") {
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ checkColType(spark.table("t").schema(1), VarcharType(5))
+ }
+ }
- sql("ALTER TABLE t DROP PARTITION(c='a')")
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val e = intercept[RuntimeException](sql("ALTER TABLE t DROP PARTITION(c='abcdef')"))
+ assert(e.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
}
}
+ test("varchar type values length check: partitioned columns of other types") {
Review comment:
done
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,134 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
+
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
}
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ sql(s"ALTER TABLE t DROP PARTITION(c='a')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: static partitioned columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t PARTITION (c ='$v') VALUES ('1')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
Review comment:
According to SQL standard, for varchar type, the tailing spaces within the length limit is valid, so this case should remain as it is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31281:
URL: https://github.com/apache/spark/pull/31281
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766406906
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38998/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766389446
**[Test build #134412 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134412/testReport)** for PR 31281 at commit [`4a0f6f1`](https://github.com/apache/spark/commit/4a0f6f129281e6b906c35b7335c9e6c43d614705).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563490943
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,109 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
+
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
}
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ val e1 = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e1.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+
+ val e2 = intercept[RuntimeException](sql("ALTER TABLE t DROP PARTITION(c='abcdef')"))
+ assert(e2.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+
+ val e3 = intercept[RuntimeException](sql("ALTER TABLE t ADD PARTITION(c='abcdef')"))
+ assert(e3.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
- sql("ALTER TABLE t DROP PARTITION(c='a')")
+ test("varchar type values length check: partitioned columns dynamic") {
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ checkColType(spark.table("t").schema(1), VarcharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val e = intercept[RuntimeException](sql("ALTER TABLE t DROP PARTITION(c='abcdef')"))
Review comment:
how about ADD PARTITION? can we have one test case for ADD/RENAME/DROP PARTITION with too-long partition values?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563521153
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/PartitioningUtils.scala
##########
@@ -30,14 +35,33 @@ object PartitioningUtils {
*/
def normalizePartitionSpec[T](
partitionSpec: Map[String, T],
- partColNames: Seq[String],
+ partCols: StructType,
tblName: String,
resolver: Resolver): Map[String, T] = {
+ val rawSchema = CharVarcharUtils.getRawSchema(partCols)
val normalizedPartSpec = partitionSpec.toSeq.map { case (key, value) =>
- val normalizedKey = partColNames.find(resolver(_, key)).getOrElse {
+ val normalizedFiled = rawSchema.find(f => resolver(f.name, key)).getOrElse {
throw new AnalysisException(s"$key is not a valid partition column in table $tblName.")
}
- normalizedKey -> value
+
+ val normalizedVal = normalizedFiled.dataType match {
+ case CharType(len) if value != null && value != DEFAULT_PARTITION_NAME =>
+ val v = value match {
+ case Some(v) => Some(charTypeWriteSideCheck(v.toString, len))
Review comment:
nit:
```
case Some(str: String) => ...
case str: String => ...
case other => other
```
Then we don't need the `v.asInstanceOf[T]` at the end.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766389446
**[Test build #134412 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134412/testReport)** for PR 31281 at commit [`4a0f6f1`](https://github.com/apache/spark/commit/4a0f6f129281e6b906c35b7335c9e6c43d614705).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766646392
**[Test build #134425 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134425/testReport)** for PR 31281 at commit [`e357cf1`](https://github.com/apache/spark/commit/e357cf1d1a9409d2e34c025b550d78d8f912ea25).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-764714864
**[Test build #134336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134336/testReport)** for PR 31281 at commit [`10d03d0`](https://github.com/apache/spark/commit/10d03d0ebb15f3b8a1a402058c78d4a161cb8452).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563527472
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,134 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
+
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
}
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ sql(s"ALTER TABLE t DROP PARTITION(c='a')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: static partitioned columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t PARTITION (c ='$v') VALUES ('1')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
+ }
+
+ test("oversize char/varchar values for alter table partition operations") {
+ Seq("CHAR(5)", "VARCHAR(5)").foreach { typ =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c $typ) USING $format PARTITIONED BY (c)")
+ Seq("ADD", "DROP").foreach { op =>
+ val e = intercept[RuntimeException](sql(s"ALTER TABLE t $op PARTITION(c='abcdef')"))
+ assert(e.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+ val e1 = intercept[RuntimeException] {
+ sql(s"ALTER TABLE t PARTITION (c='abcdef') RENAME TO PARTITION (c='2')")
+ }
+ assert(e1.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ val e2 = intercept[RuntimeException] {
+ sql(s"ALTER TABLE t PARTITION (c='1') RENAME TO PARTITION (c='abcdef')")
+ }
+ assert(e2.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+ }
+ }
+
+ test("varchar type values length check: partitioned columns dynamic") {
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ checkColType(spark.table("t").schema(1), VarcharType(5))
+ }
+ }
- sql("ALTER TABLE t DROP PARTITION(c='a')")
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
+
+ withTable("t") {
Review comment:
we can remove this as it's already in other tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765262287
**[Test build #134359 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134359/testReport)** for PR 31281 at commit [`6957941`](https://github.com/apache/spark/commit/69579415e752f1b647ab1782f98b33c5c2607219).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563488269
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/PartitioningUtils.scala
##########
@@ -30,14 +35,33 @@ object PartitioningUtils {
*/
def normalizePartitionSpec[T](
partitionSpec: Map[String, T],
- partColNames: Seq[String],
+ partCols: StructType,
tblName: String,
resolver: Resolver): Map[String, T] = {
+ val rawSchema = CharVarcharUtils.getRawSchema(partCols)
val normalizedPartSpec = partitionSpec.toSeq.map { case (key, value) =>
- val normalizedKey = partColNames.find(resolver(_, key)).getOrElse {
+ val normalizedFiled = rawSchema.find(f => resolver(f.name, key)).getOrElse {
throw new AnalysisException(s"$key is not a valid partition column in table $tblName.")
}
- normalizedKey -> value
+
+ val normalizedVal = normalizedFiled.dataType match {
+ case CharType(len) if value != null && value != DEFAULT_PARTITION_NAME =>
+ val v = value match {
+ case Some(v) => Some(charTypeWriteSideCheck(v.toString, len))
+ case None => None
+ case other => charTypeWriteSideCheck(other.toString, len)
Review comment:
I think we will never reach this branch. We can either fail here, or return `other` as it is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562446776
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
##########
@@ -54,8 +51,6 @@ public static UTF8String varcharTypeWriteSideCheck(UTF8String inputStr, int limi
if (numChars <= limit) {
return inputStr;
} else {
- // Trailing spaces do not count in the length check. We need to retain the trailing spaces
- // (truncate to length N), as there is no read-time padding for varchar type.
int maxAllowedNumTailSpaces = numChars - limit;
Review comment:
sgtm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562100157
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
##########
@@ -54,8 +51,6 @@ public static UTF8String varcharTypeWriteSideCheck(UTF8String inputStr, int limi
if (numChars <= limit) {
return inputStr;
} else {
- // Trailing spaces do not count in the length check. We need to retain the trailing spaces
- // (truncate to length N), as there is no read-time padding for varchar type.
int maxAllowedNumTailSpaces = numChars - limit;
Review comment:
we can probably create a private method for the logic below to save duplicated code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766174321
**[Test build #134404 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134404/testReport)** for PR 31281 at commit [`04c6b52`](https://github.com/apache/spark/commit/04c6b52581da032d4ad242875da5cba262f1d747).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562099377
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
##########
@@ -22,28 +22,25 @@
public class CharVarcharCodegenUtils {
private static final UTF8String SPACE = UTF8String.fromString(" ");
- /**
- * Trailing spaces do not count in the length check. We don't need to retain the trailing
- * spaces, as we will pad char type columns/fields at read time.
- */
public static UTF8String charTypeWriteSideCheck(UTF8String inputStr, int limit) {
if (inputStr == null) {
return null;
} else {
- UTF8String trimmed = inputStr.trimRight();
- if (trimmed.numChars() > limit) {
- throw new RuntimeException("Exceeds char type length limitation: " + limit);
+ int numChars = inputStr.numChars();
+ if (numChars == limit) {
+ return inputStr;
+ } else if (numChars <= limit) {
+ return inputStr.rpad(limit, SPACE);
+ } else {
+ int maxAllowedNumTailSpaces = numChars - limit;
Review comment:
nit: `numTailSpacesToTrim` seems better
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
##########
@@ -54,8 +51,6 @@ public static UTF8String varcharTypeWriteSideCheck(UTF8String inputStr, int limi
if (numChars <= limit) {
return inputStr;
} else {
- // Trailing spaces do not count in the length check. We need to retain the trailing spaces
- // (truncate to length N), as there is no read-time padding for varchar type.
int maxAllowedNumTailSpaces = numChars - limit;
Review comment:
ditto
##########
File path: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java
##########
@@ -54,8 +51,6 @@ public static UTF8String varcharTypeWriteSideCheck(UTF8String inputStr, int limi
if (numChars <= limit) {
return inputStr;
} else {
- // Trailing spaces do not count in the length check. We need to retain the trailing spaces
- // (truncate to length N), as there is no read-time padding for varchar type.
int maxAllowedNumTailSpaces = numChars - limit;
Review comment:
we can probably create a private method for the logic below to save duplicated code.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -56,7 +56,7 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
checkColType(spark.table("t").schema(1), CharType(5))
- sql("ALTER TABLE t DROP PARTITION(c='a')")
+ sql("ALTER TABLE t DROP PARTITION(c='a ')")
Review comment:
is it the same with hive? or we should also add padding for char type column in partition spec?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766712019
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39011/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765594773
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563492693
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/HiveCharVarcharTestSuite.scala
##########
@@ -50,6 +51,35 @@ class HiveCharVarcharTestSuite extends CharVarcharTestSuite with TestHiveSinglet
assert(rest.contains("CHAR(5)"))
}
}
+
+ // TODO(SPARK-34203): Move this too super class when the ticket gets fixed
Review comment:
`too` -> `to`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765427989
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134367/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766406906
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766746908
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134425/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766611889
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39007/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766742322
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39017/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766394750
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38998/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765254344
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765262287
**[Test build #134359 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134359/testReport)** for PR 31281 at commit [`6957941`](https://github.com/apache/spark/commit/69579415e752f1b647ab1782f98b33c5c2607219).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r562455020
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
##########
@@ -222,16 +155,14 @@ object CharVarcharUtils extends Logging {
classOf[CharVarcharCodegenUtils],
StringType,
"charTypeWriteSideCheck",
- expr :: Literal(length) :: Nil,
- propagateNull = false)
+ expr :: Literal(length) :: Nil)
Review comment:
good point~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766406906
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765521503
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38960/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-764761422
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38923/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563493450
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/HiveCharVarcharTestSuite.scala
##########
@@ -50,6 +51,35 @@ class HiveCharVarcharTestSuite extends CharVarcharTestSuite with TestHiveSinglet
assert(rest.contains("CHAR(5)"))
}
}
+
+ // TODO(SPARK-34203): Move this too super class when the ticket gets fixed
+ test("char type values should be padded or trimmed: static partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t PARTITION (c ='$v') VALUES ('1')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
+ }
+
+ test("SPARK-34192: Know issue of hive for tailing spaces") {
Review comment:
Can we put the code snippet in the JIRA as a way to reproduce the bug, and remove this test?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563483203
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/PartitioningUtils.scala
##########
@@ -30,14 +35,34 @@ object PartitioningUtils {
*/
def normalizePartitionSpec[T](
partitionSpec: Map[String, T],
- partColNames: Seq[String],
+ partCols: StructType,
tblName: String,
resolver: Resolver): Map[String, T] = {
val normalizedPartSpec = partitionSpec.toSeq.map { case (key, value) =>
- val normalizedKey = partColNames.find(resolver(_, key)).getOrElse {
- throw new AnalysisException(s"$key is not a valid partition column in table $tblName.")
+ val normalizedFiled = CharVarcharUtils.getRawSchema(partCols)
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765253002
**[Test build #134355 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134355/testReport)** for PR 31281 at commit [`4235420`](https://github.com/apache/spark/commit/42354200aa4c6898588bbd2233ad28cfff7554a1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766497990
ping @cloud-fan @maropu @HyukjinKwon thanks for reviewing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-764711672
cc @cloud-fan @maropu thanks for reviewing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563523300
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,134 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
+
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
Review comment:
To be complete, we need to make sure we can drop the partition with `a`, `a `, ..., or `a (with 4 spaces)`. That's too many tests, and most users would just write `a` as they know the padding semantic. I think we only need to test `ALTER TABLE t DROP PARTITION(c='a')` here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765227116
**[Test build #134357 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134357/testReport)** for PR 31281 at commit [`afa3e3a`](https://github.com/apache/spark/commit/afa3e3a1113e0edf470286f35c9dfe1c9a142ff8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765427989
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134367/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765227116
**[Test build #134357 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134357/testReport)** for PR 31281 at commit [`afa3e3a`](https://github.com/apache/spark/commit/afa3e3a1113e0edf470286f35c9dfe1c9a142ff8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765189914
**[Test build #134355 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134355/testReport)** for PR 31281 at commit [`4235420`](https://github.com/apache/spark/commit/42354200aa4c6898588bbd2233ad28cfff7554a1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-767007988
This is a much simpler solution and makes sense to me: almost all databases enforce the data constraints at the write side, Spark should expect the data sources or vendors to enforce it at the write side, instead of doing read side checks.
This changes the storage of char type (because of the write side padding), which makes more sense to better preserve the char type semantic. Usually, we can't change storage afterward, but fortunately, Spark 3.1 is not released yet. It's a good chance to get this merged and fix the perf regression regarding CBO and predicate pushdown.
Thanks, merging to master/3.1!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766389446
**[Test build #134412 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134412/testReport)** for PR 31281 at commit [`4a0f6f1`](https://github.com/apache/spark/commit/4a0f6f129281e6b906c35b7335c9e6c43d614705).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-764761350
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38923/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563478821
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/PartitioningUtils.scala
##########
@@ -30,14 +35,34 @@ object PartitioningUtils {
*/
def normalizePartitionSpec[T](
partitionSpec: Map[String, T],
- partColNames: Seq[String],
+ partCols: StructType,
tblName: String,
resolver: Resolver): Map[String, T] = {
val normalizedPartSpec = partitionSpec.toSeq.map { case (key, value) =>
- val normalizedKey = partColNames.find(resolver(_, key)).getOrElse {
- throw new AnalysisException(s"$key is not a valid partition column in table $tblName.")
+ val normalizedFiled = CharVarcharUtils.getRawSchema(partCols)
Review comment:
nit: we don't need to call `getRawSchema` in the loop body. We can do `val partSchema = ...` at the beginning.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/PartitioningUtils.scala
##########
@@ -30,14 +35,33 @@ object PartitioningUtils {
*/
def normalizePartitionSpec[T](
partitionSpec: Map[String, T],
- partColNames: Seq[String],
+ partCols: StructType,
tblName: String,
resolver: Resolver): Map[String, T] = {
+ val rawSchema = CharVarcharUtils.getRawSchema(partCols)
val normalizedPartSpec = partitionSpec.toSeq.map { case (key, value) =>
- val normalizedKey = partColNames.find(resolver(_, key)).getOrElse {
+ val normalizedFiled = rawSchema.find(f => resolver(f.name, key)).getOrElse {
throw new AnalysisException(s"$key is not a valid partition column in table $tblName.")
}
- normalizedKey -> value
+
+ val normalizedVal = normalizedFiled.dataType match {
+ case CharType(len) if value != null && value != DEFAULT_PARTITION_NAME =>
+ val v = value match {
+ case Some(v) => Some(charTypeWriteSideCheck(v.toString, len))
+ case None => None
+ case other => charTypeWriteSideCheck(other.toString, len)
Review comment:
I think we will never reach this branch. We can either fail here, or return `other` as it is.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/PartitioningUtils.scala
##########
@@ -30,14 +35,33 @@ object PartitioningUtils {
*/
def normalizePartitionSpec[T](
partitionSpec: Map[String, T],
- partColNames: Seq[String],
+ partCols: StructType,
tblName: String,
resolver: Resolver): Map[String, T] = {
+ val rawSchema = CharVarcharUtils.getRawSchema(partCols)
val normalizedPartSpec = partitionSpec.toSeq.map { case (key, value) =>
- val normalizedKey = partColNames.find(resolver(_, key)).getOrElse {
+ val normalizedFiled = rawSchema.find(f => resolver(f.name, key)).getOrElse {
throw new AnalysisException(s"$key is not a valid partition column in table $tblName.")
}
- normalizedKey -> value
+
+ val normalizedVal = normalizedFiled.dataType match {
+ case CharType(len) if value != null && value != DEFAULT_PARTITION_NAME =>
+ val v = value match {
+ case Some(v) => Some(charTypeWriteSideCheck(v.toString, len))
+ case None => None
+ case other => charTypeWriteSideCheck(other.toString, len)
+ }
+ v.asInstanceOf[T]
+ case VarcharType(len) if value != null && value != DEFAULT_PARTITION_NAME =>
+ val v = value match {
+ case Some(v) => Some(varcharTypeWriteSideCheck(v.toString, len))
+ case None => None
+ case other => varcharTypeWriteSideCheck(other.toString, len)
Review comment:
ditto
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -465,13 +465,13 @@ case class AlterTableAddPartitionCommand(
override def run(sparkSession: SparkSession): Seq[Row] = {
val catalog = sparkSession.sessionState.catalog
- val table = catalog.getTableMetadata(tableName)
+ val table = catalog.getTableRawMetadata(tableName)
Review comment:
do we need to make this change? `PartitioningUtils.normalizePartitionSpec` will restore the raw schema of partition columns.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -521,19 +521,19 @@ case class AlterTableRenamePartitionCommand(
override def run(sparkSession: SparkSession): Seq[Row] = {
val catalog = sparkSession.sessionState.catalog
- val table = catalog.getTableMetadata(tableName)
+ val table = catalog.getTableRawMetadata(tableName)
Review comment:
ditto
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,109 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
+
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
}
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ val e1 = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e1.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
Review comment:
nit: we should test that we can always drop partition with `c='a'`.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,109 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
+
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
Review comment:
I think we already tested it in the length check write tests.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,109 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
+
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
}
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ val e1 = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
Review comment:
ditto
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,109 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
+
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
}
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ val e1 = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e1.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+
+ val e2 = intercept[RuntimeException](sql("ALTER TABLE t DROP PARTITION(c='abcdef')"))
+ assert(e2.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+
+ val e3 = intercept[RuntimeException](sql("ALTER TABLE t ADD PARTITION(c='abcdef')"))
+ assert(e3.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
- sql("ALTER TABLE t DROP PARTITION(c='a')")
+ test("varchar type values length check: partitioned columns dynamic") {
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ checkColType(spark.table("t").schema(1), VarcharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
Review comment:
ditto
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,109 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
+
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
}
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ val e1 = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e1.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+
+ val e2 = intercept[RuntimeException](sql("ALTER TABLE t DROP PARTITION(c='abcdef')"))
+ assert(e2.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+
+ val e3 = intercept[RuntimeException](sql("ALTER TABLE t ADD PARTITION(c='abcdef')"))
+ assert(e3.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
- sql("ALTER TABLE t DROP PARTITION(c='a')")
+ test("varchar type values length check: partitioned columns dynamic") {
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ checkColType(spark.table("t").schema(1), VarcharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val e = intercept[RuntimeException](sql("ALTER TABLE t DROP PARTITION(c='abcdef')"))
Review comment:
how about ADD PARTITION? can we have one test case for ADD/RENAME/DROP PARTITION with too-long partition values?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,109 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
+
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
}
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ val e1 = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e1.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+
+ val e2 = intercept[RuntimeException](sql("ALTER TABLE t DROP PARTITION(c='abcdef')"))
+ assert(e2.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+
+ val e3 = intercept[RuntimeException](sql("ALTER TABLE t ADD PARTITION(c='abcdef')"))
+ assert(e3.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
- sql("ALTER TABLE t DROP PARTITION(c='a')")
+ test("varchar type values length check: partitioned columns dynamic") {
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ checkColType(spark.table("t").schema(1), VarcharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val e = intercept[SparkException](sql("INSERT OVERWRITE t VALUES ('1', 'abcdef')"))
+ assert(e.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val e = intercept[RuntimeException](sql("ALTER TABLE t DROP PARTITION(c='abcdef')"))
+ assert(e.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
}
}
+ test("varchar type values length check: partitioned columns of other types") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ Seq(1, 10, 100, 1000, 10000).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', $v)")
+ sql(s"ALTER TABLE t DROP PARTITION(c=$v)")
+ checkAnswer(spark.table("t"), Nil)
+ }
+
+ val e1 = intercept[SparkException](sql(s"INSERT OVERWRITE t VALUES ('1', 100000)"))
+ assert(e1.getCause.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+
+ val e2 = intercept[RuntimeException](sql("ALTER TABLE t DROP PARTITION(c='100000')"))
Review comment:
this is not "other types"
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -988,7 +989,15 @@ private[hive] class HiveClientImpl(
private[hive] object HiveClientImpl extends Logging {
/** Converts the native StructField to Hive's FieldSchema. */
def toHiveColumn(c: StructField): FieldSchema = {
- val typeString = HiveVoidType.replaceVoidType(c.dataType).catalogString
+ val typeString = if (c.metadata.contains(CHAR_VARCHAR_TYPE_STRING_METADATA_KEY)) {
Review comment:
where do we hide the char/varchar from schema before reaching here?
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/HiveCharVarcharTestSuite.scala
##########
@@ -50,6 +51,35 @@ class HiveCharVarcharTestSuite extends CharVarcharTestSuite with TestHiveSinglet
assert(rest.contains("CHAR(5)"))
}
}
+
+ // TODO(SPARK-34203): Move this too super class when the ticket gets fixed
Review comment:
`too` -> `to`
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/HiveCharVarcharTestSuite.scala
##########
@@ -50,6 +51,35 @@ class HiveCharVarcharTestSuite extends CharVarcharTestSuite with TestHiveSinglet
assert(rest.contains("CHAR(5)"))
}
}
+
+ // TODO(SPARK-34203): Move this too super class when the ticket gets fixed
Review comment:
BTW the jira is about null partition value, while this test doesn't cover null partition at all.
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/HiveCharVarcharTestSuite.scala
##########
@@ -50,6 +51,35 @@ class HiveCharVarcharTestSuite extends CharVarcharTestSuite with TestHiveSinglet
assert(rest.contains("CHAR(5)"))
}
}
+
+ // TODO(SPARK-34203): Move this too super class when the ticket gets fixed
+ test("char type values should be padded or trimmed: static partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t PARTITION (c ='$v') VALUES ('1')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
+ }
+
+ test("SPARK-34192: Know issue of hive for tailing spaces") {
Review comment:
Can we put the code snippet in the JIRA as a way to reproduce the bug, and remove this test?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/util/PartitioningUtils.scala
##########
@@ -30,14 +35,33 @@ object PartitioningUtils {
*/
def normalizePartitionSpec[T](
partitionSpec: Map[String, T],
- partColNames: Seq[String],
+ partCols: StructType,
tblName: String,
resolver: Resolver): Map[String, T] = {
+ val rawSchema = CharVarcharUtils.getRawSchema(partCols)
val normalizedPartSpec = partitionSpec.toSeq.map { case (key, value) =>
- val normalizedKey = partColNames.find(resolver(_, key)).getOrElse {
+ val normalizedFiled = rawSchema.find(f => resolver(f.name, key)).getOrElse {
throw new AnalysisException(s"$key is not a valid partition column in table $tblName.")
}
- normalizedKey -> value
+
+ val normalizedVal = normalizedFiled.dataType match {
+ case CharType(len) if value != null && value != DEFAULT_PARTITION_NAME =>
+ val v = value match {
+ case Some(v) => Some(charTypeWriteSideCheck(v.toString, len))
Review comment:
nit:
```
case Some(str: String) => ...
case str: String => ...
case other => other
```
Then we don't need the `v.asInstanceOf[T]` at the end.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,134 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
+
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
Review comment:
To be complete, we need to make sure we can drop the partition with `a`, `a `, ..., or `a `. That's too many tests, and most users would just write `a` as they know the padding semantic. I think we only need to test `ALTER TABLE t DROP PARTITION(c='a')` here.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,134 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
+
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
Review comment:
To be complete, we need to make sure we can drop the partition with `a`, `a `, ..., or `a (with 4 spaces)`. That's too many tests, and most users would just write `a` as they know the padding semantic. I think we only need to test `ALTER TABLE t DROP PARTITION(c='a')` here.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,134 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
+
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
}
+
+ withTable("t") {
Review comment:
If we change https://github.com/apache/spark/pull/31281/files#r563523300, we can remove this.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,134 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
+
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
}
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ sql(s"ALTER TABLE t DROP PARTITION(c='a')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: static partitioned columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t PARTITION (c ='$v') VALUES ('1')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
Review comment:
ditto, we only need to test `PARTITION(c='a')`
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,134 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
+
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
}
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ sql(s"ALTER TABLE t DROP PARTITION(c='a')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: static partitioned columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t PARTITION (c ='$v') VALUES ('1')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
+ }
+
+ test("oversize char/varchar values for alter table partition operations") {
+ Seq("CHAR(5)", "VARCHAR(5)").foreach { typ =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c $typ) USING $format PARTITIONED BY (c)")
+ Seq("ADD", "DROP").foreach { op =>
+ val e = intercept[RuntimeException](sql(s"ALTER TABLE t $op PARTITION(c='abcdef')"))
+ assert(e.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+ val e1 = intercept[RuntimeException] {
+ sql(s"ALTER TABLE t PARTITION (c='abcdef') RENAME TO PARTITION (c='2')")
+ }
+ assert(e1.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ val e2 = intercept[RuntimeException] {
+ sql(s"ALTER TABLE t PARTITION (c='1') RENAME TO PARTITION (c='abcdef')")
+ }
+ assert(e2.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+ }
+ }
+
+ test("varchar type values length check: partitioned columns dynamic") {
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ checkColType(spark.table("t").schema(1), VarcharType(5))
+ }
+ }
- sql("ALTER TABLE t DROP PARTITION(c='a')")
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
+
+ withTable("t") {
Review comment:
we can remove this as it's already in other tests
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -37,31 +37,134 @@ trait CharVarcharTestSuite extends QueryTest with SQLTestUtils {
assert(CharVarcharUtils.getRawType(f.metadata) == Some(dt))
}
- test("char type values should be padded: top-level columns") {
+ test("char type values should be padded or trimmed: top-level columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+
+ sql("INSERT OVERWRITE t VALUES ('1', null)")
+ checkAnswer(spark.table("t"), Row("1", null))
+ }
+ }
+
+ test("char type values should be padded or trimmed: partitioned columns") {
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT OVERWRITE t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
}
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ sql(s"ALTER TABLE t DROP PARTITION(c='a')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
}
- test("char type values should be padded: partitioned columns") {
+ test("char type values should be padded or trimmed: static partitioned columns") {
withTable("t") {
sql(s"CREATE TABLE t(i STRING, c CHAR(5)) USING $format PARTITIONED BY (c)")
- sql("INSERT INTO t VALUES ('1', 'a')")
- checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
- checkColType(spark.table("t").schema(1), CharType(5))
+ (0 to 5).map(n => "a" + " " * n).foreach { v =>
+ sql(s"INSERT INTO t PARTITION (c ='$v') VALUES ('1')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * 4))
+ checkColType(spark.table("t").schema(1), CharType(5))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
+ }
+
+ test("oversize char/varchar values for alter table partition operations") {
+ Seq("CHAR(5)", "VARCHAR(5)").foreach { typ =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c $typ) USING $format PARTITIONED BY (c)")
+ Seq("ADD", "DROP").foreach { op =>
+ val e = intercept[RuntimeException](sql(s"ALTER TABLE t $op PARTITION(c='abcdef')"))
+ assert(e.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+ val e1 = intercept[RuntimeException] {
+ sql(s"ALTER TABLE t PARTITION (c='abcdef') RENAME TO PARTITION (c='2')")
+ }
+ assert(e1.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ val e2 = intercept[RuntimeException] {
+ sql(s"ALTER TABLE t PARTITION (c='1') RENAME TO PARTITION (c='abcdef')")
+ }
+ assert(e2.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
+ }
+ }
+ }
+
+ test("varchar type values length check: partitioned columns dynamic") {
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ checkColType(spark.table("t").schema(1), VarcharType(5))
+ }
+ }
- sql("ALTER TABLE t DROP PARTITION(c='a')")
+ (0 to 5).foreach { n =>
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val v = "a" + " " * n
+ sql(s"INSERT INTO t VALUES ('1', '$v')")
+ checkAnswer(spark.table("t"), Row("1", "a" + " " * math.min(n, 4)))
+ sql(s"ALTER TABLE t DROP PARTITION(c='$v')")
+ checkAnswer(spark.table("t"), Nil)
+ }
+ }
+
+ withTable("t") {
+ sql(s"CREATE TABLE t(i STRING, c VARCHAR(5)) USING $format PARTITIONED BY (c)")
+ val e = intercept[RuntimeException](sql("ALTER TABLE t DROP PARTITION(c='abcdef')"))
+ assert(e.getMessage.contains("Exceeds char/varchar type length limitation: 5"))
sql("INSERT OVERWRITE t VALUES ('1', null)")
checkAnswer(spark.table("t"), Row("1", null))
}
}
+ test("varchar type values length check: partitioned columns of other types") {
Review comment:
can we test both char and varchar?
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -988,7 +989,15 @@ private[hive] class HiveClientImpl(
private[hive] object HiveClientImpl extends Logging {
/** Converts the native StructField to Hive's FieldSchema. */
def toHiveColumn(c: StructField): FieldSchema = {
- val typeString = HiveVoidType.replaceVoidType(c.dataType).catalogString
+ val typeString = if (c.metadata.contains(CHAR_VARCHAR_TYPE_STRING_METADATA_KEY)) {
+ // For Hive Serde, we still need to to restore the raw type for char and varchar type.
+ // When reading data in parquet, orc, or avro file format with string type for char,
+ // the tailing spaces may lost if we are not going to pad it.
+ c.metadata.getString(CHAR_VARCHAR_TYPE_STRING_METADATA_KEY)
Review comment:
we can add a `getRawTypeString` in `CharVarcharUtils`, instead of exposing `CHAR_VARCHAR_TYPE_STRING_METADATA_KEY`?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
##########
@@ -31,6 +31,8 @@ object CharVarcharUtils extends Logging {
private val CHAR_VARCHAR_TYPE_STRING_METADATA_KEY = "__CHAR_VARCHAR_TYPE_STRING"
+ def getRawTypeString: String = CHAR_VARCHAR_TYPE_STRING_METADATA_KEY
Review comment:
it should be `def getRawTypeString(metadata: Metadata): Option[String] = ...`
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
##########
@@ -31,6 +31,8 @@ object CharVarcharUtils extends Logging {
private val CHAR_VARCHAR_TYPE_STRING_METADATA_KEY = "__CHAR_VARCHAR_TYPE_STRING"
+ def getRawTypeString: String = CHAR_VARCHAR_TYPE_STRING_METADATA_KEY
Review comment:
and `getRawType` can call this method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31281:
URL: https://github.com/apache/spark/pull/31281#discussion_r563726969
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
##########
@@ -31,6 +31,8 @@ object CharVarcharUtils extends Logging {
private val CHAR_VARCHAR_TYPE_STRING_METADATA_KEY = "__CHAR_VARCHAR_TYPE_STRING"
+ def getRawTypeString: String = CHAR_VARCHAR_TYPE_STRING_METADATA_KEY
Review comment:
it should be `def getRawTypeString(metadata: Metadata): Option[String] = ...`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-766758670
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39017/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31281: [SPARK-34192][SQL] Move char padding to write side and remove length check on read side too
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31281:
URL: https://github.com/apache/spark/pull/31281#issuecomment-765328417
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38945/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org