You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/16 10:38:55 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #34616: [SPARK-37344][SQL] spark-sql cli will keep `\` when match `\;` after spark3
AngersZhuuuu opened a new pull request #34616:
URL: https://github.com/apache/spark/pull/34616
### What changes were proposed in this pull request?
Before Spark 3, if we pass a SQL like
`select split('Spark SQL', '\;')`
to spark-sql, after process, it will actually execute
`select split('Spark SQL', ';')`
spark-sql with verbose
```
[info] 2021-11-16 18:05:34.86 - stdout> spark-sql> select split('dawdawdawd','\;');
[info] 2021-11-16 18:05:34.875 - stdout> select split('dawdawdawd',';')
```
But after Spark 3
It will execute `select split('Spark SQL', '\;')` as origin.
spark-sql with verbose
```
[info] 2021-11-16 18:05:34.86 - stdout> spark-sql> select split('dawdawdawd','\;');
[info] 2021-11-16 18:05:34.875 - stdout> select split('dawdawdawd','\;')
```
In this PR we doc this change.
### Why are the changes needed?
Update migration guide
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Not need
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-970143586
**[Test build #145276 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145276/testReport)** for PR 34616 at commit [`a9b1cf4`](https://github.com/apache/spark/commit/a9b1cf46d6808264527bee6e25eb64fdf3727ec3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-971124705
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145299/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #34616:
URL: https://github.com/apache/spark/pull/34616#discussion_r752814021
##########
File path: docs/sql-migration-guide.md
##########
@@ -338,6 +338,12 @@ license: |
- In Spark 3.0, datetime pattern letter `F` is **aligned day of week in month** that represents the concept of the count of days within the period of a week where the weeks are aligned to the start of the month. In Spark version 2.4 and earlier, it is **week of month** that represents the concept of the count of weeks within the month where weeks start on a fixed day-of-week, e.g. `2020-07-30` is 30 days (4 weeks and 2 days) after the first day of the month, so `date_format(date '2020-07-30', 'F')` returns 2 in Spark 3.0, but as a week count in Spark 2.x, it returns 5 because it locates in the 5th week of July 2020, where week one is 2020-07-01 to 07-04.
+ - In Spark version 2.4 and below, `spark-sql` interface will chop `\` when query match `\;`. In Spark 3.0, `spark-sql` interface will not chop `\` when query match `\;`. See [HIVE-15297](https://issues.apache.org/jira/browse/HIVE-15297) for more details. For example:
Review comment:
> What is the behavior of `bin/spark-shell`?
When we write
```
select split('dawdawdawd','\\\\\\\\;');
```
in the shell, it will actually execute
```
select split('dawdawdawd','\\\\;');
```
Spark2 and spark3 behavior is same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-971142968
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49769/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #34616:
URL: https://github.com/apache/spark/pull/34616
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-970224647
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49746/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-972464626
Ping @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #34616:
URL: https://github.com/apache/spark/pull/34616#discussion_r756540258
##########
File path: docs/sql-migration-guide.md
##########
@@ -338,6 +338,12 @@ license: |
- In Spark 3.0, datetime pattern letter `F` is **aligned day of week in month** that represents the concept of the count of days within the period of a week where the weeks are aligned to the start of the month. In Spark version 2.4 and earlier, it is **week of month** that represents the concept of the count of weeks within the month where weeks start on a fixed day-of-week, e.g. `2020-07-30` is 30 days (4 weeks and 2 days) after the first day of the month, so `date_format(date '2020-07-30', 'F')` returns 2 in Spark 3.0, but as a week count in Spark 2.x, it returns 5 because it locates in the 5th week of July 2020, where week one is 2020-07-01 to 07-04.
+ - In Spark version 2.4 and below, `spark-sql` interface will chop `\` when query match `\;`. In Spark 3.0, `spark-sql` interface will not chop `\` when query match `\;`. See [HIVE-15297](https://issues.apache.org/jira/browse/HIVE-15297) for more details. For example:
Review comment:
And this pr to explain the change of spark-sql, will check the inconsistent later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-970208833
do you know which change introduced this change? I think we should better make sure if it was intended or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-970294319
> do you know which pr introduced this behaviour change? I think we should better make sure if it was intended or not.
It should be caused by https://github.com/apache/hive/commit/65a65826a0d351a3d918bdb98595bdd106d37adb#diff-79277c3cfeb5bc38066fbbe2b90dcee5c870100b8b1e106d53ed0d56817bd0ee
jira ID : https://issues.apache.org/jira/browse/HIVE-15297
In origin logic
https://github.com/apache/hive/blob/b1ca4e3cdb90932189d35a1e20e02a7bd0334920/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java#L358-L368
It will directly split command by `;` then if end char is `\` then chop it.
In this pr, it change the behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-971114976
Screenshot added in PR desc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-971172222
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49769/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-971172130
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49769/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-970143586
**[Test build #145276 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145276/testReport)** for PR 34616 at commit [`a9b1cf4`](https://github.com/apache/spark/commit/a9b1cf46d6808264527bee6e25eb64fdf3727ec3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-970141412
@dongjoon-hyun @HyukjinKwon @wangyum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #34616:
URL: https://github.com/apache/spark/pull/34616#discussion_r756540118
##########
File path: docs/sql-migration-guide.md
##########
@@ -338,6 +338,12 @@ license: |
- In Spark 3.0, datetime pattern letter `F` is **aligned day of week in month** that represents the concept of the count of days within the period of a week where the weeks are aligned to the start of the month. In Spark version 2.4 and earlier, it is **week of month** that represents the concept of the count of weeks within the month where weeks start on a fixed day-of-week, e.g. `2020-07-30` is 30 days (4 weeks and 2 days) after the first day of the month, so `date_format(date '2020-07-30', 'F')` returns 2 in Spark 3.0, but as a week count in Spark 2.x, it returns 5 because it locates in the 5th week of July 2020, where week one is 2020-07-01 to 07-04.
+ - In Spark version 2.4 and below, `spark-sql` interface will chop `\` when query match `\;`. In Spark 3.0, `spark-sql` interface will not chop `\` when query match `\;`. See [HIVE-15297](https://issues.apache.org/jira/browse/HIVE-15297) for more details. For example:
Review comment:
The behavior of bin/spark-shell and bin/spark-sql is inconsisten for all the version ==, but saprk-shell's behavior is same for both spark2 and spark3
In spark-2.4.7
```
spark-sql> explain extended select split('dawdawdawd','\\\\\\\\;');
== Parsed Logical Plan ==
'Project [unresolvedalias('split(dawdawdawd, \\\;), None)]
+- OneRowRelation
== Analyzed Logical Plan ==
split(dawdawdawd, \\\;): array<string>
Project [split(dawdawdawd, \\\;) AS split(dawdawdawd, \\\;)#2]
+- OneRowRelation
== Optimized Logical Plan ==
Project [[dawdawdawd] AS split(dawdawdawd, \\\;)#2]
+- OneRowRelation
== Physical Plan ==
*(1) Project [[dawdawdawd] AS split(dawdawdawd, \\\;)#2]
+- Scan OneRowRelation[]
Time taken: 1.613 seconds, Fetched 1 row(s)
```
```
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.7-sdi-015
/_/
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_232)
Type in expressions to have them evaluated.
Type :help for more information.
scala> spark.sql("select split('dawdawdawd','\\\\\\\\;')").explain(true)
== Parsed Logical Plan ==
'Project [unresolvedalias('split(dawdawdawd, \\;), None)]
+- OneRowRelation
== Analyzed Logical Plan ==
split(dawdawdawd, \\;): array<string>
Project [split(dawdawdawd, \\;) AS split(dawdawdawd, \\;)#3]
+- OneRowRelation
== Optimized Logical Plan ==
Project [[dawdawdawd] AS split(dawdawdawd, \\;)#3]
+- OneRowRelation
== Physical Plan ==
*(1) Project [[dawdawdawd] AS split(dawdawdawd, \\;)#3]
+- Scan OneRowRelation[]
```
In spark-3.1.2
```
spark-sql> explain extended select split('dawdawdawd','\\\\\\\\;');
== Parsed Logical Plan ==
'Project [unresolvedalias('split(dawdawdawd, \\\\;), None)]
+- OneRowRelation
== Analyzed Logical Plan ==
split(dawdawdawd, \\\\;, -1): array<string>
Project [split(dawdawdawd, \\\\;, -1) AS split(dawdawdawd, \\\\;, -1)#5]
+- OneRowRelation
== Optimized Logical Plan ==
Project [[dawdawdawd] AS split(dawdawdawd, \\\\;, -1)#5]
+- OneRowRelation
== Physical Plan ==
*(1) Project [[dawdawdawd] AS split(dawdawdawd, \\\\;, -1)#5]
+- *(1) Scan OneRowRelation[]
Time taken: 2.468 seconds, Fetched 1 row(s)
spark-sql>
```
```
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.1.2-sdi-011
/_/
Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_232)
Type in expressions to have them evaluated.
Type :help for more information.
scala> spark.sql("select split('dawdawdawd','\\\\\\\\;')").explain(true)
== Parsed Logical Plan ==
'Project [unresolvedalias('split(dawdawdawd, \\;), None)]
+- OneRowRelation
== Analyzed Logical Plan ==
split(dawdawdawd, \\;, -1): array<string>
Project [split(dawdawdawd, \\;, -1) AS split(dawdawdawd, \\;, -1)#0]
+- OneRowRelation
== Optimized Logical Plan ==
Project [[dawdawdawd] AS split(dawdawdawd, \\;, -1)#0]
+- OneRowRelation
== Physical Plan ==
*(1) Project [[dawdawdawd] AS split(dawdawdawd, \\;, -1)#0]
+- *(1) Scan OneRowRelation[]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-971124705
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145299/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-970224647
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49746/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-971118641
**[Test build #145299 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145299/testReport)** for PR 34616 at commit [`3ca6e3d`](https://github.com/apache/spark/commit/3ca6e3d2b94ccee55018b644441cd9c1bd37c3f9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-973839962
@wangyum Any more suggestion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-995389792
ping @wangyum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-997564223
ping @wangyum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu removed a comment on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu removed a comment on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-995389792
ping @wangyum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-971118641
**[Test build #145299 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145299/testReport)** for PR 34616 at commit [`3ca6e3d`](https://github.com/apache/spark/commit/3ca6e3d2b94ccee55018b644441cd9c1bd37c3f9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu edited a comment on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu edited a comment on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-970141412
ping @dongjoon-hyun @HyukjinKwon @wangyum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #34616:
URL: https://github.com/apache/spark/pull/34616#discussion_r750836303
##########
File path: docs/sql-migration-guide.md
##########
@@ -338,6 +338,12 @@ license: |
- In Spark 3.0, datetime pattern letter `F` is **aligned day of week in month** that represents the concept of the count of days within the period of a week where the weeks are aligned to the start of the month. In Spark version 2.4 and earlier, it is **week of month** that represents the concept of the count of weeks within the month where weeks start on a fixed day-of-week, e.g. `2020-07-30` is 30 days (4 weeks and 2 days) after the first day of the month, so `date_format(date '2020-07-30', 'F')` returns 2 in Spark 3.0, but as a week count in Spark 2.x, it returns 5 because it locates in the 5th week of July 2020, where week one is 2020-07-01 to 07-04.
+ - In Spark version 2.4 and below, `spark-sql` interface will chop `\` when query match `\;`. In Spark 3.0, `spark-sql` interface will not chop `\` when query match `\;`. See [HIVE-15297](https://issues.apache.org/jira/browse/HIVE-15297) for more details. For example:
Review comment:
What is the behavior of `bin/spark-shell`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-1083833196
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-971124551
**[Test build #145299 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145299/testReport)** for PR 34616 at commit [`3ca6e3d`](https://github.com/apache/spark/commit/3ca6e3d2b94ccee55018b644441cd9c1bd37c3f9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-970178504
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145276/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-970178504
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145276/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-970208833
do you know which pr introduced this behaviour change? I think we should better make sure if it was intended or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-972806636
I'll leave it to @wangyum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #34616:
URL: https://github.com/apache/spark/pull/34616#discussion_r756488226
##########
File path: docs/sql-migration-guide.md
##########
@@ -338,6 +338,12 @@ license: |
- In Spark 3.0, datetime pattern letter `F` is **aligned day of week in month** that represents the concept of the count of days within the period of a week where the weeks are aligned to the start of the month. In Spark version 2.4 and earlier, it is **week of month** that represents the concept of the count of weeks within the month where weeks start on a fixed day-of-week, e.g. `2020-07-30` is 30 days (4 weeks and 2 days) after the first day of the month, so `date_format(date '2020-07-30', 'F')` returns 2 in Spark 3.0, but as a week count in Spark 2.x, it returns 5 because it locates in the 5th week of July 2020, where week one is 2020-07-01 to 07-04.
+ - In Spark version 2.4 and below, `spark-sql` interface will chop `\` when query match `\;`. In Spark 3.0, `spark-sql` interface will not chop `\` when query match `\;`. See [HIVE-15297](https://issues.apache.org/jira/browse/HIVE-15297) for more details. For example:
Review comment:
So the behavior of `bin/spark-shell` and `bin/spark-sql` is inconsistent since Spark 3.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #34616:
URL: https://github.com/apache/spark/pull/34616#discussion_r750837788
##########
File path: docs/sql-migration-guide.md
##########
@@ -338,6 +338,12 @@ license: |
- In Spark 3.0, datetime pattern letter `F` is **aligned day of week in month** that represents the concept of the count of days within the period of a week where the weeks are aligned to the start of the month. In Spark version 2.4 and earlier, it is **week of month** that represents the concept of the count of weeks within the month where weeks start on a fixed day-of-week, e.g. `2020-07-30` is 30 days (4 weeks and 2 days) after the first day of the month, so `date_format(date '2020-07-30', 'F')` returns 2 in Spark 3.0, but as a week count in Spark 2.x, it returns 5 because it locates in the 5th week of July 2020, where week one is 2020-07-01 to 07-04.
+ - In Spark version 2.4 and below, `spark-sql` interface will chop `\` when query match `\;`. In Spark 3.0, `spark-sql` interface will not chop `\` when query match `\;`. See [HIVE-15297](https://issues.apache.org/jira/browse/HIVE-15297) for more details. For example:
Review comment:
> What is the behavior of `bin/spark-shell`?
`bin/spark-shell` won't be impact by this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-971172222
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49769/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-970179549
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49746/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-970151944
**[Test build #145276 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145276/testReport)** for PR 34616 at commit [`a9b1cf4`](https://github.com/apache/spark/commit/a9b1cf46d6808264527bee6e25eb64fdf3727ec3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34616: [SPARK-37344][SQL][DOC] spark-sql cli will keep `\` when match `\;` after spark3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34616:
URL: https://github.com/apache/spark/pull/34616#issuecomment-970224611
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49746/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org