You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/26 03:43:25 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
AngersZhuuuu opened a new pull request #32343:
URL: https://github.com/apache/spark/pull/32343
### What changes were proposed in this pull request?
Add note in migration guide about DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
### Why are the changes needed?
Add note
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Not need
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826757435
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137954/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826494988
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826511251
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42454/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826738910
**[Test build #137954 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137954/testReport)** for PR 32343 at commit [`e3650b2`](https://github.com/apache/spark/commit/e3650b2ce80640b334c96e529972347d52202dc7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826779799
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42475/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32343:
URL: https://github.com/apache/spark/pull/32343#discussion_r620288411
##########
File path: docs/sql-migration-guide.md
##########
@@ -83,6 +83,8 @@ license: |
- In Spark 3.2, the unit-to-unit interval literals like `INTERVAL '1-1' YEAR TO MONTH` are converted to ANSI interval types: `YearMonthIntervalType` or `DayTimeIntervalType`. In Spark 3.1 and earlier, such interval literals are converted to `CalendarIntervalType`. To restore the behavior before Spark 3.2, you can set `spark.sql.legacy.interval.enabled` to `true`.
+ - In Spark 3.2, Spark supports `DayTimeIntervalType` and `YearMonthIntervalType` as inputs and outputs of `TRANSFORM` clause in Hive `SERDE` mode, the behavior is different between Hive `SERDE` mode and `ROW FORMAT DELIMITED` mode when this two types are used as inputs. In Hive `SERDE` mode, `DayTimeIntervalType` column is converted to `HiveIntervalDayTime`, its string format is `[+|-]?d h:m:s.n`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?d h:m:s.n' DAY TO TIME`. In Hive `SERDE` mode, `YearMonthIntervalType` column is converted to `HiveIntervalYearMonth`, its string format is `[+|-]?y-m`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?y-m' YEAR TO MONTH`.
Review comment:
> Sorry, one more iteration:
> this two types -> these two types
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32343:
URL: https://github.com/apache/spark/pull/32343#discussion_r620288676
##########
File path: docs/sql-migration-guide.md
##########
@@ -83,6 +83,8 @@ license: |
- In Spark 3.2, the unit-to-unit interval literals like `INTERVAL '1-1' YEAR TO MONTH` are converted to ANSI interval types: `YearMonthIntervalType` or `DayTimeIntervalType`. In Spark 3.1 and earlier, such interval literals are converted to `CalendarIntervalType`. To restore the behavior before Spark 3.2, you can set `spark.sql.legacy.interval.enabled` to `true`.
+ - In Spark 3.2, Spark supports `DayTimeIntervalType` and `YearMonthIntervalType` as inputs and outputs of `TRANSFORM` clause in Hive `SERDE` mode, the behavior is different between Hive `SERDE` mode and `ROW FORMAT DELIMITED` mode when this two types are used as inputs. In Hive `SERDE` mode, `DayTimeIntervalType` column is converted to `HiveIntervalDayTime`, its string format is `[+|-]?d h:m:s.n`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?d h:m:s.n' DAY TO TIME`. In Hive `SERDE` mode, `YearMonthIntervalType` column is converted to `HiveIntervalYearMonth`, its string format is `[+|-]?y-m`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?y-m' YEAR TO MONTH`.
Review comment:
> `INTERVAL [-|+]?'[-|+]?d h:m:s.n' DAY TO TIME`
> `INTERVAL [-|+]?'[-|+]?y-m' YEAR TO MONTH`
>
> I guess the sign is inside of `''`, and the string literal cannot have `+`.
current may be more accurate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826834127
**[Test build #137959 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137959/testReport)** for PR 32343 at commit [`79f005f`](https://github.com/apache/spark/commit/79f005fc6650c0a562464e9d506e7c672fe1fd55).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826842954
**[Test build #137959 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137959/testReport)** for PR 32343 at commit [`79f005f`](https://github.com/apache/spark/commit/79f005fc6650c0a562464e9d506e7c672fe1fd55).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826490914
**[Test build #137933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137933/testReport)** for PR 32343 at commit [`93825df`](https://github.com/apache/spark/commit/93825df81399e2851096f44fc83c52bfab99403f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826494988
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32343:
URL: https://github.com/apache/spark/pull/32343#discussion_r620250581
##########
File path: docs/sql-migration-guide.md
##########
@@ -83,6 +83,8 @@ license: |
- In Spark 3.2, the unit-to-unit interval literals like `INTERVAL '1-1' YEAR TO MONTH` are converted to ANSI interval types: `YearMonthIntervalType` or `DayTimeIntervalType`. In Spark 3.1 and earlier, such interval literals are converted to `CalendarIntervalType`. To restore the behavior before Spark 3.2, you can set `spark.sql.legacy.interval.enabled` to `true`.
+ - In Spark 3.2, Spark supports `DayTimeIntervalType` and `YearMonthIntervalType` as inputs and outputs of `TRANSFORM` clause in Hive `SERDE` mode, the behavior is different between Hive `SERDE` mode and `ROW FORMAT DELIMITED` mode when this two types are used as inputs. In Hive `SERDE` mode, `DayTimeIntervalType` column is converted to `HiveIntervalDayTime`, it's string format is `[+|-]?d h:m:s.n`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?d h:m:s.n' DAY TO TIME`. In Hive `SERDE` mode, `YearMonthIntervalType` column is converted to `HiveIntervalYearMonth`, it's string format is `[+|-]?y-m`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?y-m' YEAR TO MONTH`.
Review comment:
it's string format is `[+|-]?y-m` -> its string format is `[+|-]?y-m`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826494988
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137933/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32343:
URL: https://github.com/apache/spark/pull/32343#discussion_r620249733
##########
File path: docs/sql-migration-guide.md
##########
@@ -83,6 +83,8 @@ license: |
- In Spark 3.2, the unit-to-unit interval literals like `INTERVAL '1-1' YEAR TO MONTH` are converted to ANSI interval types: `YearMonthIntervalType` or `DayTimeIntervalType`. In Spark 3.1 and earlier, such interval literals are converted to `CalendarIntervalType`. To restore the behavior before Spark 3.2, you can set `spark.sql.legacy.interval.enabled` to `true`.
+ - In Spark 3.2, Spark supports `DayTimeIntervalType` and `YearMonthIntervalType` as inputs and outputs of `TRANSFORM` clause in Hive `SERDE` mode, the behavior is different between Hive `SERDE` mode and `ROW FORMAT DELIMITED` mode when this two types are used as inputs. In Hive `SERDE` mode, `DayTimeIntervalType` column is converted to `HiveIntervalDayTime`, it's string format is `[+|-]?d h:m:s.n`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?d h:m:s.n' DAY TO TIME`. In Hive `SERDE` mode, `YearMonthIntervalType` column is converted to `HiveIntervalYearMonth`, it's string format is `[+|-]?y-m`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?y-m' YEAR TO MONTH`.
Review comment:
`it's string format is `[+|-]?d h:m:s.n` -> `its string format is `[+|-]?d h:m:s.n``
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826833555
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32343:
URL: https://github.com/apache/spark/pull/32343#discussion_r620159386
##########
File path: docs/sql-migration-guide.md
##########
@@ -83,6 +83,8 @@ license: |
- In Spark 3.2, the unit-to-unit interval literals like `INTERVAL '1-1' YEAR TO MONTH` are converted to ANSI interval types: `YearMonthIntervalType` or `DayTimeIntervalType`. In Spark 3.1 and earlier, such interval literals are converted to `CalendarIntervalType`. To restore the behavior before Spark 3.2, you can set `spark.sql.legacy.interval.enabled` to `true`.
+ - In Spark 3.2, Spark support `DayTimeIntervalType` and `YearMonthIntervalType` as inputs and outputs of `TRANSFORM` clause in Hive `SERDE` mode, the behavior is different between Hive `SERDE` mode and `ROW FORMAT DELIMITED` mode when this two types as inputs. In Hive `SERDE` mode, `DayTimeIntervalType` column is converting to `HiveIntervalDayTime`, it's string format is `[+|-]?d h:m:s.n`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?d h:m:s.n' DAY TO TIME`. In Hive `SERDE` mode, `YearMonthIntervalType` column is converting to `HiveIntervalYearMonth`, it's string format is `[+|-]?y-m`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?y-m' YEAR TO MONTH`.
Review comment:
Spark support -> Spark support**s**
##########
File path: docs/sql-migration-guide.md
##########
@@ -83,6 +83,8 @@ license: |
- In Spark 3.2, the unit-to-unit interval literals like `INTERVAL '1-1' YEAR TO MONTH` are converted to ANSI interval types: `YearMonthIntervalType` or `DayTimeIntervalType`. In Spark 3.1 and earlier, such interval literals are converted to `CalendarIntervalType`. To restore the behavior before Spark 3.2, you can set `spark.sql.legacy.interval.enabled` to `true`.
+ - In Spark 3.2, Spark support `DayTimeIntervalType` and `YearMonthIntervalType` as inputs and outputs of `TRANSFORM` clause in Hive `SERDE` mode, the behavior is different between Hive `SERDE` mode and `ROW FORMAT DELIMITED` mode when this two types as inputs. In Hive `SERDE` mode, `DayTimeIntervalType` column is converting to `HiveIntervalDayTime`, it's string format is `[+|-]?d h:m:s.n`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?d h:m:s.n' DAY TO TIME`. In Hive `SERDE` mode, `YearMonthIntervalType` column is converting to `HiveIntervalYearMonth`, it's string format is `[+|-]?y-m`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?y-m' YEAR TO MONTH`.
Review comment:
when this two types as inputs -> when this two types are used as inputs
##########
File path: docs/sql-migration-guide.md
##########
@@ -83,6 +83,8 @@ license: |
- In Spark 3.2, the unit-to-unit interval literals like `INTERVAL '1-1' YEAR TO MONTH` are converted to ANSI interval types: `YearMonthIntervalType` or `DayTimeIntervalType`. In Spark 3.1 and earlier, such interval literals are converted to `CalendarIntervalType`. To restore the behavior before Spark 3.2, you can set `spark.sql.legacy.interval.enabled` to `true`.
+ - In Spark 3.2, Spark support `DayTimeIntervalType` and `YearMonthIntervalType` as inputs and outputs of `TRANSFORM` clause in Hive `SERDE` mode, the behavior is different between Hive `SERDE` mode and `ROW FORMAT DELIMITED` mode when this two types as inputs. In Hive `SERDE` mode, `DayTimeIntervalType` column is converting to `HiveIntervalDayTime`, it's string format is `[+|-]?d h:m:s.n`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?d h:m:s.n' DAY TO TIME`. In Hive `SERDE` mode, `YearMonthIntervalType` column is converting to `HiveIntervalYearMonth`, it's string format is `[+|-]?y-m`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?y-m' YEAR TO MONTH`.
Review comment:
column is converting to -> column is converted to
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826480244
FYI @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826871540
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826895249
+1, LGTM. Merging to master.
Thank you, @AngersZhuuuu .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32343:
URL: https://github.com/apache/spark/pull/32343#discussion_r620258575
##########
File path: docs/sql-migration-guide.md
##########
@@ -83,6 +83,8 @@ license: |
- In Spark 3.2, the unit-to-unit interval literals like `INTERVAL '1-1' YEAR TO MONTH` are converted to ANSI interval types: `YearMonthIntervalType` or `DayTimeIntervalType`. In Spark 3.1 and earlier, such interval literals are converted to `CalendarIntervalType`. To restore the behavior before Spark 3.2, you can set `spark.sql.legacy.interval.enabled` to `true`.
+ - In Spark 3.2, Spark supports `DayTimeIntervalType` and `YearMonthIntervalType` as inputs and outputs of `TRANSFORM` clause in Hive `SERDE` mode, the behavior is different between Hive `SERDE` mode and `ROW FORMAT DELIMITED` mode when this two types are used as inputs. In Hive `SERDE` mode, `DayTimeIntervalType` column is converted to `HiveIntervalDayTime`, it's string format is `[+|-]?d h:m:s.n`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?d h:m:s.n' DAY TO TIME`. In Hive `SERDE` mode, `YearMonthIntervalType` column is converted to `HiveIntervalYearMonth`, it's string format is `[+|-]?y-m`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?y-m' YEAR TO MONTH`.
Review comment:
DOne
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826877776
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42481/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826728096
**[Test build #137954 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137954/testReport)** for PR 32343 at commit [`e3650b2`](https://github.com/apache/spark/commit/e3650b2ce80640b334c96e529972347d52202dc7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32343:
URL: https://github.com/apache/spark/pull/32343#discussion_r620266749
##########
File path: docs/sql-migration-guide.md
##########
@@ -83,6 +83,8 @@ license: |
- In Spark 3.2, the unit-to-unit interval literals like `INTERVAL '1-1' YEAR TO MONTH` are converted to ANSI interval types: `YearMonthIntervalType` or `DayTimeIntervalType`. In Spark 3.1 and earlier, such interval literals are converted to `CalendarIntervalType`. To restore the behavior before Spark 3.2, you can set `spark.sql.legacy.interval.enabled` to `true`.
+ - In Spark 3.2, Spark supports `DayTimeIntervalType` and `YearMonthIntervalType` as inputs and outputs of `TRANSFORM` clause in Hive `SERDE` mode, the behavior is different between Hive `SERDE` mode and `ROW FORMAT DELIMITED` mode when this two types are used as inputs. In Hive `SERDE` mode, `DayTimeIntervalType` column is converted to `HiveIntervalDayTime`, its string format is `[+|-]?d h:m:s.n`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?d h:m:s.n' DAY TO TIME`. In Hive `SERDE` mode, `YearMonthIntervalType` column is converted to `HiveIntervalYearMonth`, its string format is `[+|-]?y-m`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?y-m' YEAR TO MONTH`.
Review comment:
INTERVAL [-|+]?'[-|+]?d h:m:s.n' DAY TO TIME
INTERVAL [-|+]?'[-|+]?y-m' YEAR TO MONTH
I guess the sign is inside of `''`, and the string literal cannot have `+`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826833603
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42479/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32343:
URL: https://github.com/apache/spark/pull/32343#discussion_r620178072
##########
File path: docs/sql-migration-guide.md
##########
@@ -83,6 +83,8 @@ license: |
- In Spark 3.2, the unit-to-unit interval literals like `INTERVAL '1-1' YEAR TO MONTH` are converted to ANSI interval types: `YearMonthIntervalType` or `DayTimeIntervalType`. In Spark 3.1 and earlier, such interval literals are converted to `CalendarIntervalType`. To restore the behavior before Spark 3.2, you can set `spark.sql.legacy.interval.enabled` to `true`.
+ - In Spark 3.2, Spark support `DayTimeIntervalType` and `YearMonthIntervalType` as inputs and outputs of `TRANSFORM` clause in Hive `SERDE` mode, the behavior is different between Hive `SERDE` mode and `ROW FORMAT DELIMITED` mode when this two types as inputs. In Hive `SERDE` mode, `DayTimeIntervalType` column is converting to `HiveIntervalDayTime`, it's string format is `[+|-]?d h:m:s.n`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?d h:m:s.n' DAY TO TIME`. In Hive `SERDE` mode, `YearMonthIntervalType` column is converting to `HiveIntervalYearMonth`, it's string format is `[+|-]?y-m`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?y-m' YEAR TO MONTH`.
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826494988
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137933/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826833603
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42479/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826490914
**[Test build #137933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137933/testReport)** for PR 32343 at commit [`93825df`](https://github.com/apache/spark/commit/93825df81399e2851096f44fc83c52bfab99403f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32343:
URL: https://github.com/apache/spark/pull/32343#discussion_r620249733
##########
File path: docs/sql-migration-guide.md
##########
@@ -83,6 +83,8 @@ license: |
- In Spark 3.2, the unit-to-unit interval literals like `INTERVAL '1-1' YEAR TO MONTH` are converted to ANSI interval types: `YearMonthIntervalType` or `DayTimeIntervalType`. In Spark 3.1 and earlier, such interval literals are converted to `CalendarIntervalType`. To restore the behavior before Spark 3.2, you can set `spark.sql.legacy.interval.enabled` to `true`.
+ - In Spark 3.2, Spark supports `DayTimeIntervalType` and `YearMonthIntervalType` as inputs and outputs of `TRANSFORM` clause in Hive `SERDE` mode, the behavior is different between Hive `SERDE` mode and `ROW FORMAT DELIMITED` mode when this two types are used as inputs. In Hive `SERDE` mode, `DayTimeIntervalType` column is converted to `HiveIntervalDayTime`, it's string format is `[+|-]?d h:m:s.n`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?d h:m:s.n' DAY TO TIME`. In Hive `SERDE` mode, `YearMonthIntervalType` column is converted to `HiveIntervalYearMonth`, it's string format is `[+|-]?y-m`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?y-m' YEAR TO MONTH`.
Review comment:
it's string format is `[+|-]?d h:m:s.n` -> its string format is `[+|-]?d h:m:s.n`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826793018
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42475/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk closed pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
MaxGekk closed pull request #32343:
URL: https://github.com/apache/spark/pull/32343
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826775751
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42475/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826490914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826490914
**[Test build #137933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137933/testReport)** for PR 32343 at commit [`93825df`](https://github.com/apache/spark/commit/93825df81399e2851096f44fc83c52bfab99403f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826728096
**[Test build #137954 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137954/testReport)** for PR 32343 at commit [`e3650b2`](https://github.com/apache/spark/commit/e3650b2ce80640b334c96e529972347d52202dc7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32343:
URL: https://github.com/apache/spark/pull/32343#discussion_r620266749
##########
File path: docs/sql-migration-guide.md
##########
@@ -83,6 +83,8 @@ license: |
- In Spark 3.2, the unit-to-unit interval literals like `INTERVAL '1-1' YEAR TO MONTH` are converted to ANSI interval types: `YearMonthIntervalType` or `DayTimeIntervalType`. In Spark 3.1 and earlier, such interval literals are converted to `CalendarIntervalType`. To restore the behavior before Spark 3.2, you can set `spark.sql.legacy.interval.enabled` to `true`.
+ - In Spark 3.2, Spark supports `DayTimeIntervalType` and `YearMonthIntervalType` as inputs and outputs of `TRANSFORM` clause in Hive `SERDE` mode, the behavior is different between Hive `SERDE` mode and `ROW FORMAT DELIMITED` mode when this two types are used as inputs. In Hive `SERDE` mode, `DayTimeIntervalType` column is converted to `HiveIntervalDayTime`, its string format is `[+|-]?d h:m:s.n`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?d h:m:s.n' DAY TO TIME`. In Hive `SERDE` mode, `YearMonthIntervalType` column is converted to `HiveIntervalYearMonth`, its string format is `[+|-]?y-m`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?y-m' YEAR TO MONTH`.
Review comment:
`INTERVAL [-|+]?'[-|+]?d h:m:s.n' DAY TO TIME`
`INTERVAL [-|+]?'[-|+]?y-m' YEAR TO MONTH`
I guess the sign is inside of `''`, and the string literal cannot have `+`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826843228
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137959/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826510299
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826757435
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137954/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826877776
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42481/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826480244
FYI @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826843228
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137959/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826793018
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42475/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826834127
**[Test build #137959 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137959/testReport)** for PR 32343 at commit [`79f005f`](https://github.com/apache/spark/commit/79f005fc6650c0a562464e9d506e7c672fe1fd55).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826494848
**[Test build #137933 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137933/testReport)** for PR 32343 at commit [`93825df`](https://github.com/apache/spark/commit/93825df81399e2851096f44fc83c52bfab99403f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32343:
URL: https://github.com/apache/spark/pull/32343#issuecomment-826511251
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42454/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32343: [SPARK-35220][DOCS][FOLLOWUP] DayTimeIntervalType/YearMonthIntervalType show different between Hive SerDe and row format delimited
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32343:
URL: https://github.com/apache/spark/pull/32343#discussion_r620265656
##########
File path: docs/sql-migration-guide.md
##########
@@ -83,6 +83,8 @@ license: |
- In Spark 3.2, the unit-to-unit interval literals like `INTERVAL '1-1' YEAR TO MONTH` are converted to ANSI interval types: `YearMonthIntervalType` or `DayTimeIntervalType`. In Spark 3.1 and earlier, such interval literals are converted to `CalendarIntervalType`. To restore the behavior before Spark 3.2, you can set `spark.sql.legacy.interval.enabled` to `true`.
+ - In Spark 3.2, Spark supports `DayTimeIntervalType` and `YearMonthIntervalType` as inputs and outputs of `TRANSFORM` clause in Hive `SERDE` mode, the behavior is different between Hive `SERDE` mode and `ROW FORMAT DELIMITED` mode when this two types are used as inputs. In Hive `SERDE` mode, `DayTimeIntervalType` column is converted to `HiveIntervalDayTime`, its string format is `[+|-]?d h:m:s.n`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?d h:m:s.n' DAY TO TIME`. In Hive `SERDE` mode, `YearMonthIntervalType` column is converted to `HiveIntervalYearMonth`, its string format is `[+|-]?y-m`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL [-|+]?'[-|+]?y-m' YEAR TO MONTH`.
Review comment:
Sorry, one more iteration:
this two types -> these two types
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org