You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/07 00:43:04 UTC
[GitHub] [spark] sarutak commented on a change in pull request #32949: [SPARK-35749][SPARK-35773][SQL] Parse unit list interval literals as tightest year-month/day-time interval types
sarutak commented on a change in pull request #32949:
URL: https://github.com/apache/spark/pull/32949#discussion_r664961224
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
##########
@@ -2526,10 +2526,10 @@ object DatePart {
224
> SELECT _FUNC_('SECONDS', timestamp'2019-10-01 00:00:01.000001');
1.000001
- > SELECT _FUNC_('days', interval 1 year 10 months 5 days);
+ > SELECT _FUNC_('days', interval 5 days 3 hours 7 minutes);
5
- > SELECT _FUNC_('seconds', interval 5 hours 30 seconds 1 milliseconds 1 microseconds);
Review comment:
`milliseconds` and `microseconds` are not valid ANSI interval fields are they?
##########
File path: docs/sql-migration-guide.md
##########
@@ -83,7 +83,9 @@ license: |
- In Spark 3.2, `TRANSFORM` operator can support `ArrayType/MapType/StructType` without Hive SerDe, in this mode, we use `StructsToJosn` to convert `ArrayType/MapType/StructType` column to `STRING` and use `JsonToStructs` to parse `STRING` to `ArrayType/MapType/StructType`. In Spark 3.1, Spark just support case `ArrayType/MapType/StructType` column as `STRING` but can't support parse `STRING` to `ArrayType/MapType/StructType` output columns.
- - In Spark 3.2, the unit-to-unit interval literals like `INTERVAL '1-1' YEAR TO MONTH` are converted to ANSI interval types: `YearMonthIntervalType` or `DayTimeIntervalType`. In Spark 3.1 and earlier, such interval literals are converted to `CalendarIntervalType`. To restore the behavior before Spark 3.2, you can set `spark.sql.legacy.interval.enabled` to `true`.
+ - In Spark 3.2, the unit-to-unit interval literals like `INTERVAL '1-1' YEAR TO MONTH` and the unit list interval literals like `INTERVAL '3' DAYS '1' HOUR` are converted to ANSI interval types: `YearMonthIntervalType` or `DayTimeIntervalType`. In Spark 3.1 and earlier, such interval literals are converted to `CalendarIntervalType`. To restore the behavior before Spark 3.2, you can set `spark.sql.legacy.interval.enabled` to `true`.
+
+ - In Spark 3.2, the unit list interval literals support year-month units (YEAR and MONTH), day-time units (DAY, HOUR, MINUTE and SECOND) and the others (WEEK, MILLISECOND and MICROSECOND). Within an interval literal, multiple units are allowed to be listed as long as all of the units belong to the same category like `5 YEARS 8 MONTHS` or `1 DAY 2 MINUTE`. In Spark 3.1 and earlier, any units are allowed to be listed within a literal. To restore the behavior before Spark 3.2, you can set `spark.sql.legacy.interval.enabled` to `true`.
Review comment:
Thank you for the suggestion.
BTW, `WEEK`, `MILLISECOND` and `MICROSECOND` are not valid ANSI interval units are they?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org