You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/10 16:11:01 UTC
[GitHub] [spark] MaxGekk opened a new pull request #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
MaxGekk opened a new pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524
### What changes were proposed in this pull request?
TODO
### Why are the changes needed?
TODO
### Does this PR introduce any user-facing change?
TODO
### How was this patch tested?
TODO
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584518459
**[Test build #118219 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118219/testReport)** for PR 27524 at commit [`93f3ae1`](https://github.com/apache/spark/commit/93f3ae1c32f893cc3cc22ae075753098440e76b5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584455589
Approach seems okay.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584517025
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584510518
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584506048
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22973/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377356168
##########
File path: sql/core/src/test/resources/test-data/bad_after_good.csv
##########
@@ -1,2 +1,2 @@
"good record",1999-08-01
-"bad record",1999-088-01
+"bad record",1999-088_01
Review comment:
I had to change this because `FastDateFormat` is not so strong, and can parse `1999-088-01`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377437638
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
##########
@@ -831,12 +831,11 @@ abstract class ToTimestamp
}
case StringType =>
val zid = ctx.addReferenceObj("zoneId", zoneId, classOf[ZoneId].getName)
- val locale = ctx.addReferenceObj("locale", Locale.US)
val tf = TimestampFormatter.getClass.getName.stripSuffix("$")
nullSafeCodeGen(ctx, ev, (string, format) => {
s"""
try {
- ${ev.value} = $tf$$.MODULE$$.apply($format.toString(), $zid, $locale)
+ ${ev.value} = $tf$$.MODULE$$.withStrongLegacy($format.toString(), $zid)
Review comment:
same question. we already have the `LENIENT_SIMPLE_DATE_FORMAT` enum.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377421367
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
##########
@@ -51,41 +52,76 @@ class Iso8601DateFormatter(
}
}
-class LegacyDateFormatter(pattern: String, locale: Locale) extends DateFormatter {
- @transient
- private lazy val format = FastDateFormat.getInstance(pattern, locale)
+trait LegacyDateFormatter extends DateFormatter {
+ def parseToDate(s: String): Date
+ def formatDate(d: Date): String
override def parse(s: String): Int = {
- val milliseconds = format.parse(s).getTime
+ val milliseconds = parseToDate(s).getTime
DateTimeUtils.millisToDays(milliseconds)
}
override def format(days: Int): String = {
val date = DateTimeUtils.toJavaDate(days)
- format.format(date)
+ formatDate(date)
}
}
+class LegacyFastDateFormatter(pattern: String, locale: Locale) extends LegacyDateFormatter {
+ @transient
+ private lazy val fdf = FastDateFormat.getInstance(pattern, locale)
+ override def parseToDate(s: String): Date = fdf.parse(s)
+ def formatDate(d: Date): String = fdf.format(d)
Review comment:
`override` seems missed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377421940
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
##########
@@ -90,47 +91,168 @@ class FractionTimestampFormatter(zoneId: ZoneId)
override protected lazy val formatter = DateTimeFormatterHelper.fractionFormatter
}
-class LegacyTimestampFormatter(
+/**
+ * The custom sub-class of `GregorianCalendar` is needed to get access to
+ * protected `fields` immediately after parsing. We cannot use
+ * the `get()` method because it performs normalization of the fraction
+ * part. Accordingly, the `MILLISECOND` field doesn't contain original value.
+ *
+ * Also this class allows to set raw value to the `MILLISECOND` field
+ * directly before formatting.
+ */
+class MicrosCalendar(tz: TimeZone, digitsInFraction: Int)
+ extends GregorianCalendar(tz, Locale.US) {
+ // Converts parsed `MILLISECOND` field to seconds fraction in microsecond precision.
+ // For example if the fraction pattern is `SSSS` then `digitsInFraction` = 4, and
+ // if the `MILLISECOND` field was parsed to `1234`.
+ def getMicros(): SQLTimestamp = {
+ // Append 6 zeros to the field: 1234 -> 1234000000
+ val d = fields(Calendar.MILLISECOND) * MICROS_PER_SECOND
+ // Take the first 6 digits from `d`: 1234000000 -> 123400
+ // The rest contains exactly `digitsInFraction`: `0000` = 10 ^ digitsInFraction
+ // So, the result is `(1234 * 1000000) / (10 ^ digitsInFraction)
+ d / Decimal.POW_10(digitsInFraction)
+ }
+
+ // Converts the seconds fraction in microsecond precision to a value
+ // that can be correctly formatted according to the specified fraction pattern.
+ // The method performs operations opposite to `getMicros()`.
+ def setMicros(micros: Long): Unit = {
+ val d = micros * Decimal.POW_10(digitsInFraction)
+ fields(Calendar.MILLISECOND) = (d / MICROS_PER_SECOND).toInt
+ }
+}
+
+/**
+ * An instance of the class is aimed to re-use many times. It contains helper objects
+ * `cal` which is reused between `parse()` and `format` invokes.
+ */
+class LegacyFastDateFormat(fastDateFormat: FastDateFormat) {
+ private val cal = new MicrosCalendar(
+ fastDateFormat.getTimeZone,
+ fastDateFormat.getPattern.count(_ == 'S'))
+
+ def parse(s: String): SQLTimestamp = {
+ cal.clear() // Clear the calendar because it can be re-used many times
+ if (!fastDateFormat.parse(s, new ParsePosition(0), cal)) {
+ throw new IllegalArgumentException(s"'$s' is an invalid timestamp")
+ }
+ val micros = cal.getMicros()
+ cal.set(Calendar.MILLISECOND, 0)
+ cal.getTimeInMillis * MICROS_PER_MILLIS + micros
+ }
+
+ def format(timestamp: SQLTimestamp): String = {
+ cal.setTimeInMillis(Math.floorDiv(timestamp, MICROS_PER_SECOND) * MILLIS_PER_SECOND)
+ cal.setMicros(Math.floorMod(timestamp, MICROS_PER_SECOND))
+ fastDateFormat.format(cal)
+ }
+}
+
+class LegacyFastTimestampFormatter(
pattern: String,
zoneId: ZoneId,
locale: Locale) extends TimestampFormatter {
- @transient private lazy val format =
+ @transient private lazy val fastDateFormat =
FastDateFormat.getInstance(pattern, TimeZone.getTimeZone(zoneId), locale)
+ @transient private lazy val cal = new MicrosCalendar(
+ fastDateFormat.getTimeZone,
+ fastDateFormat.getPattern.count(_ == 'S'))
Review comment:
I forgot. is this also ported from 2.4?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584518925
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584506097
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118213/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584389477
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22947/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584373443
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118175/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584506037
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584348963
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584348974
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118172/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584348963
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584510150
**[Test build #118215 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118215/testReport)** for PR 27524 at commit [`93f3ae1`](https://github.com/apache/spark/commit/93f3ae1c32f893cc3cc22ae075753098440e76b5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584373068
**[Test build #118175 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118175/testReport)** for PR 27524 at commit [`ab1d57f`](https://github.com/apache/spark/commit/ab1d57f1b131db20c32ac1169730d3e8f45f4698).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait LegacyDateFormatter extends DateFormatter `
* `class LegacyFastDateFormatter(pattern: String, locale: Locale) extends LegacyDateFormatter `
* `class LegacySimpleDateFormatter(pattern: String, locale: Locale) extends LegacyDateFormatter `
* `class LegacyFastTimestampFormatter(`
* `class LegacySimpleTimestampFormatter(`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584506088
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377354546
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
##########
@@ -2572,3 +2572,10 @@ class JsonV2Suite extends JsonSuite {
.sparkConf
.set(SQLConf.USE_V1_SOURCE_LIST, "")
}
+
+class JsonLegacyTimeParserSuite extends JsonSuite {
+ override protected def sparkConf: SparkConf =
+ super
+ .sparkConf
+ .set(SQLConf.LEGACY_TIME_PARSER_ENABLED, true)
+}
Review comment:
Will fix that next time
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584506081
**[Test build #118213 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118213/testReport)** for PR 27524 at commit [`ddf127d`](https://github.com/apache/spark/commit/ddf127dd148c445d7f8910af1c3aef9f3976d075).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377461627
##########
File path: sql/core/src/test/resources/test-data/bad_after_good.csv
##########
@@ -1,2 +1,2 @@
"good record",1999-08-01
-"bad record",1999-088-01
+"bad record",1999-088_01
Review comment:
Yes, I added `CSVLegacyTimeParserSuite` which runs entire `CSVSuite` with the legacy parser.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584506048
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22973/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584510518
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584373429
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584506088
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584463237
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118185/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584388899
**[Test build #118185 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118185/testReport)** for PR 27524 at commit [`566170a`](https://github.com/apache/spark/commit/566170a6d7581f013142b19fcfdd2d9ea4ebaf9f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584202424
**[Test build #118172 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118172/testReport)** for PR 27524 at commit [`38d90d8`](https://github.com/apache/spark/commit/38d90d8ca27361f3dddb858fcc4843d518bd6911).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584463237
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118185/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584505701
**[Test build #118213 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118213/testReport)** for PR 27524 at commit [`ddf127d`](https://github.com/apache/spark/commit/ddf127dd148c445d7f8910af1c3aef9f3976d075).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584389384
@cloud-fan @HyukjinKwon Please, look at the draft PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27524:
[SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as
legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377477732
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -541,7 +541,7 @@ object Decimal {
/** Maximum number of decimal digits a Long can represent */
val MAX_LONG_DIGITS = 18
- private val POW_10 = Array.tabulate[Long](MAX_LONG_DIGITS + 1)(i => math.pow(10, i).toLong)
+ val POW_10 = Array.tabulate[Long](MAX_LONG_DIGITS + 1)(i => math.pow(10, i).toLong)
Review comment:
`POW_10` is needed in the wrapper of `FastDateFormat` to support parsing/formatting in microsecond precision. Similar changes were made in Spark 2.4.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584203046
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22933/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584389477
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22947/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377357186
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
##########
@@ -90,47 +91,168 @@ class FractionTimestampFormatter(zoneId: ZoneId)
override protected lazy val formatter = DateTimeFormatterHelper.fractionFormatter
}
-class LegacyTimestampFormatter(
+/**
+ * The custom sub-class of `GregorianCalendar` is needed to get access to
+ * protected `fields` immediately after parsing. We cannot use
+ * the `get()` method because it performs normalization of the fraction
+ * part. Accordingly, the `MILLISECOND` field doesn't contain original value.
+ *
+ * Also this class allows to set raw value to the `MILLISECOND` field
+ * directly before formatting.
+ */
+class MicrosCalendar(tz: TimeZone, digitsInFraction: Int)
Review comment:
This is a copy-paste from 2.4
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584517033
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118215/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584463234
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27524: [SPARK-30788][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584574154
**[Test build #118232 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118232/testReport)** for PR 27524 at commit [`93f3ae1`](https://github.com/apache/spark/commit/93f3ae1c32f893cc3cc22ae075753098440e76b5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [SPARK-30788][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584574706
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22992/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584518931
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22979/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584202424
**[Test build #118172 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118172/testReport)** for PR 27524 at commit [`38d90d8`](https://github.com/apache/spark/commit/38d90d8ca27361f3dddb858fcc4843d518bd6911).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27524: [SPARK-30788][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584571559
jenkins, retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584505701
**[Test build #118213 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118213/testReport)** for PR 27524 at commit [`ddf127d`](https://github.com/apache/spark/commit/ddf127dd148c445d7f8910af1c3aef9f3976d075).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377463939
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
##########
@@ -831,12 +831,11 @@ abstract class ToTimestamp
}
case StringType =>
val zid = ctx.addReferenceObj("zoneId", zoneId, classOf[ZoneId].getName)
- val locale = ctx.addReferenceObj("locale", Locale.US)
val tf = TimestampFormatter.getClass.getName.stripSuffix("$")
nullSafeCodeGen(ctx, ev, (string, format) => {
s"""
try {
- ${ev.value} = $tf$$.MODULE$$.apply($format.toString(), $zid, $locale)
+ ${ev.value} = $tf$$.MODULE$$.withStrongLegacy($format.toString(), $zid)
Review comment:
> is it because SIMPLE_DATE_FORMAT is an enum, and tricky for the codegen part to reference?
Yes, let me figure out how to use Scala enum from Java
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524:
[SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as
legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584721908
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118232/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524:
[SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as
legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584721896
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377437730
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
##########
@@ -831,12 +831,11 @@ abstract class ToTimestamp
}
case StringType =>
val zid = ctx.addReferenceObj("zoneId", zoneId, classOf[ZoneId].getName)
- val locale = ctx.addReferenceObj("locale", Locale.US)
val tf = TimestampFormatter.getClass.getName.stripSuffix("$")
nullSafeCodeGen(ctx, ev, (string, format) => {
s"""
try {
- ${ev.value} = $tf$$.MODULE$$.apply($format.toString(), $zid, $locale)
+ ${ev.value} = $tf$$.MODULE$$.withStrongLegacy($format.toString(), $zid)
Review comment:
and method name `withStrongLegacy` looks confusing. We haven't decided to go legacy by here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584203034
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584518931
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22979/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584510150
**[Test build #118215 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118215/testReport)** for PR 27524 at commit [`93f3ae1`](https://github.com/apache/spark/commit/93f3ae1c32f893cc3cc22ae075753098440e76b5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584517033
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118215/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27524:
[SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as
legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584574154
**[Test build #118232 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118232/testReport)** for PR 27524 at commit [`93f3ae1`](https://github.com/apache/spark/commit/93f3ae1c32f893cc3cc22ae075753098440e76b5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524:
[SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as
legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584574696
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584373429
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [SPARK-30788][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584568392
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584296573
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22936/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584506097
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118213/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377421120
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
##########
@@ -831,12 +831,11 @@ abstract class ToTimestamp
}
case StringType =>
val zid = ctx.addReferenceObj("zoneId", zoneId, classOf[ZoneId].getName)
- val locale = ctx.addReferenceObj("locale", Locale.US)
val tf = TimestampFormatter.getClass.getName.stripSuffix("$")
nullSafeCodeGen(ctx, ev, (string, format) => {
s"""
try {
- ${ev.value} = $tf$$.MODULE$$.apply($format.toString(), $zid, $locale)
+ ${ev.value} = $tf$$.MODULE$$.withStrongLegacy($format.toString(), $zid)
Review comment:
quick question, why `SIMPLE_DATE_FORMAT` alone has `withStrongLegacy` - is it because `SIMPLE_DATE_FORMAT` is an enum, and tricky for the codegen part to reference? We could just simply define string variables for them too. Seems like it's better to keep it consistent to one side as this legacy code part is already pretty difficult to follow and read.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [SPARK-30788][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584721896
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584295874
**[Test build #118175 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118175/testReport)** for PR 27524 at commit [`ab1d57f`](https://github.com/apache/spark/commit/ab1d57f1b131db20c32ac1169730d3e8f45f4698).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377357685
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
##########
@@ -90,47 +91,168 @@ class FractionTimestampFormatter(zoneId: ZoneId)
override protected lazy val formatter = DateTimeFormatterHelper.fractionFormatter
}
-class LegacyTimestampFormatter(
+/**
+ * The custom sub-class of `GregorianCalendar` is needed to get access to
+ * protected `fields` immediately after parsing. We cannot use
+ * the `get()` method because it performs normalization of the fraction
+ * part. Accordingly, the `MILLISECOND` field doesn't contain original value.
+ *
+ * Also this class allows to set raw value to the `MILLISECOND` field
+ * directly before formatting.
+ */
+class MicrosCalendar(tz: TimeZone, digitsInFraction: Int)
+ extends GregorianCalendar(tz, Locale.US) {
+ // Converts parsed `MILLISECOND` field to seconds fraction in microsecond precision.
+ // For example if the fraction pattern is `SSSS` then `digitsInFraction` = 4, and
+ // if the `MILLISECOND` field was parsed to `1234`.
+ def getMicros(): SQLTimestamp = {
+ // Append 6 zeros to the field: 1234 -> 1234000000
+ val d = fields(Calendar.MILLISECOND) * MICROS_PER_SECOND
+ // Take the first 6 digits from `d`: 1234000000 -> 123400
+ // The rest contains exactly `digitsInFraction`: `0000` = 10 ^ digitsInFraction
+ // So, the result is `(1234 * 1000000) / (10 ^ digitsInFraction)
+ d / Decimal.POW_10(digitsInFraction)
+ }
+
+ // Converts the seconds fraction in microsecond precision to a value
+ // that can be correctly formatted according to the specified fraction pattern.
+ // The method performs operations opposite to `getMicros()`.
+ def setMicros(micros: Long): Unit = {
+ val d = micros * Decimal.POW_10(digitsInFraction)
+ fields(Calendar.MILLISECOND) = (d / MICROS_PER_SECOND).toInt
+ }
+}
+
+/**
+ * An instance of the class is aimed to re-use many times. It contains helper objects
+ * `cal` which is reused between `parse()` and `format` invokes.
+ */
+class LegacyFastDateFormat(fastDateFormat: FastDateFormat) {
Review comment:
Copy-pasted from 2.4 to support arbitrary second fraction, and parsing/formatting in microsecond precision using the legacy parser - `FastDateFormat` cannot do that by default.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377476826
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
##########
@@ -90,47 +91,168 @@ class FractionTimestampFormatter(zoneId: ZoneId)
override protected lazy val formatter = DateTimeFormatterHelper.fractionFormatter
}
-class LegacyTimestampFormatter(
+/**
+ * The custom sub-class of `GregorianCalendar` is needed to get access to
+ * protected `fields` immediately after parsing. We cannot use
+ * the `get()` method because it performs normalization of the fraction
+ * part. Accordingly, the `MILLISECOND` field doesn't contain original value.
+ *
+ * Also this class allows to set raw value to the `MILLISECOND` field
+ * directly before formatting.
+ */
+class MicrosCalendar(tz: TimeZone, digitsInFraction: Int)
+ extends GregorianCalendar(tz, Locale.US) {
+ // Converts parsed `MILLISECOND` field to seconds fraction in microsecond precision.
+ // For example if the fraction pattern is `SSSS` then `digitsInFraction` = 4, and
+ // if the `MILLISECOND` field was parsed to `1234`.
+ def getMicros(): SQLTimestamp = {
+ // Append 6 zeros to the field: 1234 -> 1234000000
+ val d = fields(Calendar.MILLISECOND) * MICROS_PER_SECOND
+ // Take the first 6 digits from `d`: 1234000000 -> 123400
+ // The rest contains exactly `digitsInFraction`: `0000` = 10 ^ digitsInFraction
+ // So, the result is `(1234 * 1000000) / (10 ^ digitsInFraction)
+ d / Decimal.POW_10(digitsInFraction)
+ }
+
+ // Converts the seconds fraction in microsecond precision to a value
+ // that can be correctly formatted according to the specified fraction pattern.
+ // The method performs operations opposite to `getMicros()`.
+ def setMicros(micros: Long): Unit = {
+ val d = micros * Decimal.POW_10(digitsInFraction)
+ fields(Calendar.MILLISECOND) = (d / MICROS_PER_SECOND).toInt
+ }
+}
+
+/**
+ * An instance of the class is aimed to re-use many times. It contains helper objects
+ * `cal` which is reused between `parse()` and `format` invokes.
+ */
+class LegacyFastDateFormat(fastDateFormat: FastDateFormat) {
Review comment:
I removed the class
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584203046
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22933/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584203034
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377165571
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
##########
@@ -525,170 +529,194 @@ class DateFunctionsSuite extends QueryTest with SharedSparkSession {
}
test("from_unixtime") {
- val sdf1 = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss", Locale.US)
- val fmt2 = "yyyy-MM-dd HH:mm:ss.SSS"
- val sdf2 = new SimpleDateFormat(fmt2, Locale.US)
- val fmt3 = "yy-MM-dd HH-mm-ss"
- val sdf3 = new SimpleDateFormat(fmt3, Locale.US)
- val df = Seq((1000, "yyyy-MM-dd HH:mm:ss.SSS"), (-1000, "yy-MM-dd HH-mm-ss")).toDF("a", "b")
- checkAnswer(
- df.select(from_unixtime(col("a"))),
- Seq(Row(sdf1.format(new Timestamp(1000000))), Row(sdf1.format(new Timestamp(-1000000)))))
- checkAnswer(
- df.select(from_unixtime(col("a"), fmt2)),
- Seq(Row(sdf2.format(new Timestamp(1000000))), Row(sdf2.format(new Timestamp(-1000000)))))
- checkAnswer(
- df.select(from_unixtime(col("a"), fmt3)),
- Seq(Row(sdf3.format(new Timestamp(1000000))), Row(sdf3.format(new Timestamp(-1000000)))))
- checkAnswer(
- df.selectExpr("from_unixtime(a)"),
- Seq(Row(sdf1.format(new Timestamp(1000000))), Row(sdf1.format(new Timestamp(-1000000)))))
- checkAnswer(
- df.selectExpr(s"from_unixtime(a, '$fmt2')"),
- Seq(Row(sdf2.format(new Timestamp(1000000))), Row(sdf2.format(new Timestamp(-1000000)))))
- checkAnswer(
- df.selectExpr(s"from_unixtime(a, '$fmt3')"),
- Seq(Row(sdf3.format(new Timestamp(1000000))), Row(sdf3.format(new Timestamp(-1000000)))))
+ Seq(false, true).foreach { legacyParser =>
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> legacyParser.toString) {
+ val sdf1 = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss", Locale.US)
+ val fmt2 = "yyyy-MM-dd HH:mm:ss.SSS"
+ val sdf2 = new SimpleDateFormat(fmt2, Locale.US)
+ val fmt3 = "yy-MM-dd HH-mm-ss"
+ val sdf3 = new SimpleDateFormat(fmt3, Locale.US)
+ val df = Seq((1000, "yyyy-MM-dd HH:mm:ss.SSS"), (-1000, "yy-MM-dd HH-mm-ss")).toDF("a", "b")
+ checkAnswer(
+ df.select(from_unixtime(col("a"))),
+ Seq(Row(sdf1.format(new Timestamp(1000000))), Row(sdf1.format(new Timestamp(-1000000)))))
+ checkAnswer(
+ df.select(from_unixtime(col("a"), fmt2)),
+ Seq(Row(sdf2.format(new Timestamp(1000000))), Row(sdf2.format(new Timestamp(-1000000)))))
+ checkAnswer(
+ df.select(from_unixtime(col("a"), fmt3)),
+ Seq(Row(sdf3.format(new Timestamp(1000000))), Row(sdf3.format(new Timestamp(-1000000)))))
+ checkAnswer(
+ df.selectExpr("from_unixtime(a)"),
+ Seq(Row(sdf1.format(new Timestamp(1000000))), Row(sdf1.format(new Timestamp(-1000000)))))
+ checkAnswer(
+ df.selectExpr(s"from_unixtime(a, '$fmt2')"),
+ Seq(Row(sdf2.format(new Timestamp(1000000))), Row(sdf2.format(new Timestamp(-1000000)))))
+ checkAnswer(
+ df.selectExpr(s"from_unixtime(a, '$fmt3')"),
+ Seq(Row(sdf3.format(new Timestamp(1000000))), Row(sdf3.format(new Timestamp(-1000000)))))
+ }
+ }
}
private def secs(millis: Long): Long = TimeUnit.MILLISECONDS.toSeconds(millis)
test("unix_timestamp") {
- val date1 = Date.valueOf("2015-07-24")
- val date2 = Date.valueOf("2015-07-25")
- val ts1 = Timestamp.valueOf("2015-07-24 10:00:00.3")
- val ts2 = Timestamp.valueOf("2015-07-25 02:02:02.2")
- val s1 = "2015/07/24 10:00:00.5"
- val s2 = "2015/07/25 02:02:02.6"
- val ss1 = "2015-07-24 10:00:00"
- val ss2 = "2015-07-25 02:02:02"
- val fmt = "yyyy/MM/dd HH:mm:ss.S"
- val df = Seq((date1, ts1, s1, ss1), (date2, ts2, s2, ss2)).toDF("d", "ts", "s", "ss")
- checkAnswer(df.select(unix_timestamp(col("ts"))), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.select(unix_timestamp(col("ss"))), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.select(unix_timestamp(col("d"), fmt)), Seq(
- Row(secs(date1.getTime)), Row(secs(date2.getTime))))
- checkAnswer(df.select(unix_timestamp(col("s"), fmt)), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.selectExpr("unix_timestamp(ts)"), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.selectExpr("unix_timestamp(ss)"), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.selectExpr(s"unix_timestamp(d, '$fmt')"), Seq(
- Row(secs(date1.getTime)), Row(secs(date2.getTime))))
- checkAnswer(df.selectExpr(s"unix_timestamp(s, '$fmt')"), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
-
- val x1 = "2015-07-24 10:00:00"
- val x2 = "2015-25-07 02:02:02"
- val x3 = "2015-07-24 25:02:02"
- val x4 = "2015-24-07 26:02:02"
- val ts3 = Timestamp.valueOf("2015-07-24 02:25:02")
- val ts4 = Timestamp.valueOf("2015-07-24 00:10:00")
-
- val df1 = Seq(x1, x2, x3, x4).toDF("x")
- checkAnswer(df1.select(unix_timestamp(col("x"))), Seq(
- Row(secs(ts1.getTime)), Row(null), Row(null), Row(null)))
- checkAnswer(df1.selectExpr("unix_timestamp(x)"), Seq(
- Row(secs(ts1.getTime)), Row(null), Row(null), Row(null)))
- checkAnswer(df1.select(unix_timestamp(col("x"), "yyyy-dd-MM HH:mm:ss")), Seq(
- Row(null), Row(secs(ts2.getTime)), Row(null), Row(null)))
- checkAnswer(df1.selectExpr(s"unix_timestamp(x, 'yyyy-MM-dd mm:HH:ss')"), Seq(
- Row(secs(ts4.getTime)), Row(null), Row(secs(ts3.getTime)), Row(null)))
-
- // invalid format
- checkAnswer(df1.selectExpr(s"unix_timestamp(x, 'yyyy-MM-dd aa:HH:ss')"), Seq(
- Row(null), Row(null), Row(null), Row(null)))
-
- // february
- val y1 = "2016-02-29"
- val y2 = "2017-02-29"
- val ts5 = Timestamp.valueOf("2016-02-29 00:00:00")
- val df2 = Seq(y1, y2).toDF("y")
- checkAnswer(df2.select(unix_timestamp(col("y"), "yyyy-MM-dd")), Seq(
- Row(secs(ts5.getTime)), Row(null)))
-
- val now = sql("select unix_timestamp()").collect().head.getLong(0)
- checkAnswer(
- sql(s"select cast ($now as timestamp)"),
- Row(new java.util.Date(TimeUnit.SECONDS.toMillis(now))))
+ Seq(false, true).foreach { legacyParser =>
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> legacyParser.toString) {
+ val date1 = Date.valueOf("2015-07-24")
+ val date2 = Date.valueOf("2015-07-25")
+ val ts1 = Timestamp.valueOf("2015-07-24 10:00:00.3")
+ val ts2 = Timestamp.valueOf("2015-07-25 02:02:02.2")
+ val s1 = "2015/07/24 10:00:00.5"
+ val s2 = "2015/07/25 02:02:02.6"
+ val ss1 = "2015-07-24 10:00:00"
+ val ss2 = "2015-07-25 02:02:02"
+ val fmt = "yyyy/MM/dd HH:mm:ss.S"
+ val df = Seq((date1, ts1, s1, ss1), (date2, ts2, s2, ss2)).toDF("d", "ts", "s", "ss")
+ checkAnswer(df.select(unix_timestamp(col("ts"))), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.select(unix_timestamp(col("ss"))), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.select(unix_timestamp(col("d"), fmt)), Seq(
+ Row(secs(date1.getTime)), Row(secs(date2.getTime))))
+ checkAnswer(df.select(unix_timestamp(col("s"), fmt)), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.selectExpr("unix_timestamp(ts)"), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.selectExpr("unix_timestamp(ss)"), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.selectExpr(s"unix_timestamp(d, '$fmt')"), Seq(
+ Row(secs(date1.getTime)), Row(secs(date2.getTime))))
+ checkAnswer(df.selectExpr(s"unix_timestamp(s, '$fmt')"), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+
+ val x1 = "2015-07-24 10:00:00"
+ val x2 = "2015-25-07 02:02:02"
+ val x3 = "2015-07-24 25:02:02"
+ val x4 = "2015-24-07 26:02:02"
+ val ts3 = Timestamp.valueOf("2015-07-24 02:25:02")
+ val ts4 = Timestamp.valueOf("2015-07-24 00:10:00")
+
+ val df1 = Seq(x1, x2, x3, x4).toDF("x")
+ checkAnswer(df1.select(unix_timestamp(col("x"))), Seq(
+ Row(secs(ts1.getTime)), Row(null), Row(null), Row(null)))
+ checkAnswer(df1.selectExpr("unix_timestamp(x)"), Seq(
+ Row(secs(ts1.getTime)), Row(null), Row(null), Row(null)))
+ checkAnswer(df1.select(unix_timestamp(col("x"), "yyyy-dd-MM HH:mm:ss")), Seq(
+ Row(null), Row(secs(ts2.getTime)), Row(null), Row(null)))
+ checkAnswer(df1.selectExpr(s"unix_timestamp(x, 'yyyy-MM-dd mm:HH:ss')"), Seq(
+ Row(secs(ts4.getTime)), Row(null), Row(secs(ts3.getTime)), Row(null)))
+
+ // invalid format
+ checkAnswer(df1.selectExpr(s"unix_timestamp(x, 'yyyy-MM-dd aa:HH:ss')"), Seq(
+ Row(null), Row(null), Row(null), Row(null)))
+
+ // february
+ val y1 = "2016-02-29"
+ val y2 = "2017-02-29"
+ val ts5 = Timestamp.valueOf("2016-02-29 00:00:00")
+ val df2 = Seq(y1, y2).toDF("y")
+ checkAnswer(df2.select(unix_timestamp(col("y"), "yyyy-MM-dd")), Seq(
+ Row(secs(ts5.getTime)), Row(null)))
+
+ val now = sql("select unix_timestamp()").collect().head.getLong(0)
+ checkAnswer(
+ sql(s"select cast ($now as timestamp)"),
+ Row(new java.util.Date(TimeUnit.SECONDS.toMillis(now))))
+ }
+ }
}
test("to_unix_timestamp") {
- val date1 = Date.valueOf("2015-07-24")
- val date2 = Date.valueOf("2015-07-25")
- val ts1 = Timestamp.valueOf("2015-07-24 10:00:00.3")
- val ts2 = Timestamp.valueOf("2015-07-25 02:02:02.2")
- val s1 = "2015/07/24 10:00:00.5"
- val s2 = "2015/07/25 02:02:02.6"
- val ss1 = "2015-07-24 10:00:00"
- val ss2 = "2015-07-25 02:02:02"
- val fmt = "yyyy/MM/dd HH:mm:ss.S"
- val df = Seq((date1, ts1, s1, ss1), (date2, ts2, s2, ss2)).toDF("d", "ts", "s", "ss")
- checkAnswer(df.selectExpr("to_unix_timestamp(ts)"), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.selectExpr("to_unix_timestamp(ss)"), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.selectExpr(s"to_unix_timestamp(d, '$fmt')"), Seq(
- Row(secs(date1.getTime)), Row(secs(date2.getTime))))
- checkAnswer(df.selectExpr(s"to_unix_timestamp(s, '$fmt')"), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
-
- val x1 = "2015-07-24 10:00:00"
- val x2 = "2015-25-07 02:02:02"
- val x3 = "2015-07-24 25:02:02"
- val x4 = "2015-24-07 26:02:02"
- val ts3 = Timestamp.valueOf("2015-07-24 02:25:02")
- val ts4 = Timestamp.valueOf("2015-07-24 00:10:00")
-
- val df1 = Seq(x1, x2, x3, x4).toDF("x")
- checkAnswer(df1.selectExpr("to_unix_timestamp(x)"), Seq(
- Row(secs(ts1.getTime)), Row(null), Row(null), Row(null)))
- checkAnswer(df1.selectExpr(s"to_unix_timestamp(x, 'yyyy-MM-dd mm:HH:ss')"), Seq(
- Row(secs(ts4.getTime)), Row(null), Row(secs(ts3.getTime)), Row(null)))
-
- // february
- val y1 = "2016-02-29"
- val y2 = "2017-02-29"
- val ts5 = Timestamp.valueOf("2016-02-29 00:00:00")
- val df2 = Seq(y1, y2).toDF("y")
- checkAnswer(df2.select(unix_timestamp(col("y"), "yyyy-MM-dd")), Seq(
- Row(secs(ts5.getTime)), Row(null)))
-
- // invalid format
- checkAnswer(df1.selectExpr(s"to_unix_timestamp(x, 'yyyy-MM-dd bb:HH:ss')"), Seq(
- Row(null), Row(null), Row(null), Row(null)))
+ Seq(false, true).foreach { legacyParser =>
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> legacyParser.toString) {
+ val date1 = Date.valueOf("2015-07-24")
+ val date2 = Date.valueOf("2015-07-25")
+ val ts1 = Timestamp.valueOf("2015-07-24 10:00:00.3")
+ val ts2 = Timestamp.valueOf("2015-07-25 02:02:02.2")
+ val s1 = "2015/07/24 10:00:00.5"
+ val s2 = "2015/07/25 02:02:02.6"
+ val ss1 = "2015-07-24 10:00:00"
+ val ss2 = "2015-07-25 02:02:02"
+ val fmt = "yyyy/MM/dd HH:mm:ss.S"
+ val df = Seq((date1, ts1, s1, ss1), (date2, ts2, s2, ss2)).toDF("d", "ts", "s", "ss")
+ checkAnswer(df.selectExpr("to_unix_timestamp(ts)"), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.selectExpr("to_unix_timestamp(ss)"), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.selectExpr(s"to_unix_timestamp(d, '$fmt')"), Seq(
+ Row(secs(date1.getTime)), Row(secs(date2.getTime))))
+ checkAnswer(df.selectExpr(s"to_unix_timestamp(s, '$fmt')"), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+
+ val x1 = "2015-07-24 10:00:00"
+ val x2 = "2015-25-07 02:02:02"
+ val x3 = "2015-07-24 25:02:02"
+ val x4 = "2015-24-07 26:02:02"
+ val ts3 = Timestamp.valueOf("2015-07-24 02:25:02")
+ val ts4 = Timestamp.valueOf("2015-07-24 00:10:00")
+
+ val df1 = Seq(x1, x2, x3, x4).toDF("x")
+ checkAnswer(df1.selectExpr("to_unix_timestamp(x)"), Seq(
+ Row(secs(ts1.getTime)), Row(null), Row(null), Row(null)))
+ checkAnswer(df1.selectExpr(s"to_unix_timestamp(x, 'yyyy-MM-dd mm:HH:ss')"), Seq(
+ Row(secs(ts4.getTime)), Row(null), Row(secs(ts3.getTime)), Row(null)))
+
+ // february
+ val y1 = "2016-02-29"
+ val y2 = "2017-02-29"
+ val ts5 = Timestamp.valueOf("2016-02-29 00:00:00")
+ val df2 = Seq(y1, y2).toDF("y")
+ checkAnswer(df2.select(unix_timestamp(col("y"), "yyyy-MM-dd")), Seq(
+ Row(secs(ts5.getTime)), Row(null)))
+
+ // invalid format
+ checkAnswer(df1.selectExpr(s"to_unix_timestamp(x, 'yyyy-MM-dd bb:HH:ss')"), Seq(
+ Row(null), Row(null), Row(null), Row(null)))
+ }
+ }
}
test("to_timestamp") {
- val date1 = Date.valueOf("2015-07-24")
- val date2 = Date.valueOf("2015-07-25")
- val ts_date1 = Timestamp.valueOf("2015-07-24 00:00:00")
- val ts_date2 = Timestamp.valueOf("2015-07-25 00:00:00")
- val ts1 = Timestamp.valueOf("2015-07-24 10:00:00")
- val ts2 = Timestamp.valueOf("2015-07-25 02:02:02")
- val s1 = "2015/07/24 10:00:00.5"
- val s2 = "2015/07/25 02:02:02.6"
- val ts1m = Timestamp.valueOf("2015-07-24 10:00:00.5")
- val ts2m = Timestamp.valueOf("2015-07-25 02:02:02.6")
- val ss1 = "2015-07-24 10:00:00"
- val ss2 = "2015-07-25 02:02:02"
- val fmt = "yyyy/MM/dd HH:mm:ss.S"
- val df = Seq((date1, ts1, s1, ss1), (date2, ts2, s2, ss2)).toDF("d", "ts", "s", "ss")
-
- checkAnswer(df.select(to_timestamp(col("ss"))),
- df.select(unix_timestamp(col("ss")).cast("timestamp")))
- checkAnswer(df.select(to_timestamp(col("ss"))), Seq(
- Row(ts1), Row(ts2)))
- checkAnswer(df.select(to_timestamp(col("s"), fmt)), Seq(
- Row(ts1m), Row(ts2m)))
- checkAnswer(df.select(to_timestamp(col("ts"), fmt)), Seq(
- Row(ts1), Row(ts2)))
- checkAnswer(df.select(to_timestamp(col("d"), "yyyy-MM-dd")), Seq(
- Row(ts_date1), Row(ts_date2)))
+ Seq(false, true).foreach { legacyParser =>
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> legacyParser.toString) {
+ val date1 = Date.valueOf("2015-07-24")
+ val date2 = Date.valueOf("2015-07-25")
+ val ts_date1 = Timestamp.valueOf("2015-07-24 00:00:00")
+ val ts_date2 = Timestamp.valueOf("2015-07-25 00:00:00")
+ val ts1 = Timestamp.valueOf("2015-07-24 10:00:00")
+ val ts2 = Timestamp.valueOf("2015-07-25 02:02:02")
+ val s1 = "2015/07/24 10:00:00.5"
+ val s2 = "2015/07/25 02:02:02.6"
+ val ts1m = Timestamp.valueOf("2015-07-24 10:00:00.5")
+ val ts2m = Timestamp.valueOf("2015-07-25 02:02:02.6")
+ val ss1 = "2015-07-24 10:00:00"
+ val ss2 = "2015-07-25 02:02:02"
+ val fmt = "yyyy/MM/dd HH:mm:ss.S"
+ val df = Seq((date1, ts1, s1, ss1), (date2, ts2, s2, ss2)).toDF("d", "ts", "s", "ss")
+
+ checkAnswer(df.select(to_timestamp(col("ss"))),
+ df.select(unix_timestamp(col("ss")).cast("timestamp")))
+ checkAnswer(df.select(to_timestamp(col("ss"))), Seq(
+ Row(ts1), Row(ts2)))
+ if (legacyParser) {
Review comment:
Unfortunately, `SimpleDateFormat` doesn't work correctly with the pattern `.S`. In Spark 2.4, it wasn't visible in the test because `to_timestamp` truncated results to seconds.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377164426
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
##########
@@ -525,170 +529,194 @@ class DateFunctionsSuite extends QueryTest with SharedSparkSession {
}
test("from_unixtime") {
- val sdf1 = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss", Locale.US)
- val fmt2 = "yyyy-MM-dd HH:mm:ss.SSS"
- val sdf2 = new SimpleDateFormat(fmt2, Locale.US)
- val fmt3 = "yy-MM-dd HH-mm-ss"
- val sdf3 = new SimpleDateFormat(fmt3, Locale.US)
- val df = Seq((1000, "yyyy-MM-dd HH:mm:ss.SSS"), (-1000, "yy-MM-dd HH-mm-ss")).toDF("a", "b")
- checkAnswer(
- df.select(from_unixtime(col("a"))),
- Seq(Row(sdf1.format(new Timestamp(1000000))), Row(sdf1.format(new Timestamp(-1000000)))))
- checkAnswer(
- df.select(from_unixtime(col("a"), fmt2)),
- Seq(Row(sdf2.format(new Timestamp(1000000))), Row(sdf2.format(new Timestamp(-1000000)))))
- checkAnswer(
- df.select(from_unixtime(col("a"), fmt3)),
- Seq(Row(sdf3.format(new Timestamp(1000000))), Row(sdf3.format(new Timestamp(-1000000)))))
- checkAnswer(
- df.selectExpr("from_unixtime(a)"),
- Seq(Row(sdf1.format(new Timestamp(1000000))), Row(sdf1.format(new Timestamp(-1000000)))))
- checkAnswer(
- df.selectExpr(s"from_unixtime(a, '$fmt2')"),
- Seq(Row(sdf2.format(new Timestamp(1000000))), Row(sdf2.format(new Timestamp(-1000000)))))
- checkAnswer(
- df.selectExpr(s"from_unixtime(a, '$fmt3')"),
- Seq(Row(sdf3.format(new Timestamp(1000000))), Row(sdf3.format(new Timestamp(-1000000)))))
+ Seq(false, true).foreach { legacyParser =>
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> legacyParser.toString) {
+ val sdf1 = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss", Locale.US)
+ val fmt2 = "yyyy-MM-dd HH:mm:ss.SSS"
+ val sdf2 = new SimpleDateFormat(fmt2, Locale.US)
+ val fmt3 = "yy-MM-dd HH-mm-ss"
+ val sdf3 = new SimpleDateFormat(fmt3, Locale.US)
+ val df = Seq((1000, "yyyy-MM-dd HH:mm:ss.SSS"), (-1000, "yy-MM-dd HH-mm-ss")).toDF("a", "b")
+ checkAnswer(
+ df.select(from_unixtime(col("a"))),
+ Seq(Row(sdf1.format(new Timestamp(1000000))), Row(sdf1.format(new Timestamp(-1000000)))))
+ checkAnswer(
+ df.select(from_unixtime(col("a"), fmt2)),
+ Seq(Row(sdf2.format(new Timestamp(1000000))), Row(sdf2.format(new Timestamp(-1000000)))))
+ checkAnswer(
+ df.select(from_unixtime(col("a"), fmt3)),
+ Seq(Row(sdf3.format(new Timestamp(1000000))), Row(sdf3.format(new Timestamp(-1000000)))))
+ checkAnswer(
+ df.selectExpr("from_unixtime(a)"),
+ Seq(Row(sdf1.format(new Timestamp(1000000))), Row(sdf1.format(new Timestamp(-1000000)))))
+ checkAnswer(
+ df.selectExpr(s"from_unixtime(a, '$fmt2')"),
+ Seq(Row(sdf2.format(new Timestamp(1000000))), Row(sdf2.format(new Timestamp(-1000000)))))
+ checkAnswer(
+ df.selectExpr(s"from_unixtime(a, '$fmt3')"),
+ Seq(Row(sdf3.format(new Timestamp(1000000))), Row(sdf3.format(new Timestamp(-1000000)))))
+ }
+ }
}
private def secs(millis: Long): Long = TimeUnit.MILLISECONDS.toSeconds(millis)
test("unix_timestamp") {
- val date1 = Date.valueOf("2015-07-24")
- val date2 = Date.valueOf("2015-07-25")
- val ts1 = Timestamp.valueOf("2015-07-24 10:00:00.3")
- val ts2 = Timestamp.valueOf("2015-07-25 02:02:02.2")
- val s1 = "2015/07/24 10:00:00.5"
- val s2 = "2015/07/25 02:02:02.6"
- val ss1 = "2015-07-24 10:00:00"
- val ss2 = "2015-07-25 02:02:02"
- val fmt = "yyyy/MM/dd HH:mm:ss.S"
- val df = Seq((date1, ts1, s1, ss1), (date2, ts2, s2, ss2)).toDF("d", "ts", "s", "ss")
- checkAnswer(df.select(unix_timestamp(col("ts"))), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.select(unix_timestamp(col("ss"))), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.select(unix_timestamp(col("d"), fmt)), Seq(
- Row(secs(date1.getTime)), Row(secs(date2.getTime))))
- checkAnswer(df.select(unix_timestamp(col("s"), fmt)), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.selectExpr("unix_timestamp(ts)"), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.selectExpr("unix_timestamp(ss)"), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.selectExpr(s"unix_timestamp(d, '$fmt')"), Seq(
- Row(secs(date1.getTime)), Row(secs(date2.getTime))))
- checkAnswer(df.selectExpr(s"unix_timestamp(s, '$fmt')"), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
-
- val x1 = "2015-07-24 10:00:00"
- val x2 = "2015-25-07 02:02:02"
- val x3 = "2015-07-24 25:02:02"
- val x4 = "2015-24-07 26:02:02"
- val ts3 = Timestamp.valueOf("2015-07-24 02:25:02")
- val ts4 = Timestamp.valueOf("2015-07-24 00:10:00")
-
- val df1 = Seq(x1, x2, x3, x4).toDF("x")
- checkAnswer(df1.select(unix_timestamp(col("x"))), Seq(
- Row(secs(ts1.getTime)), Row(null), Row(null), Row(null)))
- checkAnswer(df1.selectExpr("unix_timestamp(x)"), Seq(
- Row(secs(ts1.getTime)), Row(null), Row(null), Row(null)))
- checkAnswer(df1.select(unix_timestamp(col("x"), "yyyy-dd-MM HH:mm:ss")), Seq(
- Row(null), Row(secs(ts2.getTime)), Row(null), Row(null)))
- checkAnswer(df1.selectExpr(s"unix_timestamp(x, 'yyyy-MM-dd mm:HH:ss')"), Seq(
- Row(secs(ts4.getTime)), Row(null), Row(secs(ts3.getTime)), Row(null)))
-
- // invalid format
- checkAnswer(df1.selectExpr(s"unix_timestamp(x, 'yyyy-MM-dd aa:HH:ss')"), Seq(
- Row(null), Row(null), Row(null), Row(null)))
-
- // february
- val y1 = "2016-02-29"
- val y2 = "2017-02-29"
- val ts5 = Timestamp.valueOf("2016-02-29 00:00:00")
- val df2 = Seq(y1, y2).toDF("y")
- checkAnswer(df2.select(unix_timestamp(col("y"), "yyyy-MM-dd")), Seq(
- Row(secs(ts5.getTime)), Row(null)))
-
- val now = sql("select unix_timestamp()").collect().head.getLong(0)
- checkAnswer(
- sql(s"select cast ($now as timestamp)"),
- Row(new java.util.Date(TimeUnit.SECONDS.toMillis(now))))
+ Seq(false, true).foreach { legacyParser =>
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> legacyParser.toString) {
+ val date1 = Date.valueOf("2015-07-24")
+ val date2 = Date.valueOf("2015-07-25")
+ val ts1 = Timestamp.valueOf("2015-07-24 10:00:00.3")
+ val ts2 = Timestamp.valueOf("2015-07-25 02:02:02.2")
+ val s1 = "2015/07/24 10:00:00.5"
+ val s2 = "2015/07/25 02:02:02.6"
+ val ss1 = "2015-07-24 10:00:00"
+ val ss2 = "2015-07-25 02:02:02"
+ val fmt = "yyyy/MM/dd HH:mm:ss.S"
+ val df = Seq((date1, ts1, s1, ss1), (date2, ts2, s2, ss2)).toDF("d", "ts", "s", "ss")
+ checkAnswer(df.select(unix_timestamp(col("ts"))), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.select(unix_timestamp(col("ss"))), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.select(unix_timestamp(col("d"), fmt)), Seq(
+ Row(secs(date1.getTime)), Row(secs(date2.getTime))))
+ checkAnswer(df.select(unix_timestamp(col("s"), fmt)), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.selectExpr("unix_timestamp(ts)"), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.selectExpr("unix_timestamp(ss)"), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.selectExpr(s"unix_timestamp(d, '$fmt')"), Seq(
+ Row(secs(date1.getTime)), Row(secs(date2.getTime))))
+ checkAnswer(df.selectExpr(s"unix_timestamp(s, '$fmt')"), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+
+ val x1 = "2015-07-24 10:00:00"
+ val x2 = "2015-25-07 02:02:02"
+ val x3 = "2015-07-24 25:02:02"
+ val x4 = "2015-24-07 26:02:02"
+ val ts3 = Timestamp.valueOf("2015-07-24 02:25:02")
+ val ts4 = Timestamp.valueOf("2015-07-24 00:10:00")
+
+ val df1 = Seq(x1, x2, x3, x4).toDF("x")
+ checkAnswer(df1.select(unix_timestamp(col("x"))), Seq(
+ Row(secs(ts1.getTime)), Row(null), Row(null), Row(null)))
+ checkAnswer(df1.selectExpr("unix_timestamp(x)"), Seq(
+ Row(secs(ts1.getTime)), Row(null), Row(null), Row(null)))
+ checkAnswer(df1.select(unix_timestamp(col("x"), "yyyy-dd-MM HH:mm:ss")), Seq(
+ Row(null), Row(secs(ts2.getTime)), Row(null), Row(null)))
+ checkAnswer(df1.selectExpr(s"unix_timestamp(x, 'yyyy-MM-dd mm:HH:ss')"), Seq(
+ Row(secs(ts4.getTime)), Row(null), Row(secs(ts3.getTime)), Row(null)))
+
+ // invalid format
+ checkAnswer(df1.selectExpr(s"unix_timestamp(x, 'yyyy-MM-dd aa:HH:ss')"), Seq(
+ Row(null), Row(null), Row(null), Row(null)))
+
+ // february
+ val y1 = "2016-02-29"
+ val y2 = "2017-02-29"
+ val ts5 = Timestamp.valueOf("2016-02-29 00:00:00")
+ val df2 = Seq(y1, y2).toDF("y")
+ checkAnswer(df2.select(unix_timestamp(col("y"), "yyyy-MM-dd")), Seq(
+ Row(secs(ts5.getTime)), Row(null)))
+
+ val now = sql("select unix_timestamp()").collect().head.getLong(0)
+ checkAnswer(
+ sql(s"select cast ($now as timestamp)"),
+ Row(new java.util.Date(TimeUnit.SECONDS.toMillis(now))))
+ }
+ }
}
test("to_unix_timestamp") {
- val date1 = Date.valueOf("2015-07-24")
- val date2 = Date.valueOf("2015-07-25")
- val ts1 = Timestamp.valueOf("2015-07-24 10:00:00.3")
- val ts2 = Timestamp.valueOf("2015-07-25 02:02:02.2")
- val s1 = "2015/07/24 10:00:00.5"
- val s2 = "2015/07/25 02:02:02.6"
- val ss1 = "2015-07-24 10:00:00"
- val ss2 = "2015-07-25 02:02:02"
- val fmt = "yyyy/MM/dd HH:mm:ss.S"
- val df = Seq((date1, ts1, s1, ss1), (date2, ts2, s2, ss2)).toDF("d", "ts", "s", "ss")
- checkAnswer(df.selectExpr("to_unix_timestamp(ts)"), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.selectExpr("to_unix_timestamp(ss)"), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.selectExpr(s"to_unix_timestamp(d, '$fmt')"), Seq(
- Row(secs(date1.getTime)), Row(secs(date2.getTime))))
- checkAnswer(df.selectExpr(s"to_unix_timestamp(s, '$fmt')"), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
-
- val x1 = "2015-07-24 10:00:00"
- val x2 = "2015-25-07 02:02:02"
- val x3 = "2015-07-24 25:02:02"
- val x4 = "2015-24-07 26:02:02"
- val ts3 = Timestamp.valueOf("2015-07-24 02:25:02")
- val ts4 = Timestamp.valueOf("2015-07-24 00:10:00")
-
- val df1 = Seq(x1, x2, x3, x4).toDF("x")
- checkAnswer(df1.selectExpr("to_unix_timestamp(x)"), Seq(
- Row(secs(ts1.getTime)), Row(null), Row(null), Row(null)))
- checkAnswer(df1.selectExpr(s"to_unix_timestamp(x, 'yyyy-MM-dd mm:HH:ss')"), Seq(
- Row(secs(ts4.getTime)), Row(null), Row(secs(ts3.getTime)), Row(null)))
-
- // february
- val y1 = "2016-02-29"
- val y2 = "2017-02-29"
- val ts5 = Timestamp.valueOf("2016-02-29 00:00:00")
- val df2 = Seq(y1, y2).toDF("y")
- checkAnswer(df2.select(unix_timestamp(col("y"), "yyyy-MM-dd")), Seq(
- Row(secs(ts5.getTime)), Row(null)))
-
- // invalid format
- checkAnswer(df1.selectExpr(s"to_unix_timestamp(x, 'yyyy-MM-dd bb:HH:ss')"), Seq(
- Row(null), Row(null), Row(null), Row(null)))
+ Seq(false, true).foreach { legacyParser =>
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> legacyParser.toString) {
+ val date1 = Date.valueOf("2015-07-24")
+ val date2 = Date.valueOf("2015-07-25")
+ val ts1 = Timestamp.valueOf("2015-07-24 10:00:00.3")
+ val ts2 = Timestamp.valueOf("2015-07-25 02:02:02.2")
+ val s1 = "2015/07/24 10:00:00.5"
+ val s2 = "2015/07/25 02:02:02.6"
+ val ss1 = "2015-07-24 10:00:00"
+ val ss2 = "2015-07-25 02:02:02"
+ val fmt = "yyyy/MM/dd HH:mm:ss.S"
+ val df = Seq((date1, ts1, s1, ss1), (date2, ts2, s2, ss2)).toDF("d", "ts", "s", "ss")
+ checkAnswer(df.selectExpr("to_unix_timestamp(ts)"), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.selectExpr("to_unix_timestamp(ss)"), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.selectExpr(s"to_unix_timestamp(d, '$fmt')"), Seq(
+ Row(secs(date1.getTime)), Row(secs(date2.getTime))))
+ checkAnswer(df.selectExpr(s"to_unix_timestamp(s, '$fmt')"), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+
+ val x1 = "2015-07-24 10:00:00"
+ val x2 = "2015-25-07 02:02:02"
+ val x3 = "2015-07-24 25:02:02"
+ val x4 = "2015-24-07 26:02:02"
+ val ts3 = Timestamp.valueOf("2015-07-24 02:25:02")
+ val ts4 = Timestamp.valueOf("2015-07-24 00:10:00")
+
+ val df1 = Seq(x1, x2, x3, x4).toDF("x")
+ checkAnswer(df1.selectExpr("to_unix_timestamp(x)"), Seq(
+ Row(secs(ts1.getTime)), Row(null), Row(null), Row(null)))
+ checkAnswer(df1.selectExpr(s"to_unix_timestamp(x, 'yyyy-MM-dd mm:HH:ss')"), Seq(
+ Row(secs(ts4.getTime)), Row(null), Row(secs(ts3.getTime)), Row(null)))
+
+ // february
+ val y1 = "2016-02-29"
+ val y2 = "2017-02-29"
+ val ts5 = Timestamp.valueOf("2016-02-29 00:00:00")
+ val df2 = Seq(y1, y2).toDF("y")
+ checkAnswer(df2.select(unix_timestamp(col("y"), "yyyy-MM-dd")), Seq(
+ Row(secs(ts5.getTime)), Row(null)))
+
+ // invalid format
+ checkAnswer(df1.selectExpr(s"to_unix_timestamp(x, 'yyyy-MM-dd bb:HH:ss')"), Seq(
+ Row(null), Row(null), Row(null), Row(null)))
+ }
+ }
}
test("to_timestamp") {
- val date1 = Date.valueOf("2015-07-24")
- val date2 = Date.valueOf("2015-07-25")
- val ts_date1 = Timestamp.valueOf("2015-07-24 00:00:00")
- val ts_date2 = Timestamp.valueOf("2015-07-25 00:00:00")
- val ts1 = Timestamp.valueOf("2015-07-24 10:00:00")
- val ts2 = Timestamp.valueOf("2015-07-25 02:02:02")
- val s1 = "2015/07/24 10:00:00.5"
- val s2 = "2015/07/25 02:02:02.6"
- val ts1m = Timestamp.valueOf("2015-07-24 10:00:00.5")
- val ts2m = Timestamp.valueOf("2015-07-25 02:02:02.6")
- val ss1 = "2015-07-24 10:00:00"
- val ss2 = "2015-07-25 02:02:02"
- val fmt = "yyyy/MM/dd HH:mm:ss.S"
- val df = Seq((date1, ts1, s1, ss1), (date2, ts2, s2, ss2)).toDF("d", "ts", "s", "ss")
-
- checkAnswer(df.select(to_timestamp(col("ss"))),
- df.select(unix_timestamp(col("ss")).cast("timestamp")))
- checkAnswer(df.select(to_timestamp(col("ss"))), Seq(
- Row(ts1), Row(ts2)))
- checkAnswer(df.select(to_timestamp(col("s"), fmt)), Seq(
- Row(ts1m), Row(ts2m)))
- checkAnswer(df.select(to_timestamp(col("ts"), fmt)), Seq(
- Row(ts1), Row(ts2)))
- checkAnswer(df.select(to_timestamp(col("d"), "yyyy-MM-dd")), Seq(
- Row(ts_date1), Row(ts_date2)))
+ Seq(false, true).foreach { legacyParser =>
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> legacyParser.toString) {
+ val date1 = Date.valueOf("2015-07-24")
+ val date2 = Date.valueOf("2015-07-25")
+ val ts_date1 = Timestamp.valueOf("2015-07-24 00:00:00")
+ val ts_date2 = Timestamp.valueOf("2015-07-25 00:00:00")
+ val ts1 = Timestamp.valueOf("2015-07-24 10:00:00")
+ val ts2 = Timestamp.valueOf("2015-07-25 02:02:02")
+ val s1 = "2015/07/24 10:00:00.5"
+ val s2 = "2015/07/25 02:02:02.6"
+ val ts1m = Timestamp.valueOf("2015-07-24 10:00:00.5")
+ val ts2m = Timestamp.valueOf("2015-07-25 02:02:02.6")
+ val ss1 = "2015-07-24 10:00:00"
+ val ss2 = "2015-07-25 02:02:02"
+ val fmt = "yyyy/MM/dd HH:mm:ss.S"
+ val df = Seq((date1, ts1, s1, ss1), (date2, ts2, s2, ss2)).toDF("d", "ts", "s", "ss")
+
+ checkAnswer(df.select(to_timestamp(col("ss"))),
+ df.select(unix_timestamp(col("ss")).cast("timestamp")))
+ checkAnswer(df.select(to_timestamp(col("ss"))), Seq(
+ Row(ts1), Row(ts2)))
+ if (legacyParser) {
Review comment:
I had to handle legacy mode especially due to behavior change of `to_timestamp`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377476654
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
##########
@@ -831,12 +831,11 @@ abstract class ToTimestamp
}
case StringType =>
val zid = ctx.addReferenceObj("zoneId", zoneId, classOf[ZoneId].getName)
- val locale = ctx.addReferenceObj("locale", Locale.US)
val tf = TimestampFormatter.getClass.getName.stripSuffix("$")
nullSafeCodeGen(ctx, ev, (string, format) => {
s"""
try {
- ${ev.value} = $tf$$.MODULE$$.apply($format.toString(), $zid, $locale)
+ ${ev.value} = $tf$$.MODULE$$.withStrongLegacy($format.toString(), $zid)
Review comment:
I removed `withStrongLegacy` by enum
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27524: [SPARK-30788][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584516997
**[Test build #118215 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118215/testReport)** for PR 27524 at commit [`93f3ae1`](https://github.com/apache/spark/commit/93f3ae1c32f893cc3cc22ae075753098440e76b5).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [SPARK-30788][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584574696
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584463234
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584510530
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22975/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [SPARK-30788][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584568418
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118219/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584506037
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377525872
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
##########
@@ -31,6 +31,7 @@ import org.apache.spark.sql.catalyst.util.{DateTimeUtils, IntervalUtils, Timesta
import org.apache.spark.sql.catalyst.util.DateTimeConstants.NANOS_PER_SECOND
import org.apache.spark.sql.catalyst.util.DateTimeTestUtils._
import org.apache.spark.sql.catalyst.util.DateTimeUtils.TimeZoneGMT
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
Review comment:
They work fine with `SimpleDateFormat` and `lenient = false`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377356416
##########
File path: sql/core/src/test/resources/test-data/value-malformed.csv
##########
@@ -1,2 +1,2 @@
-0,2013-111-11 12:13:14
+0,2013-111_11 12:13:14
Review comment:
`2013-111-11` is valid for `FastDateFormat`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584348159
**[Test build #118172 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118172/testReport)** for PR 27524 at commit [`38d90d8`](https://github.com/apache/spark/commit/38d90d8ca27361f3dddb858fcc4843d518bd6911).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait LegacyTimestampFormatter extends TimestampFormatter `
* `class LegacyFastDateFormatter(`
* `class LegacySimpleDateFormatter(`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27524: [SPARK-30788][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584720253
**[Test build #118232 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118232/testReport)** for PR 27524 at commit [`93f3ae1`](https://github.com/apache/spark/commit/93f3ae1c32f893cc3cc22ae075753098440e76b5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524:
[SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as
legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584568418
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118219/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377356416
##########
File path: sql/core/src/test/resources/test-data/value-malformed.csv
##########
@@ -1,2 +1,2 @@
-0,2013-111-11 12:13:14
+0,2013-111_11 12:13:14
Review comment:
`2013-111_11` is valid for `FastDateFormat`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377477732
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -541,7 +541,7 @@ object Decimal {
/** Maximum number of decimal digits a Long can represent */
val MAX_LONG_DIGITS = 18
- private val POW_10 = Array.tabulate[Long](MAX_LONG_DIGITS + 1)(i => math.pow(10, i).toLong)
+ val POW_10 = Array.tabulate[Long](MAX_LONG_DIGITS + 1)(i => math.pow(10, i).toLong)
Review comment:
`POW_10` is need in the wrapper of `FastDateFormat` to support parsing/formatting in microsecond precision. Similar changes were made in Spark 2.4.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584296573
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22936/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27524: [SPARK-30788][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584568037
**[Test build #118219 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118219/testReport)** for PR 27524 at commit [`93f3ae1`](https://github.com/apache/spark/commit/93f3ae1c32f893cc3cc22ae075753098440e76b5).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27524: [SPARK-30788][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-585177161
thanks, merging to master/3.0!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584373443
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118175/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584389466
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27524:
[SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as
legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377529775
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
##########
@@ -525,170 +529,194 @@ class DateFunctionsSuite extends QueryTest with SharedSparkSession {
}
test("from_unixtime") {
- val sdf1 = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss", Locale.US)
- val fmt2 = "yyyy-MM-dd HH:mm:ss.SSS"
- val sdf2 = new SimpleDateFormat(fmt2, Locale.US)
- val fmt3 = "yy-MM-dd HH-mm-ss"
- val sdf3 = new SimpleDateFormat(fmt3, Locale.US)
- val df = Seq((1000, "yyyy-MM-dd HH:mm:ss.SSS"), (-1000, "yy-MM-dd HH-mm-ss")).toDF("a", "b")
- checkAnswer(
- df.select(from_unixtime(col("a"))),
- Seq(Row(sdf1.format(new Timestamp(1000000))), Row(sdf1.format(new Timestamp(-1000000)))))
- checkAnswer(
- df.select(from_unixtime(col("a"), fmt2)),
- Seq(Row(sdf2.format(new Timestamp(1000000))), Row(sdf2.format(new Timestamp(-1000000)))))
- checkAnswer(
- df.select(from_unixtime(col("a"), fmt3)),
- Seq(Row(sdf3.format(new Timestamp(1000000))), Row(sdf3.format(new Timestamp(-1000000)))))
- checkAnswer(
- df.selectExpr("from_unixtime(a)"),
- Seq(Row(sdf1.format(new Timestamp(1000000))), Row(sdf1.format(new Timestamp(-1000000)))))
- checkAnswer(
- df.selectExpr(s"from_unixtime(a, '$fmt2')"),
- Seq(Row(sdf2.format(new Timestamp(1000000))), Row(sdf2.format(new Timestamp(-1000000)))))
- checkAnswer(
- df.selectExpr(s"from_unixtime(a, '$fmt3')"),
- Seq(Row(sdf3.format(new Timestamp(1000000))), Row(sdf3.format(new Timestamp(-1000000)))))
+ Seq(false, true).foreach { legacyParser =>
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> legacyParser.toString) {
+ val sdf1 = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss", Locale.US)
+ val fmt2 = "yyyy-MM-dd HH:mm:ss.SSS"
+ val sdf2 = new SimpleDateFormat(fmt2, Locale.US)
+ val fmt3 = "yy-MM-dd HH-mm-ss"
+ val sdf3 = new SimpleDateFormat(fmt3, Locale.US)
+ val df = Seq((1000, "yyyy-MM-dd HH:mm:ss.SSS"), (-1000, "yy-MM-dd HH-mm-ss")).toDF("a", "b")
+ checkAnswer(
+ df.select(from_unixtime(col("a"))),
+ Seq(Row(sdf1.format(new Timestamp(1000000))), Row(sdf1.format(new Timestamp(-1000000)))))
+ checkAnswer(
+ df.select(from_unixtime(col("a"), fmt2)),
+ Seq(Row(sdf2.format(new Timestamp(1000000))), Row(sdf2.format(new Timestamp(-1000000)))))
+ checkAnswer(
+ df.select(from_unixtime(col("a"), fmt3)),
+ Seq(Row(sdf3.format(new Timestamp(1000000))), Row(sdf3.format(new Timestamp(-1000000)))))
+ checkAnswer(
+ df.selectExpr("from_unixtime(a)"),
+ Seq(Row(sdf1.format(new Timestamp(1000000))), Row(sdf1.format(new Timestamp(-1000000)))))
+ checkAnswer(
+ df.selectExpr(s"from_unixtime(a, '$fmt2')"),
+ Seq(Row(sdf2.format(new Timestamp(1000000))), Row(sdf2.format(new Timestamp(-1000000)))))
+ checkAnswer(
+ df.selectExpr(s"from_unixtime(a, '$fmt3')"),
+ Seq(Row(sdf3.format(new Timestamp(1000000))), Row(sdf3.format(new Timestamp(-1000000)))))
+ }
+ }
}
private def secs(millis: Long): Long = TimeUnit.MILLISECONDS.toSeconds(millis)
test("unix_timestamp") {
- val date1 = Date.valueOf("2015-07-24")
- val date2 = Date.valueOf("2015-07-25")
- val ts1 = Timestamp.valueOf("2015-07-24 10:00:00.3")
- val ts2 = Timestamp.valueOf("2015-07-25 02:02:02.2")
- val s1 = "2015/07/24 10:00:00.5"
- val s2 = "2015/07/25 02:02:02.6"
- val ss1 = "2015-07-24 10:00:00"
- val ss2 = "2015-07-25 02:02:02"
- val fmt = "yyyy/MM/dd HH:mm:ss.S"
- val df = Seq((date1, ts1, s1, ss1), (date2, ts2, s2, ss2)).toDF("d", "ts", "s", "ss")
- checkAnswer(df.select(unix_timestamp(col("ts"))), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.select(unix_timestamp(col("ss"))), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.select(unix_timestamp(col("d"), fmt)), Seq(
- Row(secs(date1.getTime)), Row(secs(date2.getTime))))
- checkAnswer(df.select(unix_timestamp(col("s"), fmt)), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.selectExpr("unix_timestamp(ts)"), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.selectExpr("unix_timestamp(ss)"), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.selectExpr(s"unix_timestamp(d, '$fmt')"), Seq(
- Row(secs(date1.getTime)), Row(secs(date2.getTime))))
- checkAnswer(df.selectExpr(s"unix_timestamp(s, '$fmt')"), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
-
- val x1 = "2015-07-24 10:00:00"
- val x2 = "2015-25-07 02:02:02"
- val x3 = "2015-07-24 25:02:02"
- val x4 = "2015-24-07 26:02:02"
- val ts3 = Timestamp.valueOf("2015-07-24 02:25:02")
- val ts4 = Timestamp.valueOf("2015-07-24 00:10:00")
-
- val df1 = Seq(x1, x2, x3, x4).toDF("x")
- checkAnswer(df1.select(unix_timestamp(col("x"))), Seq(
- Row(secs(ts1.getTime)), Row(null), Row(null), Row(null)))
- checkAnswer(df1.selectExpr("unix_timestamp(x)"), Seq(
- Row(secs(ts1.getTime)), Row(null), Row(null), Row(null)))
- checkAnswer(df1.select(unix_timestamp(col("x"), "yyyy-dd-MM HH:mm:ss")), Seq(
- Row(null), Row(secs(ts2.getTime)), Row(null), Row(null)))
- checkAnswer(df1.selectExpr(s"unix_timestamp(x, 'yyyy-MM-dd mm:HH:ss')"), Seq(
- Row(secs(ts4.getTime)), Row(null), Row(secs(ts3.getTime)), Row(null)))
-
- // invalid format
- checkAnswer(df1.selectExpr(s"unix_timestamp(x, 'yyyy-MM-dd aa:HH:ss')"), Seq(
- Row(null), Row(null), Row(null), Row(null)))
-
- // february
- val y1 = "2016-02-29"
- val y2 = "2017-02-29"
- val ts5 = Timestamp.valueOf("2016-02-29 00:00:00")
- val df2 = Seq(y1, y2).toDF("y")
- checkAnswer(df2.select(unix_timestamp(col("y"), "yyyy-MM-dd")), Seq(
- Row(secs(ts5.getTime)), Row(null)))
-
- val now = sql("select unix_timestamp()").collect().head.getLong(0)
- checkAnswer(
- sql(s"select cast ($now as timestamp)"),
- Row(new java.util.Date(TimeUnit.SECONDS.toMillis(now))))
+ Seq(false, true).foreach { legacyParser =>
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> legacyParser.toString) {
+ val date1 = Date.valueOf("2015-07-24")
+ val date2 = Date.valueOf("2015-07-25")
+ val ts1 = Timestamp.valueOf("2015-07-24 10:00:00.3")
+ val ts2 = Timestamp.valueOf("2015-07-25 02:02:02.2")
+ val s1 = "2015/07/24 10:00:00.5"
+ val s2 = "2015/07/25 02:02:02.6"
+ val ss1 = "2015-07-24 10:00:00"
+ val ss2 = "2015-07-25 02:02:02"
+ val fmt = "yyyy/MM/dd HH:mm:ss.S"
+ val df = Seq((date1, ts1, s1, ss1), (date2, ts2, s2, ss2)).toDF("d", "ts", "s", "ss")
+ checkAnswer(df.select(unix_timestamp(col("ts"))), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.select(unix_timestamp(col("ss"))), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.select(unix_timestamp(col("d"), fmt)), Seq(
+ Row(secs(date1.getTime)), Row(secs(date2.getTime))))
+ checkAnswer(df.select(unix_timestamp(col("s"), fmt)), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.selectExpr("unix_timestamp(ts)"), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.selectExpr("unix_timestamp(ss)"), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.selectExpr(s"unix_timestamp(d, '$fmt')"), Seq(
+ Row(secs(date1.getTime)), Row(secs(date2.getTime))))
+ checkAnswer(df.selectExpr(s"unix_timestamp(s, '$fmt')"), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+
+ val x1 = "2015-07-24 10:00:00"
+ val x2 = "2015-25-07 02:02:02"
+ val x3 = "2015-07-24 25:02:02"
+ val x4 = "2015-24-07 26:02:02"
+ val ts3 = Timestamp.valueOf("2015-07-24 02:25:02")
+ val ts4 = Timestamp.valueOf("2015-07-24 00:10:00")
+
+ val df1 = Seq(x1, x2, x3, x4).toDF("x")
+ checkAnswer(df1.select(unix_timestamp(col("x"))), Seq(
+ Row(secs(ts1.getTime)), Row(null), Row(null), Row(null)))
+ checkAnswer(df1.selectExpr("unix_timestamp(x)"), Seq(
+ Row(secs(ts1.getTime)), Row(null), Row(null), Row(null)))
+ checkAnswer(df1.select(unix_timestamp(col("x"), "yyyy-dd-MM HH:mm:ss")), Seq(
+ Row(null), Row(secs(ts2.getTime)), Row(null), Row(null)))
+ checkAnswer(df1.selectExpr(s"unix_timestamp(x, 'yyyy-MM-dd mm:HH:ss')"), Seq(
+ Row(secs(ts4.getTime)), Row(null), Row(secs(ts3.getTime)), Row(null)))
+
+ // invalid format
+ checkAnswer(df1.selectExpr(s"unix_timestamp(x, 'yyyy-MM-dd aa:HH:ss')"), Seq(
+ Row(null), Row(null), Row(null), Row(null)))
+
+ // february
+ val y1 = "2016-02-29"
+ val y2 = "2017-02-29"
+ val ts5 = Timestamp.valueOf("2016-02-29 00:00:00")
+ val df2 = Seq(y1, y2).toDF("y")
+ checkAnswer(df2.select(unix_timestamp(col("y"), "yyyy-MM-dd")), Seq(
+ Row(secs(ts5.getTime)), Row(null)))
+
+ val now = sql("select unix_timestamp()").collect().head.getLong(0)
+ checkAnswer(
+ sql(s"select cast ($now as timestamp)"),
+ Row(new java.util.Date(TimeUnit.SECONDS.toMillis(now))))
+ }
+ }
}
test("to_unix_timestamp") {
- val date1 = Date.valueOf("2015-07-24")
- val date2 = Date.valueOf("2015-07-25")
- val ts1 = Timestamp.valueOf("2015-07-24 10:00:00.3")
- val ts2 = Timestamp.valueOf("2015-07-25 02:02:02.2")
- val s1 = "2015/07/24 10:00:00.5"
- val s2 = "2015/07/25 02:02:02.6"
- val ss1 = "2015-07-24 10:00:00"
- val ss2 = "2015-07-25 02:02:02"
- val fmt = "yyyy/MM/dd HH:mm:ss.S"
- val df = Seq((date1, ts1, s1, ss1), (date2, ts2, s2, ss2)).toDF("d", "ts", "s", "ss")
- checkAnswer(df.selectExpr("to_unix_timestamp(ts)"), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.selectExpr("to_unix_timestamp(ss)"), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
- checkAnswer(df.selectExpr(s"to_unix_timestamp(d, '$fmt')"), Seq(
- Row(secs(date1.getTime)), Row(secs(date2.getTime))))
- checkAnswer(df.selectExpr(s"to_unix_timestamp(s, '$fmt')"), Seq(
- Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
-
- val x1 = "2015-07-24 10:00:00"
- val x2 = "2015-25-07 02:02:02"
- val x3 = "2015-07-24 25:02:02"
- val x4 = "2015-24-07 26:02:02"
- val ts3 = Timestamp.valueOf("2015-07-24 02:25:02")
- val ts4 = Timestamp.valueOf("2015-07-24 00:10:00")
-
- val df1 = Seq(x1, x2, x3, x4).toDF("x")
- checkAnswer(df1.selectExpr("to_unix_timestamp(x)"), Seq(
- Row(secs(ts1.getTime)), Row(null), Row(null), Row(null)))
- checkAnswer(df1.selectExpr(s"to_unix_timestamp(x, 'yyyy-MM-dd mm:HH:ss')"), Seq(
- Row(secs(ts4.getTime)), Row(null), Row(secs(ts3.getTime)), Row(null)))
-
- // february
- val y1 = "2016-02-29"
- val y2 = "2017-02-29"
- val ts5 = Timestamp.valueOf("2016-02-29 00:00:00")
- val df2 = Seq(y1, y2).toDF("y")
- checkAnswer(df2.select(unix_timestamp(col("y"), "yyyy-MM-dd")), Seq(
- Row(secs(ts5.getTime)), Row(null)))
-
- // invalid format
- checkAnswer(df1.selectExpr(s"to_unix_timestamp(x, 'yyyy-MM-dd bb:HH:ss')"), Seq(
- Row(null), Row(null), Row(null), Row(null)))
+ Seq(false, true).foreach { legacyParser =>
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> legacyParser.toString) {
+ val date1 = Date.valueOf("2015-07-24")
+ val date2 = Date.valueOf("2015-07-25")
+ val ts1 = Timestamp.valueOf("2015-07-24 10:00:00.3")
+ val ts2 = Timestamp.valueOf("2015-07-25 02:02:02.2")
+ val s1 = "2015/07/24 10:00:00.5"
+ val s2 = "2015/07/25 02:02:02.6"
+ val ss1 = "2015-07-24 10:00:00"
+ val ss2 = "2015-07-25 02:02:02"
+ val fmt = "yyyy/MM/dd HH:mm:ss.S"
+ val df = Seq((date1, ts1, s1, ss1), (date2, ts2, s2, ss2)).toDF("d", "ts", "s", "ss")
+ checkAnswer(df.selectExpr("to_unix_timestamp(ts)"), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.selectExpr("to_unix_timestamp(ss)"), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+ checkAnswer(df.selectExpr(s"to_unix_timestamp(d, '$fmt')"), Seq(
+ Row(secs(date1.getTime)), Row(secs(date2.getTime))))
+ checkAnswer(df.selectExpr(s"to_unix_timestamp(s, '$fmt')"), Seq(
+ Row(secs(ts1.getTime)), Row(secs(ts2.getTime))))
+
+ val x1 = "2015-07-24 10:00:00"
+ val x2 = "2015-25-07 02:02:02"
+ val x3 = "2015-07-24 25:02:02"
+ val x4 = "2015-24-07 26:02:02"
+ val ts3 = Timestamp.valueOf("2015-07-24 02:25:02")
+ val ts4 = Timestamp.valueOf("2015-07-24 00:10:00")
+
+ val df1 = Seq(x1, x2, x3, x4).toDF("x")
+ checkAnswer(df1.selectExpr("to_unix_timestamp(x)"), Seq(
+ Row(secs(ts1.getTime)), Row(null), Row(null), Row(null)))
+ checkAnswer(df1.selectExpr(s"to_unix_timestamp(x, 'yyyy-MM-dd mm:HH:ss')"), Seq(
+ Row(secs(ts4.getTime)), Row(null), Row(secs(ts3.getTime)), Row(null)))
+
+ // february
+ val y1 = "2016-02-29"
+ val y2 = "2017-02-29"
+ val ts5 = Timestamp.valueOf("2016-02-29 00:00:00")
+ val df2 = Seq(y1, y2).toDF("y")
+ checkAnswer(df2.select(unix_timestamp(col("y"), "yyyy-MM-dd")), Seq(
+ Row(secs(ts5.getTime)), Row(null)))
+
+ // invalid format
+ checkAnswer(df1.selectExpr(s"to_unix_timestamp(x, 'yyyy-MM-dd bb:HH:ss')"), Seq(
+ Row(null), Row(null), Row(null), Row(null)))
+ }
+ }
}
test("to_timestamp") {
- val date1 = Date.valueOf("2015-07-24")
- val date2 = Date.valueOf("2015-07-25")
- val ts_date1 = Timestamp.valueOf("2015-07-24 00:00:00")
- val ts_date2 = Timestamp.valueOf("2015-07-25 00:00:00")
- val ts1 = Timestamp.valueOf("2015-07-24 10:00:00")
- val ts2 = Timestamp.valueOf("2015-07-25 02:02:02")
- val s1 = "2015/07/24 10:00:00.5"
- val s2 = "2015/07/25 02:02:02.6"
- val ts1m = Timestamp.valueOf("2015-07-24 10:00:00.5")
- val ts2m = Timestamp.valueOf("2015-07-25 02:02:02.6")
- val ss1 = "2015-07-24 10:00:00"
- val ss2 = "2015-07-25 02:02:02"
- val fmt = "yyyy/MM/dd HH:mm:ss.S"
- val df = Seq((date1, ts1, s1, ss1), (date2, ts2, s2, ss2)).toDF("d", "ts", "s", "ss")
-
- checkAnswer(df.select(to_timestamp(col("ss"))),
- df.select(unix_timestamp(col("ss")).cast("timestamp")))
- checkAnswer(df.select(to_timestamp(col("ss"))), Seq(
- Row(ts1), Row(ts2)))
- checkAnswer(df.select(to_timestamp(col("s"), fmt)), Seq(
- Row(ts1m), Row(ts2m)))
- checkAnswer(df.select(to_timestamp(col("ts"), fmt)), Seq(
- Row(ts1), Row(ts2)))
- checkAnswer(df.select(to_timestamp(col("d"), "yyyy-MM-dd")), Seq(
- Row(ts_date1), Row(ts_date2)))
+ Seq(false, true).foreach { legacyParser =>
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> legacyParser.toString) {
+ val date1 = Date.valueOf("2015-07-24")
+ val date2 = Date.valueOf("2015-07-25")
+ val ts_date1 = Timestamp.valueOf("2015-07-24 00:00:00")
+ val ts_date2 = Timestamp.valueOf("2015-07-25 00:00:00")
+ val ts1 = Timestamp.valueOf("2015-07-24 10:00:00")
+ val ts2 = Timestamp.valueOf("2015-07-25 02:02:02")
+ val s1 = "2015/07/24 10:00:00.5"
+ val s2 = "2015/07/25 02:02:02.6"
+ val ts1m = Timestamp.valueOf("2015-07-24 10:00:00.5")
+ val ts2m = Timestamp.valueOf("2015-07-25 02:02:02.6")
+ val ss1 = "2015-07-24 10:00:00"
+ val ss2 = "2015-07-25 02:02:02"
+ val fmt = "yyyy/MM/dd HH:mm:ss.S"
+ val df = Seq((date1, ts1, s1, ss1), (date2, ts2, s2, ss2)).toDF("d", "ts", "s", "ss")
+
+ checkAnswer(df.select(to_timestamp(col("ss"))),
+ df.select(unix_timestamp(col("ss")).cast("timestamp")))
+ checkAnswer(df.select(to_timestamp(col("ss"))), Seq(
+ Row(ts1), Row(ts2)))
+ if (legacyParser) {
Review comment:
@cloud-fan Only here, I have to modify the test to adopt it for the legacy parser.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584517025
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [SPARK-30788][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584721908
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118232/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524:
[SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as
legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584574706
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22992/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584348974
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118172/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377525133
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
##########
@@ -31,6 +31,7 @@ import org.apache.spark.sql.catalyst.util.{DateTimeUtils, IntervalUtils, Timesta
import org.apache.spark.sql.catalyst.util.DateTimeConstants.NANOS_PER_SECOND
import org.apache.spark.sql.catalyst.util.DateTimeTestUtils._
import org.apache.spark.sql.catalyst.util.DateTimeUtils.TimeZoneGMT
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
Review comment:
I wrapped the tests that are affected by:
```scala
Seq(false, true).foreach { legacyParser =>
withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> legacyParser.toString) {
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584517649
jenkins, retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584296561
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524:
[SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as
legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584568392
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584296561
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584510530
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22975/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27524:
[SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as
legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27524: [SPARK-30788][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584518459
**[Test build #118219 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118219/testReport)** for PR 27524 at commit [`93f3ae1`](https://github.com/apache/spark/commit/93f3ae1c32f893cc3cc22ae075753098440e76b5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584518925
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584389466
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584462839
**[Test build #118185 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118185/testReport)** for PR 27524 at commit [`566170a`](https://github.com/apache/spark/commit/566170a6d7581f013142b19fcfdd2d9ea4ebaf9f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377421940
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
##########
@@ -90,47 +91,168 @@ class FractionTimestampFormatter(zoneId: ZoneId)
override protected lazy val formatter = DateTimeFormatterHelper.fractionFormatter
}
-class LegacyTimestampFormatter(
+/**
+ * The custom sub-class of `GregorianCalendar` is needed to get access to
+ * protected `fields` immediately after parsing. We cannot use
+ * the `get()` method because it performs normalization of the fraction
+ * part. Accordingly, the `MILLISECOND` field doesn't contain original value.
+ *
+ * Also this class allows to set raw value to the `MILLISECOND` field
+ * directly before formatting.
+ */
+class MicrosCalendar(tz: TimeZone, digitsInFraction: Int)
+ extends GregorianCalendar(tz, Locale.US) {
+ // Converts parsed `MILLISECOND` field to seconds fraction in microsecond precision.
+ // For example if the fraction pattern is `SSSS` then `digitsInFraction` = 4, and
+ // if the `MILLISECOND` field was parsed to `1234`.
+ def getMicros(): SQLTimestamp = {
+ // Append 6 zeros to the field: 1234 -> 1234000000
+ val d = fields(Calendar.MILLISECOND) * MICROS_PER_SECOND
+ // Take the first 6 digits from `d`: 1234000000 -> 123400
+ // The rest contains exactly `digitsInFraction`: `0000` = 10 ^ digitsInFraction
+ // So, the result is `(1234 * 1000000) / (10 ^ digitsInFraction)
+ d / Decimal.POW_10(digitsInFraction)
+ }
+
+ // Converts the seconds fraction in microsecond precision to a value
+ // that can be correctly formatted according to the specified fraction pattern.
+ // The method performs operations opposite to `getMicros()`.
+ def setMicros(micros: Long): Unit = {
+ val d = micros * Decimal.POW_10(digitsInFraction)
+ fields(Calendar.MILLISECOND) = (d / MICROS_PER_SECOND).toInt
+ }
+}
+
+/**
+ * An instance of the class is aimed to re-use many times. It contains helper objects
+ * `cal` which is reused between `parse()` and `format` invokes.
+ */
+class LegacyFastDateFormat(fastDateFormat: FastDateFormat) {
+ private val cal = new MicrosCalendar(
+ fastDateFormat.getTimeZone,
+ fastDateFormat.getPattern.count(_ == 'S'))
+
+ def parse(s: String): SQLTimestamp = {
+ cal.clear() // Clear the calendar because it can be re-used many times
+ if (!fastDateFormat.parse(s, new ParsePosition(0), cal)) {
+ throw new IllegalArgumentException(s"'$s' is an invalid timestamp")
+ }
+ val micros = cal.getMicros()
+ cal.set(Calendar.MILLISECOND, 0)
+ cal.getTimeInMillis * MICROS_PER_MILLIS + micros
+ }
+
+ def format(timestamp: SQLTimestamp): String = {
+ cal.setTimeInMillis(Math.floorDiv(timestamp, MICROS_PER_SECOND) * MILLIS_PER_SECOND)
+ cal.setMicros(Math.floorMod(timestamp, MICROS_PER_SECOND))
+ fastDateFormat.format(cal)
+ }
+}
+
+class LegacyFastTimestampFormatter(
pattern: String,
zoneId: ZoneId,
locale: Locale) extends TimestampFormatter {
- @transient private lazy val format =
+ @transient private lazy val fastDateFormat =
FastDateFormat.getInstance(pattern, TimeZone.getTimeZone(zoneId), locale)
+ @transient private lazy val cal = new MicrosCalendar(
+ fastDateFormat.getTimeZone,
+ fastDateFormat.getPattern.count(_ == 'S'))
Review comment:
I forgot. is this also ported from 2.4?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27524: [WIP][SQL]
Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp
formatters
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584295874
**[Test build #118175 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118175/testReport)** for PR 27524 at commit [`ab1d57f`](https://github.com/apache/spark/commit/ab1d57f1b131db20c32ac1169730d3e8f45f4698).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377515494
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
##########
@@ -31,6 +31,7 @@ import org.apache.spark.sql.catalyst.util.{DateTimeUtils, IntervalUtils, Timesta
import org.apache.spark.sql.catalyst.util.DateTimeConstants.NANOS_PER_SECOND
import org.apache.spark.sql.catalyst.util.DateTimeTestUtils._
import org.apache.spark.sql.catalyst.util.DateTimeUtils.TimeZoneGMT
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
Review comment:
all the tests in this file are not affected by the new or legacy formatter?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27524:
[WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy
date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#discussion_r377437841
##########
File path: sql/core/src/test/resources/test-data/bad_after_good.csv
##########
@@ -1,2 +1,2 @@
"good record",1999-08-01
-"bad record",1999-088-01
+"bad record",1999-088_01
Review comment:
do we run these tests with legacy formatter?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27524: [WIP][SQL] Support
`SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27524: [WIP][SQL] Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters
URL: https://github.com/apache/spark/pull/27524#issuecomment-584388899
**[Test build #118185 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118185/testReport)** for PR 27524 at commit [`566170a`](https://github.com/apache/spark/commit/566170a6d7581f013142b19fcfdd2d9ea4ebaf9f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org