You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2018/12/27 03:10:09 UTC
[spark] branch master updated: [SPARK-26424][SQL] Use java.time API
in date/timestamp expressions
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 7c7fccf [SPARK-26424][SQL] Use java.time API in date/timestamp expressions
7c7fccf is described below
commit 7c7fccfeb5bc079fede41eb64f57ab6b1b4b9018
Author: Maxim Gekk <ma...@databricks.com>
AuthorDate: Thu Dec 27 11:09:50 2018 +0800
[SPARK-26424][SQL] Use java.time API in date/timestamp expressions
## What changes were proposed in this pull request?
In the PR, I propose to switch the `DateFormatClass`, `ToUnixTimestamp`, `FromUnixTime`, `UnixTime` on java.time API for parsing/formatting dates and timestamps. The API has been already implemented by the `Timestamp`/`DateFormatter` classes. One of benefit is those classes support parsing timestamps with microsecond precision. Old behaviour can be switched on via SQL config: `spark.sql.legacy.timeParser.enabled` (`false` by default).
## How was this patch tested?
It was tested by existing test suites - `DateFunctionsSuite`, `DateExpressionsSuite`, `JsonSuite`, `CsvSuite`, `SQLQueryTestSuite` as well as PySpark tests.
Closes #23358 from MaxGekk/new-time-cast.
Lead-authored-by: Maxim Gekk <ma...@databricks.com>
Co-authored-by: Maxim Gekk <ma...@gmail.com>
Signed-off-by: Wenchen Fan <we...@databricks.com>
---
R/pkg/R/functions.R | 8 +--
docs/sql-migration-guide-upgrade.md | 1 +
python/pyspark/sql/functions.py | 6 +-
.../spark/sql/catalyst/csv/CSVInferSchema.scala | 3 +-
.../catalyst/expressions/datetimeExpressions.scala | 82 +++++++++++++---------
.../spark/sql/catalyst/json/JsonInferSchema.scala | 3 +-
.../spark/sql/catalyst/util/DateFormatter.scala | 8 ++-
.../catalyst/util/DateTimeFormatterHelper.scala | 21 ++++--
.../spark/sql/catalyst/util/DateTimeUtils.scala | 10 ---
.../sql/catalyst/util/TimestampFormatter.scala | 22 +++++-
.../sql/catalyst/csv/UnivocityParserSuite.scala | 2 +-
.../apache/spark/sql/util/DateFormatterSuite.scala | 7 ++
.../spark/sql/util/TimestampFormatterSuite.scala | 12 ++++
.../scala/org/apache/spark/sql/functions.scala | 10 +--
.../org/apache/spark/sql/DateFunctionsSuite.scala | 2 +-
15 files changed, 122 insertions(+), 75 deletions(-)
diff --git a/R/pkg/R/functions.R b/R/pkg/R/functions.R
index f568a93..5b3cc09 100644
--- a/R/pkg/R/functions.R
+++ b/R/pkg/R/functions.R
@@ -1723,7 +1723,7 @@ setMethod("radians",
#' @details
#' \code{to_date}: Converts the column into a DateType. You may optionally specify
#' a format according to the rules in:
-#' \url{http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html}.
+#' \url{https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html}.
#' If the string cannot be parsed according to the specified format (or default),
#' the value of the column will be null.
#' By default, it follows casting rules to a DateType if the format is omitted
@@ -1819,7 +1819,7 @@ setMethod("to_csv", signature(x = "Column"),
#' @details
#' \code{to_timestamp}: Converts the column into a TimestampType. You may optionally specify
#' a format according to the rules in:
-#' \url{http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html}.
+#' \url{https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html}.
#' If the string cannot be parsed according to the specified format (or default),
#' the value of the column will be null.
#' By default, it follows casting rules to a TimestampType if the format is omitted
@@ -2240,7 +2240,7 @@ setMethod("n", signature(x = "Column"),
#' \code{date_format}: Converts a date/timestamp/string to a value of string in the format
#' specified by the date format given by the second argument. A pattern could be for instance
#' \code{dd.MM.yyyy} and could return a string like '18.03.1993'. All
-#' pattern letters of \code{java.text.SimpleDateFormat} can be used.
+#' pattern letters of \code{java.time.format.DateTimeFormatter} can be used.
#' Note: Use when ever possible specialized functions like \code{year}. These benefit from a
#' specialized implementation.
#'
@@ -2666,7 +2666,7 @@ setMethod("format_string", signature(format = "character", x = "Column"),
#' \code{from_unixtime}: Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC)
#' to a string representing the timestamp of that moment in the current system time zone in the JVM
#' in the given format.
-#' See \href{http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html}{
+#' See \href{https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html}{
#' Customizing Formats} for available options.
#'
#' @rdname column_datetime_functions
diff --git a/docs/sql-migration-guide-upgrade.md b/docs/sql-migration-guide-upgrade.md
index 1bd3b5a..c4d2157 100644
--- a/docs/sql-migration-guide-upgrade.md
+++ b/docs/sql-migration-guide-upgrade.md
@@ -39,6 +39,7 @@ displayTitle: Spark SQL Upgrading Guide
- In Spark version 2.4 and earlier, JSON datasource and JSON functions like `from_json` convert a bad JSON record to a row with all `null`s in the PERMISSIVE mode when specified schema is `StructType`. Since Spark 3.0, the returned row can contain non-`null` fields if some of JSON column values were parsed and converted to desired types successfully.
+ - Since Spark 3.0, the `unix_timestamp`, `date_format`, `to_unix_timestamp`, `from_unixtime`, `to_date`, `to_timestamp` functions use java.time API for parsing and formatting dates/timestamps from/to strings by using ISO chronology (https://docs.oracle.com/javase/8/docs/api/java/time/chrono/IsoChronology.html) based on Proleptic Gregorian calendar. In Spark version 2.4 and earlier, java.text.SimpleDateFormat and java.util.GregorianCalendar (hybrid calendar that supports both the Julian [...]
## Upgrading From Spark SQL 2.3 to 2.4
- In Spark version 2.3 and earlier, the second parameter to array_contains function is implicitly promoted to the element type of first array type parameter. This type promotion can be lossy and may cause `array_contains` function to return wrong result. This problem has been addressed in 2.4 by employing a safer type promotion mechanism. This can cause some change in behavior and are illustrated in the table below.
diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index d188de3..d2a771e 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -874,7 +874,7 @@ def date_format(date, format):
format given by the second argument.
A pattern could be for instance `dd.MM.yyyy` and could return a string like '18.03.1993'. All
- pattern letters of the Java class `java.text.SimpleDateFormat` can be used.
+ pattern letters of the Java class `java.time.format.DateTimeFormatter` can be used.
.. note:: Use when ever possible specialized functions like `year`. These benefit from a
specialized implementation.
@@ -1094,7 +1094,7 @@ def to_date(col, format=None):
"""Converts a :class:`Column` of :class:`pyspark.sql.types.StringType` or
:class:`pyspark.sql.types.TimestampType` into :class:`pyspark.sql.types.DateType`
using the optionally specified format. Specify formats according to
- `SimpleDateFormats <http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html>`_.
+ `DateTimeFormatter <https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html>`_. # noqa
By default, it follows casting rules to :class:`pyspark.sql.types.DateType` if the format
is omitted (equivalent to ``col.cast("date")``).
@@ -1119,7 +1119,7 @@ def to_timestamp(col, format=None):
"""Converts a :class:`Column` of :class:`pyspark.sql.types.StringType` or
:class:`pyspark.sql.types.TimestampType` into :class:`pyspark.sql.types.DateType`
using the optionally specified format. Specify formats according to
- `SimpleDateFormats <http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html>`_.
+ `DateTimeFormatter <https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html>`_. # noqa
By default, it follows casting rules to :class:`pyspark.sql.types.TimestampType` if the format
is omitted (equivalent to ``col.cast("timestamp")``).
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVInferSchema.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVInferSchema.scala
index 35ade13..4dd4104 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVInferSchema.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVInferSchema.scala
@@ -27,8 +27,7 @@ import org.apache.spark.sql.types._
class CSVInferSchema(val options: CSVOptions) extends Serializable {
- @transient
- private lazy val timestampParser = TimestampFormatter(
+ private val timestampParser = TimestampFormatter(
options.timestampFormat,
options.timeZone,
options.locale)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
index 45e17ae..73af0a3 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
@@ -18,8 +18,7 @@
package org.apache.spark.sql.catalyst.expressions
import java.sql.Timestamp
-import java.text.DateFormat
-import java.util.{Calendar, TimeZone}
+import java.util.{Calendar, Locale, TimeZone}
import scala.util.control.NonFatal
@@ -28,7 +27,8 @@ import org.apache.commons.lang3.StringEscapeUtils
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.codegen._
import org.apache.spark.sql.catalyst.expressions.codegen.Block._
-import org.apache.spark.sql.catalyst.util.DateTimeUtils
+import org.apache.spark.sql.catalyst.util.{DateTimeUtils, TimestampFormatter}
+import org.apache.spark.sql.catalyst.util.DateTimeUtils._
import org.apache.spark.sql.types._
import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
@@ -562,16 +562,17 @@ case class DateFormatClass(left: Expression, right: Expression, timeZoneId: Opti
copy(timeZoneId = Option(timeZoneId))
override protected def nullSafeEval(timestamp: Any, format: Any): Any = {
- val df = DateTimeUtils.newDateFormat(format.toString, timeZone)
- UTF8String.fromString(df.format(new java.util.Date(timestamp.asInstanceOf[Long] / 1000)))
+ val df = TimestampFormatter(format.toString, timeZone, Locale.US)
+ UTF8String.fromString(df.format(timestamp.asInstanceOf[Long]))
}
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
- val dtu = DateTimeUtils.getClass.getName.stripSuffix("$")
+ val tf = TimestampFormatter.getClass.getName.stripSuffix("$")
val tz = ctx.addReferenceObj("timeZone", timeZone)
+ val locale = ctx.addReferenceObj("locale", Locale.US)
defineCodeGen(ctx, ev, (timestamp, format) => {
- s"""UTF8String.fromString($dtu.newDateFormat($format.toString(), $tz)
- .format(new java.util.Date($timestamp / 1000)))"""
+ s"""UTF8String.fromString($tf.apply($format.toString(), $tz, $locale)
+ .format($timestamp))"""
})
}
@@ -612,9 +613,10 @@ case class ToUnixTimestamp(
}
/**
- * Converts time string with given pattern.
- * (see [http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html])
- * to Unix time stamp (in seconds), returns null if fail.
+ * Converts time string with given pattern to Unix time stamp (in seconds), returns null if fail.
+ * See [http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html]
+ * if SQL config spark.sql.legacy.timeParser.enabled is set to true otherwise
+ * [https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html].
* Note that hive Language Manual says it returns 0 if fail, but in fact it returns null.
* If the second parameter is missing, use "yyyy-MM-dd HH:mm:ss".
* If no parameters provided, the first parameter will be current_timestamp.
@@ -663,9 +665,9 @@ abstract class UnixTime
override def nullable: Boolean = true
private lazy val constFormat: UTF8String = right.eval().asInstanceOf[UTF8String]
- private lazy val formatter: DateFormat =
+ private lazy val formatter: TimestampFormatter =
try {
- DateTimeUtils.newDateFormat(constFormat.toString, timeZone)
+ TimestampFormatter(constFormat.toString, timeZone, Locale.US)
} catch {
case NonFatal(_) => null
}
@@ -677,16 +679,16 @@ abstract class UnixTime
} else {
left.dataType match {
case DateType =>
- DateTimeUtils.daysToMillis(t.asInstanceOf[Int], timeZone) / 1000L
+ DateTimeUtils.daysToMillis(t.asInstanceOf[Int], timeZone) / MILLIS_PER_SECOND
case TimestampType =>
- t.asInstanceOf[Long] / 1000000L
+ t.asInstanceOf[Long] / MICROS_PER_SECOND
case StringType if right.foldable =>
if (constFormat == null || formatter == null) {
null
} else {
try {
formatter.parse(
- t.asInstanceOf[UTF8String].toString).getTime / 1000L
+ t.asInstanceOf[UTF8String].toString) / MICROS_PER_SECOND
} catch {
case NonFatal(_) => null
}
@@ -698,8 +700,8 @@ abstract class UnixTime
} else {
val formatString = f.asInstanceOf[UTF8String].toString
try {
- DateTimeUtils.newDateFormat(formatString, timeZone).parse(
- t.asInstanceOf[UTF8String].toString).getTime / 1000L
+ TimestampFormatter(formatString, timeZone, Locale.US).parse(
+ t.asInstanceOf[UTF8String].toString) / MICROS_PER_SECOND
} catch {
case NonFatal(_) => null
}
@@ -712,7 +714,7 @@ abstract class UnixTime
val javaType = CodeGenerator.javaType(dataType)
left.dataType match {
case StringType if right.foldable =>
- val df = classOf[DateFormat].getName
+ val df = classOf[TimestampFormatter].getName
if (formatter == null) {
ExprCode.forNullValue(dataType)
} else {
@@ -724,24 +726,35 @@ abstract class UnixTime
$javaType ${ev.value} = ${CodeGenerator.defaultValue(dataType)};
if (!${ev.isNull}) {
try {
- ${ev.value} = $formatterName.parse(${eval1.value}.toString()).getTime() / 1000L;
+ ${ev.value} = $formatterName.parse(${eval1.value}.toString()) / 1000000L;
+ } catch (java.lang.IllegalArgumentException e) {
+ ${ev.isNull} = true;
} catch (java.text.ParseException e) {
${ev.isNull} = true;
+ } catch (java.time.format.DateTimeParseException e) {
+ ${ev.isNull} = true;
+ } catch (java.time.DateTimeException e) {
+ ${ev.isNull} = true;
}
}""")
}
case StringType =>
val tz = ctx.addReferenceObj("timeZone", timeZone)
- val dtu = DateTimeUtils.getClass.getName.stripSuffix("$")
+ val locale = ctx.addReferenceObj("locale", Locale.US)
+ val dtu = TimestampFormatter.getClass.getName.stripSuffix("$")
nullSafeCodeGen(ctx, ev, (string, format) => {
s"""
try {
- ${ev.value} = $dtu.newDateFormat($format.toString(), $tz)
- .parse($string.toString()).getTime() / 1000L;
+ ${ev.value} = $dtu.apply($format.toString(), $tz, $locale)
+ .parse($string.toString()) / 1000000L;
} catch (java.lang.IllegalArgumentException e) {
${ev.isNull} = true;
} catch (java.text.ParseException e) {
${ev.isNull} = true;
+ } catch (java.time.format.DateTimeParseException e) {
+ ${ev.isNull} = true;
+ } catch (java.time.DateTimeException e) {
+ ${ev.isNull} = true;
}
"""
})
@@ -806,9 +819,9 @@ case class FromUnixTime(sec: Expression, format: Expression, timeZoneId: Option[
copy(timeZoneId = Option(timeZoneId))
private lazy val constFormat: UTF8String = right.eval().asInstanceOf[UTF8String]
- private lazy val formatter: DateFormat =
+ private lazy val formatter: TimestampFormatter =
try {
- DateTimeUtils.newDateFormat(constFormat.toString, timeZone)
+ TimestampFormatter(constFormat.toString, timeZone, Locale.US)
} catch {
case NonFatal(_) => null
}
@@ -823,8 +836,7 @@ case class FromUnixTime(sec: Expression, format: Expression, timeZoneId: Option[
null
} else {
try {
- UTF8String.fromString(formatter.format(
- new java.util.Date(time.asInstanceOf[Long] * 1000L)))
+ UTF8String.fromString(formatter.format(time.asInstanceOf[Long] * MICROS_PER_SECOND))
} catch {
case NonFatal(_) => null
}
@@ -835,8 +847,8 @@ case class FromUnixTime(sec: Expression, format: Expression, timeZoneId: Option[
null
} else {
try {
- UTF8String.fromString(DateTimeUtils.newDateFormat(f.toString, timeZone)
- .format(new java.util.Date(time.asInstanceOf[Long] * 1000L)))
+ UTF8String.fromString(TimestampFormatter(f.toString, timeZone, Locale.US)
+ .format(time.asInstanceOf[Long] * MICROS_PER_SECOND))
} catch {
case NonFatal(_) => null
}
@@ -846,7 +858,7 @@ case class FromUnixTime(sec: Expression, format: Expression, timeZoneId: Option[
}
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
- val df = classOf[DateFormat].getName
+ val df = classOf[TimestampFormatter].getName
if (format.foldable) {
if (formatter == null) {
ExprCode.forNullValue(StringType)
@@ -859,8 +871,7 @@ case class FromUnixTime(sec: Expression, format: Expression, timeZoneId: Option[
${CodeGenerator.javaType(dataType)} ${ev.value} = ${CodeGenerator.defaultValue(dataType)};
if (!${ev.isNull}) {
try {
- ${ev.value} = UTF8String.fromString($formatterName.format(
- new java.util.Date(${t.value} * 1000L)));
+ ${ev.value} = UTF8String.fromString($formatterName.format(${t.value} * 1000000L));
} catch (java.lang.IllegalArgumentException e) {
${ev.isNull} = true;
}
@@ -868,12 +879,13 @@ case class FromUnixTime(sec: Expression, format: Expression, timeZoneId: Option[
}
} else {
val tz = ctx.addReferenceObj("timeZone", timeZone)
- val dtu = DateTimeUtils.getClass.getName.stripSuffix("$")
+ val locale = ctx.addReferenceObj("locale", Locale.US)
+ val tf = TimestampFormatter.getClass.getName.stripSuffix("$")
nullSafeCodeGen(ctx, ev, (seconds, f) => {
s"""
try {
- ${ev.value} = UTF8String.fromString($dtu.newDateFormat($f.toString(), $tz).format(
- new java.util.Date($seconds * 1000L)));
+ ${ev.value} = UTF8String.fromString($tf.apply($f.toString(), $tz, $locale).
+ format($seconds * 1000000L));
} catch (java.lang.IllegalArgumentException e) {
${ev.isNull} = true;
}"""
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala
index d1bc00c..3203e62 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala
@@ -37,8 +37,7 @@ private[sql] class JsonInferSchema(options: JSONOptions) extends Serializable {
private val decimalParser = ExprUtils.getDecimalParser(options.locale)
- @transient
- private lazy val timestampFormatter = TimestampFormatter(
+ private val timestampFormatter = TimestampFormatter(
options.timestampFormat,
options.timeZone,
options.locale)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
index 9e8d51cc..b4c9967 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
@@ -26,7 +26,7 @@ import org.apache.commons.lang3.time.FastDateFormat
import org.apache.spark.sql.internal.SQLConf
-sealed trait DateFormatter {
+sealed trait DateFormatter extends Serializable {
def parse(s: String): Int // returns days since epoch
def format(days: Int): String
}
@@ -35,7 +35,8 @@ class Iso8601DateFormatter(
pattern: String,
locale: Locale) extends DateFormatter with DateTimeFormatterHelper {
- private val formatter = buildFormatter(pattern, locale)
+ @transient
+ private lazy val formatter = buildFormatter(pattern, locale)
private val UTC = ZoneId.of("UTC")
private def toInstant(s: String): Instant = {
@@ -56,7 +57,8 @@ class Iso8601DateFormatter(
}
class LegacyDateFormatter(pattern: String, locale: Locale) extends DateFormatter {
- private val format = FastDateFormat.getInstance(pattern, locale)
+ @transient
+ private lazy val format = FastDateFormat.getInstance(pattern, locale)
override def parse(s: String): Int = {
val milliseconds = format.parse(s).getTime
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala
index b85101d..91cc57e 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala
@@ -17,27 +17,36 @@
package org.apache.spark.sql.catalyst.util
-import java.time.{Instant, LocalDateTime, ZonedDateTime, ZoneId}
-import java.time.format.{DateTimeFormatter, DateTimeFormatterBuilder}
-import java.time.temporal.{ChronoField, TemporalAccessor}
+import java.time._
+import java.time.chrono.IsoChronology
+import java.time.format.{DateTimeFormatter, DateTimeFormatterBuilder, ResolverStyle}
+import java.time.temporal.{ChronoField, TemporalAccessor, TemporalQueries}
import java.util.Locale
trait DateTimeFormatterHelper {
protected def buildFormatter(pattern: String, locale: Locale): DateTimeFormatter = {
new DateTimeFormatterBuilder()
+ .parseCaseInsensitive()
.appendPattern(pattern)
- .parseDefaulting(ChronoField.YEAR_OF_ERA, 1970)
+ .parseDefaulting(ChronoField.ERA, 1)
.parseDefaulting(ChronoField.MONTH_OF_YEAR, 1)
.parseDefaulting(ChronoField.DAY_OF_MONTH, 1)
- .parseDefaulting(ChronoField.HOUR_OF_DAY, 0)
.parseDefaulting(ChronoField.MINUTE_OF_HOUR, 0)
.parseDefaulting(ChronoField.SECOND_OF_MINUTE, 0)
.toFormatter(locale)
+ .withChronology(IsoChronology.INSTANCE)
+ .withResolverStyle(ResolverStyle.STRICT)
}
protected def toInstantWithZoneId(temporalAccessor: TemporalAccessor, zoneId: ZoneId): Instant = {
- val localDateTime = LocalDateTime.from(temporalAccessor)
+ val localTime = if (temporalAccessor.query(TemporalQueries.localTime) == null) {
+ LocalTime.ofNanoOfDay(0)
+ } else {
+ LocalTime.from(temporalAccessor)
+ }
+ val localDate = LocalDate.from(temporalAccessor)
+ val localDateTime = LocalDateTime.of(localDate, localTime)
val zonedDateTime = ZonedDateTime.of(localDateTime, zoneId)
Instant.from(zonedDateTime)
}
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
index c6dfdbf..3e5e1fb 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
@@ -111,16 +111,6 @@ object DateTimeUtils {
computedTimeZones.computeIfAbsent(timeZoneId, computeTimeZone)
}
- def newDateFormat(formatString: String, timeZone: TimeZone): DateFormat = {
- val sdf = new SimpleDateFormat(formatString, Locale.US)
- sdf.setTimeZone(timeZone)
- // Enable strict parsing, if the input date/format is invalid, it will throw an exception.
- // e.g. to parse invalid date '2016-13-12', or '2016-01-12' with invalid format 'yyyy-aa-dd',
- // an exception will be throwed.
- sdf.setLenient(false)
- sdf
- }
-
// we should use the exact day as Int, for example, (year, month, day) -> day
def millisToDays(millisUtc: Long): SQLDate = {
millisToDays(millisUtc, defaultTimeZone())
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
index eb13033..b67b2d7 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
@@ -17,7 +17,9 @@
package org.apache.spark.sql.catalyst.util
+import java.text.ParseException
import java.time._
+import java.time.format.DateTimeParseException
import java.time.temporal.TemporalQueries
import java.util.{Locale, TimeZone}
@@ -27,7 +29,19 @@ import org.apache.commons.lang3.time.FastDateFormat
import org.apache.spark.sql.internal.SQLConf
-sealed trait TimestampFormatter {
+sealed trait TimestampFormatter extends Serializable {
+ /**
+ * Parses a timestamp in a string and converts it to microseconds.
+ *
+ * @param s - string with timestamp to parse
+ * @return microseconds since epoch.
+ * @throws ParseException can be thrown by legacy parser
+ * @throws DateTimeParseException can be thrown by new parser
+ * @throws DateTimeException unable to obtain local date or time
+ */
+ @throws(classOf[ParseException])
+ @throws(classOf[DateTimeParseException])
+ @throws(classOf[DateTimeException])
def parse(s: String): Long // returns microseconds since epoch
def format(us: Long): String
}
@@ -36,7 +50,8 @@ class Iso8601TimestampFormatter(
pattern: String,
timeZone: TimeZone,
locale: Locale) extends TimestampFormatter with DateTimeFormatterHelper {
- private val formatter = buildFormatter(pattern, locale)
+ @transient
+ private lazy val formatter = buildFormatter(pattern, locale)
private def toInstant(s: String): Instant = {
val temporalAccessor = formatter.parse(s)
@@ -68,7 +83,8 @@ class LegacyTimestampFormatter(
pattern: String,
timeZone: TimeZone,
locale: Locale) extends TimestampFormatter {
- private val format = FastDateFormat.getInstance(pattern, timeZone, locale)
+ @transient
+ private lazy val format = FastDateFormat.getInstance(pattern, timeZone, locale)
protected def toMillis(s: String): Long = format.parse(s).getTime
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/csv/UnivocityParserSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/csv/UnivocityParserSuite.scala
index 2d0b0d3..4ae61bc 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/csv/UnivocityParserSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/csv/UnivocityParserSuite.scala
@@ -112,7 +112,7 @@ class UnivocityParserSuite extends SparkFunSuite with SQLHelper {
assert(parser.makeConverter("_1", BooleanType).apply("true") == true)
var timestampsOptions =
- new CSVOptions(Map("timestampFormat" -> "dd/MM/yyyy hh:mm"), false, "GMT")
+ new CSVOptions(Map("timestampFormat" -> "dd/MM/yyyy HH:mm"), false, "GMT")
parser = new UnivocityParser(StructType(Seq.empty), timestampsOptions)
val customTimestamp = "31/01/2015 00:00"
var format = FastDateFormat.getInstance(
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/util/DateFormatterSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/util/DateFormatterSuite.scala
index 019615b..2dc55e0 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/util/DateFormatterSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/util/DateFormatterSuite.scala
@@ -17,6 +17,7 @@
package org.apache.spark.sql.util
+import java.time.LocalDate
import java.util.Locale
import org.apache.spark.SparkFunSuite
@@ -89,4 +90,10 @@ class DateFormatterSuite extends SparkFunSuite with SQLHelper {
}
}
}
+
+ test("parsing date without explicit day") {
+ val formatter = DateFormatter("yyyy MMM", Locale.US)
+ val daysSinceEpoch = formatter.parse("2018 Dec")
+ assert(daysSinceEpoch === LocalDate.of(2018, 12, 1).toEpochDay)
+ }
}
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
index c110ffa..edccbb2 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
@@ -17,7 +17,9 @@
package org.apache.spark.sql.util
+import java.time.{LocalDateTime, ZoneOffset}
import java.util.{Locale, TimeZone}
+import java.util.concurrent.TimeUnit
import org.apache.spark.SparkFunSuite
import org.apache.spark.sql.catalyst.plans.SQLHelper
@@ -106,4 +108,14 @@ class TimestampFormatterSuite extends SparkFunSuite with SQLHelper {
}
}
}
+
+ test(" case insensitive parsing of am and pm") {
+ val formatter = TimestampFormatter(
+ "yyyy MMM dd hh:mm:ss a",
+ TimeZone.getTimeZone("UTC"),
+ Locale.US)
+ val micros = formatter.parse("2009 Mar 20 11:30:01 am")
+ assert(micros === TimeUnit.SECONDS.toMicros(
+ LocalDateTime.of(2009, 3, 20, 11, 30, 1).toEpochSecond(ZoneOffset.UTC)))
+ }
}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/functions.scala b/sql/core/src/main/scala/org/apache/spark/sql/functions.scala
index 33186f7..6454525 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/functions.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/functions.scala
@@ -2578,7 +2578,7 @@ object functions {
* Converts a date/timestamp/string to a value of string in the format specified by the date
* format given by the second argument.
*
- * See [[java.text.SimpleDateFormat]] for valid date and time format patterns
+ * See [[java.time.format.DateTimeFormatter]] for valid date and time format patterns
*
* @param dateExpr A date, timestamp or string. If a string, the data must be in a format that
* can be cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
@@ -2811,7 +2811,7 @@ object functions {
* representing the timestamp of that moment in the current system time zone in the given
* format.
*
- * See [[java.text.SimpleDateFormat]] for valid date and time format patterns
+ * See [[java.time.format.DateTimeFormatter]] for valid date and time format patterns
*
* @param ut A number of a type that is castable to a long, such as string or integer. Can be
* negative for timestamps before the unix epoch
@@ -2855,7 +2855,7 @@ object functions {
/**
* Converts time string with given pattern to Unix timestamp (in seconds).
*
- * See [[java.text.SimpleDateFormat]] for valid date and time format patterns
+ * See [[java.time.format.DateTimeFormatter]] for valid date and time format patterns
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
* cast to a date, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
@@ -2883,7 +2883,7 @@ object functions {
/**
* Converts time string with the given pattern to timestamp.
*
- * See [[java.text.SimpleDateFormat]] for valid date and time format patterns
+ * See [[java.time.format.DateTimeFormatter]] for valid date and time format patterns
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
* cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
@@ -2908,7 +2908,7 @@ object functions {
/**
* Converts the column into a `DateType` with a specified format
*
- * See [[java.text.SimpleDateFormat]] for valid date and time format patterns
+ * See [[java.time.format.DateTimeFormatter]] for valid date and time format patterns
*
* @param e A date, timestamp or string. If a string, the data must be in a format that can be
* cast to a date, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
index c4ec715..62bb72d 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
@@ -405,7 +405,7 @@ class DateFunctionsSuite extends QueryTest with SharedSQLContext {
Row(Date.valueOf("2014-12-31"))))
checkAnswer(
df.select(to_date(col("s"), "yyyy-MM-dd")),
- Seq(Row(Date.valueOf("2015-07-22")), Row(Date.valueOf("2014-12-31")), Row(null)))
+ Seq(Row(null), Row(Date.valueOf("2014-12-31")), Row(null)))
// now switch format
checkAnswer(
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org