You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2019/02/23 15:44:42 UTC
[spark] branch master updated: [SPARK-26903][SQL] Remove the
TimeZone cache
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new d0f2fd0 [SPARK-26903][SQL] Remove the TimeZone cache
d0f2fd0 is described below
commit d0f2fd05e1079d580ffd7d9d263f662e92849175
Author: Maxim Gekk <ma...@gmail.com>
AuthorDate: Sat Feb 23 09:44:22 2019 -0600
[SPARK-26903][SQL] Remove the TimeZone cache
## What changes were proposed in this pull request?
In the PR, I propose to convert time zone string to `TimeZone` by converting it to `ZoneId` which uses `ZoneOffset` internally. The `ZoneOffset` class of JDK 8 has a cache already: http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/java/time/ZoneOffset.java#l205 . In this way, there is no need to support cache of time zones in Spark.
The PR removes `computedTimeZones` from `DateTimeUtils`, and uses `ZoneId.of` to convert time zone id string to `ZoneId` and to `TimeZone` at the end.
## How was this patch tested?
The changes were tested by
Closes #23812 from MaxGekk/timezone-cache.
Lead-authored-by: Maxim Gekk <ma...@gmail.com>
Co-authored-by: Maxim Gekk <ma...@databricks.com>
Signed-off-by: Sean Owen <se...@databricks.com>
---
docs/sql-migration-guide-upgrade.md | 2 +
.../spark/sql/catalyst/util/DateTimeUtils.scala | 11 +-
.../expressions/DateExpressionsSuite.scala | 25 +-
sql/core/benchmarks/DateTimeBenchmark-results.txt | 312 ++++++++++-----------
4 files changed, 181 insertions(+), 169 deletions(-)
diff --git a/docs/sql-migration-guide-upgrade.md b/docs/sql-migration-guide-upgrade.md
index 1ae26e6..c201056 100644
--- a/docs/sql-migration-guide-upgrade.md
+++ b/docs/sql-migration-guide-upgrade.md
@@ -97,6 +97,8 @@ displayTitle: Spark SQL Upgrading Guide
- the JDBC options `lowerBound` and `upperBound` are converted to TimestampType/DateType values in the same way as casting strings to TimestampType/DateType values. The conversion is based on Proleptic Gregorian calendar, and time zone defined by the SQL config `spark.sql.session.timeZone`. In Spark version 2.4 and earlier, the conversion is based on the hybrid calendar (Julian + Gregorian) and on default system time zone.
+ - In Spark version 2.4 and earlier, invalid time zone ids are silently ignored and replaced by GMT time zone, for example, in the from_utc_timestamp function. Since Spark 3.0, such time zone ids are rejected, and Spark throws `java.time.DateTimeException`.
+
## Upgrading From Spark SQL 2.3 to 2.4
- In Spark version 2.3 and earlier, the second parameter to array_contains function is implicitly promoted to the element type of first array type parameter. This type promotion can be lossy and may cause `array_contains` function to return wrong result. This problem has been addressed in 2.4 by employing a safer type promotion mechanism. This can cause some change in behavior and are illustrated in the table below.
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
index 28fb6a9..5a432ba 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
@@ -22,8 +22,7 @@ import java.time._
import java.time.Year.isLeap
import java.time.temporal.IsoFields
import java.util.{Locale, TimeZone}
-import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
-import java.util.function.{Function => JFunction}
+import java.util.concurrent.TimeUnit
import scala.util.control.NonFatal
@@ -67,13 +66,9 @@ object DateTimeUtils {
def defaultTimeZone(): TimeZone = TimeZone.getDefault()
- private val computedTimeZones = new ConcurrentHashMap[String, TimeZone]
- private val computeTimeZone = new JFunction[String, TimeZone] {
- override def apply(timeZoneId: String): TimeZone = TimeZone.getTimeZone(timeZoneId)
- }
-
def getTimeZone(timeZoneId: String): TimeZone = {
- computedTimeZones.computeIfAbsent(timeZoneId, computeTimeZone)
+ val zoneId = ZoneId.of(timeZoneId, ZoneId.SHORT_IDS)
+ TimeZone.getTimeZone(zoneId)
}
// we should use the exact day as Int, for example, (year, month, day) -> day
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
index c3c29e3..ce576ec 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
@@ -819,9 +819,17 @@ class DateExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
test(null, "UTC", null)
test("2015-07-24 00:00:00", null, null)
test(null, null, null)
- // Test escaping of timezone
- GenerateUnsafeProjection.generate(
- ToUTCTimestamp(Literal(Timestamp.valueOf("2015-07-24 00:00:00")), Literal("\"quote")) :: Nil)
+ }
+
+ test("to_utc_timestamp - invalid time zone id") {
+ Seq("Invalid time zone", "\"quote", "UTC*42").foreach { invalidTz =>
+ val msg = intercept[java.time.DateTimeException] {
+ GenerateUnsafeProjection.generate(
+ ToUTCTimestamp(
+ Literal(Timestamp.valueOf("2015-07-24 00:00:00")), Literal(invalidTz)) :: Nil)
+ }.getMessage
+ assert(msg.contains(invalidTz))
+ }
}
test("from_utc_timestamp") {
@@ -842,7 +850,14 @@ class DateExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
test(null, "UTC", null)
test("2015-07-24 00:00:00", null, null)
test(null, null, null)
- // Test escaping of timezone
- GenerateUnsafeProjection.generate(FromUTCTimestamp(Literal(0), Literal("\"quote")) :: Nil)
+ }
+
+ test("from_utc_timestamp - invalid time zone id") {
+ Seq("Invalid time zone", "\"quote", "UTC*42").foreach { invalidTz =>
+ val msg = intercept[java.time.DateTimeException] {
+ GenerateUnsafeProjection.generate(FromUTCTimestamp(Literal(0), Literal(invalidTz)) :: Nil)
+ }.getMessage
+ assert(msg.contains(invalidTz))
+ }
}
}
diff --git a/sql/core/benchmarks/DateTimeBenchmark-results.txt b/sql/core/benchmarks/DateTimeBenchmark-results.txt
index bc824af..6c39f8f 100644
--- a/sql/core/benchmarks/DateTimeBenchmark-results.txt
+++ b/sql/core/benchmarks/DateTimeBenchmark-results.txt
@@ -2,327 +2,327 @@
Extract components
================================================================================================
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
cast to timestamp: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-cast to timestamp wholestage off 276 / 290 36.2 27.6 1.0X
-cast to timestamp wholestage on 254 / 267 39.4 25.4 1.1X
+cast to timestamp wholestage off 275 / 287 36.4 27.5 1.0X
+cast to timestamp wholestage on 243 / 253 41.2 24.3 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
year of timestamp: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-year of timestamp wholestage off 699 / 700 14.3 69.9 1.0X
-year of timestamp wholestage on 680 / 689 14.7 68.0 1.0X
+year of timestamp wholestage off 661 / 667 15.1 66.1 1.0X
+year of timestamp wholestage on 659 / 669 15.2 65.9 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
quarter of timestamp: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-quarter of timestamp wholestage off 848 / 864 11.8 84.8 1.0X
-quarter of timestamp wholestage on 784 / 797 12.8 78.4 1.1X
+quarter of timestamp wholestage off 820 / 822 12.2 82.0 1.0X
+quarter of timestamp wholestage on 768 / 776 13.0 76.8 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
month of timestamp: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-month of timestamp wholestage off 652 / 653 15.3 65.2 1.0X
-month of timestamp wholestage on 671 / 677 14.9 67.1 1.0X
+month of timestamp wholestage off 636 / 638 15.7 63.6 1.0X
+month of timestamp wholestage on 648 / 654 15.4 64.8 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
weekofyear of timestamp: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-weekofyear of timestamp wholestage off 1233 / 1233 8.1 123.3 1.0X
-weekofyear of timestamp wholestage on 1236 / 1240 8.1 123.6 1.0X
+weekofyear of timestamp wholestage off 1093 / 1097 9.2 109.3 1.0X
+weekofyear of timestamp wholestage on 1101 / 1107 9.1 110.1 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
day of timestamp: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-day of timestamp wholestage off 649 / 655 15.4 64.9 1.0X
-day of timestamp wholestage on 670 / 678 14.9 67.0 1.0X
+day of timestamp wholestage off 643 / 644 15.6 64.3 1.0X
+day of timestamp wholestage on 655 / 657 15.3 65.5 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
dayofyear of timestamp: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-dayofyear of timestamp wholestage off 692 / 704 14.5 69.2 1.0X
-dayofyear of timestamp wholestage on 695 / 698 14.4 69.5 1.0X
+dayofyear of timestamp wholestage off 681 / 692 14.7 68.1 1.0X
+dayofyear of timestamp wholestage on 675 / 680 14.8 67.5 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
dayofmonth of timestamp: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-dayofmonth of timestamp wholestage off 660 / 660 15.1 66.0 1.0X
-dayofmonth of timestamp wholestage on 667 / 671 15.0 66.7 1.0X
+dayofmonth of timestamp wholestage off 656 / 657 15.2 65.6 1.0X
+dayofmonth of timestamp wholestage on 651 / 658 15.4 65.1 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
dayofweek of timestamp: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-dayofweek of timestamp wholestage off 798 / 802 12.5 79.8 1.0X
-dayofweek of timestamp wholestage on 804 / 820 12.4 80.4 1.0X
+dayofweek of timestamp wholestage off 775 / 776 12.9 77.5 1.0X
+dayofweek of timestamp wholestage on 777 / 781 12.9 77.7 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
weekday of timestamp: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-weekday of timestamp wholestage off 759 / 760 13.2 75.9 1.0X
-weekday of timestamp wholestage on 774 / 813 12.9 77.4 1.0X
+weekday of timestamp wholestage off 737 / 737 13.6 73.7 1.0X
+weekday of timestamp wholestage on 737 / 739 13.6 73.7 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
hour of timestamp: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-hour of timestamp wholestage off 465 / 468 21.5 46.5 1.0X
-hour of timestamp wholestage on 443 / 451 22.6 44.3 1.0X
+hour of timestamp wholestage off 425 / 426 23.5 42.5 1.0X
+hour of timestamp wholestage on 430 / 434 23.2 43.0 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
minute of timestamp: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-minute of timestamp wholestage off 439 / 440 22.8 43.9 1.0X
-minute of timestamp wholestage on 448 / 452 22.3 44.8 1.0X
+minute of timestamp wholestage off 430 / 439 23.3 43.0 1.0X
+minute of timestamp wholestage on 436 / 438 23.0 43.6 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
second of timestamp: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-second of timestamp wholestage off 438 / 448 22.8 43.8 1.0X
-second of timestamp wholestage on 430 / 445 23.3 43.0 1.0X
+second of timestamp wholestage off 413 / 413 24.2 41.3 1.0X
+second of timestamp wholestage on 413 / 425 24.2 41.3 1.0X
================================================================================================
Current date and time
================================================================================================
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
current_date: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-current_date wholestage off 212 / 213 47.2 21.2 1.0X
-current_date wholestage on 225 / 230 44.4 22.5 0.9X
+current_date wholestage off 205 / 206 48.7 20.5 1.0X
+current_date wholestage on 219 / 224 45.8 21.9 0.9X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
current_timestamp: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-current_timestamp wholestage off 216 / 217 46.3 21.6 1.0X
-current_timestamp wholestage on 210 / 216 47.7 21.0 1.0X
+current_timestamp wholestage off 212 / 213 47.3 21.2 1.0X
+current_timestamp wholestage on 202 / 205 49.6 20.2 1.0X
================================================================================================
Date arithmetic
================================================================================================
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
cast to date: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-cast to date wholestage off 463 / 464 21.6 46.3 1.0X
-cast to date wholestage on 502 / 508 19.9 50.2 0.9X
+cast to date wholestage off 459 / 462 21.8 45.9 1.0X
+cast to date wholestage on 493 / 500 20.3 49.3 0.9X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
last_day: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-last_day wholestage off 693 / 701 14.4 69.3 1.0X
-last_day wholestage on 697 / 706 14.3 69.7 1.0X
+last_day wholestage off 680 / 686 14.7 68.0 1.0X
+last_day wholestage on 671 / 681 14.9 67.1 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
next_day: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-next_day wholestage off 553 / 570 18.1 55.3 1.0X
-next_day wholestage on 595 / 597 16.8 59.5 0.9X
+next_day wholestage off 532 / 533 18.8 53.2 1.0X
+next_day wholestage on 576 / 580 17.4 57.6 0.9X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
date_add: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-date_add wholestage off 459 / 463 21.8 45.9 1.0X
-date_add wholestage on 473 / 477 21.1 47.3 1.0X
+date_add wholestage off 445 / 446 22.5 44.5 1.0X
+date_add wholestage on 455 / 457 22.0 45.5 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
date_sub: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-date_sub wholestage off 473 / 475 21.1 47.3 1.0X
-date_sub wholestage on 479 / 497 20.9 47.9 1.0X
+date_sub wholestage off 454 / 457 22.0 45.4 1.0X
+date_sub wholestage on 455 / 458 22.0 45.5 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
add_months: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-add_months wholestage off 898 / 908 11.1 89.8 1.0X
-add_months wholestage on 898 / 906 11.1 89.8 1.0X
+add_months wholestage off 898 / 900 11.1 89.8 1.0X
+add_months wholestage on 894 / 909 11.2 89.4 1.0X
================================================================================================
Formatting dates
================================================================================================
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
format date: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-format date wholestage off 7181 / 7228 1.4 718.1 1.0X
-format date wholestage on 7303 / 7356 1.4 730.3 1.0X
+format date wholestage off 7180 / 7181 1.4 718.0 1.0X
+format date wholestage on 7051 / 7194 1.4 705.1 1.0X
================================================================================================
Formatting timestamps
================================================================================================
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
from_unixtime: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-from_unixtime wholestage off 7472 / 7476 1.3 747.2 1.0X
-from_unixtime wholestage on 7453 / 7460 1.3 745.3 1.0X
+from_unixtime wholestage off 7136 / 7163 1.4 713.6 1.0X
+from_unixtime wholestage on 7144 / 7174 1.4 714.4 1.0X
================================================================================================
Convert timestamps
================================================================================================
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
from_utc_timestamp: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-from_utc_timestamp wholestage off 894 / 895 11.2 89.4 1.0X
-from_utc_timestamp wholestage on 881 / 883 11.4 88.1 1.0X
+from_utc_timestamp wholestage off 880 / 888 11.4 88.0 1.0X
+from_utc_timestamp wholestage on 841 / 854 11.9 84.1 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
to_utc_timestamp: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-to_utc_timestamp wholestage off 909 / 910 11.0 90.9 1.0X
-to_utc_timestamp wholestage on 903 / 906 11.1 90.3 1.0X
+to_utc_timestamp wholestage off 879 / 884 11.4 87.9 1.0X
+to_utc_timestamp wholestage on 862 / 876 11.6 86.2 1.0X
================================================================================================
Intervals
================================================================================================
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
cast interval: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-cast interval wholestage off 245 / 256 40.7 24.5 1.0X
-cast interval wholestage on 227 / 230 44.0 22.7 1.1X
+cast interval wholestage off 242 / 250 41.3 24.2 1.0X
+cast interval wholestage on 221 / 223 45.3 22.1 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
datediff: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-datediff wholestage off 735 / 738 13.6 73.5 1.0X
-datediff wholestage on 687 / 689 14.6 68.7 1.1X
+datediff wholestage off 697 / 698 14.3 69.7 1.0X
+datediff wholestage on 680 / 683 14.7 68.0 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
months_between: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-months_between wholestage off 1653 / 1654 6.1 165.3 1.0X
-months_between wholestage on 1630 / 1636 6.1 163.0 1.0X
+months_between wholestage off 1675 / 1677 6.0 167.5 1.0X
+months_between wholestage on 1636 / 1649 6.1 163.6 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
window: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-window wholestage off 1636 / 1661 0.6 1635.9 1.0X
-window wholestage on 19997 / 20240 0.1 19997.4 0.1X
+window wholestage off 1600 / 1627 0.6 1599.7 1.0X
+window wholestage on 19480 / 19530 0.1 19479.6 0.1X
================================================================================================
Truncation
================================================================================================
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
date_trunc YEAR: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-date_trunc YEAR wholestage off 889 / 892 11.2 88.9 1.0X
-date_trunc YEAR wholestage on 831 / 837 12.0 83.1 1.1X
+date_trunc YEAR wholestage off 863 / 864 11.6 86.3 1.0X
+date_trunc YEAR wholestage on 812 / 814 12.3 81.2 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
date_trunc YYYY: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-date_trunc YYYY wholestage off 910 / 917 11.0 91.0 1.0X
-date_trunc YYYY wholestage on 834 / 849 12.0 83.4 1.1X
+date_trunc YYYY wholestage off 865 / 926 11.6 86.5 1.0X
+date_trunc YYYY wholestage on 811 / 820 12.3 81.1 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
date_trunc YY: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-date_trunc YY wholestage off 897 / 901 11.2 89.7 1.0X
-date_trunc YY wholestage on 836 / 841 12.0 83.6 1.1X
+date_trunc YY wholestage off 863 / 867 11.6 86.3 1.0X
+date_trunc YY wholestage on 810 / 822 12.3 81.0 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
date_trunc MON: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-date_trunc MON wholestage off 942 / 944 10.6 94.2 1.0X
-date_trunc MON wholestage on 881 / 887 11.3 88.1 1.1X
+date_trunc MON wholestage off 917 / 921 10.9 91.7 1.0X
+date_trunc MON wholestage on 857 / 860 11.7 85.7 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
date_trunc MONTH: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-date_trunc MONTH wholestage off 935 / 937 10.7 93.5 1.0X
-date_trunc MONTH wholestage on 881 / 886 11.4 88.1 1.1X
+date_trunc MONTH wholestage off 919 / 919 10.9 91.9 1.0X
+date_trunc MONTH wholestage on 862 / 863 11.6 86.2 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
date_trunc MM: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-date_trunc MM wholestage off 934 / 935 10.7 93.4 1.0X
-date_trunc MM wholestage on 878 / 880 11.4 87.8 1.1X
+date_trunc MM wholestage off 923 / 924 10.8 92.3 1.0X
+date_trunc MM wholestage on 855 / 859 11.7 85.5 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
date_trunc DAY: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-date_trunc DAY wholestage off 472 / 478 21.2 47.2 1.0X
-date_trunc DAY wholestage on 415 / 418 24.1 41.5 1.1X
+date_trunc DAY wholestage off 444 / 444 22.5 44.4 1.0X
+date_trunc DAY wholestage on 404 / 406 24.7 40.4 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
date_trunc DD: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-date_trunc DD wholestage off 472 / 485 21.2 47.2 1.0X
-date_trunc DD wholestage on 414 / 417 24.1 41.4 1.1X
+date_trunc DD wholestage off 445 / 446 22.5 44.5 1.0X
+date_trunc DD wholestage on 404 / 406 24.7 40.4 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
date_trunc HOUR: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-date_trunc HOUR wholestage off 451 / 462 22.2 45.1 1.0X
-date_trunc HOUR wholestage on 422 / 429 23.7 42.2 1.1X
+date_trunc HOUR wholestage off 462 / 464 21.6 46.2 1.0X
+date_trunc HOUR wholestage on 416 / 422 24.1 41.6 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
date_trunc MINUTE: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-date_trunc MINUTE wholestage off 291 / 291 34.4 29.1 1.0X
-date_trunc MINUTE wholestage on 268 / 272 37.3 26.8 1.1X
+date_trunc MINUTE wholestage off 294 / 294 34.0 29.4 1.0X
+date_trunc MINUTE wholestage on 258 / 266 38.8 25.8 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
date_trunc SECOND: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-date_trunc SECOND wholestage off 290 / 290 34.5 29.0 1.0X
-date_trunc SECOND wholestage on 266 / 270 37.6 26.6 1.1X
+date_trunc SECOND wholestage off 292 / 295 34.2 29.2 1.0X
+date_trunc SECOND wholestage on 271 / 276 36.9 27.1 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
date_trunc WEEK: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-date_trunc WEEK wholestage off 794 / 797 12.6 79.4 1.0X
-date_trunc WEEK wholestage on 754 / 761 13.3 75.4 1.1X
+date_trunc WEEK wholestage off 739 / 740 13.5 73.9 1.0X
+date_trunc WEEK wholestage on 712 / 715 14.0 71.2 1.0X
Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
@@ -331,86 +331,86 @@ date_trunc QUARTER: Best/Avg Time(ms) Rate(M/s) Per Ro
date_trunc QUARTER wholestage off 1465 / 1467 6.8 146.5 1.0X
date_trunc QUARTER wholestage on 1419 / 1423 7.0 141.9 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
trunc year: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-trunc year wholestage off 233 / 233 43.0 23.3 1.0X
-trunc year wholestage on 213 / 216 46.9 21.3 1.1X
+trunc year wholestage off 222 / 222 45.0 22.2 1.0X
+trunc year wholestage on 207 / 214 48.3 20.7 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
trunc yyyy: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-trunc yyyy wholestage off 232 / 233 43.1 23.2 1.0X
-trunc yyyy wholestage on 214 / 223 46.7 21.4 1.1X
+trunc yyyy wholestage off 221 / 225 45.2 22.1 1.0X
+trunc yyyy wholestage on 208 / 212 48.0 20.8 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
trunc yy: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-trunc yy wholestage off 228 / 228 43.9 22.8 1.0X
-trunc yy wholestage on 215 / 219 46.6 21.5 1.1X
+trunc yy wholestage off 221 / 222 45.3 22.1 1.0X
+trunc yy wholestage on 208 / 210 48.0 20.8 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
trunc mon: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-trunc mon wholestage off 232 / 232 43.0 23.2 1.0X
-trunc mon wholestage on 216 / 217 46.4 21.6 1.1X
+trunc mon wholestage off 231 / 239 43.3 23.1 1.0X
+trunc mon wholestage on 208 / 214 48.0 20.8 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
trunc month: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-trunc month wholestage off 233 / 233 43.0 23.3 1.0X
-trunc month wholestage on 215 / 217 46.4 21.5 1.1X
+trunc month wholestage off 222 / 222 45.1 22.2 1.0X
+trunc month wholestage on 208 / 224 48.1 20.8 1.1X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
trunc mm: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-trunc mm wholestage off 226 / 233 44.2 22.6 1.0X
-trunc mm wholestage on 214 / 220 46.7 21.4 1.1X
+trunc mm wholestage off 222 / 226 45.1 22.2 1.0X
+trunc mm wholestage on 208 / 216 48.0 20.8 1.1X
================================================================================================
Parsing
================================================================================================
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
to timestamp str: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-to timestamp str wholestage off 166 / 170 6.0 165.8 1.0X
-to timestamp str wholestage on 165 / 167 6.1 164.9 1.0X
+to timestamp str wholestage off 165 / 166 6.1 164.7 1.0X
+to timestamp str wholestage on 160 / 163 6.2 160.5 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
to_timestamp: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-to_timestamp wholestage off 1439 / 1445 0.7 1439.4 1.0X
-to_timestamp wholestage on 1366 / 1368 0.7 1365.7 1.1X
+to_timestamp wholestage off 1316 / 1320 0.8 1315.7 1.0X
+to_timestamp wholestage on 1288 / 1294 0.8 1287.5 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
to_unix_timestamp: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-to_unix_timestamp wholestage off 1386 / 1389 0.7 1385.6 1.0X
-to_unix_timestamp wholestage on 1388 / 1392 0.7 1387.8 1.0X
+to_unix_timestamp wholestage off 1295 / 1297 0.8 1295.1 1.0X
+to_unix_timestamp wholestage on 1409 / 1414 0.7 1409.2 0.9X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
to date str: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-to date str wholestage off 163 / 166 6.1 163.3 1.0X
-to date str wholestage on 160 / 163 6.2 160.3 1.0X
+to date str wholestage off 155 / 157 6.4 155.4 1.0X
+to date str wholestage on 154 / 156 6.5 154.3 1.0X
-Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-ea-b03 on Mac OS X 10.14.2
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.14.3
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
to_date: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-to_date wholestage off 1474 / 1474 0.7 1473.9 1.0X
-to_date wholestage on 1460 / 1465 0.7 1459.6 1.0X
+to_date wholestage off 1477 / 1479 0.7 1477.3 1.0X
+to_date wholestage on 1468 / 1473 0.7 1468.2 1.0X
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org