You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ma...@apache.org on 2017/03/17 00:37:09 UTC
incubator-metron git commit: METRON-764 DST bug in
metron-profiler-client Unit Tests (mattf-horton) closes
apache/incubator-metron#476
Repository: incubator-metron
Updated Branches:
refs/heads/master 68a334a87 -> f1435a6a7
METRON-764 DST bug in metron-profiler-client Unit Tests (mattf-horton) closes apache/incubator-metron#476
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/f1435a6a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/f1435a6a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/f1435a6a
Branch: refs/heads/master
Commit: f1435a6a75f016294920ac328ba79474fbf129e1
Parents: 68a334a
Author: mattf-horton <mf...@hortonworks.com>
Authored: Thu Mar 16 17:14:52 2017 -0700
Committer: YOUR NAME as In Apache <YO...@apache.org>
Committed: Thu Mar 16 17:14:52 2017 -0700
----------------------------------------------------------------------
.../metron-profiler-client/README.md | 29 ++++++++++++++++++++
.../client/window/WindowProcessorTest.java | 13 +++++++++
2 files changed, 42 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/f1435a6a/metron-analytics/metron-profiler-client/README.md
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-profiler-client/README.md b/metron-analytics/metron-profiler-client/README.md
index 656bb25..aa35136 100644
--- a/metron-analytics/metron-profiler-client/README.md
+++ b/metron-analytics/metron-profiler-client/README.md
@@ -253,6 +253,35 @@ The specifiers are a set of fixed specifiers available as part of the language:
* Countries supported are those supported in [jollyday](https://github.com/svendiedrichsen/jollyday/tree/master/src/main/resources/holidays)
* Example: `holiday:us:nyc` would be the holidays of New York City, USA
* Example: `holiday:hu` would be the holidays of Hungary
+
+**WARNING: Daylight Savings Time effects**
+
+While Universal Time (UTC) is nice and constant, many servers are set to local timezones that enable Daylight Savings Time (DST).
+This means that twice a year, on DST transition weekends, "Sunday" is either 23 or 25 hours long. However, durations specified
+as "7 days ago" are always interpreted as "7*24 hours ago". This can lead to some surprising effects when using days of the week
+as inclusion or exclusion specifiers.
+
+For example, the profile window specified by the phrase "30 minute window every 24 hours from 7 days ago"
+will always have 7 thirty-minute intervals, and these will normally occur on 5 weekdays and 2 weekend days.
+However, if you invoke this window at 12:15am any day during the week following the start of DST, you will get
+these intervals (supposing you start early on a Wednesday morning):
+```
+Tuesday 12:15am-12:45am (yesterday)
+Monday 12:15am-12:45am
+Saturday 11:15pm-11:45pm (skipped Sunday!)
+Friday 11:15pm-11:45pm
+Thursday 11:15pm-11:45pm
+Wednesday 11:15pm-11:45pm
+Tuesday 11:15pm-11:45pm
+```
+
+Sunday got skipped over because it was only 23 hours long; that is, there were 24 hours between Saturday 11:15pm and Monday 12:15am.
+So if you specified "excluding weekends", you would get 6 days' intervals instead of the expected 5. There are multiple variations
+on this theme.
+
+Remember that the underlying time is kept in UTC, so the data is always correct. It is only when attempting to interpret UTC as
+local time, date, and day, that these confusions may occur. They may be eliminated by setting your server timezone to UTC, or otherwise
+disabling DST.
**Examples**
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/f1435a6a/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/window/WindowProcessorTest.java
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/window/WindowProcessorTest.java b/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/window/WindowProcessorTest.java
index 94f9e04..79212e2 100644
--- a/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/window/WindowProcessorTest.java
+++ b/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/window/WindowProcessorTest.java
@@ -29,6 +29,13 @@ import java.util.Date;
import java.util.List;
import java.util.concurrent.TimeUnit;
+/**
+ * On any test case where we explicitly include or exclude days of the week, in 24hr periods,
+ * we need to understand that on Daylight Savings Time (DST) transition weekends,
+ * Sunday is either 23 or 25 hours long. This leads to surprising correct results,
+ * that may fail the assert criteria, if one starts the hour before or after midnight.
+ * Thus in such test cases, we force now.setHours(6).
+ */
public class WindowProcessorTest {
@Test
@@ -129,6 +136,7 @@ public class WindowProcessorTest {
Gotta be 2 tuesdays in 14 days.
*/
Date now = new Date();
+ now.setHours(6); //avoid DST impacts if near Midnight
List<Range<Long>> intervals = w.toIntervals(now.getTime());
Assert.assertEquals(2, intervals.size());
}
@@ -140,6 +148,7 @@ public class WindowProcessorTest {
Gotta be 2 days with the same dow in 14 days.
*/
Date now = new Date();
+ now.setHours(6); //avoid DST impacts if near Midnight
List<Range<Long>> intervals = w.toIntervals(now.getTime());
Assert.assertEquals(2, intervals.size());
}
@@ -162,6 +171,7 @@ public class WindowProcessorTest {
Window w = WindowProcessor.process("30 minute window every 24 hours from 7 days ago including saturdays excluding weekends");
Date now = new Date();
+ now.setHours(6); //avoid DST impacts if near Midnight
List<Range<Long>> intervals = w.toIntervals(now.getTime());
Assert.assertEquals(0, intervals.size());
}
@@ -171,6 +181,7 @@ public class WindowProcessorTest {
Window w = WindowProcessor.process("30 minute window every 24 hours from 7 days ago excluding weekends");
Date now = new Date();
+ now.setHours(6); //avoid DST impacts if near Midnight
List<Range<Long>> intervals = w.toIntervals(now.getTime());
Assert.assertEquals(5, intervals.size());
}
@@ -198,6 +209,7 @@ public class WindowProcessorTest {
Window w = WindowProcessor.process("1 hour window every 24 hours starting from 56 days ago including this day of the week");
Date now = new Date();
+ now.setHours(6); //avoid DST impacts if near Midnight
List<Range<Long>> intervals = w.toIntervals(now.getTime());
Assert.assertEquals(8, intervals.size());
}
@@ -208,6 +220,7 @@ public class WindowProcessorTest {
Window w = WindowProcessor.process("30 minute window every 24 hours from 7 days ago excluding weekdays");
Date now = new Date();
+ now.setHours(6); //avoid DST impacts if near Midnight
List<Range<Long>> intervals = w.toIntervals(now.getTime());
Assert.assertEquals(2, intervals.size());
}