You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by ka...@apache.org on 2016/12/18 22:32:19 UTC

[1/3] storm git commit: [STORM-2238] Add Timestamp extractor for windowed bolt

Repository: storm
Updated Branches:
  refs/heads/1.x-branch 3c6abc8f7 -> 7701bf4ea


[STORM-2238] Add Timestamp extractor for windowed bolt

Added a timestamp extractor so that the tuple timestamp can be derived from the tuple than directly
mapping it to a field within the tuple.


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/032592a8
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/032592a8
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/032592a8

Branch: refs/heads/1.x-branch
Commit: 032592a892356e36fd3848b1f1692099eed90d28
Parents: cce525b
Author: Arun Mahadevan <ar...@apache.org>
Authored: Thu Dec 8 21:18:12 2016 +0530
Committer: Arun Mahadevan <ar...@apache.org>
Committed: Sun Dec 18 23:11:52 2016 +0530

----------------------------------------------------------------------
 docs/Windowing.md                               | 16 ++++++-
 .../clj/org/apache/storm/daemon/executor.clj    |  1 -
 storm-core/src/jvm/org/apache/storm/Config.java | 14 ++----
 .../apache/storm/topology/IWindowedBolt.java    | 11 +++++
 .../topology/TupleFieldTimestampExtractor.java  | 48 ++++++++++++++++++++
 .../storm/topology/WindowedBoltExecutor.java    | 12 +++--
 .../topology/base/BaseStatefulWindowedBolt.java | 10 ++++
 .../storm/topology/base/BaseWindowedBolt.java   | 22 ++++++++-
 .../storm/windowing/TimestampExtractor.java     | 35 ++++++++++++++
 .../topology/WindowedBoltExecutorTest.java      | 11 +++--
 10 files changed, 156 insertions(+), 24 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/docs/Windowing.md
----------------------------------------------------------------------
diff --git a/docs/Windowing.md b/docs/Windowing.md
index 82212ea..18f6525 100644
--- a/docs/Windowing.md
+++ b/docs/Windowing.md
@@ -146,8 +146,20 @@ public BaseWindowedBolt withTimestampField(String fieldName)
 ```
 
 The value for the above `fieldName` will be looked up from the incoming tuple and considered for windowing calculations. 
-If the field is not present in the tuple an exception will be thrown. Along with the timestamp field name, a time lag parameter 
-can also be specified which indicates the max time limit for tuples with out of order timestamps. 
+If the field is not present in the tuple an exception will be thrown. Alternatively a [TimestampExtractor](../storm-core/src/jvm/org/apache/storm/windowing/TimestampExtractor.java) can be used to
+derive a timestamp value from a tuple (e.g. extract timestamp from a nested field within the tuple).
+
+```java
+/**
+* Specify the timestamp extractor implementation.
+*
+* @param timestampExtractor the {@link TimestampExtractor} implementation
+*/
+public BaseWindowedBolt withTimestampExtractor(TimestampExtractor timestampExtractor)
+```
+
+
+Along with the timestamp field name/extractor, a time lag parameter can also be specified which indicates the max time limit for tuples with out of order timestamps.
 
 ```java
 /**

http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/storm-core/src/clj/org/apache/storm/daemon/executor.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/org/apache/storm/daemon/executor.clj b/storm-core/src/clj/org/apache/storm/daemon/executor.clj
index 3b20b84..95406ec 100644
--- a/storm-core/src/clj/org/apache/storm/daemon/executor.clj
+++ b/storm-core/src/clj/org/apache/storm/daemon/executor.clj
@@ -172,7 +172,6 @@
                         TOPOLOGY-BOLTS-WINDOW-LENGTH-DURATION-MS
                         TOPOLOGY-BOLTS-SLIDING-INTERVAL-COUNT
                         TOPOLOGY-BOLTS-SLIDING-INTERVAL-DURATION-MS
-                        TOPOLOGY-BOLTS-TUPLE-TIMESTAMP-FIELD-NAME
                         TOPOLOGY-BOLTS-LATE-TUPLE-STREAM
                         TOPOLOGY-BOLTS-TUPLE-TIMESTAMP-MAX-LAG-MS
                         TOPOLOGY-BOLTS-MESSAGE-ID-FIELD-NAME

http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/storm-core/src/jvm/org/apache/storm/Config.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/org/apache/storm/Config.java b/storm-core/src/jvm/org/apache/storm/Config.java
index 0854f35..41b3f67 100644
--- a/storm-core/src/jvm/org/apache/storm/Config.java
+++ b/storm-core/src/jvm/org/apache/storm/Config.java
@@ -1902,14 +1902,6 @@ public class Config extends HashMap<String, Object> {
     @isPositiveNumber
     public static final String TOPOLOGY_BOLTS_SLIDING_INTERVAL_DURATION_MS = "topology.bolts.window.sliding.interval.duration.ms";
 
-    /*
-     * Bolt-specific configuration for windowed bolts to specify the name of the field in the tuple that holds
-     * the timestamp (e.g. the ts when the tuple was actually generated). If this config is specified and the
-     * field is not present in the incoming tuple, a java.lang.IllegalArgumentException will be thrown.
-     */
-    @isString
-    public static final String TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_FIELD_NAME = "topology.bolts.tuple.timestamp.field.name";
-
     /**
      * Bolt-specific configuration for windowed bolts to specify the name of the stream on which late tuples are
      * going to be emitted. This configuration should only be used from the BaseWindowedBolt.withLateTupleStream builder
@@ -1918,10 +1910,10 @@ public class Config extends HashMap<String, Object> {
     @isString
     public static final String TOPOLOGY_BOLTS_LATE_TUPLE_STREAM = "topology.bolts.late.tuple.stream";
 
-    /*
+    /**
      * Bolt-specific configuration for windowed bolts to specify the maximum time lag of the tuple timestamp
      * in milliseconds. It means that the tuple timestamps cannot be out of order by more than this amount.
-     * This config will be effective only if the TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_FIELD_NAME is also specified.
+     * This config will be effective only if {@link org.apache.storm.windowing.TimestampExtractor} is specified.
      */
     @isInteger
     @isPositiveNumber
@@ -1930,7 +1922,7 @@ public class Config extends HashMap<String, Object> {
     /*
      * Bolt-specific configuration for windowed bolts to specify the time interval for generating
      * watermark events. Watermark event tracks the progress of time when tuple timestamp is used.
-     * This config is effective only if TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_FIELD_NAME is also specified.
+     * This config is effective only if {@link org.apache.storm.windowing.TimestampExtractor} is specified.
      */
     @isInteger
     @isPositiveNumber

http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/storm-core/src/jvm/org/apache/storm/topology/IWindowedBolt.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/org/apache/storm/topology/IWindowedBolt.java b/storm-core/src/jvm/org/apache/storm/topology/IWindowedBolt.java
index 3fda395..b0ff262 100644
--- a/storm-core/src/jvm/org/apache/storm/topology/IWindowedBolt.java
+++ b/storm-core/src/jvm/org/apache/storm/topology/IWindowedBolt.java
@@ -19,6 +19,7 @@ package org.apache.storm.topology;
 
 import org.apache.storm.task.OutputCollector;
 import org.apache.storm.task.TopologyContext;
+import org.apache.storm.windowing.TimestampExtractor;
 import org.apache.storm.windowing.TupleWindow;
 
 import java.util.Map;
@@ -32,9 +33,19 @@ public interface IWindowedBolt extends IComponent {
      * that while emitting, the tuples are automatically anchored to the tuples in the inputWindow.
      */
     void prepare(Map stormConf, TopologyContext context, OutputCollector collector);
+
     /**
      * Process the tuple window and optionally emit new tuples based on the tuples in the input window.
      */
     void execute(TupleWindow inputWindow);
+
     void cleanup();
+
+    /**
+     * Return a {@link TimestampExtractor} for extracting timestamps from a
+     * tuple for event time based processing, or null for processing time.
+     *
+     * @return the timestamp extractor
+     */
+    TimestampExtractor getTimestampExtractor();
 }

http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/storm-core/src/jvm/org/apache/storm/topology/TupleFieldTimestampExtractor.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/org/apache/storm/topology/TupleFieldTimestampExtractor.java b/storm-core/src/jvm/org/apache/storm/topology/TupleFieldTimestampExtractor.java
new file mode 100644
index 0000000..4b4ac18
--- /dev/null
+++ b/storm-core/src/jvm/org/apache/storm/topology/TupleFieldTimestampExtractor.java
@@ -0,0 +1,48 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.topology;
+
+import org.apache.storm.tuple.Tuple;
+import org.apache.storm.windowing.TimestampExtractor;
+
+/**
+ * A {@link TimestampExtractor} that extracts timestamp from a specific field in the tuple.
+ */
+public final class TupleFieldTimestampExtractor implements TimestampExtractor {
+    private final String fieldName;
+
+    private TupleFieldTimestampExtractor(String fieldName) {
+        this.fieldName = fieldName;
+    }
+
+    @Override
+    public long extractTimestamp(Tuple tuple) {
+        return tuple.getLongByField(fieldName);
+    }
+
+    public static TupleFieldTimestampExtractor of(String fieldName) {
+        return new TupleFieldTimestampExtractor(fieldName);
+    }
+
+    @Override
+    public String toString() {
+        return "TupleFieldTimestampExtractor{" +
+                "fieldName='" + fieldName + '\'' +
+                '}';
+    }
+}

http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/storm-core/src/jvm/org/apache/storm/topology/WindowedBoltExecutor.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/org/apache/storm/topology/WindowedBoltExecutor.java b/storm-core/src/jvm/org/apache/storm/topology/WindowedBoltExecutor.java
index 3f245e1..fd98274 100644
--- a/storm-core/src/jvm/org/apache/storm/topology/WindowedBoltExecutor.java
+++ b/storm-core/src/jvm/org/apache/storm/topology/WindowedBoltExecutor.java
@@ -23,6 +23,7 @@ import org.apache.storm.spout.CheckpointSpout;
 import org.apache.storm.task.IOutputCollector;
 import org.apache.storm.task.OutputCollector;
 import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.base.BaseWindowedBolt;
 import org.apache.storm.tuple.Fields;
 import org.apache.storm.tuple.Tuple;
 import org.apache.storm.tuple.Values;
@@ -31,6 +32,7 @@ import org.apache.storm.windowing.CountTriggerPolicy;
 import org.apache.storm.windowing.EvictionPolicy;
 import org.apache.storm.windowing.TimeEvictionPolicy;
 import org.apache.storm.windowing.TimeTriggerPolicy;
+import org.apache.storm.windowing.TimestampExtractor;
 import org.apache.storm.windowing.TriggerPolicy;
 import org.apache.storm.windowing.TupleWindowImpl;
 import org.apache.storm.windowing.WaterMarkEventGenerator;
@@ -65,7 +67,7 @@ public class WindowedBoltExecutor implements IRichBolt {
     private transient WindowLifecycleListener<Tuple> listener;
     private transient WindowManager<Tuple> windowManager;
     private transient int maxLagMs;
-    private transient String tupleTsFieldName;
+    private TimestampExtractor timestampExtractor;
     private transient String lateTupleStream;
     private transient TriggerPolicy<Tuple> triggerPolicy;
     private transient EvictionPolicy<Tuple> evictionPolicy;
@@ -74,6 +76,7 @@ public class WindowedBoltExecutor implements IRichBolt {
 
     public WindowedBoltExecutor(IWindowedBolt bolt) {
         this.bolt = bolt;
+        timestampExtractor = bolt.getTimestampExtractor();
     }
 
     private int getTopologyTimeoutMillis(Map stormConf) {
@@ -165,8 +168,7 @@ public class WindowedBoltExecutor implements IRichBolt {
             slidingIntervalCount = new Count(1);
         }
         // tuple ts
-        if (stormConf.containsKey(Config.TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_FIELD_NAME)) {
-            tupleTsFieldName = (String) stormConf.get(Config.TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_FIELD_NAME);
+        if (timestampExtractor != null) {
             // late tuple stream
             lateTupleStream = (String) stormConf.get(Config.TOPOLOGY_BOLTS_LATE_TUPLE_STREAM);
             if (lateTupleStream != null) {
@@ -229,7 +231,7 @@ public class WindowedBoltExecutor implements IRichBolt {
     }
 
     private boolean isTupleTs() {
-        return tupleTsFieldName != null;
+        return timestampExtractor != null;
     }
 
     private TriggerPolicy<Tuple> getTriggerPolicy(Count slidingIntervalCount, Duration slidingIntervalDuration,
@@ -279,7 +281,7 @@ public class WindowedBoltExecutor implements IRichBolt {
     @Override
     public void execute(Tuple input) {
         if (isTupleTs()) {
-            long ts = input.getLongByField(tupleTsFieldName);
+            long ts = timestampExtractor.extractTimestamp(input);
             if (waterMarkEventGenerator.track(input.getSourceGlobalStreamId(), ts)) {
                 windowManager.add(input, ts);
             } else {

http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/storm-core/src/jvm/org/apache/storm/topology/base/BaseStatefulWindowedBolt.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/org/apache/storm/topology/base/BaseStatefulWindowedBolt.java b/storm-core/src/jvm/org/apache/storm/topology/base/BaseStatefulWindowedBolt.java
index 0c35b9d..7e80830 100644
--- a/storm-core/src/jvm/org/apache/storm/topology/base/BaseStatefulWindowedBolt.java
+++ b/storm-core/src/jvm/org/apache/storm/topology/base/BaseStatefulWindowedBolt.java
@@ -20,6 +20,7 @@ package org.apache.storm.topology.base;
 import org.apache.storm.Config;
 import org.apache.storm.state.State;
 import org.apache.storm.topology.IStatefulWindowedBolt;
+import org.apache.storm.windowing.TimestampExtractor;
 
 public abstract class BaseStatefulWindowedBolt<T extends State> extends BaseWindowedBolt implements IStatefulWindowedBolt<T> {
     /**
@@ -107,6 +108,15 @@ public abstract class BaseStatefulWindowedBolt<T extends State> extends BaseWind
      * {@inheritDoc}
      */
     @Override
+    public BaseStatefulWindowedBolt<T> withTimestampExtractor(TimestampExtractor timestampExtractor) {
+        super.withTimestampExtractor(timestampExtractor);
+        return this;
+    }
+
+    /**
+     * {@inheritDoc}
+     */
+    @Override
     public BaseStatefulWindowedBolt<T> withLateTupleStream(String streamName) {
         super.withLateTupleStream(streamName);
         return this;

http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/storm-core/src/jvm/org/apache/storm/topology/base/BaseWindowedBolt.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/org/apache/storm/topology/base/BaseWindowedBolt.java b/storm-core/src/jvm/org/apache/storm/topology/base/BaseWindowedBolt.java
index b454407..2e2af0d 100644
--- a/storm-core/src/jvm/org/apache/storm/topology/base/BaseWindowedBolt.java
+++ b/storm-core/src/jvm/org/apache/storm/topology/base/BaseWindowedBolt.java
@@ -22,6 +22,8 @@ import org.apache.storm.task.OutputCollector;
 import org.apache.storm.task.TopologyContext;
 import org.apache.storm.topology.IWindowedBolt;
 import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.topology.TupleFieldTimestampExtractor;
+import org.apache.storm.windowing.TimestampExtractor;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -33,6 +35,7 @@ public abstract class BaseWindowedBolt implements IWindowedBolt {
     private static final Logger LOG = LoggerFactory.getLogger(BaseWindowedBolt.class);
 
     protected final transient Map<String, Object> windowConfiguration;
+    protected TimestampExtractor timestampExtractor;
 
     /**
      * Holds a count value for count based windows and sliding intervals.
@@ -250,10 +253,27 @@ public abstract class BaseWindowedBolt implements IWindowedBolt {
      * @param fieldName the name of the field that contains the timestamp
      */
     public BaseWindowedBolt withTimestampField(String fieldName) {
-        windowConfiguration.put(Config.TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_FIELD_NAME, fieldName);
+        return withTimestampExtractor(TupleFieldTimestampExtractor.of(fieldName));
+    }
+
+    /**
+     * Specify the timestamp extractor implementation.
+     *
+     * @param timestampExtractor the {@link TimestampExtractor} implementation
+     */
+    public BaseWindowedBolt withTimestampExtractor(TimestampExtractor timestampExtractor) {
+        if (this.timestampExtractor != null) {
+            throw new IllegalArgumentException("Window is already configured with a timestamp extractor: " + timestampExtractor);
+        }
+        this.timestampExtractor = timestampExtractor;
         return this;
     }
 
+    @Override
+    public TimestampExtractor getTimestampExtractor() {
+        return timestampExtractor;
+    }
+
     /**
      * Specify a stream id on which late tuples are going to be emitted. They are going to be accessible via the
      * {@link org.apache.storm.topology.WindowedBoltExecutor#LATE_TUPLE_FIELD} field.

http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/storm-core/src/jvm/org/apache/storm/windowing/TimestampExtractor.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/org/apache/storm/windowing/TimestampExtractor.java b/storm-core/src/jvm/org/apache/storm/windowing/TimestampExtractor.java
new file mode 100644
index 0000000..1690535
--- /dev/null
+++ b/storm-core/src/jvm/org/apache/storm/windowing/TimestampExtractor.java
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.windowing;
+
+import org.apache.storm.tuple.Tuple;
+
+import java.io.Serializable;
+
+/**
+ * Interface to be implemented for extracting timestamp from a tuple.
+ */
+public interface TimestampExtractor extends Serializable {
+    /**
+     * Return the tuple timestamp indicating the time when the event happened.
+     *
+     * @param tuple the tuple
+     * @return the timestamp
+     */
+    long extractTimestamp(Tuple tuple);
+}

http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/storm-core/test/jvm/org/apache/storm/topology/WindowedBoltExecutorTest.java
----------------------------------------------------------------------
diff --git a/storm-core/test/jvm/org/apache/storm/topology/WindowedBoltExecutorTest.java b/storm-core/test/jvm/org/apache/storm/topology/WindowedBoltExecutorTest.java
index f6822ce..a051a6b 100644
--- a/storm-core/test/jvm/org/apache/storm/topology/WindowedBoltExecutorTest.java
+++ b/storm-core/test/jvm/org/apache/storm/topology/WindowedBoltExecutorTest.java
@@ -43,7 +43,10 @@ import java.util.List;
 import java.util.Map;
 
 import static org.hamcrest.core.Is.is;
-import static org.junit.Assert.*;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.fail;
 
 /**
  * Unit tests for {@link WindowedBoltExecutor}
@@ -102,12 +105,12 @@ public class WindowedBoltExecutorTest {
     @Before
     public void setUp() {
         testWindowedBolt = new TestWindowedBolt();
+        testWindowedBolt.withTimestampField("ts");
         executor = new WindowedBoltExecutor(testWindowedBolt);
         Map<String, Object> conf = new HashMap<>();
         conf.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS, 100000);
         conf.put(Config.TOPOLOGY_BOLTS_WINDOW_LENGTH_DURATION_MS, 20);
         conf.put(Config.TOPOLOGY_BOLTS_SLIDING_INTERVAL_DURATION_MS, 10);
-        conf.put(Config.TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_FIELD_NAME, "ts");
         conf.put(Config.TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_MAX_LAG_MS, 5);
         // trigger manually to avoid timing issues
         conf.put(Config.TOPOLOGY_BOLTS_WATERMARK_EVENT_INTERVAL_MS, 100000);
@@ -174,12 +177,12 @@ public class WindowedBoltExecutorTest {
         conf.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS, 100000);
         conf.put(Config.TOPOLOGY_BOLTS_WINDOW_LENGTH_DURATION_MS, 20);
         conf.put(Config.TOPOLOGY_BOLTS_SLIDING_INTERVAL_DURATION_MS, 10);
-        conf.put(Config.TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_FIELD_NAME, "ts");
         conf.put(Config.TOPOLOGY_BOLTS_LATE_TUPLE_STREAM, "$late");
         conf.put(Config.TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_MAX_LAG_MS, 5);
         conf.put(Config.TOPOLOGY_BOLTS_WATERMARK_EVENT_INTERVAL_MS, 10);
 
         testWindowedBolt = new TestWindowedBolt();
+        testWindowedBolt.withTimestampField("ts");
         executor = new WindowedBoltExecutor(testWindowedBolt);
         TopologyContext context = getTopologyContext();
         try {
@@ -194,6 +197,7 @@ public class WindowedBoltExecutorTest {
     @Test
     public void testExecuteWithLateTupleStream() throws Exception {
         testWindowedBolt = new TestWindowedBolt();
+        testWindowedBolt.withTimestampField("ts");
         executor = new WindowedBoltExecutor(testWindowedBolt);
         TopologyContext context = getTopologyContext();
         Mockito.when(context.getThisStreams()).thenReturn(new HashSet<>(Arrays.asList("default", "$late")));
@@ -203,7 +207,6 @@ public class WindowedBoltExecutorTest {
         conf.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS, 100000);
         conf.put(Config.TOPOLOGY_BOLTS_WINDOW_LENGTH_DURATION_MS, 20);
         conf.put(Config.TOPOLOGY_BOLTS_SLIDING_INTERVAL_DURATION_MS, 10);
-        conf.put(Config.TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_FIELD_NAME, "ts");
         conf.put(Config.TOPOLOGY_BOLTS_LATE_TUPLE_STREAM, "$late");
         conf.put(Config.TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_MAX_LAG_MS, 5);
         conf.put(Config.TOPOLOGY_BOLTS_WATERMARK_EVENT_INTERVAL_MS, 10);


[2/3] storm git commit: Merge branch 'STORM-2238-1.x' of https://github.com/arunmahadevan/storm into STORM-2238-1.x-merge

Posted by ka...@apache.org.
Merge branch 'STORM-2238-1.x' of https://github.com/arunmahadevan/storm into STORM-2238-1.x-merge


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/7bf95d66
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/7bf95d66
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/7bf95d66

Branch: refs/heads/1.x-branch
Commit: 7bf95d6644b294bd456e08386fe59ed1c78e0c5e
Parents: 3c6abc8 032592a
Author: Jungtaek Lim <ka...@gmail.com>
Authored: Mon Dec 19 07:28:13 2016 +0900
Committer: Jungtaek Lim <ka...@gmail.com>
Committed: Mon Dec 19 07:28:13 2016 +0900

----------------------------------------------------------------------
 docs/Windowing.md                               | 16 ++++++-
 .../clj/org/apache/storm/daemon/executor.clj    |  1 -
 storm-core/src/jvm/org/apache/storm/Config.java | 14 ++----
 .../apache/storm/topology/IWindowedBolt.java    | 11 +++++
 .../topology/TupleFieldTimestampExtractor.java  | 48 ++++++++++++++++++++
 .../storm/topology/WindowedBoltExecutor.java    | 12 +++--
 .../topology/base/BaseStatefulWindowedBolt.java | 10 ++++
 .../storm/topology/base/BaseWindowedBolt.java   | 22 ++++++++-
 .../storm/windowing/TimestampExtractor.java     | 35 ++++++++++++++
 .../topology/WindowedBoltExecutorTest.java      | 11 +++--
 10 files changed, 156 insertions(+), 24 deletions(-)
----------------------------------------------------------------------



[3/3] storm git commit: STORM-2238: CHANGELOG

Posted by ka...@apache.org.
STORM-2238: CHANGELOG


Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/7701bf4e
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/7701bf4e
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/7701bf4e

Branch: refs/heads/1.x-branch
Commit: 7701bf4ea99aa1a5ee34984c489f4de1eb495a73
Parents: 7bf95d6
Author: Jungtaek Lim <ka...@gmail.com>
Authored: Mon Dec 19 07:28:32 2016 +0900
Committer: Jungtaek Lim <ka...@gmail.com>
Committed: Mon Dec 19 07:28:32 2016 +0900

----------------------------------------------------------------------
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/storm/blob/7701bf4e/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9d5d5f4..99f58d7 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
 ## 1.1.0
+ * STORM-2238: Add Timestamp extractor for windowed bolt
  * STORM-2246: Logviewer download link has urlencoding on part of the URL
  * STORM-2235: Introduce new option: 'add remote repositories' for dependency resolver
  * STORM-2215: validate blobs are present before submitting