You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by ka...@apache.org on 2016/12/18 22:32:19 UTC
[1/3] storm git commit: [STORM-2238] Add Timestamp extractor for
windowed bolt
Repository: storm
Updated Branches:
refs/heads/1.x-branch 3c6abc8f7 -> 7701bf4ea
[STORM-2238] Add Timestamp extractor for windowed bolt
Added a timestamp extractor so that the tuple timestamp can be derived from the tuple than directly
mapping it to a field within the tuple.
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/032592a8
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/032592a8
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/032592a8
Branch: refs/heads/1.x-branch
Commit: 032592a892356e36fd3848b1f1692099eed90d28
Parents: cce525b
Author: Arun Mahadevan <ar...@apache.org>
Authored: Thu Dec 8 21:18:12 2016 +0530
Committer: Arun Mahadevan <ar...@apache.org>
Committed: Sun Dec 18 23:11:52 2016 +0530
----------------------------------------------------------------------
docs/Windowing.md | 16 ++++++-
.../clj/org/apache/storm/daemon/executor.clj | 1 -
storm-core/src/jvm/org/apache/storm/Config.java | 14 ++----
.../apache/storm/topology/IWindowedBolt.java | 11 +++++
.../topology/TupleFieldTimestampExtractor.java | 48 ++++++++++++++++++++
.../storm/topology/WindowedBoltExecutor.java | 12 +++--
.../topology/base/BaseStatefulWindowedBolt.java | 10 ++++
.../storm/topology/base/BaseWindowedBolt.java | 22 ++++++++-
.../storm/windowing/TimestampExtractor.java | 35 ++++++++++++++
.../topology/WindowedBoltExecutorTest.java | 11 +++--
10 files changed, 156 insertions(+), 24 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/docs/Windowing.md
----------------------------------------------------------------------
diff --git a/docs/Windowing.md b/docs/Windowing.md
index 82212ea..18f6525 100644
--- a/docs/Windowing.md
+++ b/docs/Windowing.md
@@ -146,8 +146,20 @@ public BaseWindowedBolt withTimestampField(String fieldName)
```
The value for the above `fieldName` will be looked up from the incoming tuple and considered for windowing calculations.
-If the field is not present in the tuple an exception will be thrown. Along with the timestamp field name, a time lag parameter
-can also be specified which indicates the max time limit for tuples with out of order timestamps.
+If the field is not present in the tuple an exception will be thrown. Alternatively a [TimestampExtractor](../storm-core/src/jvm/org/apache/storm/windowing/TimestampExtractor.java) can be used to
+derive a timestamp value from a tuple (e.g. extract timestamp from a nested field within the tuple).
+
+```java
+/**
+* Specify the timestamp extractor implementation.
+*
+* @param timestampExtractor the {@link TimestampExtractor} implementation
+*/
+public BaseWindowedBolt withTimestampExtractor(TimestampExtractor timestampExtractor)
+```
+
+
+Along with the timestamp field name/extractor, a time lag parameter can also be specified which indicates the max time limit for tuples with out of order timestamps.
```java
/**
http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/storm-core/src/clj/org/apache/storm/daemon/executor.clj
----------------------------------------------------------------------
diff --git a/storm-core/src/clj/org/apache/storm/daemon/executor.clj b/storm-core/src/clj/org/apache/storm/daemon/executor.clj
index 3b20b84..95406ec 100644
--- a/storm-core/src/clj/org/apache/storm/daemon/executor.clj
+++ b/storm-core/src/clj/org/apache/storm/daemon/executor.clj
@@ -172,7 +172,6 @@
TOPOLOGY-BOLTS-WINDOW-LENGTH-DURATION-MS
TOPOLOGY-BOLTS-SLIDING-INTERVAL-COUNT
TOPOLOGY-BOLTS-SLIDING-INTERVAL-DURATION-MS
- TOPOLOGY-BOLTS-TUPLE-TIMESTAMP-FIELD-NAME
TOPOLOGY-BOLTS-LATE-TUPLE-STREAM
TOPOLOGY-BOLTS-TUPLE-TIMESTAMP-MAX-LAG-MS
TOPOLOGY-BOLTS-MESSAGE-ID-FIELD-NAME
http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/storm-core/src/jvm/org/apache/storm/Config.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/org/apache/storm/Config.java b/storm-core/src/jvm/org/apache/storm/Config.java
index 0854f35..41b3f67 100644
--- a/storm-core/src/jvm/org/apache/storm/Config.java
+++ b/storm-core/src/jvm/org/apache/storm/Config.java
@@ -1902,14 +1902,6 @@ public class Config extends HashMap<String, Object> {
@isPositiveNumber
public static final String TOPOLOGY_BOLTS_SLIDING_INTERVAL_DURATION_MS = "topology.bolts.window.sliding.interval.duration.ms";
- /*
- * Bolt-specific configuration for windowed bolts to specify the name of the field in the tuple that holds
- * the timestamp (e.g. the ts when the tuple was actually generated). If this config is specified and the
- * field is not present in the incoming tuple, a java.lang.IllegalArgumentException will be thrown.
- */
- @isString
- public static final String TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_FIELD_NAME = "topology.bolts.tuple.timestamp.field.name";
-
/**
* Bolt-specific configuration for windowed bolts to specify the name of the stream on which late tuples are
* going to be emitted. This configuration should only be used from the BaseWindowedBolt.withLateTupleStream builder
@@ -1918,10 +1910,10 @@ public class Config extends HashMap<String, Object> {
@isString
public static final String TOPOLOGY_BOLTS_LATE_TUPLE_STREAM = "topology.bolts.late.tuple.stream";
- /*
+ /**
* Bolt-specific configuration for windowed bolts to specify the maximum time lag of the tuple timestamp
* in milliseconds. It means that the tuple timestamps cannot be out of order by more than this amount.
- * This config will be effective only if the TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_FIELD_NAME is also specified.
+ * This config will be effective only if {@link org.apache.storm.windowing.TimestampExtractor} is specified.
*/
@isInteger
@isPositiveNumber
@@ -1930,7 +1922,7 @@ public class Config extends HashMap<String, Object> {
/*
* Bolt-specific configuration for windowed bolts to specify the time interval for generating
* watermark events. Watermark event tracks the progress of time when tuple timestamp is used.
- * This config is effective only if TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_FIELD_NAME is also specified.
+ * This config is effective only if {@link org.apache.storm.windowing.TimestampExtractor} is specified.
*/
@isInteger
@isPositiveNumber
http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/storm-core/src/jvm/org/apache/storm/topology/IWindowedBolt.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/org/apache/storm/topology/IWindowedBolt.java b/storm-core/src/jvm/org/apache/storm/topology/IWindowedBolt.java
index 3fda395..b0ff262 100644
--- a/storm-core/src/jvm/org/apache/storm/topology/IWindowedBolt.java
+++ b/storm-core/src/jvm/org/apache/storm/topology/IWindowedBolt.java
@@ -19,6 +19,7 @@ package org.apache.storm.topology;
import org.apache.storm.task.OutputCollector;
import org.apache.storm.task.TopologyContext;
+import org.apache.storm.windowing.TimestampExtractor;
import org.apache.storm.windowing.TupleWindow;
import java.util.Map;
@@ -32,9 +33,19 @@ public interface IWindowedBolt extends IComponent {
* that while emitting, the tuples are automatically anchored to the tuples in the inputWindow.
*/
void prepare(Map stormConf, TopologyContext context, OutputCollector collector);
+
/**
* Process the tuple window and optionally emit new tuples based on the tuples in the input window.
*/
void execute(TupleWindow inputWindow);
+
void cleanup();
+
+ /**
+ * Return a {@link TimestampExtractor} for extracting timestamps from a
+ * tuple for event time based processing, or null for processing time.
+ *
+ * @return the timestamp extractor
+ */
+ TimestampExtractor getTimestampExtractor();
}
http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/storm-core/src/jvm/org/apache/storm/topology/TupleFieldTimestampExtractor.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/org/apache/storm/topology/TupleFieldTimestampExtractor.java b/storm-core/src/jvm/org/apache/storm/topology/TupleFieldTimestampExtractor.java
new file mode 100644
index 0000000..4b4ac18
--- /dev/null
+++ b/storm-core/src/jvm/org/apache/storm/topology/TupleFieldTimestampExtractor.java
@@ -0,0 +1,48 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.topology;
+
+import org.apache.storm.tuple.Tuple;
+import org.apache.storm.windowing.TimestampExtractor;
+
+/**
+ * A {@link TimestampExtractor} that extracts timestamp from a specific field in the tuple.
+ */
+public final class TupleFieldTimestampExtractor implements TimestampExtractor {
+ private final String fieldName;
+
+ private TupleFieldTimestampExtractor(String fieldName) {
+ this.fieldName = fieldName;
+ }
+
+ @Override
+ public long extractTimestamp(Tuple tuple) {
+ return tuple.getLongByField(fieldName);
+ }
+
+ public static TupleFieldTimestampExtractor of(String fieldName) {
+ return new TupleFieldTimestampExtractor(fieldName);
+ }
+
+ @Override
+ public String toString() {
+ return "TupleFieldTimestampExtractor{" +
+ "fieldName='" + fieldName + '\'' +
+ '}';
+ }
+}
http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/storm-core/src/jvm/org/apache/storm/topology/WindowedBoltExecutor.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/org/apache/storm/topology/WindowedBoltExecutor.java b/storm-core/src/jvm/org/apache/storm/topology/WindowedBoltExecutor.java
index 3f245e1..fd98274 100644
--- a/storm-core/src/jvm/org/apache/storm/topology/WindowedBoltExecutor.java
+++ b/storm-core/src/jvm/org/apache/storm/topology/WindowedBoltExecutor.java
@@ -23,6 +23,7 @@ import org.apache.storm.spout.CheckpointSpout;
import org.apache.storm.task.IOutputCollector;
import org.apache.storm.task.OutputCollector;
import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.base.BaseWindowedBolt;
import org.apache.storm.tuple.Fields;
import org.apache.storm.tuple.Tuple;
import org.apache.storm.tuple.Values;
@@ -31,6 +32,7 @@ import org.apache.storm.windowing.CountTriggerPolicy;
import org.apache.storm.windowing.EvictionPolicy;
import org.apache.storm.windowing.TimeEvictionPolicy;
import org.apache.storm.windowing.TimeTriggerPolicy;
+import org.apache.storm.windowing.TimestampExtractor;
import org.apache.storm.windowing.TriggerPolicy;
import org.apache.storm.windowing.TupleWindowImpl;
import org.apache.storm.windowing.WaterMarkEventGenerator;
@@ -65,7 +67,7 @@ public class WindowedBoltExecutor implements IRichBolt {
private transient WindowLifecycleListener<Tuple> listener;
private transient WindowManager<Tuple> windowManager;
private transient int maxLagMs;
- private transient String tupleTsFieldName;
+ private TimestampExtractor timestampExtractor;
private transient String lateTupleStream;
private transient TriggerPolicy<Tuple> triggerPolicy;
private transient EvictionPolicy<Tuple> evictionPolicy;
@@ -74,6 +76,7 @@ public class WindowedBoltExecutor implements IRichBolt {
public WindowedBoltExecutor(IWindowedBolt bolt) {
this.bolt = bolt;
+ timestampExtractor = bolt.getTimestampExtractor();
}
private int getTopologyTimeoutMillis(Map stormConf) {
@@ -165,8 +168,7 @@ public class WindowedBoltExecutor implements IRichBolt {
slidingIntervalCount = new Count(1);
}
// tuple ts
- if (stormConf.containsKey(Config.TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_FIELD_NAME)) {
- tupleTsFieldName = (String) stormConf.get(Config.TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_FIELD_NAME);
+ if (timestampExtractor != null) {
// late tuple stream
lateTupleStream = (String) stormConf.get(Config.TOPOLOGY_BOLTS_LATE_TUPLE_STREAM);
if (lateTupleStream != null) {
@@ -229,7 +231,7 @@ public class WindowedBoltExecutor implements IRichBolt {
}
private boolean isTupleTs() {
- return tupleTsFieldName != null;
+ return timestampExtractor != null;
}
private TriggerPolicy<Tuple> getTriggerPolicy(Count slidingIntervalCount, Duration slidingIntervalDuration,
@@ -279,7 +281,7 @@ public class WindowedBoltExecutor implements IRichBolt {
@Override
public void execute(Tuple input) {
if (isTupleTs()) {
- long ts = input.getLongByField(tupleTsFieldName);
+ long ts = timestampExtractor.extractTimestamp(input);
if (waterMarkEventGenerator.track(input.getSourceGlobalStreamId(), ts)) {
windowManager.add(input, ts);
} else {
http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/storm-core/src/jvm/org/apache/storm/topology/base/BaseStatefulWindowedBolt.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/org/apache/storm/topology/base/BaseStatefulWindowedBolt.java b/storm-core/src/jvm/org/apache/storm/topology/base/BaseStatefulWindowedBolt.java
index 0c35b9d..7e80830 100644
--- a/storm-core/src/jvm/org/apache/storm/topology/base/BaseStatefulWindowedBolt.java
+++ b/storm-core/src/jvm/org/apache/storm/topology/base/BaseStatefulWindowedBolt.java
@@ -20,6 +20,7 @@ package org.apache.storm.topology.base;
import org.apache.storm.Config;
import org.apache.storm.state.State;
import org.apache.storm.topology.IStatefulWindowedBolt;
+import org.apache.storm.windowing.TimestampExtractor;
public abstract class BaseStatefulWindowedBolt<T extends State> extends BaseWindowedBolt implements IStatefulWindowedBolt<T> {
/**
@@ -107,6 +108,15 @@ public abstract class BaseStatefulWindowedBolt<T extends State> extends BaseWind
* {@inheritDoc}
*/
@Override
+ public BaseStatefulWindowedBolt<T> withTimestampExtractor(TimestampExtractor timestampExtractor) {
+ super.withTimestampExtractor(timestampExtractor);
+ return this;
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
public BaseStatefulWindowedBolt<T> withLateTupleStream(String streamName) {
super.withLateTupleStream(streamName);
return this;
http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/storm-core/src/jvm/org/apache/storm/topology/base/BaseWindowedBolt.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/org/apache/storm/topology/base/BaseWindowedBolt.java b/storm-core/src/jvm/org/apache/storm/topology/base/BaseWindowedBolt.java
index b454407..2e2af0d 100644
--- a/storm-core/src/jvm/org/apache/storm/topology/base/BaseWindowedBolt.java
+++ b/storm-core/src/jvm/org/apache/storm/topology/base/BaseWindowedBolt.java
@@ -22,6 +22,8 @@ import org.apache.storm.task.OutputCollector;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.topology.IWindowedBolt;
import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.topology.TupleFieldTimestampExtractor;
+import org.apache.storm.windowing.TimestampExtractor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -33,6 +35,7 @@ public abstract class BaseWindowedBolt implements IWindowedBolt {
private static final Logger LOG = LoggerFactory.getLogger(BaseWindowedBolt.class);
protected final transient Map<String, Object> windowConfiguration;
+ protected TimestampExtractor timestampExtractor;
/**
* Holds a count value for count based windows and sliding intervals.
@@ -250,10 +253,27 @@ public abstract class BaseWindowedBolt implements IWindowedBolt {
* @param fieldName the name of the field that contains the timestamp
*/
public BaseWindowedBolt withTimestampField(String fieldName) {
- windowConfiguration.put(Config.TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_FIELD_NAME, fieldName);
+ return withTimestampExtractor(TupleFieldTimestampExtractor.of(fieldName));
+ }
+
+ /**
+ * Specify the timestamp extractor implementation.
+ *
+ * @param timestampExtractor the {@link TimestampExtractor} implementation
+ */
+ public BaseWindowedBolt withTimestampExtractor(TimestampExtractor timestampExtractor) {
+ if (this.timestampExtractor != null) {
+ throw new IllegalArgumentException("Window is already configured with a timestamp extractor: " + timestampExtractor);
+ }
+ this.timestampExtractor = timestampExtractor;
return this;
}
+ @Override
+ public TimestampExtractor getTimestampExtractor() {
+ return timestampExtractor;
+ }
+
/**
* Specify a stream id on which late tuples are going to be emitted. They are going to be accessible via the
* {@link org.apache.storm.topology.WindowedBoltExecutor#LATE_TUPLE_FIELD} field.
http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/storm-core/src/jvm/org/apache/storm/windowing/TimestampExtractor.java
----------------------------------------------------------------------
diff --git a/storm-core/src/jvm/org/apache/storm/windowing/TimestampExtractor.java b/storm-core/src/jvm/org/apache/storm/windowing/TimestampExtractor.java
new file mode 100644
index 0000000..1690535
--- /dev/null
+++ b/storm-core/src/jvm/org/apache/storm/windowing/TimestampExtractor.java
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.windowing;
+
+import org.apache.storm.tuple.Tuple;
+
+import java.io.Serializable;
+
+/**
+ * Interface to be implemented for extracting timestamp from a tuple.
+ */
+public interface TimestampExtractor extends Serializable {
+ /**
+ * Return the tuple timestamp indicating the time when the event happened.
+ *
+ * @param tuple the tuple
+ * @return the timestamp
+ */
+ long extractTimestamp(Tuple tuple);
+}
http://git-wip-us.apache.org/repos/asf/storm/blob/032592a8/storm-core/test/jvm/org/apache/storm/topology/WindowedBoltExecutorTest.java
----------------------------------------------------------------------
diff --git a/storm-core/test/jvm/org/apache/storm/topology/WindowedBoltExecutorTest.java b/storm-core/test/jvm/org/apache/storm/topology/WindowedBoltExecutorTest.java
index f6822ce..a051a6b 100644
--- a/storm-core/test/jvm/org/apache/storm/topology/WindowedBoltExecutorTest.java
+++ b/storm-core/test/jvm/org/apache/storm/topology/WindowedBoltExecutorTest.java
@@ -43,7 +43,10 @@ import java.util.List;
import java.util.Map;
import static org.hamcrest.core.Is.is;
-import static org.junit.Assert.*;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.fail;
/**
* Unit tests for {@link WindowedBoltExecutor}
@@ -102,12 +105,12 @@ public class WindowedBoltExecutorTest {
@Before
public void setUp() {
testWindowedBolt = new TestWindowedBolt();
+ testWindowedBolt.withTimestampField("ts");
executor = new WindowedBoltExecutor(testWindowedBolt);
Map<String, Object> conf = new HashMap<>();
conf.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS, 100000);
conf.put(Config.TOPOLOGY_BOLTS_WINDOW_LENGTH_DURATION_MS, 20);
conf.put(Config.TOPOLOGY_BOLTS_SLIDING_INTERVAL_DURATION_MS, 10);
- conf.put(Config.TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_FIELD_NAME, "ts");
conf.put(Config.TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_MAX_LAG_MS, 5);
// trigger manually to avoid timing issues
conf.put(Config.TOPOLOGY_BOLTS_WATERMARK_EVENT_INTERVAL_MS, 100000);
@@ -174,12 +177,12 @@ public class WindowedBoltExecutorTest {
conf.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS, 100000);
conf.put(Config.TOPOLOGY_BOLTS_WINDOW_LENGTH_DURATION_MS, 20);
conf.put(Config.TOPOLOGY_BOLTS_SLIDING_INTERVAL_DURATION_MS, 10);
- conf.put(Config.TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_FIELD_NAME, "ts");
conf.put(Config.TOPOLOGY_BOLTS_LATE_TUPLE_STREAM, "$late");
conf.put(Config.TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_MAX_LAG_MS, 5);
conf.put(Config.TOPOLOGY_BOLTS_WATERMARK_EVENT_INTERVAL_MS, 10);
testWindowedBolt = new TestWindowedBolt();
+ testWindowedBolt.withTimestampField("ts");
executor = new WindowedBoltExecutor(testWindowedBolt);
TopologyContext context = getTopologyContext();
try {
@@ -194,6 +197,7 @@ public class WindowedBoltExecutorTest {
@Test
public void testExecuteWithLateTupleStream() throws Exception {
testWindowedBolt = new TestWindowedBolt();
+ testWindowedBolt.withTimestampField("ts");
executor = new WindowedBoltExecutor(testWindowedBolt);
TopologyContext context = getTopologyContext();
Mockito.when(context.getThisStreams()).thenReturn(new HashSet<>(Arrays.asList("default", "$late")));
@@ -203,7 +207,6 @@ public class WindowedBoltExecutorTest {
conf.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS, 100000);
conf.put(Config.TOPOLOGY_BOLTS_WINDOW_LENGTH_DURATION_MS, 20);
conf.put(Config.TOPOLOGY_BOLTS_SLIDING_INTERVAL_DURATION_MS, 10);
- conf.put(Config.TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_FIELD_NAME, "ts");
conf.put(Config.TOPOLOGY_BOLTS_LATE_TUPLE_STREAM, "$late");
conf.put(Config.TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_MAX_LAG_MS, 5);
conf.put(Config.TOPOLOGY_BOLTS_WATERMARK_EVENT_INTERVAL_MS, 10);
[2/3] storm git commit: Merge branch 'STORM-2238-1.x' of
https://github.com/arunmahadevan/storm into STORM-2238-1.x-merge
Posted by ka...@apache.org.
Merge branch 'STORM-2238-1.x' of https://github.com/arunmahadevan/storm into STORM-2238-1.x-merge
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/7bf95d66
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/7bf95d66
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/7bf95d66
Branch: refs/heads/1.x-branch
Commit: 7bf95d6644b294bd456e08386fe59ed1c78e0c5e
Parents: 3c6abc8 032592a
Author: Jungtaek Lim <ka...@gmail.com>
Authored: Mon Dec 19 07:28:13 2016 +0900
Committer: Jungtaek Lim <ka...@gmail.com>
Committed: Mon Dec 19 07:28:13 2016 +0900
----------------------------------------------------------------------
docs/Windowing.md | 16 ++++++-
.../clj/org/apache/storm/daemon/executor.clj | 1 -
storm-core/src/jvm/org/apache/storm/Config.java | 14 ++----
.../apache/storm/topology/IWindowedBolt.java | 11 +++++
.../topology/TupleFieldTimestampExtractor.java | 48 ++++++++++++++++++++
.../storm/topology/WindowedBoltExecutor.java | 12 +++--
.../topology/base/BaseStatefulWindowedBolt.java | 10 ++++
.../storm/topology/base/BaseWindowedBolt.java | 22 ++++++++-
.../storm/windowing/TimestampExtractor.java | 35 ++++++++++++++
.../topology/WindowedBoltExecutorTest.java | 11 +++--
10 files changed, 156 insertions(+), 24 deletions(-)
----------------------------------------------------------------------
[3/3] storm git commit: STORM-2238: CHANGELOG
Posted by ka...@apache.org.
STORM-2238: CHANGELOG
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/7701bf4e
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/7701bf4e
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/7701bf4e
Branch: refs/heads/1.x-branch
Commit: 7701bf4ea99aa1a5ee34984c489f4de1eb495a73
Parents: 7bf95d6
Author: Jungtaek Lim <ka...@gmail.com>
Authored: Mon Dec 19 07:28:32 2016 +0900
Committer: Jungtaek Lim <ka...@gmail.com>
Committed: Mon Dec 19 07:28:32 2016 +0900
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/7701bf4e/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9d5d5f4..99f58d7 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
## 1.1.0
+ * STORM-2238: Add Timestamp extractor for windowed bolt
* STORM-2246: Logviewer download link has urlencoding on part of the URL
* STORM-2235: Introduce new option: 'add remote repositories' for dependency resolver
* STORM-2215: validate blobs are present before submitting