You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Parth-Brahmbhatt <gi...@git.apache.org> on 2014/07/16 19:59:40 UTC
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
GitHub user Parth-Brahmbhatt opened a pull request:
https://github.com/apache/incubator-storm/pull/195
STORM-405: Add kafka trident state so messages ca be sent to kafka topic...
...s
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Parth-Brahmbhatt/incubator-storm STORM-405
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-storm/pull/195.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #195
----
commit a277c887f16e07e5aae3093f77751f23dd6139b6
Author: Parth Brahmbhatt <br...@gmail.com>
Date: 2014-07-16T17:58:29Z
STORM-405: Add kafka trident state so messages ca be sent to kafka topics
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by Parth-Brahmbhatt <gi...@git.apache.org>.
Github user Parth-Brahmbhatt commented on the pull request:
https://github.com/apache/incubator-storm/pull/195#issuecomment-49940717
Modified the name and builder method names to be less verbose.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by Parth-Brahmbhatt <gi...@git.apache.org>.
Github user Parth-Brahmbhatt commented on the pull request:
https://github.com/apache/incubator-storm/pull/195#issuecomment-49778966
Added mapper interface that can map tuple/tridentTuple to kafkaKey and kafkaMessage with default implementation that is backwards compatible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by Parth-Brahmbhatt <gi...@git.apache.org>.
Github user Parth-Brahmbhatt commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/195#discussion_r15473369
--- Diff: external/storm-kafka/README.md ---
@@ -44,6 +44,105 @@ use Kafka 0.8.1.1 built against Scala 2.10, you would use the following dependen
Note that the ZooKeeper and log4j dependencies are excluded to prevent version conflicts with Storm's dependencies.
+##Writing to Kafka as part of your topology
+You can create an instance of storm.kafka.bolt.KafkaBolt and attach it as a component to your topology or if you
+are using trident you can use storm.kafka.trident.TridentState, storm.kafka.trident.TridentStateFactory and
+storm.kafka.trident.TridentKafkaUpdater.
+
+You need to provide implementation of following 2 interfaces
+
+###TupleToKafkaMapper and TridentTupleToKafkaMapper
+These interfaces have 2 methods defined:
+
+```java
+ K getKeyFromTuple(Tuple/TridentTuple tuple);
+ V getMessageFromTuple(Tuple/TridentTuple tuple);
+```
+
+as the name suggests these methods are called to map a tuple to kafka key and kafka message. If you just want one field
+as key and one field as value then you can use the provided FieldNameBasedTupleToKafkaKeyAndMessageMapper.java
+implementation. In the KafkaBolt the implementation always looks for a field with field name "key" and "message" for
+backward compatibility reasons.
+In the TridentKafkaState you can specify what is the field name for key and message. These should be specified while
+constructing and instance of FieldNameBasedTupleToKafkaKeyAndMessageMapper.
+
+###KafkaTopicSelector and trident KafkaTopicSelector
+This interface has only one method
+```java
+public interface KafkaTopicSelector {
+ List<String> getTopics(Tuple/TridentTuple tuple);
+}
+```
+The implementation of this interface should return topics to which the tuple's key/message mapping needs to be published
+You can return an empty list but null is not acceptable. If you have one static topic name then you can use
+DefaultTopicSelector.java and set the name of the topic in the constructor.
+
+### Specifying kafka producer properties
+You can provide all the produce properties , see http://kafka.apache.org/07/configuration.html
--- End diff --
updated
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by Parth-Brahmbhatt <gi...@git.apache.org>.
Github user Parth-Brahmbhatt commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/195#discussion_r15601737
--- Diff: external/storm-kafka/src/test/storm/kafka/TridentKafkaTopology.java ---
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package storm.kafka;
+
+import backtype.storm.Config;
+import backtype.storm.LocalCluster;
+import backtype.storm.generated.StormTopology;
+import backtype.storm.tuple.Fields;
+import backtype.storm.tuple.Values;
+import storm.kafka.trident.TridentKafkaState;
+import storm.kafka.trident.TridentKafkaStateFactory;
+import storm.kafka.trident.TridentKafkaUpdater;
+import storm.kafka.trident.mapper.FieldNameBasedTupleToKafkaMapper;
+import storm.kafka.trident.selector.DefaultTopicSelector;
+import storm.trident.Stream;
+import storm.trident.TridentTopology;
+import storm.trident.testing.FixedBatchSpout;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+
+
+public class TridentKafkaTopology {
+
+ private static StormTopology buildTopology() {
+ Fields fields = new Fields("word", "count");
+ FixedBatchSpout spout = new FixedBatchSpout(fields, 4,
+ new Values("storm", "1"),
+ new Values("trident", "1"),
+ new Values("needs", "1"),
+ new Values("javadoc", "1")
+ );
+ spout.setCycle(true);
+
+ TridentTopology topology = new TridentTopology();
+ Stream stream = topology.newStream("spout1", spout);
+
+ TridentKafkaStateFactory stateFactory = new TridentKafkaStateFactory()
+ .withKafkaTopicSelector(new DefaultTopicSelector("test"))
+ .withTridentTupleToKafkaMapper(new FieldNameBasedTupleToKafkaMapper("word", "count"));
+ stream.partitionPersist(stateFactory, fields, new TridentKafkaUpdater(), new Fields());
+
+ return topology.build();
+ }
+
+ /**
+ * To run this topology ensure you have a kafka broker running and provide connection string to broker as argument.
+ * Create a topic test with command line,
+ * kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partition 1 --topic test
+ *
+ * run this program and run the kafka consumer:
+ * afka-console-consumer.sh --zookeeper localhost:2181 --whitelist test --from-beginning
--- End diff --
fixed the type added the --topic
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by wurstmeister <gi...@git.apache.org>.
Github user wurstmeister commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/195#discussion_r15433205
--- Diff: external/storm-kafka/README.md ---
@@ -44,6 +44,105 @@ use Kafka 0.8.1.1 built against Scala 2.10, you would use the following dependen
Note that the ZooKeeper and log4j dependencies are excluded to prevent version conflicts with Storm's dependencies.
+##Writing to Kafka as part of your topology
+You can create an instance of storm.kafka.bolt.KafkaBolt and attach it as a component to your topology or if you
+are using trident you can use storm.kafka.trident.TridentState, storm.kafka.trident.TridentStateFactory and
+storm.kafka.trident.TridentKafkaUpdater.
+
+You need to provide implementation of following 2 interfaces
+
+###TupleToKafkaMapper and TridentTupleToKafkaMapper
+These interfaces have 2 methods defined:
+
+```java
+ K getKeyFromTuple(Tuple/TridentTuple tuple);
+ V getMessageFromTuple(Tuple/TridentTuple tuple);
+```
+
+as the name suggests these methods are called to map a tuple to kafka key and kafka message. If you just want one field
+as key and one field as value then you can use the provided FieldNameBasedTupleToKafkaKeyAndMessageMapper.java
+implementation. In the KafkaBolt the implementation always looks for a field with field name "key" and "message" for
+backward compatibility reasons.
+In the TridentKafkaState you can specify what is the field name for key and message. These should be specified while
+constructing and instance of FieldNameBasedTupleToKafkaKeyAndMessageMapper.
+
+###KafkaTopicSelector and trident KafkaTopicSelector
+This interface has only one method
+```java
+public interface KafkaTopicSelector {
+ List<String> getTopics(Tuple/TridentTuple tuple);
+}
+```
+The implementation of this interface should return topics to which the tuple's key/message mapping needs to be published
+You can return an empty list but null is not acceptable. If you have one static topic name then you can use
+DefaultTopicSelector.java and set the name of the topic in the constructor.
+
+### Specifying kafka producer properties
+You can provide all the produce properties , see http://kafka.apache.org/07/configuration.html
--- End diff --
The link refers to kafka 0.7, I think we should refer to http://kafka.apache.org/documentation.html#producerconfigs
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by ptgoetz <gi...@git.apache.org>.
Github user ptgoetz commented on the pull request:
https://github.com/apache/incubator-storm/pull/195#issuecomment-49932065
Overall it looks good to me. The only thing I would like to see changed is the name of the mapper interfaces and associated builder methods to something a little shorter.
For example, could `TridentTupleToKafkaKeyAndMessageMapper` be changed to something a little more concise like `TridentKafkaTupleMapper`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by wurstmeister <gi...@git.apache.org>.
Github user wurstmeister commented on the pull request:
https://github.com/apache/incubator-storm/pull/195#issuecomment-50805385
+1
Could you please double check the readme though, FieldNameBasedTupleToKafkaKeyAndMessageMapper does not exist anymore, and I don't think we need a reference to "Important configuration properties for the producer" anymore.
Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by wurstmeister <gi...@git.apache.org>.
Github user wurstmeister commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/195#discussion_r15542941
--- Diff: external/storm-kafka/README.md ---
@@ -44,6 +44,105 @@ use Kafka 0.8.1.1 built against Scala 2.10, you would use the following dependen
Note that the ZooKeeper and log4j dependencies are excluded to prevent version conflicts with Storm's dependencies.
+##Writing to Kafka as part of your topology
+You can create an instance of storm.kafka.bolt.KafkaBolt and attach it as a component to your topology or if you
+are using trident you can use storm.kafka.trident.TridentState, storm.kafka.trident.TridentStateFactory and
+storm.kafka.trident.TridentKafkaUpdater.
+
+You need to provide implementation of following 2 interfaces
+
+###TupleToKafkaMapper and TridentTupleToKafkaMapper
+These interfaces have 2 methods defined:
+
+```java
+ K getKeyFromTuple(Tuple/TridentTuple tuple);
+ V getMessageFromTuple(Tuple/TridentTuple tuple);
+```
+
+as the name suggests these methods are called to map a tuple to kafka key and kafka message. If you just want one field
+as key and one field as value then you can use the provided FieldNameBasedTupleToKafkaKeyAndMessageMapper.java
+implementation. In the KafkaBolt the implementation always looks for a field with field name "key" and "message" for
+backward compatibility reasons.
+In the TridentKafkaState you can specify what is the field name for key and message. These should be specified while
+constructing and instance of FieldNameBasedTupleToKafkaKeyAndMessageMapper.
+
+###KafkaTopicSelector and trident KafkaTopicSelector
+This interface has only one method
+```java
+public interface KafkaTopicSelector {
+ List<String> getTopics(Tuple/TridentTuple tuple);
+}
+```
+The implementation of this interface should return topics to which the tuple's key/message mapping needs to be published
+You can return an empty list but null is not acceptable. If you have one static topic name then you can use
+DefaultTopicSelector.java and set the name of the topic in the constructor.
+
--- End diff --
this section of the readme does not match the code anymore. especially the fact that null is not acceptable. at the moment if the topic is null for a particular tuple, the tuple is quietly ignored.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by wurstmeister <gi...@git.apache.org>.
Github user wurstmeister commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/195#discussion_r15556066
--- Diff: external/storm-kafka/src/test/storm/kafka/TridentKafkaTopology.java ---
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package storm.kafka;
+
+import backtype.storm.Config;
+import backtype.storm.LocalCluster;
+import backtype.storm.generated.StormTopology;
+import backtype.storm.tuple.Fields;
+import backtype.storm.tuple.Values;
+import storm.kafka.trident.TridentKafkaState;
+import storm.kafka.trident.TridentKafkaStateFactory;
+import storm.kafka.trident.TridentKafkaUpdater;
+import storm.kafka.trident.mapper.FieldNameBasedTupleToKafkaMapper;
+import storm.kafka.trident.selector.DefaultTopicSelector;
+import storm.trident.Stream;
+import storm.trident.TridentTopology;
+import storm.trident.testing.FixedBatchSpout;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+
+
+public class TridentKafkaTopology {
+
+ private static StormTopology buildTopology() {
+ Fields fields = new Fields("word", "count");
+ FixedBatchSpout spout = new FixedBatchSpout(fields, 4,
+ new Values("storm", "1"),
+ new Values("trident", "1"),
+ new Values("needs", "1"),
+ new Values("javadoc", "1")
+ );
+ spout.setCycle(true);
+
+ TridentTopology topology = new TridentTopology();
+ Stream stream = topology.newStream("spout1", spout);
+
+ TridentKafkaStateFactory stateFactory = new TridentKafkaStateFactory()
+ .withKafkaTopicSelector(new DefaultTopicSelector("test"))
+ .withTridentTupleToKafkaMapper(new FieldNameBasedTupleToKafkaMapper("word", "count"));
+ stream.partitionPersist(stateFactory, fields, new TridentKafkaUpdater(), new Fields());
+
+ return topology.build();
+ }
+
+ /**
+ * To run this topology ensure you have a kafka broker running and provide connection string to broker as argument.
+ * Create a topic test with command line,
+ * kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partition 1 --topic test
+ *
+ * run this program and run the kafka consumer:
+ * afka-console-consumer.sh --zookeeper localhost:2181 --whitelist test --from-beginning
+ *
+ * you should see the messages flowing through.
+ *
+ * @param args
+ * @throws Exception
+ */
+ public static void main(String[] args) throws Exception {
+ if(args.length < 1) {
+ System.out.println("Please provide kafka broker url ,e.g. localhost:9092");
+ }
+
+ Config conf = getConfig(args[0]);
+ LocalCluster cluster = new LocalCluster();
+ cluster.submitTopology("wordCounter", conf, buildTopology());
+ Thread.sleep(60 * 1000);
+ cluster.killTopology("wordCounter");
+
+ cluster.shutdown();
+ }
+
+ private static Config getConfig(String brokerConnectionString) {
+ Config conf = new Config();
+ Map config = new HashMap();
+ Properties props = new Properties();
+ props.put("metadata.broker.list", brokerConnectionString);
+ props.put("request.required.acks", "1");
+ props.put("serializer.class", "kafka.serializer.StringEncoder");
+ conf.put(TridentKafkaState.KAFKA_BROKER_PROPERTIES, props);
+ conf.put(TridentKafkaState.TOPIC, "test");
--- End diff --
is this really required? we are using a topic selector in the state factory. (as far as i can see we don't even need the constant in TridentKafkaState anymore)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by wurstmeister <gi...@git.apache.org>.
Github user wurstmeister commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/195#discussion_r15556565
--- Diff: external/storm-kafka/src/jvm/storm/kafka/trident/TridentKafkaState.java ---
@@ -0,0 +1,99 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package storm.kafka.trident;
+
+import backtype.storm.task.OutputCollector;
+import backtype.storm.topology.FailedException;
+import kafka.javaapi.producer.Producer;
+import kafka.producer.KeyedMessage;
+import kafka.producer.ProducerConfig;
+import org.apache.commons.lang.Validate;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import storm.kafka.trident.mapper.TridentTupleToKafkaMapper;
+import storm.kafka.trident.selector.KafkaTopicSelector;
+import storm.trident.operation.TridentCollector;
+import storm.trident.state.State;
+import storm.trident.tuple.TridentTuple;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+public class TridentKafkaState implements State {
+ private static final Logger LOG = LoggerFactory.getLogger(TridentKafkaState.class);
+
+
+ public static final String TOPIC = "topic";
+ public static final String KAFKA_BROKER_PROPERTIES = "kafka.broker.properties";
+
+ private Producer producer;
+ private OutputCollector collector;
+
+ private TridentTupleToKafkaMapper mapper;
+ private KafkaTopicSelector topicSelector;
+
+ public TridentKafkaState withTridentTupleToKafkaMapper(TridentTupleToKafkaMapper mapper) {
+ this.mapper = mapper;
+ return this;
+ }
+
+ public TridentKafkaState withKafkaTopicSelector(KafkaTopicSelector selector) {
+ this.topicSelector = selector;
+ return this;
+ }
+
+ @Override
+ public void beginCommit(Long txid) {
+ LOG.debug("beginCommit is Noop.");
+ }
+
+ @Override
+ public void commit(Long txid) {
+ LOG.debug("commit is Noop.");
+ }
+
+ public void prepare(Map stormConf) {
+ Validate.notNull(mapper, "mapper can not be null");
+ Validate.notNull(topicSelector, "topicSelector can not be null");
+ Map configMap = (Map) stormConf.get(KAFKA_BROKER_PROPERTIES);
+ Properties properties = new Properties();
+ properties.putAll(configMap);
+ ProducerConfig config = new ProducerConfig(properties);
+ producer = new Producer(config);
+ }
+
+ public void updateState(List<TridentTuple> tuples, TridentCollector collector) {
+ String topic = null;
+ for (TridentTuple tuple : tuples) {
+ try {
+ topic = topicSelector.getTopic(tuple);
+
+ if(topic != null) {
+ producer.send(new KeyedMessage(topic, mapper.getKeyFromTuple(tuple),
+ mapper.getMessageFromTuple(tuple)));
+ }
--- End diff --
i think we should at least log a warning that the tuple was ignored if the topic was null (here and in KafkaBolt). otherwise, if a topic selector implementation returns null by mistake, it's going to be a bit hard to debug.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by Parth-Brahmbhatt <gi...@git.apache.org>.
Github user Parth-Brahmbhatt commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/195#discussion_r15473439
--- Diff: external/storm-kafka/src/jvm/storm/kafka/trident/TridentKafkaState.java ---
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package storm.kafka.trident;
+
+import backtype.storm.task.OutputCollector;
+import backtype.storm.topology.FailedException;
+import kafka.javaapi.producer.Producer;
+import kafka.producer.KeyedMessage;
+import kafka.producer.ProducerConfig;
+import org.apache.commons.lang.Validate;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import storm.kafka.trident.mapper.TridentTupleToKafkaMapper;
+import storm.kafka.trident.selector.KafkaTopicSelector;
+import storm.trident.operation.TridentCollector;
+import storm.trident.state.State;
+import storm.trident.tuple.TridentTuple;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+public class TridentKafkaState implements State {
+ private static final Logger LOG = LoggerFactory.getLogger(TridentKafkaState.class);
+
+
+ public static final String TOPIC = "topic";
+ public static final String KAFKA_BROKER_PROPERTIES = "kafka.broker.properties";
+
+ private Producer producer;
+ private OutputCollector collector;
+
+ private TridentTupleToKafkaMapper mapper;
+ private KafkaTopicSelector topicSelector;
+
+ public TridentKafkaState withTridentTupleToKafkaMapper(TridentTupleToKafkaMapper mapper) {
+ this.mapper = mapper;
+ return this;
+ }
+
+ public TridentKafkaState withKafkaTopicSelector(KafkaTopicSelector selector) {
+ this.topicSelector = selector;
+ return this;
+ }
+
+ @Override
+ public void beginCommit(Long txid) {
+ LOG.debug("beginCommit is Noop.");
+ }
+
+ @Override
+ public void commit(Long txid) {
+ LOG.debug("commit is Noop.");
+ }
+
+ public void prepare(Map stormConf) {
+ Validate.notNull(mapper, "mapper can not be null");
+ Validate.notNull(topicSelector, "topicSelector can not be null");
+ Map configMap = (Map) stormConf.get(KAFKA_BROKER_PROPERTIES);
+ Properties properties = new Properties();
+ properties.putAll(configMap);
+ ProducerConfig config = new ProducerConfig(properties);
+ producer = new Producer(config);
+ }
+
+ public void updateState(List<TridentTuple> tuples, TridentCollector collector) {
+ for (TridentTuple tuple : tuples) {
+ try {
+ for(String topic : topicSelector.getTopics(tuple)) {
+ producer.send(new KeyedMessage(topic, mapper.getKeyFromTuple(tuple),
+ mapper.getMessageFromTuple(tuple)));
+ }
+ } catch (Exception ex) {
--- End diff --
updated to have a single topic.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by ptgoetz <gi...@git.apache.org>.
Github user ptgoetz commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/195#discussion_r15313891
--- Diff: external/storm-kafka/src/test/storm/kafka/TestUtils.java ---
@@ -34,4 +49,51 @@ public static GlobalPartitionInformation buildPartitionInfo(int numPartitions, i
return globalPartitionInformation;
}
+ public static SimpleConsumer getKafkaConsumer(KafkaTestBroker broker) {
+ BrokerHosts brokerHosts = getBrokerHosts(broker);
+ KafkaConfig kafkaConfig = new KafkaConfig(brokerHosts, TOPIC);
+ SimpleConsumer simpleConsumer = new SimpleConsumer("localhost", broker.getPort(), 60000, 1024, "testClient");
+ return simpleConsumer;
+ }
+
+ public static KafkaConfig getKafkaConfig(KafkaTestBroker broker) {
+ BrokerHosts brokerHosts = getBrokerHosts(broker);
+ KafkaConfig kafkaConfig = new KafkaConfig(brokerHosts, TOPIC);
+ return kafkaConfig;
+ }
+
+ private static BrokerHosts getBrokerHosts(KafkaTestBroker broker) {
+ GlobalPartitionInformation globalPartitionInformation = new GlobalPartitionInformation();
+ globalPartitionInformation.addPartition(0, Broker.fromString(broker.getBrokerConnectionString()));
+ return new StaticHosts(globalPartitionInformation);
+ }
+
+ public static Config getConfig(String brokerConnectionString) {
+ Config config = new Config();
+ Properties props = new Properties();
+ props.put("metadata.broker.list", brokerConnectionString);
+ props.put("request.required.acks", "1");
+ props.put("serializer.class", "kafka.serializer.StringEncoder");
--- End diff --
Would it make sense to make the serializer class configurable? I can imagine cases where users would want to use other serialization schemes as they can with the spouts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by Parth-Brahmbhatt <gi...@git.apache.org>.
Github user Parth-Brahmbhatt commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/195#discussion_r15601734
--- Diff: external/storm-kafka/src/jvm/storm/kafka/trident/TridentKafkaState.java ---
@@ -0,0 +1,99 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package storm.kafka.trident;
+
+import backtype.storm.task.OutputCollector;
+import backtype.storm.topology.FailedException;
+import kafka.javaapi.producer.Producer;
+import kafka.producer.KeyedMessage;
+import kafka.producer.ProducerConfig;
+import org.apache.commons.lang.Validate;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import storm.kafka.trident.mapper.TridentTupleToKafkaMapper;
+import storm.kafka.trident.selector.KafkaTopicSelector;
+import storm.trident.operation.TridentCollector;
+import storm.trident.state.State;
+import storm.trident.tuple.TridentTuple;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+public class TridentKafkaState implements State {
+ private static final Logger LOG = LoggerFactory.getLogger(TridentKafkaState.class);
+
+
+ public static final String TOPIC = "topic";
+ public static final String KAFKA_BROKER_PROPERTIES = "kafka.broker.properties";
+
+ private Producer producer;
+ private OutputCollector collector;
+
+ private TridentTupleToKafkaMapper mapper;
+ private KafkaTopicSelector topicSelector;
+
+ public TridentKafkaState withTridentTupleToKafkaMapper(TridentTupleToKafkaMapper mapper) {
+ this.mapper = mapper;
+ return this;
+ }
+
+ public TridentKafkaState withKafkaTopicSelector(KafkaTopicSelector selector) {
+ this.topicSelector = selector;
+ return this;
+ }
+
+ @Override
+ public void beginCommit(Long txid) {
+ LOG.debug("beginCommit is Noop.");
+ }
+
+ @Override
+ public void commit(Long txid) {
+ LOG.debug("commit is Noop.");
+ }
+
+ public void prepare(Map stormConf) {
+ Validate.notNull(mapper, "mapper can not be null");
+ Validate.notNull(topicSelector, "topicSelector can not be null");
+ Map configMap = (Map) stormConf.get(KAFKA_BROKER_PROPERTIES);
+ Properties properties = new Properties();
+ properties.putAll(configMap);
+ ProducerConfig config = new ProducerConfig(properties);
+ producer = new Producer(config);
+ }
+
+ public void updateState(List<TridentTuple> tuples, TridentCollector collector) {
+ String topic = null;
+ for (TridentTuple tuple : tuples) {
+ try {
+ topic = topicSelector.getTopic(tuple);
+
+ if(topic != null) {
+ producer.send(new KeyedMessage(topic, mapper.getKeyFromTuple(tuple),
+ mapper.getMessageFromTuple(tuple)));
+ }
--- End diff --
done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-storm/pull/195
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by Parth-Brahmbhatt <gi...@git.apache.org>.
Github user Parth-Brahmbhatt commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/195#discussion_r15601742
--- Diff: external/storm-kafka/src/test/storm/kafka/TridentKafkaTopology.java ---
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package storm.kafka;
+
+import backtype.storm.Config;
+import backtype.storm.LocalCluster;
+import backtype.storm.generated.StormTopology;
+import backtype.storm.tuple.Fields;
+import backtype.storm.tuple.Values;
+import storm.kafka.trident.TridentKafkaState;
+import storm.kafka.trident.TridentKafkaStateFactory;
+import storm.kafka.trident.TridentKafkaUpdater;
+import storm.kafka.trident.mapper.FieldNameBasedTupleToKafkaMapper;
+import storm.kafka.trident.selector.DefaultTopicSelector;
+import storm.trident.Stream;
+import storm.trident.TridentTopology;
+import storm.trident.testing.FixedBatchSpout;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+
+
+public class TridentKafkaTopology {
+
+ private static StormTopology buildTopology() {
+ Fields fields = new Fields("word", "count");
+ FixedBatchSpout spout = new FixedBatchSpout(fields, 4,
+ new Values("storm", "1"),
+ new Values("trident", "1"),
+ new Values("needs", "1"),
+ new Values("javadoc", "1")
+ );
+ spout.setCycle(true);
+
+ TridentTopology topology = new TridentTopology();
+ Stream stream = topology.newStream("spout1", spout);
+
+ TridentKafkaStateFactory stateFactory = new TridentKafkaStateFactory()
+ .withKafkaTopicSelector(new DefaultTopicSelector("test"))
+ .withTridentTupleToKafkaMapper(new FieldNameBasedTupleToKafkaMapper("word", "count"));
+ stream.partitionPersist(stateFactory, fields, new TridentKafkaUpdater(), new Fields());
+
+ return topology.build();
+ }
+
+ /**
+ * To run this topology ensure you have a kafka broker running and provide connection string to broker as argument.
+ * Create a topic test with command line,
+ * kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partition 1 --topic test
+ *
+ * run this program and run the kafka consumer:
+ * afka-console-consumer.sh --zookeeper localhost:2181 --whitelist test --from-beginning
+ *
+ * you should see the messages flowing through.
+ *
+ * @param args
+ * @throws Exception
+ */
+ public static void main(String[] args) throws Exception {
+ if(args.length < 1) {
+ System.out.println("Please provide kafka broker url ,e.g. localhost:9092");
+ }
+
+ Config conf = getConfig(args[0]);
+ LocalCluster cluster = new LocalCluster();
+ cluster.submitTopology("wordCounter", conf, buildTopology());
+ Thread.sleep(60 * 1000);
+ cluster.killTopology("wordCounter");
+
+ cluster.shutdown();
+ }
+
+ private static Config getConfig(String brokerConnectionString) {
+ Config conf = new Config();
+ Map config = new HashMap();
+ Properties props = new Properties();
+ props.put("metadata.broker.list", brokerConnectionString);
+ props.put("request.required.acks", "1");
+ props.put("serializer.class", "kafka.serializer.StringEncoder");
+ conf.put(TridentKafkaState.KAFKA_BROKER_PROPERTIES, props);
+ conf.put(TridentKafkaState.TOPIC, "test");
--- End diff --
removed unused constant.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by wurstmeister <gi...@git.apache.org>.
Github user wurstmeister commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/195#discussion_r15556292
--- Diff: external/storm-kafka/src/test/storm/kafka/TridentKafkaTopology.java ---
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package storm.kafka;
+
+import backtype.storm.Config;
+import backtype.storm.LocalCluster;
+import backtype.storm.generated.StormTopology;
+import backtype.storm.tuple.Fields;
+import backtype.storm.tuple.Values;
+import storm.kafka.trident.TridentKafkaState;
+import storm.kafka.trident.TridentKafkaStateFactory;
+import storm.kafka.trident.TridentKafkaUpdater;
+import storm.kafka.trident.mapper.FieldNameBasedTupleToKafkaMapper;
+import storm.kafka.trident.selector.DefaultTopicSelector;
+import storm.trident.Stream;
+import storm.trident.TridentTopology;
+import storm.trident.testing.FixedBatchSpout;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+
+
+public class TridentKafkaTopology {
+
+ private static StormTopology buildTopology() {
+ Fields fields = new Fields("word", "count");
+ FixedBatchSpout spout = new FixedBatchSpout(fields, 4,
+ new Values("storm", "1"),
+ new Values("trident", "1"),
+ new Values("needs", "1"),
+ new Values("javadoc", "1")
+ );
+ spout.setCycle(true);
+
+ TridentTopology topology = new TridentTopology();
+ Stream stream = topology.newStream("spout1", spout);
+
+ TridentKafkaStateFactory stateFactory = new TridentKafkaStateFactory()
+ .withKafkaTopicSelector(new DefaultTopicSelector("test"))
+ .withTridentTupleToKafkaMapper(new FieldNameBasedTupleToKafkaMapper("word", "count"));
+ stream.partitionPersist(stateFactory, fields, new TridentKafkaUpdater(), new Fields());
+
+ return topology.build();
+ }
+
+ /**
+ * To run this topology ensure you have a kafka broker running and provide connection string to broker as argument.
+ * Create a topic test with command line,
+ * kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partition 1 --topic test
+ *
+ * run this program and run the kafka consumer:
+ * afka-console-consumer.sh --zookeeper localhost:2181 --whitelist test --from-beginning
--- End diff --
small typo, also I think --topic (rather than --whitelist) would be sufficient for this example
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by wurstmeister <gi...@git.apache.org>.
Github user wurstmeister commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/195#discussion_r15433274
--- Diff: external/storm-kafka/src/jvm/storm/kafka/trident/TridentKafkaState.java ---
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package storm.kafka.trident;
+
+import backtype.storm.task.OutputCollector;
+import backtype.storm.topology.FailedException;
+import kafka.javaapi.producer.Producer;
+import kafka.producer.KeyedMessage;
+import kafka.producer.ProducerConfig;
+import org.apache.commons.lang.Validate;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import storm.kafka.trident.mapper.TridentTupleToKafkaMapper;
+import storm.kafka.trident.selector.KafkaTopicSelector;
+import storm.trident.operation.TridentCollector;
+import storm.trident.state.State;
+import storm.trident.tuple.TridentTuple;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+public class TridentKafkaState implements State {
+ private static final Logger LOG = LoggerFactory.getLogger(TridentKafkaState.class);
+
+
+ public static final String TOPIC = "topic";
+ public static final String KAFKA_BROKER_PROPERTIES = "kafka.broker.properties";
+
+ private Producer producer;
+ private OutputCollector collector;
+
+ private TridentTupleToKafkaMapper mapper;
+ private KafkaTopicSelector topicSelector;
+
+ public TridentKafkaState withTridentTupleToKafkaMapper(TridentTupleToKafkaMapper mapper) {
+ this.mapper = mapper;
+ return this;
+ }
+
+ public TridentKafkaState withKafkaTopicSelector(KafkaTopicSelector selector) {
+ this.topicSelector = selector;
+ return this;
+ }
+
+ @Override
+ public void beginCommit(Long txid) {
+ LOG.debug("beginCommit is Noop.");
+ }
+
+ @Override
+ public void commit(Long txid) {
+ LOG.debug("commit is Noop.");
+ }
+
+ public void prepare(Map stormConf) {
+ Validate.notNull(mapper, "mapper can not be null");
+ Validate.notNull(topicSelector, "topicSelector can not be null");
+ Map configMap = (Map) stormConf.get(KAFKA_BROKER_PROPERTIES);
+ Properties properties = new Properties();
+ properties.putAll(configMap);
+ ProducerConfig config = new ProducerConfig(properties);
+ producer = new Producer(config);
+ }
+
+ public void updateState(List<TridentTuple> tuples, TridentCollector collector) {
+ for (TridentTuple tuple : tuples) {
+ try {
+ for(String topic : topicSelector.getTopics(tuple)) {
+ producer.send(new KeyedMessage(topic, mapper.getKeyFromTuple(tuple),
+ mapper.getMessageFromTuple(tuple)));
+ }
+ } catch (Exception ex) {
--- End diff --
do we really need to be able to handle multiple topics here? if we have 2 topics and we can successfully send the message to the first one but fail to send it to the second one we'd throw a FailedException resulting in inconsistencies between the topics.
Also the error message does not indicate which topic actually failed (and the single quote is not closed ;) )
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by Parth-Brahmbhatt <gi...@git.apache.org>.
Github user Parth-Brahmbhatt commented on the pull request:
https://github.com/apache/incubator-storm/pull/195#issuecomment-51365572
Updated the readme and went through it one more time, hopefully I did not miss anything. I think the producer properties are still useful as we still allow them in both the bolt and trident state so I have kept them in readme.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by Parth-Brahmbhatt <gi...@git.apache.org>.
Github user Parth-Brahmbhatt commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/195#discussion_r15545039
--- Diff: external/storm-kafka/README.md ---
@@ -44,6 +44,105 @@ use Kafka 0.8.1.1 built against Scala 2.10, you would use the following dependen
Note that the ZooKeeper and log4j dependencies are excluded to prevent version conflicts with Storm's dependencies.
+##Writing to Kafka as part of your topology
+You can create an instance of storm.kafka.bolt.KafkaBolt and attach it as a component to your topology or if you
+are using trident you can use storm.kafka.trident.TridentState, storm.kafka.trident.TridentStateFactory and
+storm.kafka.trident.TridentKafkaUpdater.
+
+You need to provide implementation of following 2 interfaces
+
+###TupleToKafkaMapper and TridentTupleToKafkaMapper
+These interfaces have 2 methods defined:
+
+```java
+ K getKeyFromTuple(Tuple/TridentTuple tuple);
+ V getMessageFromTuple(Tuple/TridentTuple tuple);
+```
+
+as the name suggests these methods are called to map a tuple to kafka key and kafka message. If you just want one field
+as key and one field as value then you can use the provided FieldNameBasedTupleToKafkaKeyAndMessageMapper.java
+implementation. In the KafkaBolt the implementation always looks for a field with field name "key" and "message" for
+backward compatibility reasons.
+In the TridentKafkaState you can specify what is the field name for key and message. These should be specified while
+constructing and instance of FieldNameBasedTupleToKafkaKeyAndMessageMapper.
+
+###KafkaTopicSelector and trident KafkaTopicSelector
+This interface has only one method
+```java
+public interface KafkaTopicSelector {
+ List<String> getTopics(Tuple/TridentTuple tuple);
+}
+```
+The implementation of this interface should return topics to which the tuple's key/message mapping needs to be published
+You can return an empty list but null is not acceptable. If you have one static topic name then you can use
+DefaultTopicSelector.java and set the name of the topic in the constructor.
+
--- End diff --
fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by ptgoetz <gi...@git.apache.org>.
Github user ptgoetz commented on a diff in the pull request:
https://github.com/apache/incubator-storm/pull/195#discussion_r15313963
--- Diff: external/storm-kafka/src/test/storm/kafka/TestUtils.java ---
@@ -34,4 +49,51 @@ public static GlobalPartitionInformation buildPartitionInfo(int numPartitions, i
return globalPartitionInformation;
}
+ public static SimpleConsumer getKafkaConsumer(KafkaTestBroker broker) {
+ BrokerHosts brokerHosts = getBrokerHosts(broker);
+ KafkaConfig kafkaConfig = new KafkaConfig(brokerHosts, TOPIC);
+ SimpleConsumer simpleConsumer = new SimpleConsumer("localhost", broker.getPort(), 60000, 1024, "testClient");
+ return simpleConsumer;
+ }
+
+ public static KafkaConfig getKafkaConfig(KafkaTestBroker broker) {
+ BrokerHosts brokerHosts = getBrokerHosts(broker);
+ KafkaConfig kafkaConfig = new KafkaConfig(brokerHosts, TOPIC);
+ return kafkaConfig;
+ }
+
+ private static BrokerHosts getBrokerHosts(KafkaTestBroker broker) {
+ GlobalPartitionInformation globalPartitionInformation = new GlobalPartitionInformation();
+ globalPartitionInformation.addPartition(0, Broker.fromString(broker.getBrokerConnectionString()));
+ return new StaticHosts(globalPartitionInformation);
+ }
+
+ public static Config getConfig(String brokerConnectionString) {
+ Config config = new Config();
+ Properties props = new Properties();
+ props.put("metadata.broker.list", brokerConnectionString);
+ props.put("request.required.acks", "1");
+ props.put("serializer.class", "kafka.serializer.StringEncoder");
--- End diff --
Never mind the last comment. I failed to notice that this is a test utility.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-storm pull request: STORM-405: Add kafka trident state s...
Posted by ptgoetz <gi...@git.apache.org>.
Github user ptgoetz commented on the pull request:
https://github.com/apache/incubator-storm/pull/195#issuecomment-49947897
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---