[GitHub] storm pull request #1808: STORM-2225: change spout config to be simpler.

[srdo <gi...@git.apache.org>]

Github user srdo commented on a diff in the pull request:

    https://github.com/apache/storm/pull/1808#discussion_r91348040
  
    --- Diff: docs/storm-kafka-client.md ---
    @@ -1,90 +1,222 @@
    -#Storm Kafka Spout with New Kafka Consumer API
    +#Storm Kafka integration using the kafka-client jar
     
    -Apache Storm Spout implementation to consume data from Apache Kafka versions 0.10 onwards (please see [Apache Kafka Version Compatibility] (#compatibility)). 
    +##Compatability
     
    -The Apache Storm Spout allows clients to consume data from Kafka starting at offsets as defined by the offset strategy specified in `FirstPollOffsetStrategy`. 
    -In case of failure, the Kafka Spout will re-start consuming messages from the offset that matches the chosen `FirstPollOffsetStrategy`.
    +Apache Kafka versions 0.10 onwards (please see [Apache Kafka Version Compatibility] (#compatibility)). 
     
    -The Kafka Spout implementation allows you to specify the stream (`KafkaSpoutStream`) associated with each topic or topic wildcard. `KafkaSpoutStream` represents the stream and output fields. For named topics use `KafkaSpoutStreamsNamedTopics`, and for topic wildcards use `KafkaSpoutStreamsWildcardTopics`. 
    +##Writing to Kafka as part of your topology
    +You can create an instance of org.apache.storm.kafka.bolt.KafkaBolt and attach it as a component to your topology or if you
    +are using trident you can use org.apache.storm.kafka.trident.TridentState, org.apache.storm.kafka.trident.TridentStateFactory and
    +org.apache.storm.kafka.trident.TridentKafkaUpdater.
     
    -The `KafkaSpoutTuplesBuilder` wraps all the logic that builds `Tuple`s from `ConsumerRecord`s. The logic is provided by the user through implementing the appropriate number of `KafkaSpoutTupleBuilder` instances. For named topics use `KafkaSpoutTuplesBuilderNamedTopics`, and for topic wildcard use `KafkaSpoutTuplesBuilderWildcardTopics`.
    +You need to provide implementations for the following 2 interfaces
     
    -Multiple topics and topic wildcards can use the same `KafkaSpoutTupleBuilder` implementation, as long as the logic to build `Tuple`s from `ConsumerRecord`s is identical.
    +###TupleToKafkaMapper and TridentTupleToKafkaMapper
    +These interfaces have 2 methods defined:
     
    +```java
    +    K getKeyFromTuple(Tuple/TridentTuple tuple);
    +    V getMessageFromTuple(Tuple/TridentTuple tuple);
    +```
    +
    +As the name suggests, these methods are called to map a tuple to a Kafka key and a Kafka message. If you just want one field
    +as key and one field as value, then you can use the provided FieldNameBasedTupleToKafkaMapper.java
    +implementation. In the KafkaBolt, the implementation always looks for a field with field name "key" and "message" if you
    +use the default constructor to construct FieldNameBasedTupleToKafkaMapper for backward compatibility
    +reasons. Alternatively you could also specify a different key and message field by using the non default constructor.
    +In the TridentKafkaState you must specify what is the field name for key and message as there is no default constructor.
    +These should be specified while constructing and instance of FieldNameBasedTupleToKafkaMapper.
    +
    +###KafkaTopicSelector and trident KafkaTopicSelector
    +This interface has only one method
    +```java
    +public interface KafkaTopicSelector {
    +    String getTopics(Tuple/TridentTuple tuple);
    +}
    +```
    +The implementation of this interface should return the topic to which the tuple's key/message mapping needs to be published
    +You can return a null and the message will be ignored. If you have one static topic name then you can use
    +DefaultTopicSelector.java and set the name of the topic in the constructor.
    +`FieldNameTopicSelector` and `FieldIndexTopicSelector` use to support decided which topic should to push message from tuple.
    +User could specify the field name or field index in tuple ,selector will use this value as topic name which to publish message.
    +When the topic name not found , `KafkaBolt` will write messages into default topic .
    --- End diff --
    
    "not found" -> "is not found"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---