Review Request 52598: FLUME-2999 - Kafka channel and sink should enable statically assigned partition per event via header

[Tristan Stevens <tr...@cloudera.com>]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52598/
-----------------------------------------------------------

Review request for Flume.


Repository: flume-git


Description
-------

This feature is useful for anyone who needs greater control of which partitions are being written to - normally in a situation where multiple Flume agents are being deployed in order to horizontally scale, or alternatively if there is a scenario where there is a skew in data that might lead to one or more partitions hotspotting.
We also have the ability to specify custom partitions on to the Kafka Producer itself using the kafka.* configuration properties.

The Kafka Producer provides the ability to set the partition ID using the following constructor (https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html#ProducerRecord%28java.lang.String,%20java.lang.Integer,%20K,%20V%29 ), this is just a matter of providing the option to use this constructor.

This is specified in one of two ways: either via the staticPartition configuration property, which means that every message goes to the specified partition, or via the partitionHeader configuration property, which directs the implementation to retrieve the partitionId from one of the event headers.


Diffs
-----

  flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java 66b553a 
  flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannelConfiguration.java 3ab807b 
  flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java 57c0b28 
  flume-ng-doc/sphinx/FlumeUserGuide.rst ab71d38 
  flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java 89bdd84 
  flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkConstants.java 1bf380c 
  flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java 76eca37 
  flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/util/KafkaConsumer.java d5dfbd6 

Diff: https://reviews.apache.org/r/52598/diff/


Testing
-------

Unit testing done for both Kafka Channel and Kafka Sink.


Thanks,

Tristan Stevens