You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by bo...@apache.org on 2016/01/22 23:08:06 UTC
[1/3] storm git commit: [STORM-1486] Fix storm-kafa documentation
Repository: storm
Updated Branches:
refs/heads/master f8168cdec -> 34a289858
[STORM-1486] Fix storm-kafa documentation
Fix storm-kafka documentation to reflect changes from STORM-1220 and also
document the version incompatiblity of pre 1.x kafka-spout with storm 1.0 and later.
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/32258ebe
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/32258ebe
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/32258ebe
Branch: refs/heads/master
Commit: 32258ebe1fb98908a985757702ddd41a5d3d0abc
Parents: d052d5a
Author: Arun Mahadevan <ai...@hortonworks.com>
Authored: Fri Jan 22 15:30:04 2016 +0530
Committer: Arun Mahadevan <ai...@hortonworks.com>
Committed: Fri Jan 22 15:35:18 2016 +0530
----------------------------------------------------------------------
external/storm-kafka/README.md | 17 ++++++++++++-----
1 file changed, 12 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/32258ebe/external/storm-kafka/README.md
----------------------------------------------------------------------
diff --git a/external/storm-kafka/README.md b/external/storm-kafka/README.md
index 3a86cf0..5a34b55 100644
--- a/external/storm-kafka/README.md
+++ b/external/storm-kafka/README.md
@@ -93,26 +93,33 @@ The KafkaConfig class also has bunch of public variables that controls your appl
Most of them are self explanatory except MultiScheme.
###MultiScheme
-MultiScheme is an interface that dictates how the byte[] consumed from Kafka gets transformed into a storm tuple. It
+MultiScheme is an interface that dictates how the ByteBuffer consumed from Kafka gets transformed into a storm tuple. It
also controls the naming of your output field.
```java
- public Iterable<List<Object>> deserialize(byte[] ser);
+ public Iterable<List<Object>> deserialize(ByteBuffer ser);
public Fields getOutputFields();
```
-The default `RawMultiScheme` just takes the `byte[]` and returns a tuple with `byte[]` as is. The name of the outputField is "bytes". There are alternative implementations like `SchemeAsMultiScheme` and `KeyValueSchemeAsMultiScheme` which can convert the `byte[]` to `String`.
+The default `RawMultiScheme` just takes the `ByteBuffer` and returns a tuple with the ByteBuffer converted to a `byte[]`. The name of the outputField is "bytes". There are alternative implementations like `SchemeAsMultiScheme` and `KeyValueSchemeAsMultiScheme` which can convert the `ByteBuffer` to `String`.
There is also an extension of `SchemeAsMultiScheme`, `MessageMetadataSchemeAsMultiScheme`,
-which has an additional deserialize method that accepts the message `byte[]` in addition to the `Partition` and `offset` associated with the message.
+which has an additional deserialize method that accepts the message `ByteBuffer` in addition to the `Partition` and `offset` associated with the message.
```java
-public Iterable<List<Object>> deserializeMessageWithMetadata(byte[] message, Partition partition, long offset)
+public Iterable<List<Object>> deserializeMessageWithMetadata(ByteBuffer message, Partition partition, long offset)
```
This is useful for auditing/replaying messages from arbitrary points on a Kafka topic, saving the partition and offset of each message of a discrete stream instead of persisting the entire message.
+#### Version incompatibility
+In Storm versions prior to 1.0, the MultiScheme methods accepted a `byte[]` instead of `ByteBuffer`. The `MultScheme` and the related
+Scheme apis were changed in version 1.0 to accept a ByteBuffer instead of a byte[].
+
+This means that pre 1.0 kafka spouts will not work with Storm versions 1.0 and higher. While running topologies in Storm version 1.0
+and higher, it must be ensured that the storm-kafka version is at least 1.0. Pre 1.0 shaded topology jars that bundles
+storm-kafka classes must be rebuilt with storm-kafka version 1.0 for running in clusters with storm 1.0 and higher.
### Examples
[2/3] storm git commit: Merge branch 'STORM-1486' of
https://github.com/arunmahadevan/storm into STORM-1486
Posted by bo...@apache.org.
Merge branch 'STORM-1486' of https://github.com/arunmahadevan/storm into STORM-1486
STORM-1486: Fix storm-kafa documentation
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/6d850a81
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/6d850a81
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/6d850a81
Branch: refs/heads/master
Commit: 6d850a81f062a4aba8e52e57217bc8045f385f35
Parents: f8168cd 32258eb
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Fri Jan 22 16:06:16 2016 -0600
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Fri Jan 22 16:06:16 2016 -0600
----------------------------------------------------------------------
external/storm-kafka/README.md | 17 ++++++++++++-----
1 file changed, 12 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
[3/3] storm git commit: Added STORM-1486 to Changelog
Posted by bo...@apache.org.
Added STORM-1486 to Changelog
Project: http://git-wip-us.apache.org/repos/asf/storm/repo
Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/34a28985
Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/34a28985
Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/34a28985
Branch: refs/heads/master
Commit: 34a2898580c803d50b5e681123d342f8c957fc1f
Parents: 6d850a8
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Authored: Fri Jan 22 16:06:47 2016 -0600
Committer: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Committed: Fri Jan 22 16:06:47 2016 -0600
----------------------------------------------------------------------
CHANGELOG.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/storm/blob/34a28985/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f4442ff..aa675ad 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -3,6 +3,7 @@
* STORM-1478: make bolts getComponentConfiguration method cleaner/simpler
## 1.0.0
+ * STORM-1486: Fix storm-kafa documentation
* STORM-1214: add javadoc for Trident Streams and Operations
* STORM-1450: Fix minor bugs and refactor code in ResourceAwareScheduler
* STORM-1452: Fixes profiling/debugging out of the box