You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by nabhanelrahman <gi...@git.apache.org> on 2016/06/08 18:21:58 UTC
[GitHub] storm pull request #1474: support for wildcard topics ( storm-kafka-client )
GitHub user nabhanelrahman opened a pull request:
https://github.com/apache/storm/pull/1474
support for wildcard topics ( storm-kafka-client )
these changes allow you to define a KafkaSpout that subscribes to a wildcard of topics. For example clickstreams.*.log. This is already supported in external/storm-kafka ( 0.8.x ); adding this feature to external/storm-kafka-client ( 0.9.x ).
+ Added a new public constructer to KafkaSpoutStream that takes in a java.util.regex.Pattern. So instead of creating a KakfaSpoutStreams. You can create a single KafkaSpoutStream that takes in a wildcard. This is because a kafkaConsumer can only subscribe to a single regex pattern.
+ Added a test called KafkaSpoutWildcardTopologyMain.java that demonstrates how to setup a KakfaSpout that subscribes to a wildcard of topics.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/publica-project/storm storm-kafka-client_wildcard_spout
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1474.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1474
----
commit 0c539a9feeab80d177183b6b8815010b9446018f
Author: nabhanelrahman <na...@publica-project.com>
Date: 2016-06-08T18:19:01Z
support for wildcard topics ( storm-kafka-client )
these changes allow you to define a KafkaSpout that subscribes to a wildcard of topics. For example clickstreams.*.log. This is already supported in external/storm-kafka ( 0.8.x ); adding this feature to external/storm-kafka-client ( 0.9.x ).
+ Added a new public constructer to KafkaSpoutStream that takes in a java.util.regex.Pattern. So instead of creating a KakfaSpoutStreams. You can create a single KafkaSpoutStream that takes in a wildcard. This is because a kafkaConsumer can only subscribe to a single regex pattern.
+ Added a test called KafkaSpoutWildcardTopologyMain.java that demonstrates how to setup a KakfaSpout that subscribes to a wildcard of topics.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1474: support for wildcard topics ( storm-kafka-client )
Posted by hmcl <gi...@git.apache.org>.
Github user hmcl commented on the issue:
https://github.com/apache/storm/pull/1474
@nabhanelrahman your patch enlightened that it would be better to add the `emit` and `getOutputFields` methods to `KafkaSpoutStream`. Furthermore, there were a couple of design decisions that would be ideal to do at this point in order to reuse code, and keep it clean and modular. Since it would be too verbose to go over those discussions in here, I just created a pull request with the changes that I believe are adequate to address the wildcard topic pattern requirement. This is the pull request: https://github.com/apache/storm/pull/1516
Although I had this in my todo list, since you already worked on it, it's totally fine for you to get the credit for it. Hence feel free to assign the JIRA to you, and recommit the PR with your username. Please mention that we worked on this together, and it's all good. Thanks for your understanding.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1474: support for wildcard topics ( storm-kafka-client )
Posted by nabhanelrahman <gi...@git.apache.org>.
Github user nabhanelrahman closed the pull request at:
https://github.com/apache/storm/pull/1474
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1474: support for wildcard topics ( storm-kafka-client )
Posted by nabhanelrahman <gi...@git.apache.org>.
Github user nabhanelrahman commented on a diff in the pull request:
https://github.com/apache/storm/pull/1474#discussion_r66368331
--- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java ---
@@ -72,8 +72,10 @@
private transient boolean initialized; // Flag indicating that the spout is still undergoing initialization process.
// Initialization is only complete after the first call to KafkaSpoutConsumerRebalanceListener.onPartitionsAssigned()
+ private KafkaSpoutStream kafkaSpoutStream;
private KafkaSpoutStreams kafkaSpoutStreams; // Object that wraps all the logic to declare output fields and emit tuples
private transient KafkaSpoutTuplesBuilder<K, V> tuplesBuilder; // Object that contains the logic to build tuples for each ConsumerRecord
+ private transient KafkaSpoutTupleBuilder<K, V> tupleBuilder;
--- End diff --
the subscribe method() https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#subscribe(java.util.regex.Pattern,%20org.apache.kafka.clients.consumer.ConsumerRebalanceListener)
only supports one regex pattern. TuplesBuilder is used to define multiple topics, which is not applicable for this case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1474: support for wildcard topics ( storm-kafka-client )
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on a diff in the pull request:
https://github.com/apache/storm/pull/1474#discussion_r66339764
--- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java ---
@@ -72,8 +72,10 @@
private transient boolean initialized; // Flag indicating that the spout is still undergoing initialization process.
// Initialization is only complete after the first call to KafkaSpoutConsumerRebalanceListener.onPartitionsAssigned()
+ private KafkaSpoutStream kafkaSpoutStream;
--- End diff --
any reason to not handle this using kafkaSpoutStreams
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1474: support for wildcard topics ( storm-kafka-client )
Posted by nabhanelrahman <gi...@git.apache.org>.
Github user nabhanelrahman commented on the issue:
https://github.com/apache/storm/pull/1474
@hmcl, I integrated pull #1516 and tested it, and it looks good. The class naming looks a lot more intuitive. I resubmitted pull #1516 as #1518. I only have read-only permission in JIRA ( https://issues.apache.org/jira/browse/STORM-1930 ). So I can't do any changes their. My JIRA handle is "nabhan". I am closing out the pull request since #1518 is now the one we want.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1474: support for wildcard topics ( storm-kafka-client )
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the issue:
https://github.com/apache/storm/pull/1474
@nabhanelrahman I still think naming of those variables very confusing and doesn't tell the intended purpose.
@hmcl can you take a look at the pr.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1474: support for wildcard topics ( storm-kafka-client )
Posted by nabhanelrahman <gi...@git.apache.org>.
Github user nabhanelrahman commented on a diff in the pull request:
https://github.com/apache/storm/pull/1474#discussion_r66368337
--- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java ---
@@ -72,8 +72,10 @@
private transient boolean initialized; // Flag indicating that the spout is still undergoing initialization process.
// Initialization is only complete after the first call to KafkaSpoutConsumerRebalanceListener.onPartitionsAssigned()
+ private KafkaSpoutStream kafkaSpoutStream;
--- End diff --
KafkaSpoutStreams is a collection of KafkaSpoutStream because you can have multiple topics for the kafkaconsumer.subscribe(List<String topics... )
That is not the case of kafkaconsumer.subscribe(Pattern ). You can only have one pattern per Spout, so it doesn't make sense to put in Streams instead of Stream
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1474: support for wildcard topics ( storm-kafka-client )
Posted by nabhanelrahman <gi...@git.apache.org>.
Github user nabhanelrahman commented on the issue:
https://github.com/apache/storm/pull/1474
@harshach I can rename the variables:
private KafkaSpoutStream kafkaWildcardSpoutStream or kafkaPatternSpoutStream
and
private transient KafkaSpoutTupleBuilder<K, V> tupleWildcardBuilder or tuplePatternBuilder
what do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1474: support for wildcard topics ( storm-kafka-client )
Posted by hmcl <gi...@git.apache.org>.
Github user hmcl commented on the issue:
https://github.com/apache/storm/pull/1474
@harshach @nabhanelrahman looking at this now. Somehow I missed this PR earlier when going throw the PRs for reviewing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1474: support for wildcard topics ( storm-kafka-client )
Posted by nabhanelrahman <gi...@git.apache.org>.
Github user nabhanelrahman commented on the issue:
https://github.com/apache/storm/pull/1474
@harshach, just reaching out on the status of this pull request? Does it look good?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1474: support for wildcard topics ( storm-kafka-client )
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on a diff in the pull request:
https://github.com/apache/storm/pull/1474#discussion_r66339711
--- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java ---
@@ -72,8 +72,10 @@
private transient boolean initialized; // Flag indicating that the spout is still undergoing initialization process.
// Initialization is only complete after the first call to KafkaSpoutConsumerRebalanceListener.onPartitionsAssigned()
+ private KafkaSpoutStream kafkaSpoutStream;
private KafkaSpoutStreams kafkaSpoutStreams; // Object that wraps all the logic to declare output fields and emit tuples
private transient KafkaSpoutTuplesBuilder<K, V> tuplesBuilder; // Object that contains the logic to build tuples for each ConsumerRecord
+ private transient KafkaSpoutTupleBuilder<K, V> tupleBuilder;
--- End diff --
can't we just tuplesBuilder for this . Any reason to have another tupleBuilder.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---