[jira] [Commented] (STORM-817) Kafka Wildcard Topic Support

["Mike Treadway (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345162#comment-15345162 ] 

Mike Treadway commented on STORM-817:
-------------------------------------

Hi there, I'm seeing an issue with this implementation and I'm wondering how it ever worked with Trident.

The OpaquePartitionedTridentSpoutExecutor assumes that the partition state directories are direct children of the transaction directory. When wildcard support is enabled, the state directories are grandchildren of the transaction directory. For example, without wildcard support, it all works and the directory structure looks like this:

{stream}/user/partition_0

With wildcard support enabled, the partition id goes from 'partition_0' to '{topic}/partition_0', so the directory structure looks like this:

{stream}/user/{topic}/partition_0

The implementation inside OpaquePartitionedTridentSpoutExecutor.commit() assumes that the partition state is directly under user because it does '_state.list("")', which is incorrect.

Is anyone else seeing this?

> Kafka Wildcard Topic Support
> ----------------------------
>
>                 Key: STORM-817
>                 URL: https://issues.apache.org/jira/browse/STORM-817
>             Project: Apache Storm
>          Issue Type: New Feature
>          Components: storm-kafka
>            Reporter: Sumit Chawla
>            Assignee: Sumit Chawla
>
> Creating a feature request for supporting Wildcard Topic's for Kafka Spout.  
> We want to be able to run a aggregation stream for data coming from all tenants. Tenants get added dynamically. So new kafka topics get created. All the topics will be matching a regex pattern. 
> example:
> clickstream:tenant1:log
> clickstream:tenant2:log
> clickstream:tenant3:log
> Storm code should be able to perform auto-discovery, and should be able to to fetch from newly created topics in run time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)