You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Amber Kulkarni <am...@gmail.com> on 2016/07/26 12:40:49 UTC
Trident (Opaque Transactional Spout): multiple topologies sourcing
from same kafka topic.
So I want multiple storm topologies to read from the same kafka topic.
So suppose topic contains: A B C (data)
I want both the topologies to get A,B,C
But currently what is happening is messaged are getting distributed.(eg.
topology 1 is getting A and other topology is getting B and C)
How do I achieve this is trident.
I am using OpaqueTridentKafkaSpout.
I think what needs to be done is both topologies need to specify different
consumer group ids.
But could not find how to specify them in trident(TridentKafkaConfig)
Any ideas ?
Regards,
Amber Kulkarni
Re: Trident (Opaque Transactional Spout): multiple topologies
sourcing from same kafka topic.
Posted by Amber Kulkarni <am...@gmail.com>.
Ohh yes, this worked. Checked zkCli it created dir by name of the
TridentTopology#newStream
Thanks a ton!
On Wed, Jul 27, 2016 at 2:51 PM, Satish Duggana <sa...@gmail.com>
wrote:
> You can try setting different txnId values for spout which is used in
> maintaining opaque transactional spout's state in ZK. I guess you may have
> same txnId for both of those topologies.
>
> TridentTopology#newStream(txnId, spout);
>
>
>
> On Wed, Jul 27, 2016 at 2:20 PM, Satish Duggana <sa...@gmail.com>
> wrote:
>
>> Hi Amber,
>> I thought you wanted to set clientId to be used. I missed other part of
>> the question to set consumer group to consume messages within that topology.
>>
>>
>>
>> On Wed, Jul 27, 2016 at 1:03 PM, Amber Kulkarni <
>> amber.kulkarni89@gmail.com> wrote:
>>
>>> Tried deploying 2 topologies using above change, did not work. Still
>>> both are not getting same data from kafka.
>>> I think 3rd param is client id and not consumer group id. any ideas ?
>>>
>>> On Wed, Jul 27, 2016 at 12:21 PM, Satish Duggana <
>>> satish.duggana@gmail.com> wrote:
>>>
>>>> You can use below constructor and use respective clientId.
>>>>
>>>> TridentKafkaConfig(BrokerHosts hosts, String topic, String clientId)
>>>>
>>>> Can not you see that constructor in the version that you are using? I
>>>> guess you should have that.
>>>>
>>>> Thanks,
>>>> Satish.
>>>>
>>>>
>>>> On Wed, Jul 27, 2016 at 11:42 AM, Amber Kulkarni <
>>>> amber.kulkarni89@gmail.com> wrote:
>>>>
>>>>> Hey,
>>>>>
>>>>> Cannot find a way to specify consumer ids to spout in trident.
>>>>> Can you suggest a way to do this.
>>>>> What I have tried: TridentKafkaConfig does not seem to have a
>>>>> constructor which takes group id,
>>>>> whereas Spoutconfig has the provision for id.
>>>>> I am using Storm 0.9.6
>>>>>
>>>>>
>>>>> On Tue, Jul 26, 2016 at 6:12 PM, Stephen Powis <sp...@salesforce.com>
>>>>> wrote:
>>>>>
>>>>>> Do you have unique consumer Ids for each spout in each topology?
>>>>>>
>>>>>> On Tue, Jul 26, 2016 at 8:40 AM, Amber Kulkarni <
>>>>>> amber.kulkarni89@gmail.com> wrote:
>>>>>>
>>>>>>> So I want multiple storm topologies to read from the same kafka
>>>>>>> topic.
>>>>>>>
>>>>>>> So suppose topic contains: A B C (data)
>>>>>>>
>>>>>>> I want both the topologies to get A,B,C
>>>>>>>
>>>>>>> But currently what is happening is messaged are getting
>>>>>>> distributed.(eg. topology 1 is getting A and other topology is getting B
>>>>>>> and C)
>>>>>>>
>>>>>>> How do I achieve this is trident.
>>>>>>>
>>>>>>> I am using OpaqueTridentKafkaSpout.
>>>>>>>
>>>>>>> I think what needs to be done is both topologies need to specify
>>>>>>> different consumer group ids.
>>>>>>>
>>>>>>> But could not find how to specify them in trident(TridentKafkaConfig)
>>>>>>>
>>>>>>> Any ideas ?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Amber Kulkarni
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Amber Kulkarni
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Amber Kulkarni
>>>
>>
>>
>
--
Regards,
Amber Kulkarni
Re: Trident (Opaque Transactional Spout): multiple topologies
sourcing from same kafka topic.
Posted by Satish Duggana <sa...@gmail.com>.
You can try setting different txnId values for spout which is used in
maintaining opaque transactional spout's state in ZK. I guess you may have
same txnId for both of those topologies.
TridentTopology#newStream(txnId, spout);
On Wed, Jul 27, 2016 at 2:20 PM, Satish Duggana <sa...@gmail.com>
wrote:
> Hi Amber,
> I thought you wanted to set clientId to be used. I missed other part of
> the question to set consumer group to consume messages within that topology.
>
>
>
> On Wed, Jul 27, 2016 at 1:03 PM, Amber Kulkarni <
> amber.kulkarni89@gmail.com> wrote:
>
>> Tried deploying 2 topologies using above change, did not work. Still both
>> are not getting same data from kafka.
>> I think 3rd param is client id and not consumer group id. any ideas ?
>>
>> On Wed, Jul 27, 2016 at 12:21 PM, Satish Duggana <
>> satish.duggana@gmail.com> wrote:
>>
>>> You can use below constructor and use respective clientId.
>>>
>>> TridentKafkaConfig(BrokerHosts hosts, String topic, String clientId)
>>>
>>> Can not you see that constructor in the version that you are using? I
>>> guess you should have that.
>>>
>>> Thanks,
>>> Satish.
>>>
>>>
>>> On Wed, Jul 27, 2016 at 11:42 AM, Amber Kulkarni <
>>> amber.kulkarni89@gmail.com> wrote:
>>>
>>>> Hey,
>>>>
>>>> Cannot find a way to specify consumer ids to spout in trident.
>>>> Can you suggest a way to do this.
>>>> What I have tried: TridentKafkaConfig does not seem to have a
>>>> constructor which takes group id,
>>>> whereas Spoutconfig has the provision for id.
>>>> I am using Storm 0.9.6
>>>>
>>>>
>>>> On Tue, Jul 26, 2016 at 6:12 PM, Stephen Powis <sp...@salesforce.com>
>>>> wrote:
>>>>
>>>>> Do you have unique consumer Ids for each spout in each topology?
>>>>>
>>>>> On Tue, Jul 26, 2016 at 8:40 AM, Amber Kulkarni <
>>>>> amber.kulkarni89@gmail.com> wrote:
>>>>>
>>>>>> So I want multiple storm topologies to read from the same kafka topic.
>>>>>>
>>>>>> So suppose topic contains: A B C (data)
>>>>>>
>>>>>> I want both the topologies to get A,B,C
>>>>>>
>>>>>> But currently what is happening is messaged are getting
>>>>>> distributed.(eg. topology 1 is getting A and other topology is getting B
>>>>>> and C)
>>>>>>
>>>>>> How do I achieve this is trident.
>>>>>>
>>>>>> I am using OpaqueTridentKafkaSpout.
>>>>>>
>>>>>> I think what needs to be done is both topologies need to specify
>>>>>> different consumer group ids.
>>>>>>
>>>>>> But could not find how to specify them in trident(TridentKafkaConfig)
>>>>>>
>>>>>> Any ideas ?
>>>>>>
>>>>>> Regards,
>>>>>> Amber Kulkarni
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Amber Kulkarni
>>>>
>>>
>>>
>>
>>
>> --
>> Regards,
>> Amber Kulkarni
>>
>
>
Re: Trident (Opaque Transactional Spout): multiple topologies
sourcing from same kafka topic.
Posted by Satish Duggana <sa...@gmail.com>.
Hi Amber,
I thought you wanted to set clientId to be used. I missed other part of
the question to set consumer group to consume messages within that topology.
On Wed, Jul 27, 2016 at 1:03 PM, Amber Kulkarni <am...@gmail.com>
wrote:
> Tried deploying 2 topologies using above change, did not work. Still both
> are not getting same data from kafka.
> I think 3rd param is client id and not consumer group id. any ideas ?
>
> On Wed, Jul 27, 2016 at 12:21 PM, Satish Duggana <satish.duggana@gmail.com
> > wrote:
>
>> You can use below constructor and use respective clientId.
>>
>> TridentKafkaConfig(BrokerHosts hosts, String topic, String clientId)
>>
>> Can not you see that constructor in the version that you are using? I
>> guess you should have that.
>>
>> Thanks,
>> Satish.
>>
>>
>> On Wed, Jul 27, 2016 at 11:42 AM, Amber Kulkarni <
>> amber.kulkarni89@gmail.com> wrote:
>>
>>> Hey,
>>>
>>> Cannot find a way to specify consumer ids to spout in trident.
>>> Can you suggest a way to do this.
>>> What I have tried: TridentKafkaConfig does not seem to have a
>>> constructor which takes group id,
>>> whereas Spoutconfig has the provision for id.
>>> I am using Storm 0.9.6
>>>
>>>
>>> On Tue, Jul 26, 2016 at 6:12 PM, Stephen Powis <sp...@salesforce.com>
>>> wrote:
>>>
>>>> Do you have unique consumer Ids for each spout in each topology?
>>>>
>>>> On Tue, Jul 26, 2016 at 8:40 AM, Amber Kulkarni <
>>>> amber.kulkarni89@gmail.com> wrote:
>>>>
>>>>> So I want multiple storm topologies to read from the same kafka topic.
>>>>>
>>>>> So suppose topic contains: A B C (data)
>>>>>
>>>>> I want both the topologies to get A,B,C
>>>>>
>>>>> But currently what is happening is messaged are getting
>>>>> distributed.(eg. topology 1 is getting A and other topology is getting B
>>>>> and C)
>>>>>
>>>>> How do I achieve this is trident.
>>>>>
>>>>> I am using OpaqueTridentKafkaSpout.
>>>>>
>>>>> I think what needs to be done is both topologies need to specify
>>>>> different consumer group ids.
>>>>>
>>>>> But could not find how to specify them in trident(TridentKafkaConfig)
>>>>>
>>>>> Any ideas ?
>>>>>
>>>>> Regards,
>>>>> Amber Kulkarni
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Amber Kulkarni
>>>
>>
>>
>
>
> --
> Regards,
> Amber Kulkarni
>
Re: Trident (Opaque Transactional Spout): multiple topologies
sourcing from same kafka topic.
Posted by Amber Kulkarni <am...@gmail.com>.
Tried deploying 2 topologies using above change, did not work. Still both
are not getting same data from kafka.
I think 3rd param is client id and not consumer group id. any ideas ?
On Wed, Jul 27, 2016 at 12:21 PM, Satish Duggana <sa...@gmail.com>
wrote:
> You can use below constructor and use respective clientId.
>
> TridentKafkaConfig(BrokerHosts hosts, String topic, String clientId)
>
> Can not you see that constructor in the version that you are using? I
> guess you should have that.
>
> Thanks,
> Satish.
>
>
> On Wed, Jul 27, 2016 at 11:42 AM, Amber Kulkarni <
> amber.kulkarni89@gmail.com> wrote:
>
>> Hey,
>>
>> Cannot find a way to specify consumer ids to spout in trident.
>> Can you suggest a way to do this.
>> What I have tried: TridentKafkaConfig does not seem to have a
>> constructor which takes group id,
>> whereas Spoutconfig has the provision for id.
>> I am using Storm 0.9.6
>>
>>
>> On Tue, Jul 26, 2016 at 6:12 PM, Stephen Powis <sp...@salesforce.com>
>> wrote:
>>
>>> Do you have unique consumer Ids for each spout in each topology?
>>>
>>> On Tue, Jul 26, 2016 at 8:40 AM, Amber Kulkarni <
>>> amber.kulkarni89@gmail.com> wrote:
>>>
>>>> So I want multiple storm topologies to read from the same kafka topic.
>>>>
>>>> So suppose topic contains: A B C (data)
>>>>
>>>> I want both the topologies to get A,B,C
>>>>
>>>> But currently what is happening is messaged are getting
>>>> distributed.(eg. topology 1 is getting A and other topology is getting B
>>>> and C)
>>>>
>>>> How do I achieve this is trident.
>>>>
>>>> I am using OpaqueTridentKafkaSpout.
>>>>
>>>> I think what needs to be done is both topologies need to specify
>>>> different consumer group ids.
>>>>
>>>> But could not find how to specify them in trident(TridentKafkaConfig)
>>>>
>>>> Any ideas ?
>>>>
>>>> Regards,
>>>> Amber Kulkarni
>>>>
>>>
>>>
>>
>>
>> --
>> Regards,
>> Amber Kulkarni
>>
>
>
--
Regards,
Amber Kulkarni
Re: Trident (Opaque Transactional Spout): multiple topologies
sourcing from same kafka topic.
Posted by Satish Duggana <sa...@gmail.com>.
You can use below constructor and use respective clientId.
TridentKafkaConfig(BrokerHosts hosts, String topic, String clientId)
Can not you see that constructor in the version that you are using? I guess
you should have that.
Thanks,
Satish.
On Wed, Jul 27, 2016 at 11:42 AM, Amber Kulkarni <amber.kulkarni89@gmail.com
> wrote:
> Hey,
>
> Cannot find a way to specify consumer ids to spout in trident.
> Can you suggest a way to do this.
> What I have tried: TridentKafkaConfig does not seem to have a constructor
> which takes group id,
> whereas Spoutconfig has the provision for id.
> I am using Storm 0.9.6
>
>
> On Tue, Jul 26, 2016 at 6:12 PM, Stephen Powis <sp...@salesforce.com>
> wrote:
>
>> Do you have unique consumer Ids for each spout in each topology?
>>
>> On Tue, Jul 26, 2016 at 8:40 AM, Amber Kulkarni <
>> amber.kulkarni89@gmail.com> wrote:
>>
>>> So I want multiple storm topologies to read from the same kafka topic.
>>>
>>> So suppose topic contains: A B C (data)
>>>
>>> I want both the topologies to get A,B,C
>>>
>>> But currently what is happening is messaged are getting distributed.(eg.
>>> topology 1 is getting A and other topology is getting B and C)
>>>
>>> How do I achieve this is trident.
>>>
>>> I am using OpaqueTridentKafkaSpout.
>>>
>>> I think what needs to be done is both topologies need to specify
>>> different consumer group ids.
>>>
>>> But could not find how to specify them in trident(TridentKafkaConfig)
>>>
>>> Any ideas ?
>>>
>>> Regards,
>>> Amber Kulkarni
>>>
>>
>>
>
>
> --
> Regards,
> Amber Kulkarni
>
Re: Trident (Opaque Transactional Spout): multiple topologies
sourcing from same kafka topic.
Posted by Amber Kulkarni <am...@gmail.com>.
Hey,
Cannot find a way to specify consumer ids to spout in trident.
Can you suggest a way to do this.
What I have tried: TridentKafkaConfig does not seem to have a constructor
which takes group id,
whereas Spoutconfig has the provision for id.
I am using Storm 0.9.6
On Tue, Jul 26, 2016 at 6:12 PM, Stephen Powis <sp...@salesforce.com>
wrote:
> Do you have unique consumer Ids for each spout in each topology?
>
> On Tue, Jul 26, 2016 at 8:40 AM, Amber Kulkarni <
> amber.kulkarni89@gmail.com> wrote:
>
>> So I want multiple storm topologies to read from the same kafka topic.
>>
>> So suppose topic contains: A B C (data)
>>
>> I want both the topologies to get A,B,C
>>
>> But currently what is happening is messaged are getting distributed.(eg.
>> topology 1 is getting A and other topology is getting B and C)
>>
>> How do I achieve this is trident.
>>
>> I am using OpaqueTridentKafkaSpout.
>>
>> I think what needs to be done is both topologies need to specify
>> different consumer group ids.
>>
>> But could not find how to specify them in trident(TridentKafkaConfig)
>>
>> Any ideas ?
>>
>> Regards,
>> Amber Kulkarni
>>
>
>
--
Regards,
Amber Kulkarni
Re: Trident (Opaque Transactional Spout): multiple topologies
sourcing from same kafka topic.
Posted by Stephen Powis <sp...@salesforce.com>.
Do you have unique consumer Ids for each spout in each topology?
On Tue, Jul 26, 2016 at 8:40 AM, Amber Kulkarni <am...@gmail.com>
wrote:
> So I want multiple storm topologies to read from the same kafka topic.
>
> So suppose topic contains: A B C (data)
>
> I want both the topologies to get A,B,C
>
> But currently what is happening is messaged are getting distributed.(eg.
> topology 1 is getting A and other topology is getting B and C)
>
> How do I achieve this is trident.
>
> I am using OpaqueTridentKafkaSpout.
>
> I think what needs to be done is both topologies need to specify different
> consumer group ids.
>
> But could not find how to specify them in trident(TridentKafkaConfig)
>
> Any ideas ?
>
> Regards,
> Amber Kulkarni
>