You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Ashwin Chandra Putta <as...@gmail.com> on 2016/01/28 01:24:31 UTC
read from multiple kafka topics
Hi,
Does the kafka input operator support reading from multiple kafka topics?
What is the suggested approach to deal with reading from multiple topics?
--
Regards,
Ashwin.
Re: read from multiple kafka topics
Posted by Thomas Weise <th...@gmail.com>.
Takes an array of topics (although there is a PR that will move the
splitting of multiple topics into the operator):
https://github.com/apache/incubator-apex-malhar/blob/devel-3/kafka/src/main/java/org/apache/apex/malhar/kafka/AbstractKafkaInputOperator.java#L369
On Wed, Jan 27, 2016 at 5:30 PM, Ashwin Chandra Putta <
ashwinchandrap@gmail.com> wrote:
> Thanks Thomas. How do I specify multiple topics on the operator?
>
> On Wed, Jan 27, 2016 at 5:15 PM, Thomas Weise <th...@gmail.com>
> wrote:
>
>> The new operator for Kafka 0.9 can read multiple topics.
>>
>>
>> On Wed, Jan 27, 2016 at 5:02 PM, Munagala Ramanath <ra...@datatorrent.com>
>> wrote:
>>
>>> Since a single AbstractKafkaInputOperator has a single KafkaConsumer and
>>> the latter has only a single topic,
>>> it appears that the only option is to have one input operator per topic.
>>>
>>> There also seems to be some machinery to get a reasonable mapping of
>>> input operator partitions to
>>> topic partitions, so this also points to needing an input operator per
>>> topic.
>>>
>>> Ram
>>>
>>> On Wed, Jan 27, 2016 at 4:24 PM, Ashwin Chandra Putta <
>>> ashwinchandrap@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Does the kafka input operator support reading from multiple kafka
>>>> topics? What is the suggested approach to deal with reading from multiple
>>>> topics?
>>>>
>>>> --
>>>>
>>>> Regards,
>>>> Ashwin.
>>>>
>>>
>>>
>>
>
>
> --
>
> Regards,
> Ashwin.
>
Re: read from multiple kafka topics
Posted by Ashwin Chandra Putta <as...@gmail.com>.
Thanks Thomas. How do I specify multiple topics on the operator?
On Wed, Jan 27, 2016 at 5:15 PM, Thomas Weise <th...@gmail.com>
wrote:
> The new operator for Kafka 0.9 can read multiple topics.
>
>
> On Wed, Jan 27, 2016 at 5:02 PM, Munagala Ramanath <ra...@datatorrent.com>
> wrote:
>
>> Since a single AbstractKafkaInputOperator has a single KafkaConsumer and
>> the latter has only a single topic,
>> it appears that the only option is to have one input operator per topic.
>>
>> There also seems to be some machinery to get a reasonable mapping of
>> input operator partitions to
>> topic partitions, so this also points to needing an input operator per
>> topic.
>>
>> Ram
>>
>> On Wed, Jan 27, 2016 at 4:24 PM, Ashwin Chandra Putta <
>> ashwinchandrap@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Does the kafka input operator support reading from multiple kafka
>>> topics? What is the suggested approach to deal with reading from multiple
>>> topics?
>>>
>>> --
>>>
>>> Regards,
>>> Ashwin.
>>>
>>
>>
>
--
Regards,
Ashwin.
Re: read from multiple kafka topics
Posted by Thomas Weise <th...@gmail.com>.
The new operator for Kafka 0.9 can read multiple topics.
On Wed, Jan 27, 2016 at 5:02 PM, Munagala Ramanath <ra...@datatorrent.com>
wrote:
> Since a single AbstractKafkaInputOperator has a single KafkaConsumer and
> the latter has only a single topic,
> it appears that the only option is to have one input operator per topic.
>
> There also seems to be some machinery to get a reasonable mapping of input
> operator partitions to
> topic partitions, so this also points to needing an input operator per
> topic.
>
> Ram
>
> On Wed, Jan 27, 2016 at 4:24 PM, Ashwin Chandra Putta <
> ashwinchandrap@gmail.com> wrote:
>
>> Hi,
>>
>> Does the kafka input operator support reading from multiple kafka topics?
>> What is the suggested approach to deal with reading from multiple topics?
>>
>> --
>>
>> Regards,
>> Ashwin.
>>
>
>
Re: read from multiple kafka topics
Posted by Munagala Ramanath <ra...@datatorrent.com>.
Since a single AbstractKafkaInputOperator has a single KafkaConsumer and
the latter has only a single topic,
it appears that the only option is to have one input operator per topic.
There also seems to be some machinery to get a reasonable mapping of input
operator partitions to
topic partitions, so this also points to needing an input operator per
topic.
Ram
On Wed, Jan 27, 2016 at 4:24 PM, Ashwin Chandra Putta <
ashwinchandrap@gmail.com> wrote:
> Hi,
>
> Does the kafka input operator support reading from multiple kafka topics?
> What is the suggested approach to deal with reading from multiple topics?
>
> --
>
> Regards,
> Ashwin.
>