You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ratha v <vi...@gmail.com> on 2016/10/09 23:49:15 UTC
How to merge two topics in kafka?
Hi all;
I have two topics in the broker. Without consuming from one topic, I want
to merge both topics, and will consume messages from the second topic.
It is because, I have two processes, one process, pushes messages to topic
A. And the second process once finished processing, it wants to merge both
topicA and TopicB. Then another process will consume messages from the
merged topic.
How can I merge both topics in high level kafka APIs. I use Kafka 0.10.0.1.
Thanks.
--
-Ratha
http://vvratha.blogspot.com/
Re: How to merge two topics in kafka?
Posted by Ratha v <vi...@gmail.com>.
I checked my targetTopic for available messages, but it says '0' . What
might cause issue here to merge two topics with custom type messages ?
On 11 October 2016 at 14:44, Ratha v <vi...@gmail.com> wrote:
> Thanks, this demo works perfectly for string messages.
>
> I have custom messageType defined( a java pojo class). And i have SerDe
> implemented for that.
> Now after merging sourceTopic-->Target Topic,
> I could not consume the messages..Means, Consumer does not return any
> messages.
>
> What might be the cause?
>
> On 10 October 2016 at 17:54, Sachin Mittal <sj...@gmail.com> wrote:
>
>> Check this example
>> https://github.com/apache/kafka/blob/trunk/streams/examples/
>> src/main/java/org/apache/kafka/streams/examples/pipe/PipeDemo.java
>>
>> On Mon, Oct 10, 2016 at 11:34 AM, Ratha v <vi...@gmail.com> wrote:
>>
>> > Hi Sachin;
>> > I went through the KStream/KTable Documentation. My scenario is very
>> > simple..I want to merge two topics( ie: Send messages available in the
>> > topic A -->topic B , in my case i'll be having only single message in
>> that
>> > topicA)
>> >
>> > Do I need Stateful processing (KStream)?
>> >
>> > Thanks.
>> >
>> > On 10 October 2016 at 13:10, Ratha v <vi...@gmail.com> wrote:
>> >
>> > > Thank you.I'll try the solution.
>> > > But in the highlevel consumer API, topics will be created
>> automatically?
>> > ,
>> > > We are not using zookeeper?
>> > >
>> > > On 10 October 2016 at 12:34, Sachin Mittal <sj...@gmail.com>
>> wrote:
>> > >
>> > >> You can use topicA.leftJoin (topicB).to (new-topic).
>> > >>
>> > >> You can the consume message from that new topic via second process.
>> Note
>> > >> you need to create all three topics in zookeeper first.
>> > >>
>> > >> On 10 Oct 2016 5:19 a.m., "Ratha v" <vi...@gmail.com> wrote:
>> > >>
>> > >> Hi all;
>> > >>
>> > >> I have two topics in the broker. Without consuming from one topic, I
>> > want
>> > >> to merge both topics, and will consume messages from the second
>> topic.
>> > >>
>> > >> It is because, I have two processes, one process, pushes messages to
>> > topic
>> > >> A. And the second process once finished processing, it wants to merge
>> > both
>> > >> topicA and TopicB. Then another process will consume messages from
>> the
>> > >> merged topic.
>> > >>
>> > >> How can I merge both topics in high level kafka APIs. I use Kafka
>> > >> 0.10.0.1.
>> > >>
>> > >>
>> > >> Thanks.
>> > >> --
>> > >> -Ratha
>> > >> http://vvratha.blogspot.com/
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > > -Ratha
>> > > http://vvratha.blogspot.com/
>> > >
>> >
>> >
>> >
>> > --
>> > -Ratha
>> > http://vvratha.blogspot.com/
>> >
>>
>
>
>
> --
> -Ratha
> http://vvratha.blogspot.com/
>
--
-Ratha
http://vvratha.blogspot.com/
Re: How to merge two topics in kafka?
Posted by Ratha v <vi...@gmail.com>.
Thanks, this demo works perfectly for string messages.
I have custom messageType defined( a java pojo class). And i have SerDe
implemented for that.
Now after merging sourceTopic-->Target Topic,
I could not consume the messages..Means, Consumer does not return any
messages.
What might be the cause?
On 10 October 2016 at 17:54, Sachin Mittal <sj...@gmail.com> wrote:
> Check this example
> https://github.com/apache/kafka/blob/trunk/streams/
> examples/src/main/java/org/apache/kafka/streams/examples/
> pipe/PipeDemo.java
>
> On Mon, Oct 10, 2016 at 11:34 AM, Ratha v <vi...@gmail.com> wrote:
>
> > Hi Sachin;
> > I went through the KStream/KTable Documentation. My scenario is very
> > simple..I want to merge two topics( ie: Send messages available in the
> > topic A -->topic B , in my case i'll be having only single message in
> that
> > topicA)
> >
> > Do I need Stateful processing (KStream)?
> >
> > Thanks.
> >
> > On 10 October 2016 at 13:10, Ratha v <vi...@gmail.com> wrote:
> >
> > > Thank you.I'll try the solution.
> > > But in the highlevel consumer API, topics will be created
> automatically?
> > ,
> > > We are not using zookeeper?
> > >
> > > On 10 October 2016 at 12:34, Sachin Mittal <sj...@gmail.com> wrote:
> > >
> > >> You can use topicA.leftJoin (topicB).to (new-topic).
> > >>
> > >> You can the consume message from that new topic via second process.
> Note
> > >> you need to create all three topics in zookeeper first.
> > >>
> > >> On 10 Oct 2016 5:19 a.m., "Ratha v" <vi...@gmail.com> wrote:
> > >>
> > >> Hi all;
> > >>
> > >> I have two topics in the broker. Without consuming from one topic, I
> > want
> > >> to merge both topics, and will consume messages from the second topic.
> > >>
> > >> It is because, I have two processes, one process, pushes messages to
> > topic
> > >> A. And the second process once finished processing, it wants to merge
> > both
> > >> topicA and TopicB. Then another process will consume messages from the
> > >> merged topic.
> > >>
> > >> How can I merge both topics in high level kafka APIs. I use Kafka
> > >> 0.10.0.1.
> > >>
> > >>
> > >> Thanks.
> > >> --
> > >> -Ratha
> > >> http://vvratha.blogspot.com/
> > >>
> > >
> > >
> > >
> > > --
> > > -Ratha
> > > http://vvratha.blogspot.com/
> > >
> >
> >
> >
> > --
> > -Ratha
> > http://vvratha.blogspot.com/
> >
>
--
-Ratha
http://vvratha.blogspot.com/
Re: How to merge two topics in kafka?
Posted by Sachin Mittal <sj...@gmail.com>.
Check this example
https://github.com/apache/kafka/blob/trunk/streams/examples/src/main/java/org/apache/kafka/streams/examples/pipe/PipeDemo.java
On Mon, Oct 10, 2016 at 11:34 AM, Ratha v <vi...@gmail.com> wrote:
> Hi Sachin;
> I went through the KStream/KTable Documentation. My scenario is very
> simple..I want to merge two topics( ie: Send messages available in the
> topic A -->topic B , in my case i'll be having only single message in that
> topicA)
>
> Do I need Stateful processing (KStream)?
>
> Thanks.
>
> On 10 October 2016 at 13:10, Ratha v <vi...@gmail.com> wrote:
>
> > Thank you.I'll try the solution.
> > But in the highlevel consumer API, topics will be created automatically?
> ,
> > We are not using zookeeper?
> >
> > On 10 October 2016 at 12:34, Sachin Mittal <sj...@gmail.com> wrote:
> >
> >> You can use topicA.leftJoin (topicB).to (new-topic).
> >>
> >> You can the consume message from that new topic via second process. Note
> >> you need to create all three topics in zookeeper first.
> >>
> >> On 10 Oct 2016 5:19 a.m., "Ratha v" <vi...@gmail.com> wrote:
> >>
> >> Hi all;
> >>
> >> I have two topics in the broker. Without consuming from one topic, I
> want
> >> to merge both topics, and will consume messages from the second topic.
> >>
> >> It is because, I have two processes, one process, pushes messages to
> topic
> >> A. And the second process once finished processing, it wants to merge
> both
> >> topicA and TopicB. Then another process will consume messages from the
> >> merged topic.
> >>
> >> How can I merge both topics in high level kafka APIs. I use Kafka
> >> 0.10.0.1.
> >>
> >>
> >> Thanks.
> >> --
> >> -Ratha
> >> http://vvratha.blogspot.com/
> >>
> >
> >
> >
> > --
> > -Ratha
> > http://vvratha.blogspot.com/
> >
>
>
>
> --
> -Ratha
> http://vvratha.blogspot.com/
>
Re: How to merge two topics in kafka?
Posted by Ratha v <vi...@gmail.com>.
Hi Sachin;
I went through the KStream/KTable Documentation. My scenario is very
simple..I want to merge two topics( ie: Send messages available in the
topic A -->topic B , in my case i'll be having only single message in that
topicA)
Do I need Stateful processing (KStream)?
Thanks.
On 10 October 2016 at 13:10, Ratha v <vi...@gmail.com> wrote:
> Thank you.I'll try the solution.
> But in the highlevel consumer API, topics will be created automatically? ,
> We are not using zookeeper?
>
> On 10 October 2016 at 12:34, Sachin Mittal <sj...@gmail.com> wrote:
>
>> You can use topicA.leftJoin (topicB).to (new-topic).
>>
>> You can the consume message from that new topic via second process. Note
>> you need to create all three topics in zookeeper first.
>>
>> On 10 Oct 2016 5:19 a.m., "Ratha v" <vi...@gmail.com> wrote:
>>
>> Hi all;
>>
>> I have two topics in the broker. Without consuming from one topic, I want
>> to merge both topics, and will consume messages from the second topic.
>>
>> It is because, I have two processes, one process, pushes messages to topic
>> A. And the second process once finished processing, it wants to merge both
>> topicA and TopicB. Then another process will consume messages from the
>> merged topic.
>>
>> How can I merge both topics in high level kafka APIs. I use Kafka
>> 0.10.0.1.
>>
>>
>> Thanks.
>> --
>> -Ratha
>> http://vvratha.blogspot.com/
>>
>
>
>
> --
> -Ratha
> http://vvratha.blogspot.com/
>
--
-Ratha
http://vvratha.blogspot.com/
Re: How to merge two topics in kafka?
Posted by Ratha v <vi...@gmail.com>.
Thank you.I'll try the solution.
But in the highlevel consumer API, topics will be created automatically? ,
We are not using zookeeper?
On 10 October 2016 at 12:34, Sachin Mittal <sj...@gmail.com> wrote:
> You can use topicA.leftJoin (topicB).to (new-topic).
>
> You can the consume message from that new topic via second process. Note
> you need to create all three topics in zookeeper first.
>
> On 10 Oct 2016 5:19 a.m., "Ratha v" <vi...@gmail.com> wrote:
>
> Hi all;
>
> I have two topics in the broker. Without consuming from one topic, I want
> to merge both topics, and will consume messages from the second topic.
>
> It is because, I have two processes, one process, pushes messages to topic
> A. And the second process once finished processing, it wants to merge both
> topicA and TopicB. Then another process will consume messages from the
> merged topic.
>
> How can I merge both topics in high level kafka APIs. I use Kafka 0.10.0.1.
>
>
> Thanks.
> --
> -Ratha
> http://vvratha.blogspot.com/
>
--
-Ratha
http://vvratha.blogspot.com/
Re: How to merge two topics in kafka?
Posted by Sachin Mittal <sj...@gmail.com>.
You can use topicA.leftJoin (topicB).to (new-topic).
You can the consume message from that new topic via second process. Note
you need to create all three topics in zookeeper first.
On 10 Oct 2016 5:19 a.m., "Ratha v" <vi...@gmail.com> wrote:
Hi all;
I have two topics in the broker. Without consuming from one topic, I want
to merge both topics, and will consume messages from the second topic.
It is because, I have two processes, one process, pushes messages to topic
A. And the second process once finished processing, it wants to merge both
topicA and TopicB. Then another process will consume messages from the
merged topic.
How can I merge both topics in high level kafka APIs. I use Kafka 0.10.0.1.
Thanks.
--
-Ratha
http://vvratha.blogspot.com/