You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Tianji Li <sk...@gmail.com> on 2017/04/01 14:26:57 UTC
Kafka Streams: Is it possible to pause/resume consuming a topic?
Hi there,
Say a processor that is consuming topic A and producing into topic B, and
somehow the processing takes long time, is it possible to pause the
consuming from topic A, and later on resume?
Or does it make sense to do so? If not, what are the options to resolve
this issue?
Thanks
Tianji
Re: Kafka Streams: Is it possible to pause/resume consuming a topic?
Posted by Tianji Li <sk...@gmail.com>.
Perfect, thanks so much Eno!
On Sat, Apr 1, 2017 at 12:44 PM, Eno Thereska <en...@gmail.com>
wrote:
> Sure, here is an example of pausing:
> https://github.com/apache/kafka/blob/trunk/streams/src/
> main/java/org/apache/kafka/streams/processor/internals/
> StreamTask.java#L165 <https://github.com/apache/
> kafka/blob/trunk/streams/src/main/java/org/apache/kafka/
> streams/processor/internals/StreamTask.java#L165>
>
> And resuming:
> https://github.com/apache/kafka/blob/trunk/streams/src/
> main/java/org/apache/kafka/streams/processor/internals/
> StreamTask.java#L215 <https://github.com/apache/
> kafka/blob/trunk/streams/src/main/java/org/apache/kafka/
> streams/processor/internals/StreamTask.java#L215>
>
> Cheers
> Eno
> > On 1 Apr 2017, at 17:14, Tianji Li <sk...@gmail.com> wrote:
> >
> > Hi Eno,
> >
> > Could you point to me where in code this is happening please?
> >
> > Thanks
> > Tianji
> >
> > On Sat, Apr 1, 2017 at 11:45 AM, Eno Thereska <en...@gmail.com>
> > wrote:
> >
> >> Tianji,
> >>
> >> You shouldn’t have to worry about pausing and resuming the consumer,
> since
> >> that happens internally automatically.
> >>
> >> Eno
> >>
> >>> On Apr 1, 2017, at 3:26 PM, Tianji Li <sk...@gmail.com> wrote:
> >>>
> >>> Hi there,
> >>>
> >>> Say a processor that is consuming topic A and producing into topic B,
> and
> >>> somehow the processing takes long time, is it possible to pause the
> >>> consuming from topic A, and later on resume?
> >>>
> >>> Or does it make sense to do so? If not, what are the options to resolve
> >>> this issue?
> >>>
> >>> Thanks
> >>> Tianji
> >>
> >>
>
>
Re: Kafka Streams: Is it possible to pause/resume consuming a topic?
Posted by Eno Thereska <en...@gmail.com>.
Sure, here is an example of pausing:
https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java#L165 <https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java#L165>
And resuming:
https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java#L215 <https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java#L215>
Cheers
Eno
> On 1 Apr 2017, at 17:14, Tianji Li <sk...@gmail.com> wrote:
>
> Hi Eno,
>
> Could you point to me where in code this is happening please?
>
> Thanks
> Tianji
>
> On Sat, Apr 1, 2017 at 11:45 AM, Eno Thereska <en...@gmail.com>
> wrote:
>
>> Tianji,
>>
>> You shouldn’t have to worry about pausing and resuming the consumer, since
>> that happens internally automatically.
>>
>> Eno
>>
>>> On Apr 1, 2017, at 3:26 PM, Tianji Li <sk...@gmail.com> wrote:
>>>
>>> Hi there,
>>>
>>> Say a processor that is consuming topic A and producing into topic B, and
>>> somehow the processing takes long time, is it possible to pause the
>>> consuming from topic A, and later on resume?
>>>
>>> Or does it make sense to do so? If not, what are the options to resolve
>>> this issue?
>>>
>>> Thanks
>>> Tianji
>>
>>
Re: Kafka Streams: Is it possible to pause/resume consuming a topic?
Posted by Tianji Li <sk...@gmail.com>.
Hi Eno,
Could you point to me where in code this is happening please?
Thanks
Tianji
On Sat, Apr 1, 2017 at 11:45 AM, Eno Thereska <en...@gmail.com>
wrote:
> Tianji,
>
> You shouldn’t have to worry about pausing and resuming the consumer, since
> that happens internally automatically.
>
> Eno
>
> > On Apr 1, 2017, at 3:26 PM, Tianji Li <sk...@gmail.com> wrote:
> >
> > Hi there,
> >
> > Say a processor that is consuming topic A and producing into topic B, and
> > somehow the processing takes long time, is it possible to pause the
> > consuming from topic A, and later on resume?
> >
> > Or does it make sense to do so? If not, what are the options to resolve
> > this issue?
> >
> > Thanks
> > Tianji
>
>
Re: Kafka Streams: Is it possible to pause/resume consuming a topic?
Posted by Eno Thereska <en...@gmail.com>.
Tianji,
You shouldn’t have to worry about pausing and resuming the consumer, since that happens internally automatically.
Eno
> On Apr 1, 2017, at 3:26 PM, Tianji Li <sk...@gmail.com> wrote:
>
> Hi there,
>
> Say a processor that is consuming topic A and producing into topic B, and
> somehow the processing takes long time, is it possible to pause the
> consuming from topic A, and later on resume?
>
> Or does it make sense to do so? If not, what are the options to resolve
> this issue?
>
> Thanks
> Tianji