You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Amit Joshi <ma...@gmail.com> on 2020/08/27 17:59:44 UTC
[Spark Kafka Structured Streaming] Adding partition and topic to the
kafka dynamically
Hi All,
I am trying to understand the effect of adding topics and partitions to a
topic in kafka, which is being consumed by spark structured streaming
applications.
Do we have to restart the spark structured streaming application to read
from the newly added topic?
Do we have to restart the spark structured streaming application to read
from the newly added partition to a topic?
Kafka consumers have a meta data refresh property that works without
restarting.
Thanks advance.
Regards
Amit Joshi
Re: [Spark Kafka Structured Streaming] Adding partition and topic to
the kafka dynamically
Posted by Gabor Somogyi <ga...@gmail.com>.
Hi Amit,
The answer is no.
G
On Fri, Aug 28, 2020 at 9:16 AM Jungtaek Lim <ka...@gmail.com>
wrote:
> Hi Amit,
>
> if I remember correctly, you don't need to restart the query to reflect
> the newly added topic and partition, if your subscription covers the topic
> (like subscribe pattern). Please try it out.
>
> Hope this helps.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> On Fri, Aug 28, 2020 at 1:56 PM Amit Joshi <ma...@gmail.com>
> wrote:
>
>> Any pointers will be appreciated.
>>
>> On Thursday, August 27, 2020, Amit Joshi <ma...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> I am trying to understand the effect of adding topics and partitions to
>>> a topic in kafka, which is being consumed by spark structured streaming
>>> applications.
>>>
>>> Do we have to restart the spark structured streaming application to read
>>> from the newly added topic?
>>> Do we have to restart the spark structured streaming application to read
>>> from the newly added partition to a topic?
>>>
>>> Kafka consumers have a meta data refresh property that works without
>>> restarting.
>>>
>>> Thanks advance.
>>>
>>> Regards
>>> Amit Joshi
>>>
>>
Re: [Spark Kafka Structured Streaming] Adding partition and topic to
the kafka dynamically
Posted by Amit Joshi <ma...@gmail.com>.
Hi Jungtaek,
Thanks for the input. I did tried and it worked.
I got confused earlier after reading some blogs.
Regards
Amit
On Friday, August 28, 2020, Jungtaek Lim <ka...@gmail.com>
wrote:
> Hi Amit,
>
> if I remember correctly, you don't need to restart the query to reflect
> the newly added topic and partition, if your subscription covers the topic
> (like subscribe pattern). Please try it out.
>
> Hope this helps.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> On Fri, Aug 28, 2020 at 1:56 PM Amit Joshi <ma...@gmail.com>
> wrote:
>
>> Any pointers will be appreciated.
>>
>> On Thursday, August 27, 2020, Amit Joshi <ma...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> I am trying to understand the effect of adding topics and partitions to
>>> a topic in kafka, which is being consumed by spark structured streaming
>>> applications.
>>>
>>> Do we have to restart the spark structured streaming application to read
>>> from the newly added topic?
>>> Do we have to restart the spark structured streaming application to read
>>> from the newly added partition to a topic?
>>>
>>> Kafka consumers have a meta data refresh property that works without
>>> restarting.
>>>
>>> Thanks advance.
>>>
>>> Regards
>>> Amit Joshi
>>>
>>
Re: [Spark Kafka Structured Streaming] Adding partition and topic to
the kafka dynamically
Posted by Jungtaek Lim <ka...@gmail.com>.
Hi Amit,
if I remember correctly, you don't need to restart the query to reflect the
newly added topic and partition, if your subscription covers the topic
(like subscribe pattern). Please try it out.
Hope this helps.
Thanks,
Jungtaek Lim (HeartSaVioR)
On Fri, Aug 28, 2020 at 1:56 PM Amit Joshi <ma...@gmail.com>
wrote:
> Any pointers will be appreciated.
>
> On Thursday, August 27, 2020, Amit Joshi <ma...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I am trying to understand the effect of adding topics and partitions to a
>> topic in kafka, which is being consumed by spark structured streaming
>> applications.
>>
>> Do we have to restart the spark structured streaming application to read
>> from the newly added topic?
>> Do we have to restart the spark structured streaming application to read
>> from the newly added partition to a topic?
>>
>> Kafka consumers have a meta data refresh property that works without
>> restarting.
>>
>> Thanks advance.
>>
>> Regards
>> Amit Joshi
>>
>
Re: [Spark Kafka Structured Streaming] Adding partition and topic to
the kafka dynamically
Posted by Amit Joshi <ma...@gmail.com>.
Any pointers will be appreciated.
On Thursday, August 27, 2020, Amit Joshi <ma...@gmail.com> wrote:
> Hi All,
>
> I am trying to understand the effect of adding topics and partitions to a
> topic in kafka, which is being consumed by spark structured streaming
> applications.
>
> Do we have to restart the spark structured streaming application to read
> from the newly added topic?
> Do we have to restart the spark structured streaming application to read
> from the newly added partition to a topic?
>
> Kafka consumers have a meta data refresh property that works without
> restarting.
>
> Thanks advance.
>
> Regards
> Amit Joshi
>