You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Suraj Puvvada <su...@traceable.ai> on 2020/04/20 22:01:22 UTC

Changing number of partitions for a topic

Hello,

I have a flink job that reads from a source topic that currently has 4
partitions and I need to increase the partition count to 8.

Do you need to restart the job for that to take effect ?
How does it work in case there is persistent state (like a window operator)
involved ?

Any design documents on how partition mapping works would be very helpful.

Thanks
Suraj

Re: Changing number of partitions for a topic

Posted by Suraj Puvvada <su...@traceable.ai>.

Thanks, I'll check it out.

On Mon, Apr 20, 2020 at 6:23 PM Benchao Li <li...@gmail.com> wrote:

> Hi Suraj,
>
> There is a config option[1] to enable partition discovery, which is
> disabled by default.
> The community discussed to enable it by default[2], but only aims to the
> new Source API.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#partition-discovery
> [2] https://issues.apache.org/jira/browse/FLINK-15703
>
> Suraj Puvvada <su...@traceable.ai> 于2020年4月21日周二 上午6:01写道：
>
>> Hello,
>>
>> I have a flink job that reads from a source topic that currently has 4
>> partitions and I need to increase the partition count to 8.
>>
>> Do you need to restart the job for that to take effect ?
>> How does it work in case there is persistent state (like a window
>> operator) involved ?
>>
>> Any design documents on how partition mapping works would be very
>> helpful.
>>
>> Thanks
>> Suraj
>>
>
>
> --
>
> Benchao Li
> School of Electronics Engineering and Computer Science, Peking University
> Tel:+86-15650713730
> Email: libenchao@gmail.com; libenchao@pku.edu.cn
>
>

Re: Changing number of partitions for a topic

Posted by Benchao Li <li...@gmail.com>.

Hi Suraj,

There is a config option[1] to enable partition discovery, which is
disabled by default.
The community discussed to enable it by default[2], but only aims to the
new Source API.

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#partition-discovery
[2] https://issues.apache.org/jira/browse/FLINK-15703

Suraj Puvvada <su...@traceable.ai> 于2020年4月21日周二 上午6:01写道：

> Hello,
>
> I have a flink job that reads from a source topic that currently has 4
> partitions and I need to increase the partition count to 8.
>
> Do you need to restart the job for that to take effect ?
> How does it work in case there is persistent state (like a window
> operator) involved ?
>
> Any design documents on how partition mapping works would be very helpful.
>
> Thanks
> Suraj
>


-- 

Benchao Li
School of Electronics Engineering and Computer Science, Peking University
Tel:+86-15650713730
Email: libenchao@gmail.com; libenchao@pku.edu.cn