You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Abhishek Agarwal <ab...@gmail.com> on 2016/08/29 07:00:58 UTC
Dynamic scaling in flink
Is it possible to upscale or downscale a flink application without
re-deploying (similar to rebalancing in storm)?
--
Regards,
Abhishek Agarwal
Re: Dynamic scaling in flink
Posted by Robert Metzger <rm...@apache.org>.
Hi,
this JIRA is a good starting point:
https://issues.apache.org/jira/browse/FLINK-3755
If you don't care about processing guarantees and you are using a stateless
streaming job, you can implement a simple Kafka consumer that uses Kafka's
consumer group mechanism. I recently implemented such a Kafka consumer if
you want to use it as a base for your work:
https://github.com/rmetzger/scratch/blob/kafka-group-consumer/src/main/java/com/dataartisans/FlinkKafkaGroupConsumer.java
Regards,
Robert
On Mon, Aug 29, 2016 at 11:54 AM, Abhishek Agarwal <ab...@gmail.com>
wrote:
> Thanks Stephan. I understand guaranteeing exactly once semantics with the
> dynamic scaling is tough. If I were to let go of the exactly once
> requirement, is it not possible in current version? It would be really
> great if you can point me to the JIRA tracking this work.
>
> On Mon, Aug 29, 2016 at 2:30 PM, Stephan Ewen <se...@apache.org> wrote:
>
>> Hi!
>>
>> There is a lot of work in progress on that feature, and it looks like you
>> can expect the next version to have some upscale/downscale feature that
>> maintains exactly-once semantics.
>>
>> Stephan
>>
>>
>> On Mon, Aug 29, 2016 at 9:00 AM, Abhishek Agarwal <ab...@gmail.com>
>> wrote:
>>
>>> Is it possible to upscale or downscale a flink application without
>>> re-deploying (similar to rebalancing in storm)?
>>>
>>> --
>>> Regards,
>>> Abhishek Agarwal
>>>
>>>
>>
>
>
> --
> Regards,
> Abhishek Agarwal
>
>
Re: Dynamic scaling in flink
Posted by Abhishek Agarwal <ab...@gmail.com>.
Thanks Stephan. I understand guaranteeing exactly once semantics with the
dynamic scaling is tough. If I were to let go of the exactly once
requirement, is it not possible in current version? It would be really
great if you can point me to the JIRA tracking this work.
On Mon, Aug 29, 2016 at 2:30 PM, Stephan Ewen <se...@apache.org> wrote:
> Hi!
>
> There is a lot of work in progress on that feature, and it looks like you
> can expect the next version to have some upscale/downscale feature that
> maintains exactly-once semantics.
>
> Stephan
>
>
> On Mon, Aug 29, 2016 at 9:00 AM, Abhishek Agarwal <ab...@gmail.com>
> wrote:
>
>> Is it possible to upscale or downscale a flink application without
>> re-deploying (similar to rebalancing in storm)?
>>
>> --
>> Regards,
>> Abhishek Agarwal
>>
>>
>
--
Regards,
Abhishek Agarwal
Re: Dynamic scaling in flink
Posted by Stephan Ewen <se...@apache.org>.
Hi!
There is a lot of work in progress on that feature, and it looks like you
can expect the next version to have some upscale/downscale feature that
maintains exactly-once semantics.
Stephan
On Mon, Aug 29, 2016 at 9:00 AM, Abhishek Agarwal <ab...@gmail.com>
wrote:
> Is it possible to upscale or downscale a flink application without
> re-deploying (similar to rebalancing in storm)?
>
> --
> Regards,
> Abhishek Agarwal
>
>