Live configuration change

[Ning Shi <ni...@gmail.com>]

In the job I'm implementing, there are a couple of configuration
variables that I wnat to change at runtime, such as rate limit at the
Kafka source. I know it's possible to use a control stream and join it
with the normal stream to configure things in certain operators, but
this doesn't work for the source. Is there any other way to configure
settings at runtime?

Thanks,

--
Ning

Re: Live configuration change

[Ning Shi <ni...@gmail.com>]

> On Nov 6, 2018, at 4:22 PM, Elias Levy <fe...@gmail.com> wrote:
> 
> Also note that there is a pending PR to allow the Cassandra sink to back pressure, so that the cluster does not get overwhelmed.

Yes, I’ve been following the development on that pull request. Unfortunately, we have to go live very soon so we can’t wait to leverage that, but it’s definitely a very nice feature to have.

Thanks,

—
Ning

Re: Live configuration change

[Elias Levy <fe...@gmail.com>]

Also note that there is a pending PR to allow the Cassandra sink to back
pressure, so that the cluster does not get overwhelmed.

On Tue, Nov 6, 2018 at 12:46 PM Ning Shi <ni...@gmail.com> wrote:

> > for rate limiting, would quota at Kafka brokers help?
>
> Thanks, Steven. This looks very promising. I'll try it out.
>
> --
> Ning
>

Re: Live configuration change

[Ning Shi <ni...@gmail.com>]

> for rate limiting, would quota at Kafka brokers help?

Thanks, Steven. This looks very promising. I'll try it out.

--
Ning

Re: Live configuration change

[Steven Wu <st...@gmail.com>]

for rate limiting, would quota at Kafka brokers help?

On Tue, Nov 6, 2018 at 10:29 AM Ning Shi <ni...@gmail.com> wrote:

> On Tue, Nov 06, 2018 at 07:44:50PM +0200, Nicos Maris wrote:
> > Ning can you provide another example except for rate limiting?
>
> Our main use case and concern is rate limiting because without it we
> could potentially overwhelm downstream systems (Cassandra) when the job
> plays catch up or replay events from Kafka. The consequence of that is
> that exceptions will be thrown in the Cassandra sink causing the whole
> job to restart and end up in the same situation over and over.
>
> With that said, we do have other use cases such as doing canary
> deployment. We'd like to start processing events for a subset of users,
> then expand it to all if things look good. Without live configuration,
> we have to take a savepoint and restart the job, which will cause the
> job to play catch up at the beginning, potentially overwhelming
> downstream system without rate limiting.
>
> Hope the use cases described above clarifies.
>
> Thanks,
>
> --
> Ning
>

Re: Live configuration change

[Ning Shi <ni...@gmail.com>]

On Tue, Nov 06, 2018 at 07:44:50PM +0200, Nicos Maris wrote:
> Ning can you provide another example except for rate limiting?

Our main use case and concern is rate limiting because without it we
could potentially overwhelm downstream systems (Cassandra) when the job
plays catch up or replay events from Kafka. The consequence of that is
that exceptions will be thrown in the Cassandra sink causing the whole
job to restart and end up in the same situation over and over.

With that said, we do have other use cases such as doing canary
deployment. We'd like to start processing events for a subset of users,
then expand it to all if things look good. Without live configuration,
we have to take a savepoint and restart the job, which will cause the
job to play catch up at the beginning, potentially overwhelming
downstream system without rate limiting.

Hope the use cases described above clarifies.

Thanks,

--
Ning

Re: Live configuration change

[Nicos Maris <ni...@gmail.com>]

Ning can you provide another example except for rate limiting?

On Tue, Nov 6, 2018, 6:20 PM Piotr Nowojski <piotr@data-artisans.com wrote:

> Hi,
>
> Sorry but none that I’m aware of. As far as I know, the only way to
> dynamically configure Kafka source would be for you to copy and modify it’s
> code.
>
> Piotrek
>
> > On 6 Nov 2018, at 15:19, Ning Shi <ni...@gmail.com> wrote:
> >
> > In the job I'm implementing, there are a couple of configuration
> > variables that I wnat to change at runtime, such as rate limit at the
> > Kafka source. I know it's possible to use a control stream and join it
> > with the normal stream to configure things in certain operators, but
> > this doesn't work for the source. Is there any other way to configure
> > settings at runtime?
> >
> > Thanks,
> >
> > --
> > Ning
>
>

Re: Live configuration change

[Piotr Nowojski <pi...@data-artisans.com>]

Hi,

Sorry but none that I’m aware of. As far as I know, the only way to dynamically configure Kafka source would be for you to copy and modify it’s code.

Piotrek

> On 6 Nov 2018, at 15:19, Ning Shi <ni...@gmail.com> wrote:
> 
> In the job I'm implementing, there are a couple of configuration
> variables that I wnat to change at runtime, such as rate limit at the
> Kafka source. I know it's possible to use a control stream and join it
> with the normal stream to configure things in certain operators, but
> this doesn't work for the source. Is there any other way to configure
> settings at runtime?
> 
> Thanks,
> 
> --
> Ning