You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Moiz S Jinia <mo...@gmail.com> on 2017/05/03 08:02:51 UTC
Programmatic management of Flink jobs
Is there an API that allows remotely adding, modifying, and cancelling
Flink jobs? Example - changing the time window of a deployed Pattern,
adding new Patterns, etc.
Whats the best way to go about this? To the end user the Pattern would
manifest as rules that can be updated anytime.
Moiz
Re: Programmatic management of Flink jobs
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
Not if it doesn’t happen to frequently. Otherwise you would spend most of your time restarting jobs.
Best,
Aljoscha
> On 9. May 2017, at 08:23, Moiz S Jinia <mo...@gmail.com> wrote:
>
> Thanks. Do you foresee any problems with canceling/starting the jobs whenever the patterns change?
>
> Moiz
>
> On Wed, May 3, 2017 at 4:57 PM, Aljoscha Krettek <aljoscha@apache.org <ma...@apache.org>> wrote:
> Hi,
> Yes, this would work even though it requires canceling/restarting the job whenever the patterns change.
>
> Best,
> Aljoscha
>
>> On 3. May 2017, at 12:36, Moiz S Jinia <moiz.jinia@gmail.com <ma...@gmail.com>> wrote:
>>
>> The kind of program I intend to submit would be one that sets up a StreamExecutionEnvironment, connects to a stream from a Kafka topic, and uses a PatternStream over the kafka events. I could have the jar for this program readily available in the "jobmanager.web.upload.dir", and use the REST API for submitting a program with some configuration params.
>>
>> Does that sound like it'd work or am I missing something?
>>
>> Moiz
>>
>> On Wed, May 3, 2017 at 3:23 PM, Moiz S Jinia <moiz.jinia@gmail.com <ma...@gmail.com>> wrote:
>> Not sure I understand Operators. What I need is to have a Pattern that starts consuming from a Kafka stream. And I need the Patterns to come and go.
>>
>> Another option that comes to mind is this -
>> The Patterns I'll need are well known in advance. Only certain parameters such as the time duration of the within clause, and maybe certain filter confitions of the where clause need tweaking. So I could pre-deploy the Patterns (or jobs) and start or stop them (with parameters).
>>
>> Does that sound feasible?
>>
>> On Wed, May 3, 2017 at 3:15 PM, Aljoscha Krettek <aljoscha@apache.org <ma...@apache.org>> wrote:
>> What would the pattern be added to. An existing custom operator?
>>
>> The REST interface only allows for managing the lifecycle of a job, not modifying their graph structure.
>>
>>> On 3. May 2017, at 11:43, Moiz S Jinia <moiz.jinia@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> Thanks for the references. Looking at the REST API, would adding new Patterns not work via this?
>>>
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs>
>>>
>>> Moiz
>>>
>>> On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek <aljoscha@apache.org <ma...@apache.org>> wrote:
>>> Hi,
>>> For managing a Job you can either use the bin/flink command-line tool or the Rest API [1]. As for dynamically adding patterns, that’s outside of the scope of Flink right now. There are, however, some users that implemented this on top of Flink, see for example RBEA [2]. The basic idea is to use a ConnectedStream where one input is the main input and the other input is a control stream that updates the existing patterns.
>>>
>>> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html>
>>> [2] https://techblog.king.com/rbea-scalable-real-time-analytics-king/ <https://techblog.king.com/rbea-scalable-real-time-analytics-king/>
>>>
>>>> On 3. May 2017, at 10:02, Moiz S Jinia <moiz.jinia@gmail.com <ma...@gmail.com>> wrote:
>>>>
>>>> Is there an API that allows remotely adding, modifying, and cancelling Flink jobs? Example - changing the time window of a deployed Pattern, adding new Patterns, etc.
>>>>
>>>> Whats the best way to go about this? To the end user the Pattern would manifest as rules that can be updated anytime.
>>>>
>>>>
>>>> Moiz
>>>
>>>
>>
>>
>>
>
>
Re: Programmatic management of Flink jobs
Posted by Moiz S Jinia <mo...@gmail.com>.
Thanks. Do you foresee any problems with canceling/starting the jobs
whenever the patterns change?
Moiz
On Wed, May 3, 2017 at 4:57 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> Hi,
> Yes, this would work even though it requires canceling/restarting the job
> whenever the patterns change.
>
> Best,
> Aljoscha
>
> On 3. May 2017, at 12:36, Moiz S Jinia <mo...@gmail.com> wrote:
>
> The kind of program I intend to submit would be one that sets up
> a StreamExecutionEnvironment, connects to a stream from a Kafka topic, and
> uses a PatternStream over the kafka events. I could have the jar for this
> program readily available in the "jobmanager.web.upload.dir", and use the
> REST API for submitting a program with some configuration params.
>
> Does that sound like it'd work or am I missing something?
>
> Moiz
>
> On Wed, May 3, 2017 at 3:23 PM, Moiz S Jinia <mo...@gmail.com> wrote:
>
>> Not sure I understand Operators. What I need is to have a Pattern that
>> starts consuming from a Kafka stream. And I need the Patterns to come and
>> go.
>>
>> Another option that comes to mind is this -
>> The Patterns I'll need are well known in advance. Only certain parameters
>> such as the time duration of the within clause, and maybe certain filter
>> confitions of the where clause need tweaking. So I could pre-deploy the
>> Patterns (or jobs) and start or stop them (with parameters).
>>
>> Does that sound feasible?
>>
>> On Wed, May 3, 2017 at 3:15 PM, Aljoscha Krettek <al...@apache.org>
>> wrote:
>>
>>> What would the pattern be added to. An existing custom operator?
>>>
>>> The REST interface only allows for managing the lifecycle of a job, not
>>> modifying their graph structure.
>>>
>>> On 3. May 2017, at 11:43, Moiz S Jinia <mo...@gmail.com> wrote:
>>>
>>> Thanks for the references. Looking at the REST API, would adding new
>>> Patterns not work via this?
>>>
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/
>>> monitoring/rest_api.html#submitting-programs
>>>
>>> Moiz
>>>
>>> On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek <al...@apache.org>
>>> wrote:
>>>
>>>> Hi,
>>>> For managing a Job you can either use the bin/flink command-line tool
>>>> or the Rest API [1]. As for dynamically adding patterns, that’s outside of
>>>> the scope of Flink right now. There are, however, some users that
>>>> implemented this on top of Flink, see for example RBEA [2]. The basic idea
>>>> is to use a ConnectedStream where one input is the main input and the other
>>>> input is a control stream that updates the existing patterns.
>>>>
>>>> [1] https://ci.apache.org/projects/flink/flink-docs-release-
>>>> 1.3/monitoring/rest_api.html
>>>> [2] https://techblog.king.com/rbea-scalable-real-time-analytics-king/
>>>>
>>>> On 3. May 2017, at 10:02, Moiz S Jinia <mo...@gmail.com> wrote:
>>>>
>>>> Is there an API that allows remotely adding, modifying, and cancelling
>>>> Flink jobs? Example - changing the time window of a deployed Pattern,
>>>> adding new Patterns, etc.
>>>>
>>>> Whats the best way to go about this? To the end user the Pattern would
>>>> manifest as rules that can be updated anytime.
>>>>
>>>>
>>>> Moiz
>>>>
>>>>
>>>>
>>>
>>>
>>
>
>
Re: Programmatic management of Flink jobs
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
Yes, this would work even though it requires canceling/restarting the job whenever the patterns change.
Best,
Aljoscha
> On 3. May 2017, at 12:36, Moiz S Jinia <mo...@gmail.com> wrote:
>
> The kind of program I intend to submit would be one that sets up a StreamExecutionEnvironment, connects to a stream from a Kafka topic, and uses a PatternStream over the kafka events. I could have the jar for this program readily available in the "jobmanager.web.upload.dir", and use the REST API for submitting a program with some configuration params.
>
> Does that sound like it'd work or am I missing something?
>
> Moiz
>
> On Wed, May 3, 2017 at 3:23 PM, Moiz S Jinia <moiz.jinia@gmail.com <ma...@gmail.com>> wrote:
> Not sure I understand Operators. What I need is to have a Pattern that starts consuming from a Kafka stream. And I need the Patterns to come and go.
>
> Another option that comes to mind is this -
> The Patterns I'll need are well known in advance. Only certain parameters such as the time duration of the within clause, and maybe certain filter confitions of the where clause need tweaking. So I could pre-deploy the Patterns (or jobs) and start or stop them (with parameters).
>
> Does that sound feasible?
>
> On Wed, May 3, 2017 at 3:15 PM, Aljoscha Krettek <aljoscha@apache.org <ma...@apache.org>> wrote:
> What would the pattern be added to. An existing custom operator?
>
> The REST interface only allows for managing the lifecycle of a job, not modifying their graph structure.
>
>> On 3. May 2017, at 11:43, Moiz S Jinia <moiz.jinia@gmail.com <ma...@gmail.com>> wrote:
>>
>> Thanks for the references. Looking at the REST API, would adding new Patterns not work via this?
>>
>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs>
>>
>> Moiz
>>
>> On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek <aljoscha@apache.org <ma...@apache.org>> wrote:
>> Hi,
>> For managing a Job you can either use the bin/flink command-line tool or the Rest API [1]. As for dynamically adding patterns, that’s outside of the scope of Flink right now. There are, however, some users that implemented this on top of Flink, see for example RBEA [2]. The basic idea is to use a ConnectedStream where one input is the main input and the other input is a control stream that updates the existing patterns.
>>
>> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html>
>> [2] https://techblog.king.com/rbea-scalable-real-time-analytics-king/ <https://techblog.king.com/rbea-scalable-real-time-analytics-king/>
>>
>>> On 3. May 2017, at 10:02, Moiz S Jinia <moiz.jinia@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> Is there an API that allows remotely adding, modifying, and cancelling Flink jobs? Example - changing the time window of a deployed Pattern, adding new Patterns, etc.
>>>
>>> Whats the best way to go about this? To the end user the Pattern would manifest as rules that can be updated anytime.
>>>
>>>
>>> Moiz
>>
>>
>
>
>
Re: Programmatic management of Flink jobs
Posted by Moiz S Jinia <mo...@gmail.com>.
The kind of program I intend to submit would be one that sets up
a StreamExecutionEnvironment, connects to a stream from a Kafka topic, and
uses a PatternStream over the kafka events. I could have the jar for this
program readily available in the "jobmanager.web.upload.dir", and use the
REST API for submitting a program with some configuration params.
Does that sound like it'd work or am I missing something?
Moiz
On Wed, May 3, 2017 at 3:23 PM, Moiz S Jinia <mo...@gmail.com> wrote:
> Not sure I understand Operators. What I need is to have a Pattern that
> starts consuming from a Kafka stream. And I need the Patterns to come and
> go.
>
> Another option that comes to mind is this -
> The Patterns I'll need are well known in advance. Only certain parameters
> such as the time duration of the within clause, and maybe certain filter
> confitions of the where clause need tweaking. So I could pre-deploy the
> Patterns (or jobs) and start or stop them (with parameters).
>
> Does that sound feasible?
>
> On Wed, May 3, 2017 at 3:15 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
>> What would the pattern be added to. An existing custom operator?
>>
>> The REST interface only allows for managing the lifecycle of a job, not
>> modifying their graph structure.
>>
>> On 3. May 2017, at 11:43, Moiz S Jinia <mo...@gmail.com> wrote:
>>
>> Thanks for the references. Looking at the REST API, would adding new
>> Patterns not work via this?
>>
>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/
>> monitoring/rest_api.html#submitting-programs
>>
>> Moiz
>>
>> On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek <al...@apache.org>
>> wrote:
>>
>>> Hi,
>>> For managing a Job you can either use the bin/flink command-line tool or
>>> the Rest API [1]. As for dynamically adding patterns, that’s outside of the
>>> scope of Flink right now. There are, however, some users that implemented
>>> this on top of Flink, see for example RBEA [2]. The basic idea is to use a
>>> ConnectedStream where one input is the main input and the other input is a
>>> control stream that updates the existing patterns.
>>>
>>> [1] https://ci.apache.org/projects/flink/flink-docs-release-
>>> 1.3/monitoring/rest_api.html
>>> [2] https://techblog.king.com/rbea-scalable-real-time-analytics-king/
>>>
>>> On 3. May 2017, at 10:02, Moiz S Jinia <mo...@gmail.com> wrote:
>>>
>>> Is there an API that allows remotely adding, modifying, and cancelling
>>> Flink jobs? Example - changing the time window of a deployed Pattern,
>>> adding new Patterns, etc.
>>>
>>> Whats the best way to go about this? To the end user the Pattern would
>>> manifest as rules that can be updated anytime.
>>>
>>>
>>> Moiz
>>>
>>>
>>>
>>
>>
>
Re: Programmatic management of Flink jobs
Posted by Moiz S Jinia <mo...@gmail.com>.
Not sure I understand Operators. What I need is to have a Pattern that
starts consuming from a Kafka stream. And I need the Patterns to come and
go.
Another option that comes to mind is this -
The Patterns I'll need are well known in advance. Only certain parameters
such as the time duration of the within clause, and maybe certain filter
confitions of the where clause need tweaking. So I could pre-deploy the
Patterns (or jobs) and start or stop them (with parameters).
Does that sound feasible?
On Wed, May 3, 2017 at 3:15 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> What would the pattern be added to. An existing custom operator?
>
> The REST interface only allows for managing the lifecycle of a job, not
> modifying their graph structure.
>
> On 3. May 2017, at 11:43, Moiz S Jinia <mo...@gmail.com> wrote:
>
> Thanks for the references. Looking at the REST API, would adding new
> Patterns not work via this?
>
> https://ci.apache.org/projects/flink/flink-docs-
> release-1.3/monitoring/rest_api.html#submitting-programs
>
> Moiz
>
> On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
>> Hi,
>> For managing a Job you can either use the bin/flink command-line tool or
>> the Rest API [1]. As for dynamically adding patterns, that’s outside of the
>> scope of Flink right now. There are, however, some users that implemented
>> this on top of Flink, see for example RBEA [2]. The basic idea is to use a
>> ConnectedStream where one input is the main input and the other input is a
>> control stream that updates the existing patterns.
>>
>> [1] https://ci.apache.org/projects/flink/flink-docs-release-
>> 1.3/monitoring/rest_api.html
>> [2] https://techblog.king.com/rbea-scalable-real-time-analytics-king/
>>
>> On 3. May 2017, at 10:02, Moiz S Jinia <mo...@gmail.com> wrote:
>>
>> Is there an API that allows remotely adding, modifying, and cancelling
>> Flink jobs? Example - changing the time window of a deployed Pattern,
>> adding new Patterns, etc.
>>
>> Whats the best way to go about this? To the end user the Pattern would
>> manifest as rules that can be updated anytime.
>>
>>
>> Moiz
>>
>>
>>
>
>
Re: Programmatic management of Flink jobs
Posted by Aljoscha Krettek <al...@apache.org>.
What would the pattern be added to. An existing custom operator?
The REST interface only allows for managing the lifecycle of a job, not modifying their graph structure.
> On 3. May 2017, at 11:43, Moiz S Jinia <mo...@gmail.com> wrote:
>
> Thanks for the references. Looking at the REST API, would adding new Patterns not work via this?
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs>
>
> Moiz
>
> On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek <aljoscha@apache.org <ma...@apache.org>> wrote:
> Hi,
> For managing a Job you can either use the bin/flink command-line tool or the Rest API [1]. As for dynamically adding patterns, that’s outside of the scope of Flink right now. There are, however, some users that implemented this on top of Flink, see for example RBEA [2]. The basic idea is to use a ConnectedStream where one input is the main input and the other input is a control stream that updates the existing patterns.
>
> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html>
> [2] https://techblog.king.com/rbea-scalable-real-time-analytics-king/ <https://techblog.king.com/rbea-scalable-real-time-analytics-king/>
>
>> On 3. May 2017, at 10:02, Moiz S Jinia <moiz.jinia@gmail.com <ma...@gmail.com>> wrote:
>>
>> Is there an API that allows remotely adding, modifying, and cancelling Flink jobs? Example - changing the time window of a deployed Pattern, adding new Patterns, etc.
>>
>> Whats the best way to go about this? To the end user the Pattern would manifest as rules that can be updated anytime.
>>
>>
>> Moiz
>
>
Re: Programmatic management of Flink jobs
Posted by Moiz S Jinia <mo...@gmail.com>.
Thanks for the references. Looking at the REST API, would adding new
Patterns not work via this?
https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs
Moiz
On Wed, May 3, 2017 at 2:52 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> Hi,
> For managing a Job you can either use the bin/flink command-line tool or
> the Rest API [1]. As for dynamically adding patterns, that’s outside of the
> scope of Flink right now. There are, however, some users that implemented
> this on top of Flink, see for example RBEA [2]. The basic idea is to use a
> ConnectedStream where one input is the main input and the other input is a
> control stream that updates the existing patterns.
>
> [1] https://ci.apache.org/projects/flink/flink-docs-
> release-1.3/monitoring/rest_api.html
> [2] https://techblog.king.com/rbea-scalable-real-time-analytics-king/
>
> On 3. May 2017, at 10:02, Moiz S Jinia <mo...@gmail.com> wrote:
>
> Is there an API that allows remotely adding, modifying, and cancelling
> Flink jobs? Example - changing the time window of a deployed Pattern,
> adding new Patterns, etc.
>
> Whats the best way to go about this? To the end user the Pattern would
> manifest as rules that can be updated anytime.
>
>
> Moiz
>
>
>
Re: Programmatic management of Flink jobs
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
For managing a Job you can either use the bin/flink command-line tool or the Rest API [1]. As for dynamically adding patterns, that’s outside of the scope of Flink right now. There are, however, some users that implemented this on top of Flink, see for example RBEA [2]. The basic idea is to use a ConnectedStream where one input is the main input and the other input is a control stream that updates the existing patterns.
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html>
[2] https://techblog.king.com/rbea-scalable-real-time-analytics-king/ <https://techblog.king.com/rbea-scalable-real-time-analytics-king/>
> On 3. May 2017, at 10:02, Moiz S Jinia <mo...@gmail.com> wrote:
>
> Is there an API that allows remotely adding, modifying, and cancelling Flink jobs? Example - changing the time window of a deployed Pattern, adding new Patterns, etc.
>
> Whats the best way to go about this? To the end user the Pattern would manifest as rules that can be updated anytime.
>
>
> Moiz