Flink Queue Scheduling (JobManager)

[Vikram Saxena <vi...@gmail.com>]

Hi

I am reading and learning about Flink and I have tried to implement some
Flink Jobs.

In my application I have 2 Flink Jobs which I want to run in parallel.
Of course, as I understand I can have the task slots divided so that each
one can run concurrently.

But, is there a possibility for scheduling jobs only to be processed when
the JobManager has resources free(task slots) and not throw back
(NoResourceAvailableException) ??

The reason I ask this is because I have 2 Flink Jobs, one which would be
always running in short intervals and other which would be in ad hoc basis.

I do not want to have my task slots divided just for this ad hoc job,
So I wanted to submit this adhoc job and the Job Manager can schedule it
whenever it has free task slots ?


Cheers,
Vikram

Re: Flink Queue Scheduling (JobManager)

[Vikram Saxena <vi...@gmail.com>]

Yes, I have tried using separate Yarn queues for this, but I have my
doubts.

Here is what I am trying to do:
I have 2 Flink Jobs

JobA : Regular Job running every x minutes.
JobB : User requested adhoc job

Tried1.

Sol A :
Have 2 queues on Yarn with 95 : 5 resource distribution

Sol B:
Have parallelism control the concurrent execution i.e. divide total task
slots in 95:5 ratio

In both the above solutions I would be dedicating a chunk of resources to
 Job B.
I want Job A to run will all my resources and only if Job B is submitted,
it should get some resources to process and not throw back
NoResourceException.

I hope I am able to explain my problem :)


VG,
VIkram

On Wed, Apr 27, 2016 at 2:13 PM, Flavio Pompermaier <po...@okkam.it>
wrote:

> That would be definitely interesting but I think that at the moment the
> only way to achieve that is to exploit YARN for that..
> An integration with some job-workflow engine (like Apache Oozie and Apache
> Falcon) would also be very useful! I tried to wrote on Apache Oozie mailing
> list if there is any interest in integrating with apache Flink but I didn't
> receive any feedback yet :(
>
> Best,
> Flavio
>
> On Wed, Apr 27, 2016 at 1:00 PM, Vikram Saxena <vi...@gmail.com>
> wrote:
>
>> Hi
>>
>> I am reading and learning about Flink and I have tried to implement some
>> Flink Jobs.
>>
>> In my application I have 2 Flink Jobs which I want to run in parallel.
>> Of course, as I understand I can have the task slots divided so that each
>> one can run concurrently.
>>
>> But, is there a possibility for scheduling jobs only to be processed when
>> the JobManager has resources free(task slots) and not throw back
>> (NoResourceAvailableException) ??
>>
>> The reason I ask this is because I have 2 Flink Jobs, one which would be
>> always running in short intervals and other which would be in ad hoc basis.
>>
>> I do not want to have my task slots divided just for this ad hoc job,
>> So I wanted to submit this adhoc job and the Job Manager can schedule it
>> whenever it has free task slots ?
>>
>>
>> Cheers,
>> Vikram
>>
>
>

Re: Flink Queue Scheduling (JobManager)

[Flavio Pompermaier <po...@okkam.it>]

That would be definitely interesting but I think that at the moment the
only way to achieve that is to exploit YARN for that..
An integration with some job-workflow engine (like Apache Oozie and Apache
Falcon) would also be very useful! I tried to wrote on Apache Oozie mailing
list if there is any interest in integrating with apache Flink but I didn't
receive any feedback yet :(

Best,
Flavio

On Wed, Apr 27, 2016 at 1:00 PM, Vikram Saxena <vi...@gmail.com> wrote:

> Hi
>
> I am reading and learning about Flink and I have tried to implement some
> Flink Jobs.
>
> In my application I have 2 Flink Jobs which I want to run in parallel.
> Of course, as I understand I can have the task slots divided so that each
> one can run concurrently.
>
> But, is there a possibility for scheduling jobs only to be processed when
> the JobManager has resources free(task slots) and not throw back
> (NoResourceAvailableException) ??
>
> The reason I ask this is because I have 2 Flink Jobs, one which would be
> always running in short intervals and other which would be in ad hoc basis.
>
> I do not want to have my task slots divided just for this ad hoc job,
> So I wanted to submit this adhoc job and the Job Manager can schedule it
> whenever it has free task slots ?
>
>
> Cheers,
> Vikram
>