You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by mingleizhang <zm...@163.com> on 2018/02/05 07:38:37 UTC

Why does jobmanager running needs slot ?

I find some codes in flink does not make sense to me. Like in some classes below


JobMasterGateway.java has a offerSlots method which means Offers the given slots to the job manager. I was wondering why a jobmanager running should need slots ?
TaskExecutor.java has a offerSlotsToJobManager method which means offer slots to jobmanager.


Above both are confuse me. I just know that Task running needs slots which support by a taskManager. Does anyone let me why what does jobmanager needs slots mean ?


Thanks in advance.
Rice.

Re: Why does jobmanager running needs slot ?

Posted by Piotr Nowojski <pi...@data-artisans.com>.

Yes, but this issue is still a part of the FLIP-6 work.

Piotrek

> On 5 Feb 2018, at 11:01, mingleizhang <zm...@163.com> wrote:
> 
> I found a website: https://issues.apache.org/jira/browse/FLINK-4360 <https://issues.apache.org/jira/browse/FLINK-4360> implemented this before.
> 
> Rice.
> 
> 
> 
> 
> 
> At 2018-02-05 17:56:49, "Piotr Nowojski" <pi...@data-artisans.com> wrote:
> It seems so - but I’m saying this only basing on a annotations when this method was added (in the last couple of months). I’m not that much familiar with those code parts.
> 
> Piotrek
> 
>> On 5 Feb 2018, at 10:51, mingleizhang <zml13856086071@163.com <ma...@163.com>> wrote:
>> 
>>  Makes sense to me now. Is it a new design at FLIP6 ?
>> 
>> Rice.
>> 
>> 
>> 
>> 
>> 
>> At 2018-02-05 17:49:05, "Piotr Nowojski" <piotr@data-artisans.com <ma...@data-artisans.com>> wrote:
>> I might be wrong but I think it is other way around and the naming of this method is correct - it does exactly what it says. TaskManager comes with some predefined task slots and it is the one that is offering them to a JobManager. JobManager can use those slots offers to (later!) schedule tasks. (#offerSlotsToJobManager() is being called during TaskManager initialisation).
>> 
>> Piotrek
>> 
>>> On 5 Feb 2018, at 10:44, mingleizhang <zml13856086071@163.com <ma...@163.com>> wrote:
>>> 
>>> Yes. Thanks Piotrek. Of course. So, TaskExecutor#offerSlotsToJobManager sounds confuse to me. It might be better to rename it to requestSlotsFromJobManager. I dont know whether it is sounds OKay for that. I just feel like offerSlotToJobManager sounds strange.. What do you think of this ?
>>> 
>>> Rice.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> At 2018-02-05 17:30:32, "Piotr Nowojski" <piotr@data-artisans.com <ma...@data-artisans.com>> wrote:
>>> org.apache.flink.runtime.jobmaster.JobMaster#offerSlots is a receiver side of an RPC call that is being initiated on the sender side: org.apache.flink.runtime.taskexecutor.TaskExecutor#offerSlotsToJobManager.
>>> 
>>> In other words, JobMasterGateway.offerSlots is called by a TaskManager and it is a way how TaskManager is advertising his slots to a JobManager.
>>> 
>>> Piotrek
>>> 
>>>> On 5 Feb 2018, at 08:38, mingleizhang <zml13856086071@163.com <ma...@163.com>> wrote:
>>>> 
>>>> I find some codes in flink does not make sense to me. Like in some classes below
>>>> 
>>>> JobMasterGateway.java has a offerSlots method which means Offers the given slots to the job manager. I was wondering why a jobmanager running should need slots ?
>>>> TaskExecutor.java has a offerSlotsToJobManager method which means offer slots to jobmanager.
>>>> 
>>>> Above both are confuse me. I just know that Task running needs slots which support by a taskManager. Does anyone let me why what does jobmanager needs slots mean ?
>>>> 
>>>> Thanks in advance.
>>>> Rice.
>>>> 
>>>>  
>>>> 
>>>> 
>>>>  
>>> 
>>> 
>>> 
>>>  
>> 
>> 
>> 
>>  
> 
> 
> 
>

Re:Re: Why does jobmanager running needs slot ?

Posted by mingleizhang <zm...@163.com>.

I found a website: https://issues.apache.org/jira/browse/FLINK-4360 implemented this before.

Rice.

At 2018-02-05 17:56:49, "Piotr Nowojski" <pi...@data-artisans.com> wrote:

It seems so - but I’m saying this only basing on a annotations when this method was added (in the last couple of months). I’m not that much familiar with those code parts.

Piotrek

On 5 Feb 2018, at 10:51, mingleizhang <zm...@163.com> wrote:

Makes sense to me now. Is it a new design at FLIP6 ?

Rice.

At 2018-02-05 17:49:05, "Piotr Nowojski" <pi...@data-artisans.com> wrote:
I might be wrong but I think it is other way around and the naming of this method is correct - it does exactly what it says. TaskManager comes with some predefined task slots and it is the one that is offering them to a JobManager. JobManager can use those slots offers to (later!) schedule tasks. (#offerSlotsToJobManager() is being called during TaskManager initialisation).

Piotrek

On 5 Feb 2018, at 10:44, mingleizhang <zm...@163.com> wrote:

Yes. Thanks Piotrek. Of course. So, TaskExecutor#offerSlotsToJobManager sounds confuse to me. It might be better to rename it to requestSlotsFromJobManager. I dont know whether it is sounds OKay for that. I just feel like offerSlotToJobManager sounds strange.. What do you think of this ?

Rice.

At 2018-02-05 17:30:32, "Piotr Nowojski" <pi...@data-artisans.com> wrote:
org.apache.flink.runtime.jobmaster.JobMaster#offerSlots is a receiver side of an RPC call that is being initiated on the sender side: org.apache.flink.runtime.taskexecutor.TaskExecutor#offerSlotsToJobManager.

In other words, JobMasterGateway.offerSlots is called by a TaskManager and it is a way how TaskManager is advertising his slots to a JobManager.

Piotrek

On 5 Feb 2018, at 08:38, mingleizhang <zm...@163.com> wrote:

I find some codes in flink does not make sense to me. Like in some classes below

JobMasterGateway.java has a offerSlots method which means Offers the given slots to the job manager. I was wondering why a jobmanager running should need slots ?
TaskExecutor.java has a offerSlotsToJobManager method which means offer slots to jobmanager.

Above both are confuse me. I just know that Task running needs slots which support by a taskManager. Does anyone let me why what does jobmanager needs slots mean ?

Thanks in advance.
Rice.

Re: Why does jobmanager running needs slot ?

Posted by Piotr Nowojski <pi...@data-artisans.com>.

It seems so - but I’m saying this only basing on a annotations when this method was added (in the last couple of months). I’m not that much familiar with those code parts.

Piotrek

> On 5 Feb 2018, at 10:51, mingleizhang <zm...@163.com> wrote:
> 
>  Makes sense to me now. Is it a new design at FLIP6 ?
> 
> Rice.
> 
> 
> 
> 
> 
> At 2018-02-05 17:49:05, "Piotr Nowojski" <pi...@data-artisans.com> wrote:
> I might be wrong but I think it is other way around and the naming of this method is correct - it does exactly what it says. TaskManager comes with some predefined task slots and it is the one that is offering them to a JobManager. JobManager can use those slots offers to (later!) schedule tasks. (#offerSlotsToJobManager() is being called during TaskManager initialisation).
> 
> Piotrek
> 
>> On 5 Feb 2018, at 10:44, mingleizhang <zml13856086071@163.com <ma...@163.com>> wrote:
>> 
>> Yes. Thanks Piotrek. Of course. So, TaskExecutor#offerSlotsToJobManager sounds confuse to me. It might be better to rename it to requestSlotsFromJobManager. I dont know whether it is sounds OKay for that. I just feel like offerSlotToJobManager sounds strange.. What do you think of this ?
>> 
>> Rice.
>> 
>> 
>> 
>> 
>> 
>> At 2018-02-05 17:30:32, "Piotr Nowojski" <piotr@data-artisans.com <ma...@data-artisans.com>> wrote:
>> org.apache.flink.runtime.jobmaster.JobMaster#offerSlots is a receiver side of an RPC call that is being initiated on the sender side: org.apache.flink.runtime.taskexecutor.TaskExecutor#offerSlotsToJobManager.
>> 
>> In other words, JobMasterGateway.offerSlots is called by a TaskManager and it is a way how TaskManager is advertising his slots to a JobManager.
>> 
>> Piotrek
>> 
>>> On 5 Feb 2018, at 08:38, mingleizhang <zml13856086071@163.com <ma...@163.com>> wrote:
>>> 
>>> I find some codes in flink does not make sense to me. Like in some classes below
>>> 
>>> JobMasterGateway.java has a offerSlots method which means Offers the given slots to the job manager. I was wondering why a jobmanager running should need slots ?
>>> TaskExecutor.java has a offerSlotsToJobManager method which means offer slots to jobmanager.
>>> 
>>> Above both are confuse me. I just know that Task running needs slots which support by a taskManager. Does anyone let me why what does jobmanager needs slots mean ?
>>> 
>>> Thanks in advance.
>>> Rice.
>>> 
>>>  
>>> 
>>> 
>>>  
>> 
>> 
>> 
>>  
> 
> 
> 
>

Re:Re: Why does jobmanager running needs slot ?

Posted by mingleizhang <zm...@163.com>.

Makes sense to me now. Is it a new design at FLIP6 ?

Rice.

Piotrek

On 5 Feb 2018, at 10:44, mingleizhang <zm...@163.com> wrote:

Rice.

In other words, JobMasterGateway.offerSlots is called by a TaskManager and it is a way how TaskManager is advertising his slots to a JobManager.

Piotrek

On 5 Feb 2018, at 08:38, mingleizhang <zm...@163.com> wrote:

I find some codes in flink does not make sense to me. Like in some classes below

Above both are confuse me. I just know that Task running needs slots which support by a taskManager. Does anyone let me why what does jobmanager needs slots mean ?

Thanks in advance.
Rice.

Re: Why does jobmanager running needs slot ?

Posted by Piotr Nowojski <pi...@data-artisans.com>.

I might be wrong but I think it is other way around and the naming of this method is correct - it does exactly what it says. TaskManager comes with some predefined task slots and it is the one that is offering them to a JobManager. JobManager can use those slots offers to (later!) schedule tasks. (#offerSlotsToJobManager() is being called during TaskManager initialisation).

Piotrek

> On 5 Feb 2018, at 10:44, mingleizhang <zm...@163.com> wrote:
> 
> Yes. Thanks Piotrek. Of course. So, TaskExecutor#offerSlotsToJobManager sounds confuse to me. It might be better to rename it to requestSlotsFromJobManager. I dont know whether it is sounds OKay for that. I just feel like offerSlotToJobManager sounds strange.. What do you think of this ?
> 
> Rice.
> 
> 
> 
> 
> 
> At 2018-02-05 17:30:32, "Piotr Nowojski" <pi...@data-artisans.com> wrote:
> org.apache.flink.runtime.jobmaster.JobMaster#offerSlots is a receiver side of an RPC call that is being initiated on the sender side: org.apache.flink.runtime.taskexecutor.TaskExecutor#offerSlotsToJobManager.
> 
> In other words, JobMasterGateway.offerSlots is called by a TaskManager and it is a way how TaskManager is advertising his slots to a JobManager.
> 
> Piotrek
> 
>> On 5 Feb 2018, at 08:38, mingleizhang <zml13856086071@163.com <ma...@163.com>> wrote:
>> 
>> I find some codes in flink does not make sense to me. Like in some classes below
>> 
>> JobMasterGateway.java has a offerSlots method which means Offers the given slots to the job manager. I was wondering why a jobmanager running should need slots ?
>> TaskExecutor.java has a offerSlotsToJobManager method which means offer slots to jobmanager.
>> 
>> Above both are confuse me. I just know that Task running needs slots which support by a taskManager. Does anyone let me why what does jobmanager needs slots mean ?
>> 
>> Thanks in advance.
>> Rice.
>> 
>>  
>> 
>> 
>>  
> 
> 
> 
>

Re:Re: Why does jobmanager running needs slot ?

Posted by mingleizhang <zm...@163.com>.

Yes. Thanks Piotrek. Of course. So, TaskExecutor#offerSlotsToJobManager sounds confuse to me. It might be better to rename it to requestSlotsFromJobManager. I dont know whether it is sounds OKay for that. I just feel like offerSlotToJobManager sounds strange.. What do you think of this ?


Rice.






At 2018-02-05 17:30:32, "Piotr Nowojski" <pi...@data-artisans.com> wrote:
org.apache.flink.runtime.jobmaster.JobMaster#offerSlots is a receiver side of an RPC call that is being initiated on the sender side: org.apache.flink.runtime.taskexecutor.TaskExecutor#offerSlotsToJobManager.


In other words, JobMasterGateway.offerSlots is called by a TaskManager and it is a way how TaskManager is advertising his slots to a JobManager.


Piotrek



On 5 Feb 2018, at 08:38, mingleizhang <zm...@163.com> wrote:


I find some codes in flink does not make sense to me. Like in some classes below


JobMasterGateway.java has a offerSlots method which means Offers the given slots to the job manager. I was wondering why a jobmanager running should need slots ?
TaskExecutor.java has a offerSlotsToJobManager method which means offer slots to jobmanager.


Above both are confuse me. I just know that Task running needs slots which support by a taskManager. Does anyone let me why what does jobmanager needs slots mean ?


Thanks in advance.
Rice.

Re: Why does jobmanager running needs slot ?

Posted by Piotr Nowojski <pi...@data-artisans.com>.

org.apache.flink.runtime.jobmaster.JobMaster#offerSlots is a receiver side of an RPC call that is being initiated on the sender side: org.apache.flink.runtime.taskexecutor.TaskExecutor#offerSlotsToJobManager.

In other words, JobMasterGateway.offerSlots is called by a TaskManager and it is a way how TaskManager is advertising his slots to a JobManager.

Piotrek

> On 5 Feb 2018, at 08:38, mingleizhang <zm...@163.com> wrote:
> 
> I find some codes in flink does not make sense to me. Like in some classes below
> 
> JobMasterGateway.java has a offerSlots method which means Offers the given slots to the job manager. I was wondering why a jobmanager running should need slots ?
> TaskExecutor.java has a offerSlotsToJobManager method which means offer slots to jobmanager.
> 
> Above both are confuse me. I just know that Task running needs slots which support by a taskManager. Does anyone let me why what does jobmanager needs slots mean ?
> 
> Thanks in advance.
> Rice.
> 
>  
> 
> 
>