[GitHub] [flink] xintongsong commented on issue #9058: [FLINK-13166] Add support for batch slot requests to SlotPoolImpl

[GitBox <gi...@apache.org>]

xintongsong commented on issue #9058: [FLINK-13166] Add support for batch slot requests to SlotPoolImpl
URL: https://github.com/apache/flink/pull/9058#issuecomment-509967863
 
 
   Hi @tillrohrmann, thanks for opening this PR.
   
   I have one concern about the way we handle batch slot requests. 
   
   > - Batch slot request don't time out if the SlotPool contains at least one allocated slot
   which can fulfill the pending slot request
   > - Batch slot request don't react to the failAllocation signal from the ResourceManager
   > - Batch slot request don't fail if the slot request to the resource manager fails
   
   From above descriptions, it seems to me that you are assuming a batch request the is not fulfilled in the first place can only wait for another slot in the slot pool to be freed. I'm not sure about this.
   
   I think we need to consider the possibility that a pending slot request in slot pool can also be satisfied by the resource manager after the slot request to RM failed in the first place.
   - When there are multiple jobs running in the cluster, there might be resources taken by other jobs becoming available. While for streaming jobs there may not be many cases that users run multiple jobs in one cluster, it is probably not that rare for batch jobs.
   - If we think of fine grained resource profiles, we might be able to regenerate new slots from the resources returned by existing slots. E.g., if we have 1 slot with 100MB and 2 slots with 50MB managed memory in the slot pool, and lots of tasks that requires 100MB managed memory to run. There is a chance that when tasks in the two 50MB slots finished, the two slots are freed and then the resource manager is able to allocate another 100MB slot from the task executor.
   
   Therefore, I would suggest to make the slot pool retry requesting slot from the resource manager for batch slot requests before their timeouts.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services