[GitHub] [flink] xintongsong commented on pull request #14612: [FLINK-20864][runtime] Introduce the DEFAULT resource profile for the…

[GitBox <gi...@apache.org>]

xintongsong commented on pull request #14612:
URL: https://github.com/apache/flink/pull/14612#issuecomment-758605289


   @KarmaGYZ 
   I'm not entirely sure about introducing the concept `DEFAULT` and changing the matching rules for `UNKNOWN` to be only matched by `DEFAULT`.
   
   First of all, it complicates the concepts. Currently, `ResourceProfile` is already complex and hard for new comers to understand.
     * It is used for describing both requirement and resource.
     * `UNKNOWN` can only be used for requirement
     * `ANY` can only be used for resource
     * `isMatching` can only be called as `resource.isMatching(requirement)`
    
   I think we might want to have two separate class `ResourceRequirement` and `ResourceCapacity` for the above two concepts, and make `ResourceProfile` a field of them that purely describes the resource value (like primitive types or `MemorySize`). In this way, we can have special values like `ResourceRequirement#UNKNOWN` and `ResourceCapacity#ANY` whose `ResourceProfile` field is `null`. However, this is not necessary for this PR. We can open a separate jira ticket to discuss it.
   
   I see how matching unknown requests only to the default slots may help in fine-grained resource management. When there are both fine-grained and unknown requests, the fine-grained requests may create slots with very different resources. Once these slots are reused for the unknown requirements, performance of tasks with unknown requirements could be very unstable, depending on how many resource the reused slot has.
   
   However, I would consider having fine-grained and unknown resource requirements together as a follow-up improvement. I'd rather not touching the matching rules for the unknown requirements for a future improvement, at the risk that the existing corse-grained resource management might also be affected.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org