You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jiaxin Shan (Jira)" <ji...@apache.org> on 2019/10/18 08:46:00 UTC
[jira] [Commented] (SPARK-28022) k8s pod affinity to achieve cloud
native friendly autoscaling
[ https://issues.apache.org/jira/browse/SPARK-28022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954398#comment-16954398 ]
Jiaxin Shan commented on SPARK-28022:
-------------------------------------
I don't quite understand the use case here. Sound like you want to put your executor as close as possible.
Kubernetes has the native support for node affinity and pod affinity but they're a little bit different even it make you pods sit close at some level.
# node selector or node affinity
k8s scheduler put your application of subset of nodes pool. The problem is if you have a large pool of certain nodes, it won't do bin pack inside target node group. In cloud environment, if you have autoscaler enable, it will guarantee resource is utilized.
# pod affinity.
k8s scheduler will try to find the qualified pod and try to put following pods to the same node.
My question is can both of these address your use case?
> k8s pod affinity to achieve cloud native friendly autoscaling
> --------------------------------------------------------------
>
> Key: SPARK-28022
> URL: https://issues.apache.org/jira/browse/SPARK-28022
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes
> Affects Versions: 3.0.0
> Reporter: Henry Yu
> Priority: Major
>
> Hi, in order to achieve cloud native friendly autoscaling , I propose to add a pod affinity feature.
> Traditionally, when we use spark in fix size yarn cluster, it make sense to spread containers to every node.
> Coming to cloud native resource manage, we want to release node when we don't need it any more.
> Pod affinity feature counts to place all pods of certain application to some nodes instead of all nodes.
> By the way, using pod template is not a good choice, adding application id to pod affinity term when submit is more robust.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org