[GitHub] [spark] attilapiros commented on a change in pull request #34924: [SPARK-37145][K8S] Add support for extending user feature steps with configuration

[GitBox <gi...@apache.org>]

attilapiros commented on a change in pull request #34924:
URL: https://github.com/apache/spark/pull/34924#discussion_r793930121



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -381,4 +382,32 @@ object KubernetesUtils extends Logging {
       }
     }
   }
+
+  @Since("3.3.0")
+  def loadFeatureStep(conf: KubernetesConf, className: String): KubernetesFeatureConfigStep = {
+    val constructors = Utils.classForName(className).getConstructors

Review comment:
       I was thinking about how we could avoid reflections here and my idea is the following:
   1) Extending  `KubernetesFeatureConfigStep` with a new method `def init(config: KubernetesConf) {}` 
   2) Introducing two new traits (`KubernetesDriverFeatureConfigStep`/`KubernetesExecutorFeatureConfigStep`) where the `init` accepts `KubernetesDriverConf` and `KubernetesExecutorConf` respectively. The `init` in the derived classes this way is not overriding the init of `KubernetesFeatureConfigStep`. So we cannot create a common `loadFeatureStep` where `KubernetesConf` is passed but it can be a function with generic parameter.
   
   And finally we can extend the doc of `KUBERNETES_DRIVER_POD_FEATURE_STEPS` and `KUBERNETES_EXECUTOR_POD_FEATURE_STEPS` to mention those two specific traits.
   
   This way the custom feature step's API will contain a clue that configs can be used to initialize. 
   
   WDYT?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org