[GitHub] [spark] AndrewXWei commented on pull request #42843: [KUBERNETES] Add ephemeral storage configs for spark on kubernetes

["AndrewXWei (via GitHub)" <gi...@apache.org>]

AndrewXWei commented on PR #42843:
URL: https://github.com/apache/spark/pull/42843#issuecomment-1710967750

   > Thank you for making a PR, @AndrewXWei .
   > 
   > However, we recommend to use pod templates for fine tuning because we cannot have one to one mapping in `spark.kubernetes.*` configuration for all K8s features.
   > 
   > ```
   > --conf spark.kubernetes.driver.podTemplateFile=s3a://bucket/driver.yml
   > --conf spark.kubernetes.executor.podTemplateFile=s3a://bucket/executor.yml
   > ```
   > 
   > Did you try to use pod template configurations?  
   
   Hi, @dongjoon-hyun Thanks for your comment.   
   
   I understand that we can not have all k8s features in spark configuration. 
   We are building a platform for users to submit their data pipelines easily.   
   
   Using pod template can solve the problem, and we do have one. Pod template works well for constant variables, but some variables vary from job to job (notably cpu, memory, volume, ephemeral-storage, etc.). It would requires extra effort to maintain these variable configs (both reading the code and keeping the records).   
   
   When we submit jobs to spark on kubernetes, it would be helpful to have these common variable configurations to use.    
   
   Anyway, thanks for your review.  
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org