[GitHub] [spark] pan3793 commented on pull request #38205: [SPARK-40747][CORE] Support setting driver log url using env vars on other resource managers

[GitBox <gi...@apache.org>]

pan3793 commented on PR #38205:
URL: https://github.com/apache/spark/pull/38205#issuecomment-1274887897

   To make it flexible, I'm planning to add the variable substitution support to the log URL just like SPARK-26311 does, and let the cluster manager exposes some attributes.
   
   For example, suppose exposing `APP_ID`, `KUBERNETES_POD_NAME` in K8s, then the user could integrate to external log service easily.
   
   ```
   spark-submit \
     --master=k8s://api-server:1234 \
     --deployMode=cluster \
     --conf spark.kubernetes.driverEnv.SPARK_DRIVER_LOG_URL_log=https://spark-log-svc:8080/?app_id={{APP_ID}}&pod_name={{KUBERNETES_POD_NAME}} \
     --conf spark.executorEnv.SPARK_LOG_URL_log=https://spark-log-svc:8080/?app_id={{APP_ID}}&pod_name={{KUBERNETES_POD_NAME}}
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org