[GitHub] [spark] skonto opened a new pull request #26161: [SPARK-27900][K8s] Add jvm oom flag in cluster mode

[GitBox <gi...@apache.org>]

skonto opened a new pull request #26161: [SPARK-27900][K8s] Add jvm oom flag in cluster mode
URL: https://github.com/apache/spark/pull/26161
 
 
   
   ### What changes were proposed in this pull request?
   
   - Adds a flag to make the driver exit in case of an oom error in cluster mode (by default enabled).
   - Adds integration tests for supporting de-activation of the flag if the user wishes.
   adds verbose flag support within the driver's container.
   Note: This follows the discussion here: #24796. Without this pods on K8s will keep running although Spark has failed. In addition current behavior creates a problem to the Spark Operator and any other operator as it cannot detect failure at the K8s level.
   
   ### Why are the changes needed?
   See for details Spark-27900
   
   ### How was this patch tested?
   Manually by launching SparkPi with a large number 100000000 which leads to an oom due to the large number of tasks allocated.
   kubectl logs spark-pi-driver -n spark
   ```
   19/10/18 09:47:02 INFO KubernetesClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
   19/10/18 09:47:02 INFO BlockManagerMasterEndpoint: Registering block manager 172.17.0.6:33435 with 413.9 MiB RAM, BlockManagerId(2, 172.17.0.6, 33435, None)
   #
   # java.lang.OutOfMemoryError: Java heap space
   # -XX:OnOutOfMemoryError="kill -9 %p"
   #   Executing /bin/sh -c "kill -9 14"...
   ```
   Also there are two integration tests for the K8s resource manager. Integration tests might be needed for the other managers.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org