[GitHub] [spark] holdenk commented on pull request #40383: [SPARK-42762][K8S] Improve logging in K8s on disconnect when using StatefulSets

["holdenk (via GitHub)" <gi...@apache.org>]

holdenk commented on PR #40383:
URL: https://github.com/apache/spark/pull/40383#issuecomment-1465064951

   So we only keep it in the window between an executor requesting it's ID and it disconnecting to restabmudh a proper connection (generally <1s). Given it's a few bytes of information per outstanding executor without an ID assigned yet I'm not super worried about the memory usage.
   
   We could also close the connection from the driver side instead I suppose but that might cause a race condition with the executor (would have to check how we handle disconnects and if we process outstanding messages).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org