[GitHub] vanzin commented on issue #23801: [SPARK-26891][YARN] Fixing flaky test in YarnSchedulerBackendSuite

[GitBox <gi...@apache.org>]

vanzin commented on issue #23801: [SPARK-26891][YARN] Fixing flaky test in YarnSchedulerBackendSuite
URL: https://github.com/apache/spark/pull/23801#issuecomment-464173703
 
 
   Hmm, I think I understand where this is coming from. But if that's the case, then the following test in the same suite is also flaky, for the same reason.
   
   I think the problem is here:
   
   ```
   val driverEndpoint = rpcEnv.setupEndpoint(ENDPOINT_NAME, createDriverEndpoint())
   ```
   
   When the test instantiates a `YarnSchedulerBackend`, that line is overwriting the endpoint in the running `SparkContext`, so that any messages now are sent to the test scheduler instead. That's where the multi-threaded access comes from.
   
   So I think the best thing here would be to avoid having a live `SparkContext` if possible. If not, then figure out another way to avoid the situation above.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org