[GitHub] [spark] gaborgsomogyi opened a new pull request #25075: [SPARK-28261][CORE] Fix client reuse test

[GitBox <gi...@apache.org>]

gaborgsomogyi opened a new pull request #25075: [SPARK-28261][CORE] Fix client reuse test
URL: https://github.com/apache/spark/pull/25075
 
 
   ## What changes were proposed in this pull request?
   
   There is the following code in [TransportClientFactory#createClient](https://github.com/apache/spark/blob/master/common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java#L150)
   ```
       int clientIndex = rand.nextInt(numConnectionsPerPeer);
       TransportClient cachedClient = clientPool.clients[clientIndex];
   ```
   which choose a client from its pool randomly. If we are unlucky we might not get the max number of connections out, but less than that.
   
   To prove that I've tried out the following test:
   ```
     @Test
     public void testRandom() {
       Random rand = new Random();
       Set<Integer> clients = Collections.synchronizedSet(new HashSet<>());
       long iterCounter = 0;
       while (true) {
         iterCounter++;
         int maxConnections = 4;
         clients.clear();
         for (int i = 0; i < maxConnections * 10; i++) {
           int clientIndex = rand.nextInt(maxConnections);
           clients.add(clientIndex);
         }
         if (clients.size() != maxConnections) {
           System.err.println("Unexpected clients size (iterCounter=" + iterCounter + "): " + clients.size() + ", maxConnections: " + maxConnections);
         }
         if (iterCounter % 100000 == 0) {
           System.out.println("IterCounter: " + iterCounter);
         }
       }
     }
   ```
   
   Result:
   ```
   Unexpected clients size (iterCounter=22388): 3, maxConnections: 4
   Unexpected clients size (iterCounter=36244): 3, maxConnections: 4
   Unexpected clients size (iterCounter=85798): 3, maxConnections: 4
   IterCounter: 100000
   Unexpected clients size (iterCounter=97108): 3, maxConnections: 4
   Unexpected clients size (iterCounter=119121): 3, maxConnections: 4
   Unexpected clients size (iterCounter=129948): 3, maxConnections: 4
   Unexpected clients size (iterCounter=173736): 3, maxConnections: 4
   Unexpected clients size (iterCounter=178138): 3, maxConnections: 4
   Unexpected clients size (iterCounter=195108): 3, maxConnections: 4
   IterCounter: 200000
   Unexpected clients size (iterCounter=209006): 3, maxConnections: 4
   Unexpected clients size (iterCounter=217105): 3, maxConnections: 4
   Unexpected clients size (iterCounter=222456): 3, maxConnections: 4
   Unexpected clients size (iterCounter=226899): 3, maxConnections: 4
   Unexpected clients size (iterCounter=229101): 3, maxConnections: 4
   Unexpected clients size (iterCounter=253549): 3, maxConnections: 4
   Unexpected clients size (iterCounter=277550): 3, maxConnections: 4
   Unexpected clients size (iterCounter=289637): 3, maxConnections: 4
   ...
   ```
   
   In this PR I've adapted the test code not to have this flakyness.
   
   ## How was this patch tested?
   
   Additional (not committed test) + existing unit tests in a loop.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org