You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2019/07/08 18:10:58 UTC
[spark] branch master updated: [SPARK-28261][CORE] Fix client reuse
test
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e11a558 [SPARK-28261][CORE] Fix client reuse test
e11a558 is described below
commit e11a55827e7475aab77e8a4ea0baed7c14059908
Author: Gabor Somogyi <ga...@gmail.com>
AuthorDate: Mon Jul 8 11:10:03 2019 -0700
[SPARK-28261][CORE] Fix client reuse test
## What changes were proposed in this pull request?
There is the following code in [TransportClientFactory#createClient](https://github.com/apache/spark/blob/master/common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java#L150)
```
int clientIndex = rand.nextInt(numConnectionsPerPeer);
TransportClient cachedClient = clientPool.clients[clientIndex];
```
which choose a client from its pool randomly. If we are unlucky we might not get the max number of connections out, but less than that.
To prove that I've tried out the following test:
```java
Test
public void testRandom() {
Random rand = new Random();
Set<Integer> clients = Collections.synchronizedSet(new HashSet<>());
long iterCounter = 0;
while (true) {
iterCounter++;
int maxConnections = 4;
clients.clear();
for (int i = 0; i < maxConnections * 10; i++) {
int clientIndex = rand.nextInt(maxConnections);
clients.add(clientIndex);
}
if (clients.size() != maxConnections) {
System.err.println("Unexpected clients size (iterCounter=" + iterCounter + "): " + clients.size() + ", maxConnections: " + maxConnections);
}
if (iterCounter % 100000 == 0) {
System.out.println("IterCounter: " + iterCounter);
}
}
}
```
Result:
```
Unexpected clients size (iterCounter=22388): 3, maxConnections: 4
Unexpected clients size (iterCounter=36244): 3, maxConnections: 4
Unexpected clients size (iterCounter=85798): 3, maxConnections: 4
IterCounter: 100000
Unexpected clients size (iterCounter=97108): 3, maxConnections: 4
Unexpected clients size (iterCounter=119121): 3, maxConnections: 4
Unexpected clients size (iterCounter=129948): 3, maxConnections: 4
Unexpected clients size (iterCounter=173736): 3, maxConnections: 4
Unexpected clients size (iterCounter=178138): 3, maxConnections: 4
Unexpected clients size (iterCounter=195108): 3, maxConnections: 4
IterCounter: 200000
Unexpected clients size (iterCounter=209006): 3, maxConnections: 4
Unexpected clients size (iterCounter=217105): 3, maxConnections: 4
Unexpected clients size (iterCounter=222456): 3, maxConnections: 4
Unexpected clients size (iterCounter=226899): 3, maxConnections: 4
Unexpected clients size (iterCounter=229101): 3, maxConnections: 4
Unexpected clients size (iterCounter=253549): 3, maxConnections: 4
Unexpected clients size (iterCounter=277550): 3, maxConnections: 4
Unexpected clients size (iterCounter=289637): 3, maxConnections: 4
...
```
In this PR I've adapted the test code not to have this flakyness.
## How was this patch tested?
Additional (not committed test) + existing unit tests in a loop.
Closes #25075 from gaborgsomogyi/SPARK-28261.
Authored-by: Gabor Somogyi <ga...@gmail.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
.../test/java/org/apache/spark/network/TransportClientFactorySuite.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/common/network-common/src/test/java/org/apache/spark/network/TransportClientFactorySuite.java b/common/network-common/src/test/java/org/apache/spark/network/TransportClientFactorySuite.java
index 2c62114..b4caa87 100644
--- a/common/network-common/src/test/java/org/apache/spark/network/TransportClientFactorySuite.java
+++ b/common/network-common/src/test/java/org/apache/spark/network/TransportClientFactorySuite.java
@@ -117,7 +117,7 @@ public class TransportClientFactorySuite {
}
Assert.assertEquals(0, failed.get());
- Assert.assertEquals(clients.size(), maxConnections);
+ Assert.assertTrue(clients.size() <= maxConnections);
for (TransportClient client : clients) {
client.close();
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org