You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/09/28 02:58:34 UTC
[GitHub] [pulsar] congbobo184 opened a new pull request, #17865: [fix][flaky-test] BookKeeperClusterTestCase.setup()
congbobo184 opened a new pull request, #17865:
URL: https://github.com/apache/pulsar/pull/17865
Fixes:
- Prove https://github.com/apache/pulsar/issues/15773 can be solved
### Motivation
```
Error: Tests run: 11, Failures: 1, Errors: 0, Skipped: 3, Time elapsed: 87.06 s <<< FAILURE! - in org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorageTest
Error: setUp(org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorageTest) Time elapsed: 13.089 s <<< FAILURE!
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase.waitForConnection(ZooKeeperWatcherBase.java:159)
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$Builder.build(ZooKeeperClient.java:260)
at org.apache.bookkeeper.test.ZooKeeperUtil.restartCluster(ZooKeeperUtil.java:133)
at org.apache.bookkeeper.test.ZooKeeperUtil.startCluster(ZooKeeperUtil.java:104)
at org.apache.pulsar.packages.management.storage.bookkeeper.bookkeeper.test.BookKeeperClusterTestCase.startZKCluster(BookKeeperClusterTestCase.java:238)
at org.apache.pulsar.packages.management.storage.bookkeeper.bookkeeper.test.BookKeeperClusterTestCase.setUp(BookKeeperClusterTestCase.java:178)
at org.apache.pulsar.packages.management.storage.bookkeeper.bookkeeper.test.BookKeeperClusterTestCase.setUp(BookKeeperClusterTestCase.java:166)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:61)
at org.testng.internal.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:366)
at org.testng.internal.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:320)
at org.testng.internal.TestInvoker.runConfigMethods(TestInvoker.java:701)
at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:527)
at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174)
at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.testng.TestRunner.privateRun(TestRunner.java:764)
at org.testng.TestRunner.run(TestRunner.java:585)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
at org.testng.SuiteRunner.run(SuiteRunner.java:286)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
at org.testng.TestNG.runSuites(TestNG.java:1069)
at org.testng.TestNG.run(TestNG.java:1037)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:123)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:90)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
```
total 8 tests, skipped 3, Run 11. after skipped tests, zkCluster, it is not closed, causing the second startup of zkCluster to fail
### Modifications
add @AfterMethod(alwaysRun = true)
### Documentation
- [x] `doc-not-needed`
### Matching PR in the forked repository
PR in forked repository:
- https://github.com/congbobo184/pulsar/pull/1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed pull request #17865: [fix][flaky-test] BookKeeperClusterTestCase.setup()
Posted by GitBox <gi...@apache.org>.
codelipenghui closed pull request #17865: [fix][flaky-test] BookKeeperClusterTestCase.setup()
URL: https://github.com/apache/pulsar/pull/17865
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #17865: [fix][flaky-test] BookKeeperClusterTestCase.setup()
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on PR #17865:
URL: https://github.com/apache/pulsar/pull/17865#issuecomment-1262226389
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a diff in pull request #17865: [fix][flaky-test] BookKeeperClusterTestCase.setup()
Posted by GitBox <gi...@apache.org>.
eolivelli commented on code in PR #17865:
URL: https://github.com/apache/pulsar/pull/17865#discussion_r983494728
##########
pulsar-package-management/bookkeeper-storage/src/test/java/org/apache/pulsar/packages/management/storage/bookkeeper/bookkeeper/test/ZooKeeperUtil.java:
##########
@@ -16,120 +16,142 @@
* specific language governing permissions and limitations
* under the License.
*/
-/**
- * This file is derived from ZooKeeperUtil from Apache BookKeeper
- * http://bookkeeper.apache.org
- */
package org.apache.pulsar.packages.management.storage.bookkeeper.bookkeeper.test;
+import java.io.File;
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.bookkeeper.test.ZooKeeperCluster;
+import org.apache.bookkeeper.util.IOUtils;
import org.apache.bookkeeper.zookeeper.ZooKeeperClient;
import org.apache.commons.io.FileUtils;
-import org.apache.zookeeper.CreateMode;
-import org.apache.zookeeper.ZooDefs.Ids;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.server.NIOServerCnxnFactory;
import org.apache.zookeeper.server.ZooKeeperServer;
import org.apache.zookeeper.test.ClientBase;
+import org.junit.Assert;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import org.testng.Assert;
-
-import java.io.File;
-import java.io.IOException;
-import java.net.InetSocketAddress;
-import java.util.concurrent.CountDownLatch;
-
-public class ZooKeeperUtil {
- static final Logger LOG = LoggerFactory.getLogger(org.apache.bookkeeper.test.ZooKeeperUtil.class);
-
- // ZooKeeper related variables
- protected static final Integer zooKeeperPort = PortManager.nextFreePort();
- private final InetSocketAddress zkaddr;
+public class ZooKeeperUtil implements ZooKeeperCluster {
+ static final Logger LOG;
+ protected Integer zooKeeperPort = 0;
+ private InetSocketAddress zkaddr;
protected ZooKeeperServer zks;
- protected ZooKeeper zkc; // zookeeper client
+ protected ZooKeeper zkc;
protected NIOServerCnxnFactory serverFactory;
- protected File ZkTmpDir;
- private final String connectString;
+ protected File zkTmpDir;
+ private String connectString;
public ZooKeeperUtil() {
- zkaddr = new InetSocketAddress(zooKeeperPort);
- connectString = "localhost:" + zooKeeperPort;
+ String loopbackIPAddr = InetAddress.getLoopbackAddress().getHostAddress();
+ this.zkaddr = new InetSocketAddress(loopbackIPAddr, 0);
+ this.connectString = loopbackIPAddr + ":" + this.zooKeeperPort;
}
public ZooKeeper getZooKeeperClient() {
- return zkc;
+ return this.zkc;
}
public String getZooKeeperConnectString() {
- return connectString;
+ return this.connectString;
+ }
+
+ public String getMetadataServiceUri() {
+ return this.getMetadataServiceUri("/ledgers");
}
- public void startServer() throws Exception {
- // create a ZooKeeper server(dataDir, dataLogDir, port)
+ public String getMetadataServiceUri(String zkLedgersRootPath) {
+ return "zk://" + this.connectString + zkLedgersRootPath;
+ }
+
+ public String getMetadataServiceUri(String zkLedgersRootPath, String type) {
+ return "zk+" + type + "://" + this.connectString + zkLedgersRootPath;
+ }
+
+ public void startCluster() throws Exception {
LOG.debug("Running ZK server");
- // ServerStats.registerAsConcrete();
ClientBase.setupTestEnv();
- ZkTmpDir = File.createTempFile("zookeeper", "test");
- ZkTmpDir.delete();
- ZkTmpDir.mkdir();
+ this.zkTmpDir = IOUtils.createTempDir("zookeeper", "test");
+ this.restartCluster();
+ this.createBKEnsemble("/ledgers");
+ }
- zks = new ZooKeeperServer(ZkTmpDir, ZkTmpDir, ZooKeeperServer.DEFAULT_TICK_TIME);
- serverFactory = new NIOServerCnxnFactory();
- serverFactory.configure(zkaddr, 100);
- serverFactory.startup(zks);
+ public void restartCluster() throws Exception {
+ this.zks = new ZooKeeperServer(this.zkTmpDir, this.zkTmpDir, 3000);
Review Comment:
as a followup work we should move to the official ZookKeeper Java API to bootstrap the server from Java (ZooKeeperServerEmbedded) or use Curator Testing Server.
Both of them provide official an tested APIs to run ZooKeeper
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #17865: [fix][flaky-test] BookKeeperClusterTestCase.setup()
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on code in PR #17865:
URL: https://github.com/apache/pulsar/pull/17865#discussion_r982330511
##########
pulsar-package-management/bookkeeper-storage/src/test/java/org/apache/pulsar/packages/management/storage/bookkeeper/bookkeeper/test/BookKeeperClusterTestCase.java:
##########
@@ -191,7 +191,7 @@ protected String getMetadataServiceUri(String ledgersRootPath) {
return zkUtil.getMetadataServiceUri(ledgersRootPath);
}
- @AfterMethod
+ @AfterMethod(alwaysRun = true)
Review Comment:
Should we need do same for `managed-ledger/src/test/java/org/apache/bookkeeper/test/BookKeeperClusterTestCase.java`
relate issue:
- https://github.com/apache/pulsar/issues/16863
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #17865: [fix][flaky-test] BookKeeperClusterTestCase.setup()
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on PR #17865:
URL: https://github.com/apache/pulsar/pull/17865#issuecomment-1260811488
same issue:
- https://github.com/apache/pulsar/issues/16860
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 merged pull request #17865: [fix][flaky-test] BookKeeperClusterTestCase.setup()
Posted by GitBox <gi...@apache.org>.
congbobo184 merged PR #17865:
URL: https://github.com/apache/pulsar/pull/17865
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #17865: [fix][flaky-test] BookKeeperClusterTestCase.setup()
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on code in PR #17865:
URL: https://github.com/apache/pulsar/pull/17865#discussion_r983496968
##########
pulsar-package-management/bookkeeper-storage/src/test/java/org/apache/pulsar/packages/management/storage/bookkeeper/bookkeeper/test/ZooKeeperUtil.java:
##########
@@ -16,120 +16,142 @@
* specific language governing permissions and limitations
* under the License.
*/
-/**
- * This file is derived from ZooKeeperUtil from Apache BookKeeper
- * http://bookkeeper.apache.org
- */
package org.apache.pulsar.packages.management.storage.bookkeeper.bookkeeper.test;
+import java.io.File;
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.bookkeeper.test.ZooKeeperCluster;
+import org.apache.bookkeeper.util.IOUtils;
import org.apache.bookkeeper.zookeeper.ZooKeeperClient;
import org.apache.commons.io.FileUtils;
-import org.apache.zookeeper.CreateMode;
-import org.apache.zookeeper.ZooDefs.Ids;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.server.NIOServerCnxnFactory;
import org.apache.zookeeper.server.ZooKeeperServer;
import org.apache.zookeeper.test.ClientBase;
+import org.junit.Assert;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import org.testng.Assert;
-
-import java.io.File;
-import java.io.IOException;
-import java.net.InetSocketAddress;
-import java.util.concurrent.CountDownLatch;
-
-public class ZooKeeperUtil {
- static final Logger LOG = LoggerFactory.getLogger(org.apache.bookkeeper.test.ZooKeeperUtil.class);
-
- // ZooKeeper related variables
- protected static final Integer zooKeeperPort = PortManager.nextFreePort();
- private final InetSocketAddress zkaddr;
+public class ZooKeeperUtil implements ZooKeeperCluster {
+ static final Logger LOG;
+ protected Integer zooKeeperPort = 0;
+ private InetSocketAddress zkaddr;
protected ZooKeeperServer zks;
- protected ZooKeeper zkc; // zookeeper client
+ protected ZooKeeper zkc;
protected NIOServerCnxnFactory serverFactory;
- protected File ZkTmpDir;
- private final String connectString;
+ protected File zkTmpDir;
+ private String connectString;
public ZooKeeperUtil() {
- zkaddr = new InetSocketAddress(zooKeeperPort);
- connectString = "localhost:" + zooKeeperPort;
+ String loopbackIPAddr = InetAddress.getLoopbackAddress().getHostAddress();
+ this.zkaddr = new InetSocketAddress(loopbackIPAddr, 0);
+ this.connectString = loopbackIPAddr + ":" + this.zooKeeperPort;
}
public ZooKeeper getZooKeeperClient() {
- return zkc;
+ return this.zkc;
}
public String getZooKeeperConnectString() {
- return connectString;
+ return this.connectString;
+ }
+
+ public String getMetadataServiceUri() {
+ return this.getMetadataServiceUri("/ledgers");
}
- public void startServer() throws Exception {
- // create a ZooKeeper server(dataDir, dataLogDir, port)
+ public String getMetadataServiceUri(String zkLedgersRootPath) {
+ return "zk://" + this.connectString + zkLedgersRootPath;
+ }
+
+ public String getMetadataServiceUri(String zkLedgersRootPath, String type) {
+ return "zk+" + type + "://" + this.connectString + zkLedgersRootPath;
+ }
+
+ public void startCluster() throws Exception {
LOG.debug("Running ZK server");
- // ServerStats.registerAsConcrete();
ClientBase.setupTestEnv();
- ZkTmpDir = File.createTempFile("zookeeper", "test");
- ZkTmpDir.delete();
- ZkTmpDir.mkdir();
+ this.zkTmpDir = IOUtils.createTempDir("zookeeper", "test");
+ this.restartCluster();
+ this.createBKEnsemble("/ledgers");
+ }
- zks = new ZooKeeperServer(ZkTmpDir, ZkTmpDir, ZooKeeperServer.DEFAULT_TICK_TIME);
- serverFactory = new NIOServerCnxnFactory();
- serverFactory.configure(zkaddr, 100);
- serverFactory.startup(zks);
+ public void restartCluster() throws Exception {
+ this.zks = new ZooKeeperServer(this.zkTmpDir, this.zkTmpDir, 3000);
Review Comment:
Good suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org