You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Michael Stack (Jira)" <ji...@apache.org> on 2020/03/24 05:22:00 UTC
[jira] [Resolved] (HBASE-23919) [Flakey Test] Standalone Zookeeper
won't start (minizookeepercluster won't come up)
[ https://issues.apache.org/jira/browse/HBASE-23919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Stack resolved HBASE-23919.
-----------------------------------
Resolution: Not A Problem
HBASE-23993 fixed this condition
> [Flakey Test] Standalone Zookeeper won't start (minizookeepercluster won't come up)
> -----------------------------------------------------------------------------------
>
> Key: HBASE-23919
> URL: https://issues.apache.org/jira/browse/HBASE-23919
> Project: HBase
> Issue Type: Bug
> Components: flakies
> Reporter: Michael Stack
> Priority: Major
>
> I've seen this on occasion across different hardwares; the standalone zookeeper won't come up and then random unit test fails in its junit startup phase as part of launching mini cluster.
> I've been trying to track this w/ a while now adding in logging, using more of zk client instead of the copy/paste we've had a long while now, and adding in logging (I've been running locally w/ zk logging set to INFO).
> It looks like this currently where the last thing out of the server launch is this....
> {code:java}
> 2020-03-02 07:57:46,129 INFO [Time-limited test] server.ZooKeeperServer(854): maxSessionTimeout set to -1 2020-03-02 07:57:46,139 INFO [Time-limited test] server.NIOServerCnxnFactory(89): binding to port 0.0.0.0/0.0.0.0:49316 2020-03-02 07:57:46,181 INFO [Time-limited test] zookeeper.MiniZooKeeperCluster(256): Started connectionTimeout=30000, dir=/Users/stack/checkouts/hbase.git/hbase-server/target/test-data/89d57393-fe97-9200-630f-7843ee406bd2/ cluster_6b4d6f67-7978-dc67-a1a3-7b1b0c0e4268/zookeeper_0, clientPort=49316, dataDir=/Users/stack/checkouts/hbase.git/hbase-server/target/test-data/89d57393-fe97-9200-630f-7843ee406bd2/cluster_6b4d6f67-7978-dc67-a1a3-7b1b0c0e4268/ zookeeper_0/version-2, dataLogDir=/Users/stack/checkouts/hbase.git/hbase-server/target/test-data/89d57393-fe97-9200-630f-7843ee406bd2/cluster_6b4d6f67-7978-dc67-a1a3-7b1b0c0e4268/zookeeper_0/version-2, tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0{code}
> ... then the client just does this over and over:
> {code:java}
> 2020-03-02 07:57:46,182 INFO [Time-limited test] client.FourLetterWordMain(65): connecting to localhost 49316 2020-03-02 07:57:46,213 INFO [Time-limited test] zookeeper.MiniZooKeeperCluster(453): localhost:49316 not up java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:209) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:161) at java.io.BufferedReader.readLine(BufferedReader.java:324) at java.io.BufferedReader.readLine(BufferedReader.java:389) at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:84) at org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.waitForServerUp(MiniZooKeeperCluster.java:442) at org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.startup(MiniZooKeeperCluster.java:259) at org.apache.hadoop.hbase.HBaseZKTestingUtility.startMiniZKCluster(HBaseZKTestingUtility.java:130) at org.apache.hadoop.hbase.HBaseZKTestingUtility.startMiniZKCluster(HBaseZKTestingUtility.java:103) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1107) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1065) at org.apache.hadoop.hbase.regionserver.TestRegionReplicasWithModifyTable.before(TestRegionReplicasWithModifyTable.java:62) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:745)
> {code}
>
> There is ZOOKEEPER-2714 but it is closed as not-reproducible and it looks a little different going by the log emissions in that it registers the client connections.
> Noting this here in case others have seen similar or there are ideas out there for how to fix.
> Here for example is how the failure might look high-level:
> {code:java}
> -------------------------------------------------------------------------------
> Test set: org.apache.hadoop.hbase.regionserver.TestRegionReplicasWithModifyTable
> -------------------------------------------------------------------------------
> Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.017 s <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRegionReplicasWithModifyTable
> org.apache.hadoop.hbase.regionserver.TestRegionReplicasWithModifyTable Time elapsed: 0.01 s <<< ERROR!
> java.io.IOException: Waiting for startup of standalone server; server isRunning=true
> at
> org.apache.hadoop.hbase.regionserver.TestRegionReplicasWithModifyTable.before(TestRegionReplicasWithModifyTable.java:62) {code}
> ... and then when you dig in, the test never even made it past cluster launch in setup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)