You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by "Parise, Jonathan" <Jo...@gd-ms.com> on 2015/06/26 15:33:17 UTC
MiniAccumuloClutser Unit Test Problems
Hello,
I have been writing some J-unit tests based on the MiniAccumuloCluster class. I'm experiencing some issues when several of the tests run back to back. Before I get into the error, let me explain how the tests work in general. Also, I am using Accumulo 1.6.2.
Each test has an @BeforeClass method that first creates a new random directory. Then makes a new MiniAccumuloCluster instance using that directory as the dir parameter. Then, I call MiniAccumuloCluster.start().
There are several @Test methods in each test class. The typical pattern for them is that they create any necessary tables, write some data into those tables and then scan to verify it was written correctly. Basically they are testing that I can serialize and deserialize various types of Objects correctly.
Then the test class as an @AfterClass method that calls MiniAccumuloCluster.stop(). It also deletes the random directory used by the previous test.
The issue I am running into is that generally the first test or two run fine. However, the third test usually gets stuck in the MiniAccumuloCluster startup. It just keeps complaining about being unable to connect. Note that if the test is run independently it passes just fine. When run back to back, I see errors like this one repeatedly:
2015-06-26 09:10:24,352 INFO [main-SendThread(localhost:47046)] zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket connection to server localhost/127.0.0.1:47046
2015-06-26 09:10:24,353 WARN [main-SendThread(localhost:47046)] zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffc86d80004 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:24,956 INFO [main-SendThread(localhost:10406)] zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket connection to server localhost/127.0.0.1:10406
2015-06-26 09:10:24,957 WARN [main-SendThread(localhost:10406)] zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffcb2390004 for server null, unexpected error, closing socket connection and attempting reconnect
2015-06-26 09:10:51,764 INFO [main-SendThread(localhost:10406)] zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket connection to server localhost/127.0.0.1:10406
2015-06-26 09:10:51,765 WARN [main-SendThread(localhost:10406)] zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffcb2390004 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:52,572 INFO [main-SendThread(localhost:47046)] zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket connection to server localhost/127.0.0.1:47046
2015-06-26 09:10:52,573 WARN [main-SendThread(localhost:47046)] zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffc86d80004 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:52,890 INFO [main-SendThread(localhost:10406)] zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket connection to server localhost/127.0.0.1:10406
2015-06-26 09:10:52,891 WARN [main-SendThread(localhost:10406)] zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffcb2390004 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:54,191 INFO [main-SendThread(localhost:10406)] zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket connection to server localhost/127.0.0.1:10406
2015-06-26 09:10:54,192 WARN [main-SendThread(localhost:10406)] zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffcb2390004 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:54,471 INFO [main-SendThread(localhost:47046)] zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket connection to server localhost/127.0.0.1:47046
2015-06-26 09:10:54,471 WARN [main-SendThread(localhost:47046)] zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffc86d80004 for server null, unexpected error, closing socket connection and attempting reconnect
In the ZooKeeperServerMain.out I see lines like this repeating several thousand times:
2015-06-26 08:39:44,810 INFO [SyncThread:0] server.NIOServerCnxn (NIOServerCnxn.java:finishSessionInit(1580)) - Established session 0x14e2fe179eb0000 with negotiated timeout 30000 for client /127.0.0.1:36311
2015-06-26 08:39:45,278 INFO [ProcessThread:-1] server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419)) - Got user-level KeeperException when processing sessionid:0x14e2fe179eb0000 type:create cxid:0x31 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf Error:KeeperErrorCode = NodeExists for /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
2015-06-26 08:39:45,300 INFO [ProcessThread:-1] server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419)) - Got user-level KeeperException when processing sessionid:0x14e2fe179eb0000 type:create cxid:0x33 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf Error:KeeperErrorCode = NodeExists for /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
I've thought about putting a Thread.sleep() call after the MiniAccumuloCluster.stop() call, but that certainly seems brittle. I'm not sure if that would improve the situation.
It seems to me that the MiniAccumuloCluster does not behave well when instances are started and stopped several times. I am running the tests through Maven using the default test behavior.
Could it be something with Maven? Maybe I need to be more explicit when telling it how to run the tests?
Anyone have any insight into what is going wrong here? In general is my usage pattern correct for MiniAccumuloCluster?
Thanks,
Jon Parise
Senior Software Engineer
Viz | General Dynamics Missons Systems
This message and/or attachments may include information subject to GD Corporate Policies 07-103 and 07-105 and is intended to be accessed only by authorized recipients. Use, storage and transmission are governed by General Dynamics and its policies. Contractual restrictions apply to third parties. Recipients should refer to the policies or contract to determine proper handling. Unauthorized review, use, disclosure or distribution is prohibited. If you are not an intended recipient, please contact the sender and destroy all copies of the original message.
Re: MiniAccumuloClutser Unit Test Problems
Posted by Josh Elser <jo...@gmail.com>.
They should all be using '0' which will cause the OS to allocate a
dynamic port. I doubt your parallelism is so high on your tests that you
exhaust the dynamic port range from the OS :)
Parise, Jonathan wrote:
> Should I be doing something to dynamically allocate the ports? I am
> using the MAC constructor that simply takes a File and a root password.
>
> From what I am seeing in the logs, it does look like ports could be the
> cause. If all the MAC instances try to use the same ports, they would
> conflict.
>
> Jon
>
> *From:*David Medinets [mailto:david.medinets@gmail.com]
> *Sent:* Friday, June 26, 2015 2:36 PM
> *To:* accumulo-user
> *Subject:* Re: MiniAccumuloClutser Unit Test Problems
>
> How are the ports dynamically allocated? If five MAC are started at the
> same time, could they be stepping on each other?
>
> On Fri, Jun 26, 2015 at 11:36 AM, Josh Elser <josh.elser@gmail.com
> <ma...@gmail.com>> wrote:
>
> Makes sense. Thanks for clarifying!
>
> Definitely send us a JIRA issue so we can track the issue and try to get
> to the bottom of it.
>
>
>
> Parise, Jonathan wrote:
>
> Yes each test has a unique directory for the MAC.
>
> I use the following code to ensure that:
>
> storageDirectory_ = new File("atf-temp" + Path.SEPARATOR + "accumulo-test-"
> + UUID.randomUUID().toString());
>
> Basically this makes a new random UUID so that I get very unique paths.
> All of the directories end up in atf-temp. This way I can clean them up
> easily.
>
> Jon
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com <ma...@gmail.com>]
> Sent: Friday, June 26, 2015 10:50 AM
> To: user@accumulo.apache.org <ma...@accumulo.apache.org>
> Subject: Re: MiniAccumuloClutser Unit Test Problems
>
> Did each test class use a unique directory for MAC?
>
> Assuming they're all unique, it sounds like a bug. I know some devs run
> our suite of integration tests in parallel which use MAC out of the box.
> If all of the above is true, please feel free to open up an issue on
> JIRA and we can look into it more there.
>
> https://issues.apache.org/jira/secure/CreateIssue!default.jspa
>
> Thanks!
>
> Parise, Jonathan wrote:
>
> Josh,
>
> Sorry for the confusion. I'll try and explain again with better terminology.
>
> I have several test classes, each of which contain several test methods.
> For each test class I have an @BeforeClass that configures and starts a
> MAC. I also have an @AfterClass that calls MAC.stop().
>
> So the flow is MAC is created, 1-n tests in the class run against it,
> MAC is stopped, repeat for next N tests.
>
> This way several @test methods in the same class use the same MAC. I
> mostly did this to make the tests run faster. I understand that having
> several test methods share a MAC could cause test state pollution, but I
> am careful to avoid that.
>
> The issues I was seeing is that if I simply ran "mvn test", the first
> few tests would pass and then eventually one of the tests would get
> stuck forever and just keep throwing connection errors.
>
> When I changed the maven configuration to force the tests to run
> serially, the issue stopped occurring.
>
> I'm not sure if there tests are expected to work when run in parallel.
> My best guess is that the tests may conflict with each other over ports
> or something like that.
>
> I'd like to understand why changing the test running behavior fixed this
> issue. Also, I think it would be good to document this somewhere. The
> MAC javadoc and also the user's guide should provide details about using
> the MAC for tests and properly configuring those tests.
>
> Does that make more sense?
>
> Jon
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com <ma...@gmail.com>]
> Sent: Friday, June 26, 2015 10:00 AM
> To: user@accumulo.apache.org <ma...@accumulo.apache.org>
> Subject: Re: MiniAccumuloClutser Unit Test Problems
>
> Jonathan,
>
> If you're not seeing consistent behavior starting and stopping a
> MiniAccumuloCluster repeatedly, that's a bug. If you can provide a code
> which shows this problem, that'd be a huge help.
>
> If you can get a list of the processes running when you see this happen
> and cross-reference it with what processes should be running, that would
> also go a long way in trying to debug this.
>
> I am a little confused to your specific situation. You said that you
> construct and start a MAC instance in a BeforeClass and stop it in an
> AfterClass, but then you said that you start and stop it for each test.
> Are you saying that after the third construction and use of a MAC, you
> see problems? Or, are you saying that you stop and start each MAC
> instance before you run the @Test methods?
>
> - Josh
>
> Parise, Jonathan wrote:
>
> Hello,
>
> I have been writing some J-unit tests based on the
> MiniAccumuloCluster class. I'm experiencing some issues when several
> of the tests run back to back. Before I get into the error, let me
> explain how the tests work in general. Also, I am using Accumulo 1.6.2.
>
> Each test has an @BeforeClass method that first creates a new random
> directory. Then makes a new MiniAccumuloCluster instance using that
> directory as the dir parameter. Then, I call MiniAccumuloCluster.start().
>
> There are several @Test methods in each test class. The typical
> pattern for them is that they create any necessary tables, write some
> data into those tables and then scan to verify it was written
> correctly. Basically they are testing that I can serialize and
> deserialize various types of Objects correctly.
>
> Then the test class as an @AfterClass method that calls
> MiniAccumuloCluster.stop(). It also deletes the random directory used
> by the previous test.
>
> The issue I am running into is that generally the first test or two
> run fine. However, the third test usually gets stuck in the
> MiniAccumuloCluster startup. It just keeps complaining about being
> unable to connect. Note that if the test is run independently it
> passes just fine. When run back to back, I see errors like this one
> repeatedly:
>
> 2015-06-26 09:10:24,352 INFO [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:47046
> <http://127.0.0.1:47046>
>
> 2015-06-26 09:10:24,353 WARN [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
> )
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:24,956 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
> <http://127.0.0.1:10406>
>
> 2015-06-26 09:10:24,957 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> 2015-06-26 09:10:51,764 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
> <http://127.0.0.1:10406>
>
> 2015-06-26 09:10:51,765 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
> )
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:52,572 INFO [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:47046
> <http://127.0.0.1:47046>
>
> 2015-06-26 09:10:52,573 WARN [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
> )
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:52,890 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
> <http://127.0.0.1:10406>
>
> 2015-06-26 09:10:52,891 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
> )
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:54,191 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
> <http://127.0.0.1:10406>
>
> 2015-06-26 09:10:54,192 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
> )
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:54,471 INFO [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:47046
> <http://127.0.0.1:47046>
>
> 2015-06-26 09:10:54,471 WARN [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> In the ZooKeeperServerMain.out I see lines like this repeating
> several thousand times:
>
> 2015-06-26 08:39:44,810 INFO [SyncThread:0] server.NIOServerCnxn
> (NIOServerCnxn.java:finishSessionInit(1580)) - Established session
> 0x14e2fe179eb0000 with negotiated timeout 30000 for client
> /127.0.0.1:36311 <http://127.0.0.1:36311>
>
> 2015-06-26 08:39:45,278 INFO [ProcessThread:-1]
> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
> - Got user-level KeeperException when processing
> sessionid:0x14e2fe179eb0000 type:create cxid:0x31
> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
> Error:KeeperErrorCode = NodeExists for
> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
>
> 2015-06-26 08:39:45,300 INFO [ProcessThread:-1]
> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
> - Got user-level KeeperException when processing
> sessionid:0x14e2fe179eb0000 type:create cxid:0x33
> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
> Error:KeeperErrorCode = NodeExists for
> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
>
> I've thought about putting a Thread.sleep() call after the
> MiniAccumuloCluster.stop() call, but that certainly seems brittle.
> I'm not sure if that would improve the situation.
>
> It seems to me that the MiniAccumuloCluster does not behave well when
> instances are started and stopped several times. I am running the
> tests through Maven using the default test behavior.
>
> Could it be something with Maven? Maybe I need to be more explicit
> when telling it how to run the tests?
>
> Anyone have any insight into what is going wrong here? In general is
> my usage pattern correct for MiniAccumuloCluster?
>
> Thanks,
>
> Jon Parise
>
> Senior Software Engineer
>
> Viz | General Dynamics Missons Systems
>
> *This message and/or attachments may include information subject to
> GD Corporate Policies 07-103 and 07-105 and is intended to be
> accessed only by authorized recipients. Use, storage and transmission
> are governed by General Dynamics and its policies. Contractual
> restrictions apply to third parties. Recipients should refer to the
> policies or contract to determine proper handling. Unauthorized
> review, use, disclosure or distribution is prohibited. If you are not
> an intended recipient, please contact the sender and destroy all
> copies of the original message.*
>
Re: MiniAccumuloClutser Unit Test Problems
Posted by Keith Turner <ke...@deenlo.com>.
On Fri, Jun 26, 2015 at 3:17 PM, Parise, Jonathan <Jonathan.Parise@gd-ms.com
> wrote:
> Should I be doing something to dynamically allocate the ports? I am
> using the MAC constructor that simply takes a File and a root password.
>
>
>
> From what I am seeing in the logs, it does look like ports could be the
> cause. If all the MAC instances try to use the same ports, they would
> conflict.
>
What did you see in the logs? Can you post the log entries or describe
them?
>
>
> Jon
>
>
>
> *From:* David Medinets [mailto:david.medinets@gmail.com]
> *Sent:* Friday, June 26, 2015 2:36 PM
> *To:* accumulo-user
>
> *Subject:* Re: MiniAccumuloClutser Unit Test Problems
>
>
>
> How are the ports dynamically allocated? If five MAC are started at the
> same time, could they be stepping on each other?
>
>
>
> On Fri, Jun 26, 2015 at 11:36 AM, Josh Elser <jo...@gmail.com> wrote:
>
> Makes sense. Thanks for clarifying!
>
> Definitely send us a JIRA issue so we can track the issue and try to get
> to the bottom of it.
>
>
>
> Parise, Jonathan wrote:
>
> Yes each test has a unique directory for the MAC.
>
> I use the following code to ensure that:
>
> storageDirectory_ = new File("atf-temp" + Path.SEPARATOR + "accumulo-test-"
> + UUID.randomUUID().toString());
>
> Basically this makes a new random UUID so that I get very unique paths.
> All of the directories end up in atf-temp. This way I can clean them up
> easily.
>
> Jon
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com]
> Sent: Friday, June 26, 2015 10:50 AM
> To: user@accumulo.apache.org
> Subject: Re: MiniAccumuloClutser Unit Test Problems
>
> Did each test class use a unique directory for MAC?
>
> Assuming they're all unique, it sounds like a bug. I know some devs run
> our suite of integration tests in parallel which use MAC out of the box.
> If all of the above is true, please feel free to open up an issue on JIRA
> and we can look into it more there.
>
> https://issues.apache.org/jira/secure/CreateIssue!default.jspa
>
> Thanks!
>
> Parise, Jonathan wrote:
>
> Josh,
>
> Sorry for the confusion. I'll try and explain again with better
> terminology.
>
> I have several test classes, each of which contain several test methods.
> For each test class I have an @BeforeClass that configures and starts a
> MAC. I also have an @AfterClass that calls MAC.stop().
>
> So the flow is MAC is created, 1-n tests in the class run against it, MAC
> is stopped, repeat for next N tests.
>
> This way several @test methods in the same class use the same MAC. I
> mostly did this to make the tests run faster. I understand that having
> several test methods share a MAC could cause test state pollution, but I am
> careful to avoid that.
>
> The issues I was seeing is that if I simply ran "mvn test", the first few
> tests would pass and then eventually one of the tests would get stuck
> forever and just keep throwing connection errors.
>
> When I changed the maven configuration to force the tests to run serially,
> the issue stopped occurring.
>
> I'm not sure if there tests are expected to work when run in parallel. My
> best guess is that the tests may conflict with each other over ports or
> something like that.
>
> I'd like to understand why changing the test running behavior fixed this
> issue. Also, I think it would be good to document this somewhere. The MAC
> javadoc and also the user's guide should provide details about using the
> MAC for tests and properly configuring those tests.
>
> Does that make more sense?
>
> Jon
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com]
> Sent: Friday, June 26, 2015 10:00 AM
> To: user@accumulo.apache.org
> Subject: Re: MiniAccumuloClutser Unit Test Problems
>
> Jonathan,
>
> If you're not seeing consistent behavior starting and stopping a
> MiniAccumuloCluster repeatedly, that's a bug. If you can provide a code
> which shows this problem, that'd be a huge help.
>
> If you can get a list of the processes running when you see this happen
> and cross-reference it with what processes should be running, that would
> also go a long way in trying to debug this.
>
> I am a little confused to your specific situation. You said that you
> construct and start a MAC instance in a BeforeClass and stop it in an
> AfterClass, but then you said that you start and stop it for each test.
> Are you saying that after the third construction and use of a MAC, you see
> problems? Or, are you saying that you stop and start each MAC instance
> before you run the @Test methods?
>
> - Josh
>
> Parise, Jonathan wrote:
>
> Hello,
>
> I have been writing some J-unit tests based on the
> MiniAccumuloCluster class. I'm experiencing some issues when several
> of the tests run back to back. Before I get into the error, let me
> explain how the tests work in general. Also, I am using Accumulo 1.6.2.
>
> Each test has an @BeforeClass method that first creates a new random
> directory. Then makes a new MiniAccumuloCluster instance using that
> directory as the dir parameter. Then, I call MiniAccumuloCluster.start().
>
> There are several @Test methods in each test class. The typical
> pattern for them is that they create any necessary tables, write some
> data into those tables and then scan to verify it was written
> correctly. Basically they are testing that I can serialize and
> deserialize various types of Objects correctly.
>
> Then the test class as an @AfterClass method that calls
> MiniAccumuloCluster.stop(). It also deletes the random directory used
> by the previous test.
>
> The issue I am running into is that generally the first test or two
> run fine. However, the third test usually gets stuck in the
> MiniAccumuloCluster startup. It just keeps complaining about being
> unable to connect. Note that if the test is run independently it
> passes just fine. When run back to back, I see errors like this one
> repeatedly:
>
> 2015-06-26 09:10:24,352 INFO [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:47046
>
> 2015-06-26 09:10:24,353 WARN [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
> )
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:24,956 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
>
> 2015-06-26 09:10:24,957 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> 2015-06-26 09:10:51,764 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
>
> 2015-06-26 09:10:51,765 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
> )
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:52,572 INFO [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:47046
>
> 2015-06-26 09:10:52,573 WARN [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
> )
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:52,890 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
>
> 2015-06-26 09:10:52,891 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
> )
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:54,191 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
>
> 2015-06-26 09:10:54,192 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
> )
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:54,471 INFO [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:47046
>
> 2015-06-26 09:10:54,471 WARN [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> In the ZooKeeperServerMain.out I see lines like this repeating
> several thousand times:
>
> 2015-06-26 08:39:44,810 INFO [SyncThread:0] server.NIOServerCnxn
> (NIOServerCnxn.java:finishSessionInit(1580)) - Established session
> 0x14e2fe179eb0000 with negotiated timeout 30000 for client
> /127.0.0.1:36311
>
> 2015-06-26 08:39:45,278 INFO [ProcessThread:-1]
> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
> - Got user-level KeeperException when processing
> sessionid:0x14e2fe179eb0000 type:create cxid:0x31
> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
> Error:KeeperErrorCode = NodeExists for
> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
>
> 2015-06-26 08:39:45,300 INFO [ProcessThread:-1]
> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
> - Got user-level KeeperException when processing
> sessionid:0x14e2fe179eb0000 type:create cxid:0x33
> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
> Error:KeeperErrorCode = NodeExists for
> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
>
> I've thought about putting a Thread.sleep() call after the
> MiniAccumuloCluster.stop() call, but that certainly seems brittle.
> I'm not sure if that would improve the situation.
>
> It seems to me that the MiniAccumuloCluster does not behave well when
> instances are started and stopped several times. I am running the
> tests through Maven using the default test behavior.
>
> Could it be something with Maven? Maybe I need to be more explicit
> when telling it how to run the tests?
>
> Anyone have any insight into what is going wrong here? In general is
> my usage pattern correct for MiniAccumuloCluster?
>
> Thanks,
>
> Jon Parise
>
> Senior Software Engineer
>
> Viz | General Dynamics Missons Systems
>
> *This message and/or attachments may include information subject to
> GD Corporate Policies 07-103 and 07-105 and is intended to be
> accessed only by authorized recipients. Use, storage and transmission
> are governed by General Dynamics and its policies. Contractual
> restrictions apply to third parties. Recipients should refer to the
> policies or contract to determine proper handling. Unauthorized
> review, use, disclosure or distribution is prohibited. If you are not
> an intended recipient, please contact the sender and destroy all
> copies of the original message.*
>
>
>
RE: MiniAccumuloClutser Unit Test Problems
Posted by "Parise, Jonathan" <Jo...@gd-ms.com>.
Should I be doing something to dynamically allocate the ports? I am using the MAC constructor that simply takes a File and a root password.
From what I am seeing in the logs, it does look like ports could be the cause. If all the MAC instances try to use the same ports, they would conflict.
Jon
From: David Medinets [mailto:david.medinets@gmail.com]
Sent: Friday, June 26, 2015 2:36 PM
To: accumulo-user
Subject: Re: MiniAccumuloClutser Unit Test Problems
How are the ports dynamically allocated? If five MAC are started at the same time, could they be stepping on each other?
On Fri, Jun 26, 2015 at 11:36 AM, Josh Elser <jo...@gmail.com>> wrote:
Makes sense. Thanks for clarifying!
Definitely send us a JIRA issue so we can track the issue and try to get to the bottom of it.
Parise, Jonathan wrote:
Yes each test has a unique directory for the MAC.
I use the following code to ensure that:
storageDirectory_ = new File("atf-temp" + Path.SEPARATOR + "accumulo-test-"
+ UUID.randomUUID().toString());
Basically this makes a new random UUID so that I get very unique paths. All of the directories end up in atf-temp. This way I can clean them up easily.
Jon
-----Original Message-----
From: Josh Elser [mailto:josh.elser@gmail.com<ma...@gmail.com>]
Sent: Friday, June 26, 2015 10:50 AM
To: user@accumulo.apache.org<ma...@accumulo.apache.org>
Subject: Re: MiniAccumuloClutser Unit Test Problems
Did each test class use a unique directory for MAC?
Assuming they're all unique, it sounds like a bug. I know some devs run our suite of integration tests in parallel which use MAC out of the box.
If all of the above is true, please feel free to open up an issue on JIRA and we can look into it more there.
https://issues.apache.org/jira/secure/CreateIssue!default.jspa
Thanks!
Parise, Jonathan wrote:
Josh,
Sorry for the confusion. I'll try and explain again with better terminology.
I have several test classes, each of which contain several test methods. For each test class I have an @BeforeClass that configures and starts a MAC. I also have an @AfterClass that calls MAC.stop().
So the flow is MAC is created, 1-n tests in the class run against it, MAC is stopped, repeat for next N tests.
This way several @test methods in the same class use the same MAC. I mostly did this to make the tests run faster. I understand that having several test methods share a MAC could cause test state pollution, but I am careful to avoid that.
The issues I was seeing is that if I simply ran "mvn test", the first few tests would pass and then eventually one of the tests would get stuck forever and just keep throwing connection errors.
When I changed the maven configuration to force the tests to run serially, the issue stopped occurring.
I'm not sure if there tests are expected to work when run in parallel. My best guess is that the tests may conflict with each other over ports or something like that.
I'd like to understand why changing the test running behavior fixed this issue. Also, I think it would be good to document this somewhere. The MAC javadoc and also the user's guide should provide details about using the MAC for tests and properly configuring those tests.
Does that make more sense?
Jon
-----Original Message-----
From: Josh Elser [mailto:josh.elser@gmail.com<ma...@gmail.com>]
Sent: Friday, June 26, 2015 10:00 AM
To: user@accumulo.apache.org<ma...@accumulo.apache.org>
Subject: Re: MiniAccumuloClutser Unit Test Problems
Jonathan,
If you're not seeing consistent behavior starting and stopping a MiniAccumuloCluster repeatedly, that's a bug. If you can provide a code which shows this problem, that'd be a huge help.
If you can get a list of the processes running when you see this happen and cross-reference it with what processes should be running, that would also go a long way in trying to debug this.
I am a little confused to your specific situation. You said that you construct and start a MAC instance in a BeforeClass and stop it in an AfterClass, but then you said that you start and stop it for each test.
Are you saying that after the third construction and use of a MAC, you see problems? Or, are you saying that you stop and start each MAC instance before you run the @Test methods?
- Josh
Parise, Jonathan wrote:
Hello,
I have been writing some J-unit tests based on the
MiniAccumuloCluster class. I'm experiencing some issues when several
of the tests run back to back. Before I get into the error, let me
explain how the tests work in general. Also, I am using Accumulo 1.6.2.
Each test has an @BeforeClass method that first creates a new random
directory. Then makes a new MiniAccumuloCluster instance using that
directory as the dir parameter. Then, I call MiniAccumuloCluster.start().
There are several @Test methods in each test class. The typical
pattern for them is that they create any necessary tables, write some
data into those tables and then scan to verify it was written
correctly. Basically they are testing that I can serialize and
deserialize various types of Objects correctly.
Then the test class as an @AfterClass method that calls
MiniAccumuloCluster.stop(). It also deletes the random directory used
by the previous test.
The issue I am running into is that generally the first test or two
run fine. However, the third test usually gets stuck in the
MiniAccumuloCluster startup. It just keeps complaining about being
unable to connect. Note that if the test is run independently it
passes just fine. When run back to back, I see errors like this one repeatedly:
2015-06-26 09:10:24,352 INFO [main-SendThread(localhost:47046)]
zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
socket connection to server localhost/127.0.0.1:47046<http://127.0.0.1:47046>
2015-06-26 09:10:24,353 WARN [main-SendThread(localhost:47046)]
zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
0x14e2ffc86d80004 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:24,956 INFO [main-SendThread(localhost:10406)]
zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
socket connection to server localhost/127.0.0.1:10406<http://127.0.0.1:10406>
2015-06-26 09:10:24,957 WARN [main-SendThread(localhost:10406)]
zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
0x14e2ffcb2390004 for server null, unexpected error, closing socket
connection and attempting reconnect
2015-06-26 09:10:51,764 INFO [main-SendThread(localhost:10406)]
zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
socket connection to server localhost/127.0.0.1:10406<http://127.0.0.1:10406>
2015-06-26 09:10:51,765 WARN [main-SendThread(localhost:10406)]
zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
0x14e2ffcb2390004 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:52,572 INFO [main-SendThread(localhost:47046)]
zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
socket connection to server localhost/127.0.0.1:47046<http://127.0.0.1:47046>
2015-06-26 09:10:52,573 WARN [main-SendThread(localhost:47046)]
zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
0x14e2ffc86d80004 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:52,890 INFO [main-SendThread(localhost:10406)]
zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
socket connection to server localhost/127.0.0.1:10406<http://127.0.0.1:10406>
2015-06-26 09:10:52,891 WARN [main-SendThread(localhost:10406)]
zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
0x14e2ffcb2390004 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:54,191 INFO [main-SendThread(localhost:10406)]
zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
socket connection to server localhost/127.0.0.1:10406<http://127.0.0.1:10406>
2015-06-26 09:10:54,192 WARN [main-SendThread(localhost:10406)]
zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
0x14e2ffcb2390004 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:54,471 INFO [main-SendThread(localhost:47046)]
zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
socket connection to server localhost/127.0.0.1:47046<http://127.0.0.1:47046>
2015-06-26 09:10:54,471 WARN [main-SendThread(localhost:47046)]
zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
0x14e2ffc86d80004 for server null, unexpected error, closing socket
connection and attempting reconnect
In the ZooKeeperServerMain.out I see lines like this repeating
several thousand times:
2015-06-26 08:39:44,810 INFO [SyncThread:0] server.NIOServerCnxn
(NIOServerCnxn.java:finishSessionInit(1580)) - Established session
0x14e2fe179eb0000 with negotiated timeout 30000 for client
/127.0.0.1:36311<http://127.0.0.1:36311>
2015-06-26 08:39:45,278 INFO [ProcessThread:-1]
server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
- Got user-level KeeperException when processing
sessionid:0x14e2fe179eb0000 type:create cxid:0x31
zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
Error:KeeperErrorCode = NodeExists for
/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
2015-06-26 08:39:45,300 INFO [ProcessThread:-1]
server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
- Got user-level KeeperException when processing
sessionid:0x14e2fe179eb0000 type:create cxid:0x33
zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
Error:KeeperErrorCode = NodeExists for
/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
I've thought about putting a Thread.sleep() call after the
MiniAccumuloCluster.stop() call, but that certainly seems brittle.
I'm not sure if that would improve the situation.
It seems to me that the MiniAccumuloCluster does not behave well when
instances are started and stopped several times. I am running the
tests through Maven using the default test behavior.
Could it be something with Maven? Maybe I need to be more explicit
when telling it how to run the tests?
Anyone have any insight into what is going wrong here? In general is
my usage pattern correct for MiniAccumuloCluster?
Thanks,
Jon Parise
Senior Software Engineer
Viz | General Dynamics Missons Systems
*This message and/or attachments may include information subject to
GD Corporate Policies 07-103 and 07-105 and is intended to be
accessed only by authorized recipients. Use, storage and transmission
are governed by General Dynamics and its policies. Contractual
restrictions apply to third parties. Recipients should refer to the
policies or contract to determine proper handling. Unauthorized
review, use, disclosure or distribution is prohibited. If you are not
an intended recipient, please contact the sender and destroy all
copies of the original message.*
Re: MiniAccumuloClutser Unit Test Problems
Posted by David Medinets <da...@gmail.com>.
How are the ports dynamically allocated? If five MAC are started at the
same time, could they be stepping on each other?
On Fri, Jun 26, 2015 at 11:36 AM, Josh Elser <jo...@gmail.com> wrote:
> Makes sense. Thanks for clarifying!
>
> Definitely send us a JIRA issue so we can track the issue and try to get
> to the bottom of it.
>
>
> Parise, Jonathan wrote:
>
>> Yes each test has a unique directory for the MAC.
>>
>> I use the following code to ensure that:
>>
>> storageDirectory_ = new File("atf-temp" + Path.SEPARATOR +
>> "accumulo-test-"
>> + UUID.randomUUID().toString());
>>
>> Basically this makes a new random UUID so that I get very unique paths.
>> All of the directories end up in atf-temp. This way I can clean them up
>> easily.
>>
>> Jon
>>
>> -----Original Message-----
>> From: Josh Elser [mailto:josh.elser@gmail.com]
>> Sent: Friday, June 26, 2015 10:50 AM
>> To: user@accumulo.apache.org
>> Subject: Re: MiniAccumuloClutser Unit Test Problems
>>
>> Did each test class use a unique directory for MAC?
>>
>> Assuming they're all unique, it sounds like a bug. I know some devs run
>> our suite of integration tests in parallel which use MAC out of the box.
>> If all of the above is true, please feel free to open up an issue on JIRA
>> and we can look into it more there.
>>
>> https://issues.apache.org/jira/secure/CreateIssue!default.jspa
>>
>> Thanks!
>>
>> Parise, Jonathan wrote:
>>
>>> Josh,
>>>
>>> Sorry for the confusion. I'll try and explain again with better
>>> terminology.
>>>
>>> I have several test classes, each of which contain several test methods.
>>> For each test class I have an @BeforeClass that configures and starts a
>>> MAC. I also have an @AfterClass that calls MAC.stop().
>>>
>>> So the flow is MAC is created, 1-n tests in the class run against it,
>>> MAC is stopped, repeat for next N tests.
>>>
>>> This way several @test methods in the same class use the same MAC. I
>>> mostly did this to make the tests run faster. I understand that having
>>> several test methods share a MAC could cause test state pollution, but I am
>>> careful to avoid that.
>>>
>>> The issues I was seeing is that if I simply ran "mvn test", the first
>>> few tests would pass and then eventually one of the tests would get stuck
>>> forever and just keep throwing connection errors.
>>>
>>> When I changed the maven configuration to force the tests to run
>>> serially, the issue stopped occurring.
>>>
>>> I'm not sure if there tests are expected to work when run in parallel.
>>> My best guess is that the tests may conflict with each other over ports or
>>> something like that.
>>>
>>> I'd like to understand why changing the test running behavior fixed this
>>> issue. Also, I think it would be good to document this somewhere. The MAC
>>> javadoc and also the user's guide should provide details about using the
>>> MAC for tests and properly configuring those tests.
>>>
>>> Does that make more sense?
>>>
>>> Jon
>>>
>>> -----Original Message-----
>>> From: Josh Elser [mailto:josh.elser@gmail.com]
>>> Sent: Friday, June 26, 2015 10:00 AM
>>> To: user@accumulo.apache.org
>>> Subject: Re: MiniAccumuloClutser Unit Test Problems
>>>
>>> Jonathan,
>>>
>>> If you're not seeing consistent behavior starting and stopping a
>>> MiniAccumuloCluster repeatedly, that's a bug. If you can provide a code
>>> which shows this problem, that'd be a huge help.
>>>
>>> If you can get a list of the processes running when you see this happen
>>> and cross-reference it with what processes should be running, that would
>>> also go a long way in trying to debug this.
>>>
>>> I am a little confused to your specific situation. You said that you
>>> construct and start a MAC instance in a BeforeClass and stop it in an
>>> AfterClass, but then you said that you start and stop it for each test.
>>> Are you saying that after the third construction and use of a MAC, you
>>> see problems? Or, are you saying that you stop and start each MAC instance
>>> before you run the @Test methods?
>>>
>>> - Josh
>>>
>>> Parise, Jonathan wrote:
>>>
>>>> Hello,
>>>>
>>>> I have been writing some J-unit tests based on the
>>>> MiniAccumuloCluster class. I'm experiencing some issues when several
>>>> of the tests run back to back. Before I get into the error, let me
>>>> explain how the tests work in general. Also, I am using Accumulo 1.6.2.
>>>>
>>>> Each test has an @BeforeClass method that first creates a new random
>>>> directory. Then makes a new MiniAccumuloCluster instance using that
>>>> directory as the dir parameter. Then, I call
>>>> MiniAccumuloCluster.start().
>>>>
>>>> There are several @Test methods in each test class. The typical
>>>> pattern for them is that they create any necessary tables, write some
>>>> data into those tables and then scan to verify it was written
>>>> correctly. Basically they are testing that I can serialize and
>>>> deserialize various types of Objects correctly.
>>>>
>>>> Then the test class as an @AfterClass method that calls
>>>> MiniAccumuloCluster.stop(). It also deletes the random directory used
>>>> by the previous test.
>>>>
>>>> The issue I am running into is that generally the first test or two
>>>> run fine. However, the third test usually gets stuck in the
>>>> MiniAccumuloCluster startup. It just keeps complaining about being
>>>> unable to connect. Note that if the test is run independently it
>>>> passes just fine. When run back to back, I see errors like this one
>>>> repeatedly:
>>>>
>>>> 2015-06-26 09:10:24,352 INFO [main-SendThread(localhost:47046)]
>>>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>>>> socket connection to server localhost/127.0.0.1:47046
>>>>
>>>> 2015-06-26 09:10:24,353 WARN [main-SendThread(localhost:47046)]
>>>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>>>> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
>>>> connection and attempting reconnect
>>>>
>>>> java.net.ConnectException: Connection refused
>>>>
>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>>
>>>> at
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
>>>> )
>>>>
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>>>
>>>> 2015-06-26 09:10:24,956 INFO [main-SendThread(localhost:10406)]
>>>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>>>> socket connection to server localhost/127.0.0.1:10406
>>>>
>>>> 2015-06-26 09:10:24,957 WARN [main-SendThread(localhost:10406)]
>>>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>>>> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
>>>> connection and attempting reconnect
>>>>
>>>> 2015-06-26 09:10:51,764 INFO [main-SendThread(localhost:10406)]
>>>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>>>> socket connection to server localhost/127.0.0.1:10406
>>>>
>>>> 2015-06-26 09:10:51,765 WARN [main-SendThread(localhost:10406)]
>>>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>>>> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
>>>> connection and attempting reconnect
>>>>
>>>> java.net.ConnectException: Connection refused
>>>>
>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>>
>>>> at
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
>>>> )
>>>>
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>>>
>>>> 2015-06-26 09:10:52,572 INFO [main-SendThread(localhost:47046)]
>>>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>>>> socket connection to server localhost/127.0.0.1:47046
>>>>
>>>> 2015-06-26 09:10:52,573 WARN [main-SendThread(localhost:47046)]
>>>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>>>> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
>>>> connection and attempting reconnect
>>>>
>>>> java.net.ConnectException: Connection refused
>>>>
>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>>
>>>> at
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
>>>> )
>>>>
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>>>
>>>> 2015-06-26 09:10:52,890 INFO [main-SendThread(localhost:10406)]
>>>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>>>> socket connection to server localhost/127.0.0.1:10406
>>>>
>>>> 2015-06-26 09:10:52,891 WARN [main-SendThread(localhost:10406)]
>>>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>>>> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
>>>> connection and attempting reconnect
>>>>
>>>> java.net.ConnectException: Connection refused
>>>>
>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>>
>>>> at
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
>>>> )
>>>>
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>>>
>>>> 2015-06-26 09:10:54,191 INFO [main-SendThread(localhost:10406)]
>>>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>>>> socket connection to server localhost/127.0.0.1:10406
>>>>
>>>> 2015-06-26 09:10:54,192 WARN [main-SendThread(localhost:10406)]
>>>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>>>> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
>>>> connection and attempting reconnect
>>>>
>>>> java.net.ConnectException: Connection refused
>>>>
>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>>
>>>> at
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
>>>> )
>>>>
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>>>
>>>> 2015-06-26 09:10:54,471 INFO [main-SendThread(localhost:47046)]
>>>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>>>> socket connection to server localhost/127.0.0.1:47046
>>>>
>>>> 2015-06-26 09:10:54,471 WARN [main-SendThread(localhost:47046)]
>>>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>>>> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
>>>> connection and attempting reconnect
>>>>
>>>> In the ZooKeeperServerMain.out I see lines like this repeating
>>>> several thousand times:
>>>>
>>>> 2015-06-26 08:39:44,810 INFO [SyncThread:0] server.NIOServerCnxn
>>>> (NIOServerCnxn.java:finishSessionInit(1580)) - Established session
>>>> 0x14e2fe179eb0000 with negotiated timeout 30000 for client
>>>> /127.0.0.1:36311
>>>>
>>>> 2015-06-26 08:39:45,278 INFO [ProcessThread:-1]
>>>> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
>>>> - Got user-level KeeperException when processing
>>>> sessionid:0x14e2fe179eb0000 type:create cxid:0x31
>>>> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
>>>> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
>>>> Error:KeeperErrorCode = NodeExists for
>>>> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
>>>>
>>>> 2015-06-26 08:39:45,300 INFO [ProcessThread:-1]
>>>> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
>>>> - Got user-level KeeperException when processing
>>>> sessionid:0x14e2fe179eb0000 type:create cxid:0x33
>>>> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
>>>> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
>>>> Error:KeeperErrorCode = NodeExists for
>>>> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
>>>>
>>>> I've thought about putting a Thread.sleep() call after the
>>>> MiniAccumuloCluster.stop() call, but that certainly seems brittle.
>>>> I'm not sure if that would improve the situation.
>>>>
>>>> It seems to me that the MiniAccumuloCluster does not behave well when
>>>> instances are started and stopped several times. I am running the
>>>> tests through Maven using the default test behavior.
>>>>
>>>> Could it be something with Maven? Maybe I need to be more explicit
>>>> when telling it how to run the tests?
>>>>
>>>> Anyone have any insight into what is going wrong here? In general is
>>>> my usage pattern correct for MiniAccumuloCluster?
>>>>
>>>> Thanks,
>>>>
>>>> Jon Parise
>>>>
>>>> Senior Software Engineer
>>>>
>>>> Viz | General Dynamics Missons Systems
>>>>
>>>> *This message and/or attachments may include information subject to
>>>> GD Corporate Policies 07-103 and 07-105 and is intended to be
>>>> accessed only by authorized recipients. Use, storage and transmission
>>>> are governed by General Dynamics and its policies. Contractual
>>>> restrictions apply to third parties. Recipients should refer to the
>>>> policies or contract to determine proper handling. Unauthorized
>>>> review, use, disclosure or distribution is prohibited. If you are not
>>>> an intended recipient, please contact the sender and destroy all
>>>> copies of the original message.*
>>>>
>>>>
Re: MiniAccumuloClutser Unit Test Problems
Posted by Josh Elser <jo...@gmail.com>.
Makes sense. Thanks for clarifying!
Definitely send us a JIRA issue so we can track the issue and try to get
to the bottom of it.
Parise, Jonathan wrote:
> Yes each test has a unique directory for the MAC.
>
> I use the following code to ensure that:
>
> storageDirectory_ = new File("atf-temp" + Path.SEPARATOR + "accumulo-test-"
> + UUID.randomUUID().toString());
>
> Basically this makes a new random UUID so that I get very unique paths. All of the directories end up in atf-temp. This way I can clean them up easily.
>
> Jon
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com]
> Sent: Friday, June 26, 2015 10:50 AM
> To: user@accumulo.apache.org
> Subject: Re: MiniAccumuloClutser Unit Test Problems
>
> Did each test class use a unique directory for MAC?
>
> Assuming they're all unique, it sounds like a bug. I know some devs run our suite of integration tests in parallel which use MAC out of the box.
> If all of the above is true, please feel free to open up an issue on JIRA and we can look into it more there.
>
> https://issues.apache.org/jira/secure/CreateIssue!default.jspa
>
> Thanks!
>
> Parise, Jonathan wrote:
>> Josh,
>>
>> Sorry for the confusion. I'll try and explain again with better terminology.
>>
>> I have several test classes, each of which contain several test methods. For each test class I have an @BeforeClass that configures and starts a MAC. I also have an @AfterClass that calls MAC.stop().
>>
>> So the flow is MAC is created, 1-n tests in the class run against it, MAC is stopped, repeat for next N tests.
>>
>> This way several @test methods in the same class use the same MAC. I mostly did this to make the tests run faster. I understand that having several test methods share a MAC could cause test state pollution, but I am careful to avoid that.
>>
>> The issues I was seeing is that if I simply ran "mvn test", the first few tests would pass and then eventually one of the tests would get stuck forever and just keep throwing connection errors.
>>
>> When I changed the maven configuration to force the tests to run serially, the issue stopped occurring.
>>
>> I'm not sure if there tests are expected to work when run in parallel. My best guess is that the tests may conflict with each other over ports or something like that.
>>
>> I'd like to understand why changing the test running behavior fixed this issue. Also, I think it would be good to document this somewhere. The MAC javadoc and also the user's guide should provide details about using the MAC for tests and properly configuring those tests.
>>
>> Does that make more sense?
>>
>> Jon
>>
>> -----Original Message-----
>> From: Josh Elser [mailto:josh.elser@gmail.com]
>> Sent: Friday, June 26, 2015 10:00 AM
>> To: user@accumulo.apache.org
>> Subject: Re: MiniAccumuloClutser Unit Test Problems
>>
>> Jonathan,
>>
>> If you're not seeing consistent behavior starting and stopping a MiniAccumuloCluster repeatedly, that's a bug. If you can provide a code which shows this problem, that'd be a huge help.
>>
>> If you can get a list of the processes running when you see this happen and cross-reference it with what processes should be running, that would also go a long way in trying to debug this.
>>
>> I am a little confused to your specific situation. You said that you construct and start a MAC instance in a BeforeClass and stop it in an AfterClass, but then you said that you start and stop it for each test.
>> Are you saying that after the third construction and use of a MAC, you see problems? Or, are you saying that you stop and start each MAC instance before you run the @Test methods?
>>
>> - Josh
>>
>> Parise, Jonathan wrote:
>>> Hello,
>>>
>>> I have been writing some J-unit tests based on the
>>> MiniAccumuloCluster class. I'm experiencing some issues when several
>>> of the tests run back to back. Before I get into the error, let me
>>> explain how the tests work in general. Also, I am using Accumulo 1.6.2.
>>>
>>> Each test has an @BeforeClass method that first creates a new random
>>> directory. Then makes a new MiniAccumuloCluster instance using that
>>> directory as the dir parameter. Then, I call MiniAccumuloCluster.start().
>>>
>>> There are several @Test methods in each test class. The typical
>>> pattern for them is that they create any necessary tables, write some
>>> data into those tables and then scan to verify it was written
>>> correctly. Basically they are testing that I can serialize and
>>> deserialize various types of Objects correctly.
>>>
>>> Then the test class as an @AfterClass method that calls
>>> MiniAccumuloCluster.stop(). It also deletes the random directory used
>>> by the previous test.
>>>
>>> The issue I am running into is that generally the first test or two
>>> run fine. However, the third test usually gets stuck in the
>>> MiniAccumuloCluster startup. It just keeps complaining about being
>>> unable to connect. Note that if the test is run independently it
>>> passes just fine. When run back to back, I see errors like this one repeatedly:
>>>
>>> 2015-06-26 09:10:24,352 INFO [main-SendThread(localhost:47046)]
>>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>>> socket connection to server localhost/127.0.0.1:47046
>>>
>>> 2015-06-26 09:10:24,353 WARN [main-SendThread(localhost:47046)]
>>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>>> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
>>> connection and attempting reconnect
>>>
>>> java.net.ConnectException: Connection refused
>>>
>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>
>>> at
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
>>> )
>>>
>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>>
>>> 2015-06-26 09:10:24,956 INFO [main-SendThread(localhost:10406)]
>>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>>> socket connection to server localhost/127.0.0.1:10406
>>>
>>> 2015-06-26 09:10:24,957 WARN [main-SendThread(localhost:10406)]
>>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>>> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
>>> connection and attempting reconnect
>>>
>>> 2015-06-26 09:10:51,764 INFO [main-SendThread(localhost:10406)]
>>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>>> socket connection to server localhost/127.0.0.1:10406
>>>
>>> 2015-06-26 09:10:51,765 WARN [main-SendThread(localhost:10406)]
>>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>>> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
>>> connection and attempting reconnect
>>>
>>> java.net.ConnectException: Connection refused
>>>
>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>
>>> at
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
>>> )
>>>
>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>>
>>> 2015-06-26 09:10:52,572 INFO [main-SendThread(localhost:47046)]
>>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>>> socket connection to server localhost/127.0.0.1:47046
>>>
>>> 2015-06-26 09:10:52,573 WARN [main-SendThread(localhost:47046)]
>>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>>> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
>>> connection and attempting reconnect
>>>
>>> java.net.ConnectException: Connection refused
>>>
>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>
>>> at
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
>>> )
>>>
>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>>
>>> 2015-06-26 09:10:52,890 INFO [main-SendThread(localhost:10406)]
>>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>>> socket connection to server localhost/127.0.0.1:10406
>>>
>>> 2015-06-26 09:10:52,891 WARN [main-SendThread(localhost:10406)]
>>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>>> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
>>> connection and attempting reconnect
>>>
>>> java.net.ConnectException: Connection refused
>>>
>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>
>>> at
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
>>> )
>>>
>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>>
>>> 2015-06-26 09:10:54,191 INFO [main-SendThread(localhost:10406)]
>>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>>> socket connection to server localhost/127.0.0.1:10406
>>>
>>> 2015-06-26 09:10:54,192 WARN [main-SendThread(localhost:10406)]
>>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>>> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
>>> connection and attempting reconnect
>>>
>>> java.net.ConnectException: Connection refused
>>>
>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>
>>> at
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
>>> )
>>>
>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>>
>>> 2015-06-26 09:10:54,471 INFO [main-SendThread(localhost:47046)]
>>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>>> socket connection to server localhost/127.0.0.1:47046
>>>
>>> 2015-06-26 09:10:54,471 WARN [main-SendThread(localhost:47046)]
>>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>>> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
>>> connection and attempting reconnect
>>>
>>> In the ZooKeeperServerMain.out I see lines like this repeating
>>> several thousand times:
>>>
>>> 2015-06-26 08:39:44,810 INFO [SyncThread:0] server.NIOServerCnxn
>>> (NIOServerCnxn.java:finishSessionInit(1580)) - Established session
>>> 0x14e2fe179eb0000 with negotiated timeout 30000 for client
>>> /127.0.0.1:36311
>>>
>>> 2015-06-26 08:39:45,278 INFO [ProcessThread:-1]
>>> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
>>> - Got user-level KeeperException when processing
>>> sessionid:0x14e2fe179eb0000 type:create cxid:0x31
>>> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
>>> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
>>> Error:KeeperErrorCode = NodeExists for
>>> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
>>>
>>> 2015-06-26 08:39:45,300 INFO [ProcessThread:-1]
>>> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
>>> - Got user-level KeeperException when processing
>>> sessionid:0x14e2fe179eb0000 type:create cxid:0x33
>>> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
>>> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
>>> Error:KeeperErrorCode = NodeExists for
>>> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
>>>
>>> I've thought about putting a Thread.sleep() call after the
>>> MiniAccumuloCluster.stop() call, but that certainly seems brittle.
>>> I'm not sure if that would improve the situation.
>>>
>>> It seems to me that the MiniAccumuloCluster does not behave well when
>>> instances are started and stopped several times. I am running the
>>> tests through Maven using the default test behavior.
>>>
>>> Could it be something with Maven? Maybe I need to be more explicit
>>> when telling it how to run the tests?
>>>
>>> Anyone have any insight into what is going wrong here? In general is
>>> my usage pattern correct for MiniAccumuloCluster?
>>>
>>> Thanks,
>>>
>>> Jon Parise
>>>
>>> Senior Software Engineer
>>>
>>> Viz | General Dynamics Missons Systems
>>>
>>> *This message and/or attachments may include information subject to
>>> GD Corporate Policies 07-103 and 07-105 and is intended to be
>>> accessed only by authorized recipients. Use, storage and transmission
>>> are governed by General Dynamics and its policies. Contractual
>>> restrictions apply to third parties. Recipients should refer to the
>>> policies or contract to determine proper handling. Unauthorized
>>> review, use, disclosure or distribution is prohibited. If you are not
>>> an intended recipient, please contact the sender and destroy all
>>> copies of the original message.*
>>>
RE: MiniAccumuloClutser Unit Test Problems
Posted by "Parise, Jonathan" <Jo...@gd-ms.com>.
Yes each test has a unique directory for the MAC.
I use the following code to ensure that:
storageDirectory_ = new File("atf-temp" + Path.SEPARATOR + "accumulo-test-"
+ UUID.randomUUID().toString());
Basically this makes a new random UUID so that I get very unique paths. All of the directories end up in atf-temp. This way I can clean them up easily.
Jon
-----Original Message-----
From: Josh Elser [mailto:josh.elser@gmail.com]
Sent: Friday, June 26, 2015 10:50 AM
To: user@accumulo.apache.org
Subject: Re: MiniAccumuloClutser Unit Test Problems
Did each test class use a unique directory for MAC?
Assuming they're all unique, it sounds like a bug. I know some devs run our suite of integration tests in parallel which use MAC out of the box.
If all of the above is true, please feel free to open up an issue on JIRA and we can look into it more there.
https://issues.apache.org/jira/secure/CreateIssue!default.jspa
Thanks!
Parise, Jonathan wrote:
> Josh,
>
> Sorry for the confusion. I'll try and explain again with better terminology.
>
> I have several test classes, each of which contain several test methods. For each test class I have an @BeforeClass that configures and starts a MAC. I also have an @AfterClass that calls MAC.stop().
>
> So the flow is MAC is created, 1-n tests in the class run against it, MAC is stopped, repeat for next N tests.
>
> This way several @test methods in the same class use the same MAC. I mostly did this to make the tests run faster. I understand that having several test methods share a MAC could cause test state pollution, but I am careful to avoid that.
>
> The issues I was seeing is that if I simply ran "mvn test", the first few tests would pass and then eventually one of the tests would get stuck forever and just keep throwing connection errors.
>
> When I changed the maven configuration to force the tests to run serially, the issue stopped occurring.
>
> I'm not sure if there tests are expected to work when run in parallel. My best guess is that the tests may conflict with each other over ports or something like that.
>
> I'd like to understand why changing the test running behavior fixed this issue. Also, I think it would be good to document this somewhere. The MAC javadoc and also the user's guide should provide details about using the MAC for tests and properly configuring those tests.
>
> Does that make more sense?
>
> Jon
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com]
> Sent: Friday, June 26, 2015 10:00 AM
> To: user@accumulo.apache.org
> Subject: Re: MiniAccumuloClutser Unit Test Problems
>
> Jonathan,
>
> If you're not seeing consistent behavior starting and stopping a MiniAccumuloCluster repeatedly, that's a bug. If you can provide a code which shows this problem, that'd be a huge help.
>
> If you can get a list of the processes running when you see this happen and cross-reference it with what processes should be running, that would also go a long way in trying to debug this.
>
> I am a little confused to your specific situation. You said that you construct and start a MAC instance in a BeforeClass and stop it in an AfterClass, but then you said that you start and stop it for each test.
> Are you saying that after the third construction and use of a MAC, you see problems? Or, are you saying that you stop and start each MAC instance before you run the @Test methods?
>
> - Josh
>
> Parise, Jonathan wrote:
>> Hello,
>>
>> I have been writing some J-unit tests based on the
>> MiniAccumuloCluster class. I'm experiencing some issues when several
>> of the tests run back to back. Before I get into the error, let me
>> explain how the tests work in general. Also, I am using Accumulo 1.6.2.
>>
>> Each test has an @BeforeClass method that first creates a new random
>> directory. Then makes a new MiniAccumuloCluster instance using that
>> directory as the dir parameter. Then, I call MiniAccumuloCluster.start().
>>
>> There are several @Test methods in each test class. The typical
>> pattern for them is that they create any necessary tables, write some
>> data into those tables and then scan to verify it was written
>> correctly. Basically they are testing that I can serialize and
>> deserialize various types of Objects correctly.
>>
>> Then the test class as an @AfterClass method that calls
>> MiniAccumuloCluster.stop(). It also deletes the random directory used
>> by the previous test.
>>
>> The issue I am running into is that generally the first test or two
>> run fine. However, the third test usually gets stuck in the
>> MiniAccumuloCluster startup. It just keeps complaining about being
>> unable to connect. Note that if the test is run independently it
>> passes just fine. When run back to back, I see errors like this one repeatedly:
>>
>> 2015-06-26 09:10:24,352 INFO [main-SendThread(localhost:47046)]
>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>> socket connection to server localhost/127.0.0.1:47046
>>
>> 2015-06-26 09:10:24,353 WARN [main-SendThread(localhost:47046)]
>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
>> connection and attempting reconnect
>>
>> java.net.ConnectException: Connection refused
>>
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
>> )
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>
>> 2015-06-26 09:10:24,956 INFO [main-SendThread(localhost:10406)]
>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>> socket connection to server localhost/127.0.0.1:10406
>>
>> 2015-06-26 09:10:24,957 WARN [main-SendThread(localhost:10406)]
>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
>> connection and attempting reconnect
>>
>> 2015-06-26 09:10:51,764 INFO [main-SendThread(localhost:10406)]
>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>> socket connection to server localhost/127.0.0.1:10406
>>
>> 2015-06-26 09:10:51,765 WARN [main-SendThread(localhost:10406)]
>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
>> connection and attempting reconnect
>>
>> java.net.ConnectException: Connection refused
>>
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
>> )
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>
>> 2015-06-26 09:10:52,572 INFO [main-SendThread(localhost:47046)]
>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>> socket connection to server localhost/127.0.0.1:47046
>>
>> 2015-06-26 09:10:52,573 WARN [main-SendThread(localhost:47046)]
>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
>> connection and attempting reconnect
>>
>> java.net.ConnectException: Connection refused
>>
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
>> )
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>
>> 2015-06-26 09:10:52,890 INFO [main-SendThread(localhost:10406)]
>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>> socket connection to server localhost/127.0.0.1:10406
>>
>> 2015-06-26 09:10:52,891 WARN [main-SendThread(localhost:10406)]
>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
>> connection and attempting reconnect
>>
>> java.net.ConnectException: Connection refused
>>
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
>> )
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>
>> 2015-06-26 09:10:54,191 INFO [main-SendThread(localhost:10406)]
>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>> socket connection to server localhost/127.0.0.1:10406
>>
>> 2015-06-26 09:10:54,192 WARN [main-SendThread(localhost:10406)]
>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
>> connection and attempting reconnect
>>
>> java.net.ConnectException: Connection refused
>>
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
>> )
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>
>> 2015-06-26 09:10:54,471 INFO [main-SendThread(localhost:47046)]
>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>> socket connection to server localhost/127.0.0.1:47046
>>
>> 2015-06-26 09:10:54,471 WARN [main-SendThread(localhost:47046)]
>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
>> connection and attempting reconnect
>>
>> In the ZooKeeperServerMain.out I see lines like this repeating
>> several thousand times:
>>
>> 2015-06-26 08:39:44,810 INFO [SyncThread:0] server.NIOServerCnxn
>> (NIOServerCnxn.java:finishSessionInit(1580)) - Established session
>> 0x14e2fe179eb0000 with negotiated timeout 30000 for client
>> /127.0.0.1:36311
>>
>> 2015-06-26 08:39:45,278 INFO [ProcessThread:-1]
>> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
>> - Got user-level KeeperException when processing
>> sessionid:0x14e2fe179eb0000 type:create cxid:0x31
>> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
>> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
>> Error:KeeperErrorCode = NodeExists for
>> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
>>
>> 2015-06-26 08:39:45,300 INFO [ProcessThread:-1]
>> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
>> - Got user-level KeeperException when processing
>> sessionid:0x14e2fe179eb0000 type:create cxid:0x33
>> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
>> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
>> Error:KeeperErrorCode = NodeExists for
>> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
>>
>> I've thought about putting a Thread.sleep() call after the
>> MiniAccumuloCluster.stop() call, but that certainly seems brittle.
>> I'm not sure if that would improve the situation.
>>
>> It seems to me that the MiniAccumuloCluster does not behave well when
>> instances are started and stopped several times. I am running the
>> tests through Maven using the default test behavior.
>>
>> Could it be something with Maven? Maybe I need to be more explicit
>> when telling it how to run the tests?
>>
>> Anyone have any insight into what is going wrong here? In general is
>> my usage pattern correct for MiniAccumuloCluster?
>>
>> Thanks,
>>
>> Jon Parise
>>
>> Senior Software Engineer
>>
>> Viz | General Dynamics Missons Systems
>>
>> *This message and/or attachments may include information subject to
>> GD Corporate Policies 07-103 and 07-105 and is intended to be
>> accessed only by authorized recipients. Use, storage and transmission
>> are governed by General Dynamics and its policies. Contractual
>> restrictions apply to third parties. Recipients should refer to the
>> policies or contract to determine proper handling. Unauthorized
>> review, use, disclosure or distribution is prohibited. If you are not
>> an intended recipient, please contact the sender and destroy all
>> copies of the original message.*
>>
Re: MiniAccumuloClutser Unit Test Problems
Posted by Josh Elser <jo...@gmail.com>.
Did each test class use a unique directory for MAC?
Assuming they're all unique, it sounds like a bug. I know some devs run
our suite of integration tests in parallel which use MAC out of the box.
If all of the above is true, please feel free to open up an issue on
JIRA and we can look into it more there.
https://issues.apache.org/jira/secure/CreateIssue!default.jspa
Thanks!
Parise, Jonathan wrote:
> Josh,
>
> Sorry for the confusion. I'll try and explain again with better terminology.
>
> I have several test classes, each of which contain several test methods. For each test class I have an @BeforeClass that configures and starts a MAC. I also have an @AfterClass that calls MAC.stop().
>
> So the flow is MAC is created, 1-n tests in the class run against it, MAC is stopped, repeat for next N tests.
>
> This way several @test methods in the same class use the same MAC. I mostly did this to make the tests run faster. I understand that having several test methods share a MAC could cause test state pollution, but I am careful to avoid that.
>
> The issues I was seeing is that if I simply ran "mvn test", the first few tests would pass and then eventually one of the tests would get stuck forever and just keep throwing connection errors.
>
> When I changed the maven configuration to force the tests to run serially, the issue stopped occurring.
>
> I'm not sure if there tests are expected to work when run in parallel. My best guess is that the tests may conflict with each other over ports or something like that.
>
> I'd like to understand why changing the test running behavior fixed this issue. Also, I think it would be good to document this somewhere. The MAC javadoc and also the user's guide should provide details about using the MAC for tests and properly configuring those tests.
>
> Does that make more sense?
>
> Jon
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com]
> Sent: Friday, June 26, 2015 10:00 AM
> To: user@accumulo.apache.org
> Subject: Re: MiniAccumuloClutser Unit Test Problems
>
> Jonathan,
>
> If you're not seeing consistent behavior starting and stopping a MiniAccumuloCluster repeatedly, that's a bug. If you can provide a code which shows this problem, that'd be a huge help.
>
> If you can get a list of the processes running when you see this happen and cross-reference it with what processes should be running, that would also go a long way in trying to debug this.
>
> I am a little confused to your specific situation. You said that you construct and start a MAC instance in a BeforeClass and stop it in an AfterClass, but then you said that you start and stop it for each test.
> Are you saying that after the third construction and use of a MAC, you see problems? Or, are you saying that you stop and start each MAC instance before you run the @Test methods?
>
> - Josh
>
> Parise, Jonathan wrote:
>> Hello,
>>
>> I have been writing some J-unit tests based on the MiniAccumuloCluster
>> class. I'm experiencing some issues when several of the tests run back
>> to back. Before I get into the error, let me explain how the tests
>> work in general. Also, I am using Accumulo 1.6.2.
>>
>> Each test has an @BeforeClass method that first creates a new random
>> directory. Then makes a new MiniAccumuloCluster instance using that
>> directory as the dir parameter. Then, I call MiniAccumuloCluster.start().
>>
>> There are several @Test methods in each test class. The typical
>> pattern for them is that they create any necessary tables, write some
>> data into those tables and then scan to verify it was written
>> correctly. Basically they are testing that I can serialize and
>> deserialize various types of Objects correctly.
>>
>> Then the test class as an @AfterClass method that calls
>> MiniAccumuloCluster.stop(). It also deletes the random directory used
>> by the previous test.
>>
>> The issue I am running into is that generally the first test or two
>> run fine. However, the third test usually gets stuck in the
>> MiniAccumuloCluster startup. It just keeps complaining about being
>> unable to connect. Note that if the test is run independently it
>> passes just fine. When run back to back, I see errors like this one repeatedly:
>>
>> 2015-06-26 09:10:24,352 INFO [main-SendThread(localhost:47046)]
>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>> socket connection to server localhost/127.0.0.1:47046
>>
>> 2015-06-26 09:10:24,353 WARN [main-SendThread(localhost:47046)]
>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
>> connection and attempting reconnect
>>
>> java.net.ConnectException: Connection refused
>>
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>
>> 2015-06-26 09:10:24,956 INFO [main-SendThread(localhost:10406)]
>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>> socket connection to server localhost/127.0.0.1:10406
>>
>> 2015-06-26 09:10:24,957 WARN [main-SendThread(localhost:10406)]
>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
>> connection and attempting reconnect
>>
>> 2015-06-26 09:10:51,764 INFO [main-SendThread(localhost:10406)]
>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>> socket connection to server localhost/127.0.0.1:10406
>>
>> 2015-06-26 09:10:51,765 WARN [main-SendThread(localhost:10406)]
>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
>> connection and attempting reconnect
>>
>> java.net.ConnectException: Connection refused
>>
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>
>> 2015-06-26 09:10:52,572 INFO [main-SendThread(localhost:47046)]
>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>> socket connection to server localhost/127.0.0.1:47046
>>
>> 2015-06-26 09:10:52,573 WARN [main-SendThread(localhost:47046)]
>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
>> connection and attempting reconnect
>>
>> java.net.ConnectException: Connection refused
>>
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>
>> 2015-06-26 09:10:52,890 INFO [main-SendThread(localhost:10406)]
>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>> socket connection to server localhost/127.0.0.1:10406
>>
>> 2015-06-26 09:10:52,891 WARN [main-SendThread(localhost:10406)]
>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
>> connection and attempting reconnect
>>
>> java.net.ConnectException: Connection refused
>>
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>
>> 2015-06-26 09:10:54,191 INFO [main-SendThread(localhost:10406)]
>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>> socket connection to server localhost/127.0.0.1:10406
>>
>> 2015-06-26 09:10:54,192 WARN [main-SendThread(localhost:10406)]
>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
>> connection and attempting reconnect
>>
>> java.net.ConnectException: Connection refused
>>
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>>
>> 2015-06-26 09:10:54,471 INFO [main-SendThread(localhost:47046)]
>> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
>> socket connection to server localhost/127.0.0.1:47046
>>
>> 2015-06-26 09:10:54,471 WARN [main-SendThread(localhost:47046)]
>> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
>> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
>> connection and attempting reconnect
>>
>> In the ZooKeeperServerMain.out I see lines like this repeating several
>> thousand times:
>>
>> 2015-06-26 08:39:44,810 INFO [SyncThread:0] server.NIOServerCnxn
>> (NIOServerCnxn.java:finishSessionInit(1580)) - Established session
>> 0x14e2fe179eb0000 with negotiated timeout 30000 for client
>> /127.0.0.1:36311
>>
>> 2015-06-26 08:39:45,278 INFO [ProcessThread:-1]
>> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
>> - Got user-level KeeperException when processing
>> sessionid:0x14e2fe179eb0000 type:create cxid:0x31
>> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
>> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
>> Error:KeeperErrorCode = NodeExists for
>> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
>>
>> 2015-06-26 08:39:45,300 INFO [ProcessThread:-1]
>> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
>> - Got user-level KeeperException when processing
>> sessionid:0x14e2fe179eb0000 type:create cxid:0x33
>> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
>> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
>> Error:KeeperErrorCode = NodeExists for
>> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
>>
>> I've thought about putting a Thread.sleep() call after the
>> MiniAccumuloCluster.stop() call, but that certainly seems brittle. I'm
>> not sure if that would improve the situation.
>>
>> It seems to me that the MiniAccumuloCluster does not behave well when
>> instances are started and stopped several times. I am running the
>> tests through Maven using the default test behavior.
>>
>> Could it be something with Maven? Maybe I need to be more explicit
>> when telling it how to run the tests?
>>
>> Anyone have any insight into what is going wrong here? In general is
>> my usage pattern correct for MiniAccumuloCluster?
>>
>> Thanks,
>>
>> Jon Parise
>>
>> Senior Software Engineer
>>
>> Viz | General Dynamics Missons Systems
>>
>> *This message and/or attachments may include information subject to GD
>> Corporate Policies 07-103 and 07-105 and is intended to be accessed
>> only by authorized recipients. Use, storage and transmission are
>> governed by General Dynamics and its policies. Contractual
>> restrictions apply to third parties. Recipients should refer to the
>> policies or contract to determine proper handling. Unauthorized
>> review, use, disclosure or distribution is prohibited. If you are not
>> an intended recipient, please contact the sender and destroy all
>> copies of the original message.*
>>
RE: MiniAccumuloClutser Unit Test Problems
Posted by "Parise, Jonathan" <Jo...@gd-ms.com>.
Yes I think that is correct.
When I changed the pom like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18</version>
<configuration>
<useUnlimitedThreads>true</useUnlimitedThreads>
<parallel>suites</parallel>
<reuseForks>false</reuseForks>
<includes>
<include>**/*Test.java</include>
<include>**/Test*.java</include>
</includes>
</configuration>
</plugin>
That is telling maven to not run them in parallel anymore, since they are all defined in the same suite.
With that change, my MAC based tests all started passing consistently.
Also, the error case I was seeing was basically the test getting stuck. It wasn’t failing, it was just spinning forever.
Jon
From: Sean Busbey [mailto:busbey@cloudera.com]
Sent: Friday, June 26, 2015 10:30 AM
To: Accumulo User List
Subject: Re: MiniAccumuloClutser Unit Test Problems
So the problem is with tests for different classes, with a MAC per class, running in parallel? if so, that sounds like a bug.
On Fri, Jun 26, 2015 at 9:14 AM, Parise, Jonathan <Jo...@gd-ms.com>> wrote:
Josh,
Sorry for the confusion. I'll try and explain again with better terminology.
I have several test classes, each of which contain several test methods. For each test class I have an @BeforeClass that configures and starts a MAC. I also have an @AfterClass that calls MAC.stop().
So the flow is MAC is created, 1-n tests in the class run against it, MAC is stopped, repeat for next N tests.
This way several @test methods in the same class use the same MAC. I mostly did this to make the tests run faster. I understand that having several test methods share a MAC could cause test state pollution, but I am careful to avoid that.
The issues I was seeing is that if I simply ran "mvn test", the first few tests would pass and then eventually one of the tests would get stuck forever and just keep throwing connection errors.
When I changed the maven configuration to force the tests to run serially, the issue stopped occurring.
I'm not sure if there tests are expected to work when run in parallel. My best guess is that the tests may conflict with each other over ports or something like that.
I'd like to understand why changing the test running behavior fixed this issue. Also, I think it would be good to document this somewhere. The MAC javadoc and also the user's guide should provide details about using the MAC for tests and properly configuring those tests.
Does that make more sense?
Jon
-----Original Message-----
From: Josh Elser [mailto:josh.elser@gmail.com<ma...@gmail.com>]
Sent: Friday, June 26, 2015 10:00 AM
To: user@accumulo.apache.org<ma...@accumulo.apache.org>
Subject: Re: MiniAccumuloClutser Unit Test Problems
Jonathan,
If you're not seeing consistent behavior starting and stopping a MiniAccumuloCluster repeatedly, that's a bug. If you can provide a code which shows this problem, that'd be a huge help.
If you can get a list of the processes running when you see this happen and cross-reference it with what processes should be running, that would also go a long way in trying to debug this.
I am a little confused to your specific situation. You said that you construct and start a MAC instance in a BeforeClass and stop it in an AfterClass, but then you said that you start and stop it for each test.
Are you saying that after the third construction and use of a MAC, you see problems? Or, are you saying that you stop and start each MAC instance before you run the @Test methods?
- Josh
Parise, Jonathan wrote:
> Hello,
>
> I have been writing some J-unit tests based on the MiniAccumuloCluster
> class. I'm experiencing some issues when several of the tests run back
> to back. Before I get into the error, let me explain how the tests
> work in general. Also, I am using Accumulo 1.6.2.
>
> Each test has an @BeforeClass method that first creates a new random
> directory. Then makes a new MiniAccumuloCluster instance using that
> directory as the dir parameter. Then, I call MiniAccumuloCluster.start().
>
> There are several @Test methods in each test class. The typical
> pattern for them is that they create any necessary tables, write some
> data into those tables and then scan to verify it was written
> correctly. Basically they are testing that I can serialize and
> deserialize various types of Objects correctly.
>
> Then the test class as an @AfterClass method that calls
> MiniAccumuloCluster.stop(). It also deletes the random directory used
> by the previous test.
>
> The issue I am running into is that generally the first test or two
> run fine. However, the third test usually gets stuck in the
> MiniAccumuloCluster startup. It just keeps complaining about being
> unable to connect. Note that if the test is run independently it
> passes just fine. When run back to back, I see errors like this one repeatedly:
>
> 2015-06-26 09:10:24,352 INFO [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:47046<http://127.0.0.1:47046>
>
> 2015-06-26 09:10:24,353 WARN [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:24,956 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406<http://127.0.0.1:10406>
>
> 2015-06-26 09:10:24,957 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> 2015-06-26 09:10:51,764 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406<http://127.0.0.1:10406>
>
> 2015-06-26 09:10:51,765 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:52,572 INFO [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:47046<http://127.0.0.1:47046>
>
> 2015-06-26 09:10:52,573 WARN [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:52,890 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406<http://127.0.0.1:10406>
>
> 2015-06-26 09:10:52,891 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:54,191 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406<http://127.0.0.1:10406>
>
> 2015-06-26 09:10:54,192 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:54,471 INFO [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:47046<http://127.0.0.1:47046>
>
> 2015-06-26 09:10:54,471 WARN [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> In the ZooKeeperServerMain.out I see lines like this repeating several
> thousand times:
>
> 2015-06-26 08:39:44,810 INFO [SyncThread:0] server.NIOServerCnxn
> (NIOServerCnxn.java:finishSessionInit(1580)) - Established session
> 0x14e2fe179eb0000 with negotiated timeout 30000 for client
> /127.0.0.1:36311<http://127.0.0.1:36311>
>
> 2015-06-26 08:39:45,278 INFO [ProcessThread:-1]
> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
> - Got user-level KeeperException when processing
> sessionid:0x14e2fe179eb0000 type:create cxid:0x31
> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
> Error:KeeperErrorCode = NodeExists for
> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
>
> 2015-06-26 08:39:45,300 INFO [ProcessThread:-1]
> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
> - Got user-level KeeperException when processing
> sessionid:0x14e2fe179eb0000 type:create cxid:0x33
> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
> Error:KeeperErrorCode = NodeExists for
> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
>
> I've thought about putting a Thread.sleep() call after the
> MiniAccumuloCluster.stop() call, but that certainly seems brittle. I'm
> not sure if that would improve the situation.
>
> It seems to me that the MiniAccumuloCluster does not behave well when
> instances are started and stopped several times. I am running the
> tests through Maven using the default test behavior.
>
> Could it be something with Maven? Maybe I need to be more explicit
> when telling it how to run the tests?
>
> Anyone have any insight into what is going wrong here? In general is
> my usage pattern correct for MiniAccumuloCluster?
>
> Thanks,
>
> Jon Parise
>
> Senior Software Engineer
>
> Viz | General Dynamics Missons Systems
>
> *This message and/or attachments may include information subject to GD
> Corporate Policies 07-103 and 07-105 and is intended to be accessed
> only by authorized recipients. Use, storage and transmission are
> governed by General Dynamics and its policies. Contractual
> restrictions apply to third parties. Recipients should refer to the
> policies or contract to determine proper handling. Unauthorized
> review, use, disclosure or distribution is prohibited. If you are not
> an intended recipient, please contact the sender and destroy all
> copies of the original message.*
>
--
Sean
Re: MiniAccumuloClutser Unit Test Problems
Posted by Sean Busbey <bu...@cloudera.com>.
So the problem is with tests for different classes, with a MAC per class,
running in parallel? if so, that sounds like a bug.
On Fri, Jun 26, 2015 at 9:14 AM, Parise, Jonathan <Jonathan.Parise@gd-ms.com
> wrote:
> Josh,
>
> Sorry for the confusion. I'll try and explain again with better
> terminology.
>
> I have several test classes, each of which contain several test methods.
> For each test class I have an @BeforeClass that configures and starts a
> MAC. I also have an @AfterClass that calls MAC.stop().
>
> So the flow is MAC is created, 1-n tests in the class run against it, MAC
> is stopped, repeat for next N tests.
>
> This way several @test methods in the same class use the same MAC. I
> mostly did this to make the tests run faster. I understand that having
> several test methods share a MAC could cause test state pollution, but I am
> careful to avoid that.
>
> The issues I was seeing is that if I simply ran "mvn test", the first few
> tests would pass and then eventually one of the tests would get stuck
> forever and just keep throwing connection errors.
>
> When I changed the maven configuration to force the tests to run serially,
> the issue stopped occurring.
>
> I'm not sure if there tests are expected to work when run in parallel. My
> best guess is that the tests may conflict with each other over ports or
> something like that.
>
> I'd like to understand why changing the test running behavior fixed this
> issue. Also, I think it would be good to document this somewhere. The MAC
> javadoc and also the user's guide should provide details about using the
> MAC for tests and properly configuring those tests.
>
> Does that make more sense?
>
> Jon
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com]
> Sent: Friday, June 26, 2015 10:00 AM
> To: user@accumulo.apache.org
> Subject: Re: MiniAccumuloClutser Unit Test Problems
>
> Jonathan,
>
> If you're not seeing consistent behavior starting and stopping a
> MiniAccumuloCluster repeatedly, that's a bug. If you can provide a code
> which shows this problem, that'd be a huge help.
>
> If you can get a list of the processes running when you see this happen
> and cross-reference it with what processes should be running, that would
> also go a long way in trying to debug this.
>
> I am a little confused to your specific situation. You said that you
> construct and start a MAC instance in a BeforeClass and stop it in an
> AfterClass, but then you said that you start and stop it for each test.
> Are you saying that after the third construction and use of a MAC, you see
> problems? Or, are you saying that you stop and start each MAC instance
> before you run the @Test methods?
>
> - Josh
>
> Parise, Jonathan wrote:
> > Hello,
> >
> > I have been writing some J-unit tests based on the MiniAccumuloCluster
> > class. I'm experiencing some issues when several of the tests run back
> > to back. Before I get into the error, let me explain how the tests
> > work in general. Also, I am using Accumulo 1.6.2.
> >
> > Each test has an @BeforeClass method that first creates a new random
> > directory. Then makes a new MiniAccumuloCluster instance using that
> > directory as the dir parameter. Then, I call MiniAccumuloCluster.start().
> >
> > There are several @Test methods in each test class. The typical
> > pattern for them is that they create any necessary tables, write some
> > data into those tables and then scan to verify it was written
> > correctly. Basically they are testing that I can serialize and
> > deserialize various types of Objects correctly.
> >
> > Then the test class as an @AfterClass method that calls
> > MiniAccumuloCluster.stop(). It also deletes the random directory used
> > by the previous test.
> >
> > The issue I am running into is that generally the first test or two
> > run fine. However, the third test usually gets stuck in the
> > MiniAccumuloCluster startup. It just keeps complaining about being
> > unable to connect. Note that if the test is run independently it
> > passes just fine. When run back to back, I see errors like this one
> repeatedly:
> >
> > 2015-06-26 09:10:24,352 INFO [main-SendThread(localhost:47046)]
> > zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> > socket connection to server localhost/127.0.0.1:47046
> >
> > 2015-06-26 09:10:24,353 WARN [main-SendThread(localhost:47046)]
> > zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> > 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> > connection and attempting reconnect
> >
> > java.net.ConnectException: Connection refused
> >
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> >
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
> >
> > 2015-06-26 09:10:24,956 INFO [main-SendThread(localhost:10406)]
> > zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> > socket connection to server localhost/127.0.0.1:10406
> >
> > 2015-06-26 09:10:24,957 WARN [main-SendThread(localhost:10406)]
> > zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> > 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> > connection and attempting reconnect
> >
> > 2015-06-26 09:10:51,764 INFO [main-SendThread(localhost:10406)]
> > zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> > socket connection to server localhost/127.0.0.1:10406
> >
> > 2015-06-26 09:10:51,765 WARN [main-SendThread(localhost:10406)]
> > zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> > 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> > connection and attempting reconnect
> >
> > java.net.ConnectException: Connection refused
> >
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> >
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
> >
> > 2015-06-26 09:10:52,572 INFO [main-SendThread(localhost:47046)]
> > zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> > socket connection to server localhost/127.0.0.1:47046
> >
> > 2015-06-26 09:10:52,573 WARN [main-SendThread(localhost:47046)]
> > zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> > 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> > connection and attempting reconnect
> >
> > java.net.ConnectException: Connection refused
> >
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> >
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
> >
> > 2015-06-26 09:10:52,890 INFO [main-SendThread(localhost:10406)]
> > zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> > socket connection to server localhost/127.0.0.1:10406
> >
> > 2015-06-26 09:10:52,891 WARN [main-SendThread(localhost:10406)]
> > zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> > 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> > connection and attempting reconnect
> >
> > java.net.ConnectException: Connection refused
> >
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> >
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
> >
> > 2015-06-26 09:10:54,191 INFO [main-SendThread(localhost:10406)]
> > zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> > socket connection to server localhost/127.0.0.1:10406
> >
> > 2015-06-26 09:10:54,192 WARN [main-SendThread(localhost:10406)]
> > zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> > 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> > connection and attempting reconnect
> >
> > java.net.ConnectException: Connection refused
> >
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> >
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
> >
> > 2015-06-26 09:10:54,471 INFO [main-SendThread(localhost:47046)]
> > zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> > socket connection to server localhost/127.0.0.1:47046
> >
> > 2015-06-26 09:10:54,471 WARN [main-SendThread(localhost:47046)]
> > zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> > 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> > connection and attempting reconnect
> >
> > In the ZooKeeperServerMain.out I see lines like this repeating several
> > thousand times:
> >
> > 2015-06-26 08:39:44,810 INFO [SyncThread:0] server.NIOServerCnxn
> > (NIOServerCnxn.java:finishSessionInit(1580)) - Established session
> > 0x14e2fe179eb0000 with negotiated timeout 30000 for client
> > /127.0.0.1:36311
> >
> > 2015-06-26 08:39:45,278 INFO [ProcessThread:-1]
> > server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
> > - Got user-level KeeperException when processing
> > sessionid:0x14e2fe179eb0000 type:create cxid:0x31
> > zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> > Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
> > Error:KeeperErrorCode = NodeExists for
> > /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
> >
> > 2015-06-26 08:39:45,300 INFO [ProcessThread:-1]
> > server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
> > - Got user-level KeeperException when processing
> > sessionid:0x14e2fe179eb0000 type:create cxid:0x33
> > zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> > Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
> > Error:KeeperErrorCode = NodeExists for
> > /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
> >
> > I've thought about putting a Thread.sleep() call after the
> > MiniAccumuloCluster.stop() call, but that certainly seems brittle. I'm
> > not sure if that would improve the situation.
> >
> > It seems to me that the MiniAccumuloCluster does not behave well when
> > instances are started and stopped several times. I am running the
> > tests through Maven using the default test behavior.
> >
> > Could it be something with Maven? Maybe I need to be more explicit
> > when telling it how to run the tests?
> >
> > Anyone have any insight into what is going wrong here? In general is
> > my usage pattern correct for MiniAccumuloCluster?
> >
> > Thanks,
> >
> > Jon Parise
> >
> > Senior Software Engineer
> >
> > Viz | General Dynamics Missons Systems
> >
> > *This message and/or attachments may include information subject to GD
> > Corporate Policies 07-103 and 07-105 and is intended to be accessed
> > only by authorized recipients. Use, storage and transmission are
> > governed by General Dynamics and its policies. Contractual
> > restrictions apply to third parties. Recipients should refer to the
> > policies or contract to determine proper handling. Unauthorized
> > review, use, disclosure or distribution is prohibited. If you are not
> > an intended recipient, please contact the sender and destroy all
> > copies of the original message.*
> >
>
--
Sean
RE: MiniAccumuloClutser Unit Test Problems
Posted by "Parise, Jonathan" <Jo...@gd-ms.com>.
Josh,
Sorry for the confusion. I'll try and explain again with better terminology.
I have several test classes, each of which contain several test methods. For each test class I have an @BeforeClass that configures and starts a MAC. I also have an @AfterClass that calls MAC.stop().
So the flow is MAC is created, 1-n tests in the class run against it, MAC is stopped, repeat for next N tests.
This way several @test methods in the same class use the same MAC. I mostly did this to make the tests run faster. I understand that having several test methods share a MAC could cause test state pollution, but I am careful to avoid that.
The issues I was seeing is that if I simply ran "mvn test", the first few tests would pass and then eventually one of the tests would get stuck forever and just keep throwing connection errors.
When I changed the maven configuration to force the tests to run serially, the issue stopped occurring.
I'm not sure if there tests are expected to work when run in parallel. My best guess is that the tests may conflict with each other over ports or something like that.
I'd like to understand why changing the test running behavior fixed this issue. Also, I think it would be good to document this somewhere. The MAC javadoc and also the user's guide should provide details about using the MAC for tests and properly configuring those tests.
Does that make more sense?
Jon
-----Original Message-----
From: Josh Elser [mailto:josh.elser@gmail.com]
Sent: Friday, June 26, 2015 10:00 AM
To: user@accumulo.apache.org
Subject: Re: MiniAccumuloClutser Unit Test Problems
Jonathan,
If you're not seeing consistent behavior starting and stopping a MiniAccumuloCluster repeatedly, that's a bug. If you can provide a code which shows this problem, that'd be a huge help.
If you can get a list of the processes running when you see this happen and cross-reference it with what processes should be running, that would also go a long way in trying to debug this.
I am a little confused to your specific situation. You said that you construct and start a MAC instance in a BeforeClass and stop it in an AfterClass, but then you said that you start and stop it for each test.
Are you saying that after the third construction and use of a MAC, you see problems? Or, are you saying that you stop and start each MAC instance before you run the @Test methods?
- Josh
Parise, Jonathan wrote:
> Hello,
>
> I have been writing some J-unit tests based on the MiniAccumuloCluster
> class. I'm experiencing some issues when several of the tests run back
> to back. Before I get into the error, let me explain how the tests
> work in general. Also, I am using Accumulo 1.6.2.
>
> Each test has an @BeforeClass method that first creates a new random
> directory. Then makes a new MiniAccumuloCluster instance using that
> directory as the dir parameter. Then, I call MiniAccumuloCluster.start().
>
> There are several @Test methods in each test class. The typical
> pattern for them is that they create any necessary tables, write some
> data into those tables and then scan to verify it was written
> correctly. Basically they are testing that I can serialize and
> deserialize various types of Objects correctly.
>
> Then the test class as an @AfterClass method that calls
> MiniAccumuloCluster.stop(). It also deletes the random directory used
> by the previous test.
>
> The issue I am running into is that generally the first test or two
> run fine. However, the third test usually gets stuck in the
> MiniAccumuloCluster startup. It just keeps complaining about being
> unable to connect. Note that if the test is run independently it
> passes just fine. When run back to back, I see errors like this one repeatedly:
>
> 2015-06-26 09:10:24,352 INFO [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:47046
>
> 2015-06-26 09:10:24,353 WARN [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:24,956 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
>
> 2015-06-26 09:10:24,957 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> 2015-06-26 09:10:51,764 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
>
> 2015-06-26 09:10:51,765 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:52,572 INFO [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:47046
>
> 2015-06-26 09:10:52,573 WARN [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:52,890 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
>
> 2015-06-26 09:10:52,891 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:54,191 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
>
> 2015-06-26 09:10:54,192 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:54,471 INFO [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:47046
>
> 2015-06-26 09:10:54,471 WARN [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> In the ZooKeeperServerMain.out I see lines like this repeating several
> thousand times:
>
> 2015-06-26 08:39:44,810 INFO [SyncThread:0] server.NIOServerCnxn
> (NIOServerCnxn.java:finishSessionInit(1580)) - Established session
> 0x14e2fe179eb0000 with negotiated timeout 30000 for client
> /127.0.0.1:36311
>
> 2015-06-26 08:39:45,278 INFO [ProcessThread:-1]
> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
> - Got user-level KeeperException when processing
> sessionid:0x14e2fe179eb0000 type:create cxid:0x31
> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
> Error:KeeperErrorCode = NodeExists for
> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
>
> 2015-06-26 08:39:45,300 INFO [ProcessThread:-1]
> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
> - Got user-level KeeperException when processing
> sessionid:0x14e2fe179eb0000 type:create cxid:0x33
> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
> Error:KeeperErrorCode = NodeExists for
> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
>
> I've thought about putting a Thread.sleep() call after the
> MiniAccumuloCluster.stop() call, but that certainly seems brittle. I'm
> not sure if that would improve the situation.
>
> It seems to me that the MiniAccumuloCluster does not behave well when
> instances are started and stopped several times. I am running the
> tests through Maven using the default test behavior.
>
> Could it be something with Maven? Maybe I need to be more explicit
> when telling it how to run the tests?
>
> Anyone have any insight into what is going wrong here? In general is
> my usage pattern correct for MiniAccumuloCluster?
>
> Thanks,
>
> Jon Parise
>
> Senior Software Engineer
>
> Viz | General Dynamics Missons Systems
>
> *This message and/or attachments may include information subject to GD
> Corporate Policies 07-103 and 07-105 and is intended to be accessed
> only by authorized recipients. Use, storage and transmission are
> governed by General Dynamics and its policies. Contractual
> restrictions apply to third parties. Recipients should refer to the
> policies or contract to determine proper handling. Unauthorized
> review, use, disclosure or distribution is prohibited. If you are not
> an intended recipient, please contact the sender and destroy all
> copies of the original message.*
>
Re: MiniAccumuloClutser Unit Test Problems
Posted by Josh Elser <jo...@gmail.com>.
Jonathan,
If you're not seeing consistent behavior starting and stopping a
MiniAccumuloCluster repeatedly, that's a bug. If you can provide a code
which shows this problem, that'd be a huge help.
If you can get a list of the processes running when you see this happen
and cross-reference it with what processes should be running, that would
also go a long way in trying to debug this.
I am a little confused to your specific situation. You said that you
construct and start a MAC instance in a BeforeClass and stop it in an
AfterClass, but then you said that you start and stop it for each test.
Are you saying that after the third construction and use of a MAC, you
see problems? Or, are you saying that you stop and start each MAC
instance before you run the @Test methods?
- Josh
Parise, Jonathan wrote:
> Hello,
>
> I have been writing some J-unit tests based on the MiniAccumuloCluster
> class. I’m experiencing some issues when several of the tests run back
> to back. Before I get into the error, let me explain how the tests work
> in general. Also, I am using Accumulo 1.6.2.
>
> Each test has an @BeforeClass method that first creates a new random
> directory. Then makes a new MiniAccumuloCluster instance using that
> directory as the dir parameter. Then, I call MiniAccumuloCluster.start().
>
> There are several @Test methods in each test class. The typical pattern
> for them is that they create any necessary tables, write some data into
> those tables and then scan to verify it was written correctly. Basically
> they are testing that I can serialize and deserialize various types of
> Objects correctly.
>
> Then the test class as an @AfterClass method that calls
> MiniAccumuloCluster.stop(). It also deletes the random directory used by
> the previous test.
>
> The issue I am running into is that generally the first test or two run
> fine. However, the third test usually gets stuck in the
> MiniAccumuloCluster startup. It just keeps complaining about being
> unable to connect. Note that if the test is run independently it passes
> just fine. When run back to back, I see errors like this one repeatedly:
>
> 2015-06-26 09:10:24,352 INFO [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:47046
>
> 2015-06-26 09:10:24,353 WARN [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:24,956 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
>
> 2015-06-26 09:10:24,957 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> 2015-06-26 09:10:51,764 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
>
> 2015-06-26 09:10:51,765 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:52,572 INFO [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:47046
>
> 2015-06-26 09:10:52,573 WARN [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:52,890 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
>
> 2015-06-26 09:10:52,891 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:54,191 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
>
> 2015-06-26 09:10:54,192 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:54,471 INFO [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:47046
>
> 2015-06-26 09:10:54,471 WARN [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> In the ZooKeeperServerMain.out I see lines like this repeating several
> thousand times:
>
> 2015-06-26 08:39:44,810 INFO [SyncThread:0] server.NIOServerCnxn
> (NIOServerCnxn.java:finishSessionInit(1580)) - Established session
> 0x14e2fe179eb0000 with negotiated timeout 30000 for client /127.0.0.1:36311
>
> 2015-06-26 08:39:45,278 INFO [ProcessThread:-1]
> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419)) -
> Got user-level KeeperException when processing
> sessionid:0x14e2fe179eb0000 type:create cxid:0x31
> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
> Error:KeeperErrorCode = NodeExists for
> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
>
> 2015-06-26 08:39:45,300 INFO [ProcessThread:-1]
> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419)) -
> Got user-level KeeperException when processing
> sessionid:0x14e2fe179eb0000 type:create cxid:0x33
> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
> Error:KeeperErrorCode = NodeExists for
> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
>
> I’ve thought about putting a Thread.sleep() call after the
> MiniAccumuloCluster.stop() call, but that certainly seems brittle. I’m
> not sure if that would improve the situation.
>
> It seems to me that the MiniAccumuloCluster does not behave well when
> instances are started and stopped several times. I am running the tests
> through Maven using the default test behavior.
>
> Could it be something with Maven? Maybe I need to be more explicit when
> telling it how to run the tests?
>
> Anyone have any insight into what is going wrong here? In general is my
> usage pattern correct for MiniAccumuloCluster?
>
> Thanks,
>
> Jon Parise
>
> Senior Software Engineer
>
> Viz | General Dynamics Missons Systems
>
> *This message and/or attachments may include information subject to GD
> Corporate Policies 07-103 and 07-105 and is intended to be accessed only
> by authorized recipients. Use, storage and transmission are governed by
> General Dynamics and its policies. Contractual restrictions apply to
> third parties. Recipients should refer to the policies or contract to
> determine proper handling. Unauthorized review, use, disclosure or
> distribution is prohibited. If you are not an intended recipient, please
> contact the sender and destroy all copies of the original message.*
>
RE: MiniAccumuloClutser Unit Test Problems
Posted by "Parise, Jonathan" <Jo...@gd-ms.com>.
I think I may have fixed this issue. I think it is caused by Maven running the tests in parallel. I changed my pom.xml to have the following configuration:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18</version>
<configuration>
<useUnlimitedThreads>true</useUnlimitedThreads>
<parallel>suites</parallel>
<reuseForks>false</reuseForks>
<includes>
<include>**/*Test.java</include>
<include>**/Test*.java</include>
</includes>
</configuration>
</plugin>
This is telling maven to use only a single thread for the execution of my entire test suite. This way nothing runs in parallel. With this I don't see the same connection issues.
I would still like to know if this general approach to writing Accumulo unit tests is good.
Thanks,
Jon Parise
From: Parise, Jonathan [mailto:Jonathan.Parise@gd-ms.com]
Sent: Friday, June 26, 2015 9:35 AM
To: user@accumulo.apache.org
Subject: MiniAccumuloClutser Unit Test Problems
Hello,
I have been writing some J-unit tests based on the MiniAccumuloCluster class. I'm experiencing some issues when several of the tests run back to back. Before I get into the error, let me explain how the tests work in general. Also, I am using Accumulo 1.6.2.
Each test has an @BeforeClass method that first creates a new random directory. Then makes a new MiniAccumuloCluster instance using that directory as the dir parameter. Then, I call MiniAccumuloCluster.start().
There are several @Test methods in each test class. The typical pattern for them is that they create any necessary tables, write some data into those tables and then scan to verify it was written correctly. Basically they are testing that I can serialize and deserialize various types of Objects correctly.
Then the test class as an @AfterClass method that calls MiniAccumuloCluster.stop(). It also deletes the random directory used by the previous test.
The issue I am running into is that generally the first test or two run fine. However, the third test usually gets stuck in the MiniAccumuloCluster startup. It just keeps complaining about being unable to connect. Note that if the test is run independently it passes just fine. When run back to back, I see errors like this one repeatedly:
2015-06-26 09:10:24,352 INFO [main-SendThread(localhost:47046)] zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket connection to server localhost/127.0.0.1:47046
2015-06-26 09:10:24,353 WARN [main-SendThread(localhost:47046)] zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffc86d80004 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:24,956 INFO [main-SendThread(localhost:10406)] zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket connection to server localhost/127.0.0.1:10406
2015-06-26 09:10:24,957 WARN [main-SendThread(localhost:10406)] zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffcb2390004 for server null, unexpected error, closing socket connection and attempting reconnect
2015-06-26 09:10:51,764 INFO [main-SendThread(localhost:10406)] zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket connection to server localhost/127.0.0.1:10406
2015-06-26 09:10:51,765 WARN [main-SendThread(localhost:10406)] zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffcb2390004 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:52,572 INFO [main-SendThread(localhost:47046)] zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket connection to server localhost/127.0.0.1:47046
2015-06-26 09:10:52,573 WARN [main-SendThread(localhost:47046)] zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffc86d80004 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:52,890 INFO [main-SendThread(localhost:10406)] zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket connection to server localhost/127.0.0.1:10406
2015-06-26 09:10:52,891 WARN [main-SendThread(localhost:10406)] zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffcb2390004 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:54,191 INFO [main-SendThread(localhost:10406)] zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket connection to server localhost/127.0.0.1:10406
2015-06-26 09:10:54,192 WARN [main-SendThread(localhost:10406)] zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffcb2390004 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2015-06-26 09:10:54,471 INFO [main-SendThread(localhost:47046)] zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening socket connection to server localhost/127.0.0.1:47046
2015-06-26 09:10:54,471 WARN [main-SendThread(localhost:47046)] zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x14e2ffc86d80004 for server null, unexpected error, closing socket connection and attempting reconnect
In the ZooKeeperServerMain.out I see lines like this repeating several thousand times:
2015-06-26 08:39:44,810 INFO [SyncThread:0] server.NIOServerCnxn (NIOServerCnxn.java:finishSessionInit(1580)) - Established session 0x14e2fe179eb0000 with negotiated timeout 30000 for client /127.0.0.1:36311
2015-06-26 08:39:45,278 INFO [ProcessThread:-1] server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419)) - Got user-level KeeperException when processing sessionid:0x14e2fe179eb0000 type:create cxid:0x31 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf Error:KeeperErrorCode = NodeExists for /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
2015-06-26 08:39:45,300 INFO [ProcessThread:-1] server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419)) - Got user-level KeeperException when processing sessionid:0x14e2fe179eb0000 type:create cxid:0x33 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf Error:KeeperErrorCode = NodeExists for /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
I've thought about putting a Thread.sleep() call after the MiniAccumuloCluster.stop() call, but that certainly seems brittle. I'm not sure if that would improve the situation.
It seems to me that the MiniAccumuloCluster does not behave well when instances are started and stopped several times. I am running the tests through Maven using the default test behavior.
Could it be something with Maven? Maybe I need to be more explicit when telling it how to run the tests?
Anyone have any insight into what is going wrong here? In general is my usage pattern correct for MiniAccumuloCluster?
Thanks,
Jon Parise
Senior Software Engineer
Viz | General Dynamics Missons Systems
This message and/or attachments may include information subject to GD Corporate Policies 07-103 and 07-105 and is intended to be accessed only by authorized recipients. Use, storage and transmission are governed by General Dynamics and its policies. Contractual restrictions apply to third parties. Recipients should refer to the policies or contract to determine proper handling. Unauthorized review, use, disclosure or distribution is prohibited. If you are not an intended recipient, please contact the sender and destroy all copies of the original message.