You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by "Cornish, Duane C." <Du...@jhuapl.edu> on 2012/09/11 23:25:31 UTC

Intermittent error connecting to Accumulo Instance

I stood up a 4 node Accumulo Cloud.  I wrote a Java interface to it and when I attempt to connect to the table through my interface, I'm getting an intermittent error that only pops up maybe 1 out of every 5 runs of the code.  I have zookeeper running on all 4 nodes but in my accumulo config file, I only mentioned the one on the Accumulo name node.  I don't know if that would be a problem or not.  Also, the zookeeper stuff is all owned by root, while hadoop and Accumulo are owned by a different non-root user.  I built this cluster from the ground up (i.e. didn't use Cloudera, etc.), so it's possible that I missed something.

I'm running Accumulo 1.4.1 with hadoop 1.0.3 and zookeeper 3.3.6.

As you can see from the error below, it is saying that I have an INVALID_INSTANCEID.  However my instance (and credentials) are static final Strings and are not changing.  I haven't had any trouble connecting to the instance via the shell.  Also, the monitor webpage works just fine and shows no errors.

Here is the error below:

Exception in thread "main" org.apache.accumulo.core.client.AccumuloSecurityException: Error INVALID_INSTANCEID - Unknown security exception
                at org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:73)
                at org.apache.accumulo.core.client.impl.ConnectorImpl.<init>(ConnectorImpl.java:75)
                at org.apache.accumulo.core.client.ZooKeeperInstance.getConnector(ZooKeeperInstance.java:218)
                at jhuapl.redd.coremax.shared.accumuloInterface.AccumuloConnector.deleteTable(AccumuloConnector.java:583)
                at jhuapl.redd.coremax.shared.accumuloInterface.TestAccumuloInterface.main(TestAccumuloInterface.java:27)
Caused by: ThriftSecurityException(user:root, code:INVALID_INSTANCEID)
                at org.apache.accumulo.core.client.impl.thrift.ClientService$authenticateUser_result.read(ClientService.java:7755)
                at org.apache.accumulo.core.client.impl.thrift.ClientService$Client.recv_authenticateUser(ClientService.java:415)
                at org.apache.accumulo.core.client.impl.thrift.ClientService$Client.authenticateUser(ClientService.java:387)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:616)
                at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
                at $Proxy0.authenticateUser(Unknown Source)
                at org.apache.accumulo.core.client.impl.ConnectorImpl$1.execute(ConnectorImpl.java:78)
                at org.apache.accumulo.core.client.impl.ConnectorImpl$1.execute(ConnectorImpl.java:75)
                at org.apache.accumulo.core.client.impl.ServerClient.executeRaw(ServerClient.java:108)
                at org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:71)
                ... 4 more

Any help would be greatly appreciated!
Thanks,
Duane

Re: Intermittent error connecting to Accumulo Instance

Posted by David Medinets <da...@gmail.com>.
Why are you running four copies of Zookeeper? The recommendation is to
run a prime number of copies (1, 3, 5, etc.). Consider cutting back to
one instance or three.

Re: Intermittent error connecting to Accumulo Instance

Posted by Adam Fuchs <af...@apache.org>.
Duane,

The INVALID_INSTANCEID message implies that your client is trying to
contact a server that thinks the instance id is something else. The 1 out
of 5 likelihood probably indicates that one out of the five server
processes (one master and four tablet servers) has some misconfiguration.
When you do a "deleteTable" operation, the client code will get the list of
active servers and pick one at random. This would seem to imply that
zookeeper is configured correctly across your cluster.

Tablet servers read their instance name and ID out of the accumulo
directory in HDFS. I wonder if this could be caused by something like HDFS
not being configured the same on one of the tablet servers, so when it
launches it doesn't get the right instance ID? (I'm thinking out loud here)

Do you see any warning or error messages propagating to your monitor page?

Adam



On Tue, Sep 11, 2012 at 5:25 PM, Cornish, Duane C. <Duane.Cornish@jhuapl.edu
> wrote:

> I stood up a 4 node Accumulo Cloud.  I wrote a Java interface to it and
> when I attempt to connect to the table through my interface, I'm getting an
> intermittent error that only pops up maybe 1 out of every 5 runs of the
> code.  I have zookeeper running on all 4 nodes but in my accumulo config
> file, I only mentioned the one on the Accumulo name node.  I don't know if
> that would be a problem or not.  Also, the zookeeper stuff is all owned by
> root, while hadoop and Accumulo are owned by a different non-root user.  I
> built this cluster from the ground up (i.e. didn't use Cloudera, etc.), so
> it's possible that I missed something.
>
> I'm running Accumulo 1.4.1 with hadoop 1.0.3 and zookeeper 3.3.6.
>
> As you can see from the error below, it is saying that I have an
> INVALID_INSTANCEID.  However my instance (and credentials) are static final
> Strings and are not changing.  I haven't had any trouble connecting to the
> instance via the shell.  Also, the monitor webpage works just fine and
> shows no errors.
>
> Here is the error below:
>
> Exception in thread "main"
> org.apache.accumulo.core.client.AccumuloSecurityException: Error
> INVALID_INSTANCEID - Unknown security exception
>                 at
> org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:73)
>                 at
> org.apache.accumulo.core.client.impl.ConnectorImpl.<init>(ConnectorImpl.java:75)
>                 at
> org.apache.accumulo.core.client.ZooKeeperInstance.getConnector(ZooKeeperInstance.java:218)
>                 at
> jhuapl.redd.coremax.shared.accumuloInterface.AccumuloConnector.deleteTable(AccumuloConnector.java:583)
>                 at
> jhuapl.redd.coremax.shared.accumuloInterface.TestAccumuloInterface.main(TestAccumuloInterface.java:27)
> Caused by: ThriftSecurityException(user:root, code:INVALID_INSTANCEID)
>                 at
> org.apache.accumulo.core.client.impl.thrift.ClientService$authenticateUser_result.read(ClientService.java:7755)
>                 at
> org.apache.accumulo.core.client.impl.thrift.ClientService$Client.recv_authenticateUser(ClientService.java:415)
>                 at
> org.apache.accumulo.core.client.impl.thrift.ClientService$Client.authenticateUser(ClientService.java:387)
>                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>                 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>                 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>                 at java.lang.reflect.Method.invoke(Method.java:616)
>                 at
> org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
>                 at $Proxy0.authenticateUser(Unknown Source)
>                 at
> org.apache.accumulo.core.client.impl.ConnectorImpl$1.execute(ConnectorImpl.java:78)
>                 at
> org.apache.accumulo.core.client.impl.ConnectorImpl$1.execute(ConnectorImpl.java:75)
>                 at
> org.apache.accumulo.core.client.impl.ServerClient.executeRaw(ServerClient.java:108)
>                 at
> org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:71)
>                 ... 4 more
>
> Any help would be greatly appreciated!
> Thanks,
> Duane
>
>

Re: Intermittent error connecting to Accumulo Instance

Posted by Josh Elser <jo...@gmail.com>.
Duane,

One ZooKeeper host should be more than enough for 5 servers.

Out of curiosity, did you configure zookeeper to be replicated? In your 
zoo.cfg, did you create "server.X" entries for each ZooKeeper and create 
the myid file in your ZooKeeper dataDir on each host? It sounds like you 
initialized 4 unique ZooKeepers, hit one ZooKeeper on init, and then hit 
another ZooKeeper when you went to connect.

See http://zookeeper.apache.org/doc/r3.3.6/zookeeperStarted.html, 
specifically "Running Replicated ZooKeeper" if you do want to run more 
than one ZooKeeper instance.

On 9/11/2012 5:25 PM, Cornish, Duane C. wrote:
> I stood up a 4 node Accumulo Cloud.  I wrote a Java interface to it and when I attempt to connect to the table through my interface, I'm getting an intermittent error that only pops up maybe 1 out of every 5 runs of the code.  I have zookeeper running on all 4 nodes but in my accumulo config file, I only mentioned the one on the Accumulo name node.  I don't know if that would be a problem or not.  Also, the zookeeper stuff is all owned by root, while hadoop and Accumulo are owned by a different non-root user.  I built this cluster from the ground up (i.e. didn't use Cloudera, etc.), so it's possible that I missed something.
>
> I'm running Accumulo 1.4.1 with hadoop 1.0.3 and zookeeper 3.3.6.
>
> As you can see from the error below, it is saying that I have an INVALID_INSTANCEID.  However my instance (and credentials) are static final Strings and are not changing.  I haven't had any trouble connecting to the instance via the shell.  Also, the monitor webpage works just fine and shows no errors.
>
> Here is the error below:
>
> Exception in thread "main" org.apache.accumulo.core.client.AccumuloSecurityException: Error INVALID_INSTANCEID - Unknown security exception
>                  at org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:73)
>                  at org.apache.accumulo.core.client.impl.ConnectorImpl.<init>(ConnectorImpl.java:75)
>                  at org.apache.accumulo.core.client.ZooKeeperInstance.getConnector(ZooKeeperInstance.java:218)
>                  at jhuapl.redd.coremax.shared.accumuloInterface.AccumuloConnector.deleteTable(AccumuloConnector.java:583)
>                  at jhuapl.redd.coremax.shared.accumuloInterface.TestAccumuloInterface.main(TestAccumuloInterface.java:27)
> Caused by: ThriftSecurityException(user:root, code:INVALID_INSTANCEID)
>                  at org.apache.accumulo.core.client.impl.thrift.ClientService$authenticateUser_result.read(ClientService.java:7755)
>                  at org.apache.accumulo.core.client.impl.thrift.ClientService$Client.recv_authenticateUser(ClientService.java:415)
>                  at org.apache.accumulo.core.client.impl.thrift.ClientService$Client.authenticateUser(ClientService.java:387)
>                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>                  at java.lang.reflect.Method.invoke(Method.java:616)
>                  at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:84)
>                  at $Proxy0.authenticateUser(Unknown Source)
>                  at org.apache.accumulo.core.client.impl.ConnectorImpl$1.execute(ConnectorImpl.java:78)
>                  at org.apache.accumulo.core.client.impl.ConnectorImpl$1.execute(ConnectorImpl.java:75)
>                  at org.apache.accumulo.core.client.impl.ServerClient.executeRaw(ServerClient.java:108)
>                  at org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java:71)
>                  ... 4 more
>
> Any help would be greatly appreciated!
> Thanks,
> Duane
>
>