You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by "Roberts, Geoffry [USA]" <Ro...@bah.com> on 2020/12/30 01:38:25 UTC
Re: [External] Re: Accumulo access question
Thanks for the quick response.
I don’t think it’s a DNS issue. I am using /etc/hosts. I’ll double check the entries. I did, however, use the hostnames when setting up the passwordless ssh.
The stack trace I put below.
16:37:33.842 [main] DEBUG org.apache.accumulo.core.rpc.SaslConnectionParams(117) - Kerberos krb5 configuration not found, setting default realm to empty
16:37:33.843 [main] TRACE org.apache.accumulo.core.rpc.ThriftUtil(365) - Opening normal transport
16:37:38.878 [main] DEBUG org.apache.accumulo.core.clientImpl.MasterClient(78) - Failed to connect to master=haz11:9999, will retry...
org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
at org.apache.thrift.transport.TSocket.open(TSocket.java:226) ~[libthrift-0.12.0.jar:0.12.0]
at org.apache.accumulo.core.rpc.ThriftUtil.createClientTransport(ThriftUtil.java:368) ~[accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.ThriftTransportPool.createNewTransport(ThriftTransportPool.java:496) ~[accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.ThriftTransportPool.getTransport(ThriftTransportPool.java:432) ~[accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.ThriftTransportPool.getTransport(ThriftTransportPool.java:410) ~[accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.rpc.ThriftUtil.getClient(ThriftUtil.java:150) ~[accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.rpc.ThriftUtil.getClientNoTimeout(ThriftUtil.java:113) ~[accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.MasterClient.getConnection(MasterClient.java:70) [accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.MasterClient.getConnectionWithRetry(MasterClient.java:47) [accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.beginFateOperation(TableOperationsImpl.java:246) [accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:352) [accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:342) [accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doTableFateOperation(TableOperationsImpl.java:1599) [accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.create(TableOperationsImpl.java:234) [accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.create(TableOperationsImpl.java:206) [accumulo-core-2.0.0.jar:2.0.0]
at gov.cdc.dhcp.hehr.dcp.cda2d4m.AccumuloComponent.initAccumulo(AccumuloComponent.java:84) [classes/:?]
at gov.cdc.dhcp.hehr.dcp.cda2d4m.CDA2D4M.run(CDA2D4M.java:79) [classes/:?]
at gov.cdc.dhcp.hehr.dcp.cda2d4m.CDA2D4M.main(CDA2D4M.java:133) [classes/:?]
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method) ~[?:?]
at sun.nio.ch.Net.connect(Net.java:493) ~[?:?]
at sun.nio.ch.Net.connect(Net.java:482) ~[?:?]
at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588) ~[?:?]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:339) ~[?:?]
at java.net.Socket.connect(Socket.java:585) ~[?:?]
at org.apache.thrift.transport.TSocket.open(TSocket.java:221) ~[libthrift-0.12.0.jar:0.12.0]
... 17 more
16:37:39.134 [main] TRACE org.apache.accumulo.core.clientImpl.ClientContext(366) - tid=1 Looking up master location in zookeeper.
16:37:39.134 [main] TRACE org.apache.accumulo.core.clientImpl.ClientContext(374) - tid=1 Found master at haz11:9999 in 0.000 secs
16:37:39.134 [main] TRACE org.apache.accumulo.core.rpc.ThriftUtil(365) - Opening normal transport
16:37:39.160 [main] DEBUG org.apache.accumulo.core.clientImpl.MasterClient(78) - Failed to connect to master=haz11:9999, will retry...
org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
at org.apache.thrift.transport.TSocket.open(TSocket.java:226) ~[libthrift-0.12.0.jar:0.12.0]
at org.apache.accumulo.core.rpc.ThriftUtil.createClientTransport(ThriftUtil.java:368) ~[accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.ThriftTransportPool.createNewTransport(ThriftTransportPool.java:496) ~[accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.ThriftTransportPool.getTransport(ThriftTransportPool.java:432) ~[accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.ThriftTransportPool.getTransport(ThriftTransportPool.java:410) ~[accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.rpc.ThriftUtil.getClient(ThriftUtil.java:150) ~[accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.rpc.ThriftUtil.getClientNoTimeout(ThriftUtil.java:113) ~[accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.MasterClient.getConnection(MasterClient.java:70) [accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.MasterClient.getConnectionWithRetry(MasterClient.java:47) [accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.beginFateOperation(TableOperationsImpl.java:246) [accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:352) [accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:342) [accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doTableFateOperation(TableOperationsImpl.java:1599) [accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.create(TableOperationsImpl.java:234) [accumulo-core-2.0.0.jar:2.0.0]
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.create(TableOperationsImpl.java:206) [accumulo-core-2.0.0.jar:2.0.0]
at gov.cdc.dhcp.hehr.dcp.cda2d4m.AccumuloComponent.initAccumulo(AccumuloComponent.java:84) [classes/:?]
at gov.cdc.dhcp.hehr.dcp.cda2d4m.CDA2D4M.run(CDA2D4M.java:79) [classes/:?]
at gov.cdc.dhcp.hehr.dcp.cda2d4m.CDA2D4M.main(CDA2D4M.java:133) [classes/:?]
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method) ~[?:?]
at sun.nio.ch.Net.connect(Net.java:493) ~[?:?]
at sun.nio.ch.Net.connect(Net.java:482) ~[?:?]
at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588) ~[?:?]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:339) ~[?:?]
at java.net.Socket.connect(Socket.java:585) ~[?:?]
at org.apache.thrift.transport.TSocket.open(TSocket.java:221) ~[libthrift-0.12.0.jar:0.12.0]
... 17 more
Geoffry Roberts
Lead Technologist
702.290.9098
roberts_geoffry@bah.com
Booz | Allen | Hamilton
BoozAllen.com
From: Christopher <ct...@apache.org>
Reply-To: "user@accumulo.apache.org" <us...@accumulo.apache.org>
Date: Tuesday, December 29, 2020 at 8:17 PM
To: accumulo-user <us...@accumulo.apache.org>
Subject: [External] Re: Accumulo access question
Is there more to the debug message? A full stack trace perhaps? It might show the specific IOException causing the problem connecting.
The two most likely causes are DNS/name service causing the server host name to not be resolved (very common), and the port (9999 in this case) being unreachable from the client (in EC2, this can happen with an inadequate security group configuration, for example).
On Tue, Dec 29, 2020, 17:04 Roberts, Geoffry [USA] <Ro...@bah.com>> wrote:
All,
I have am using Accumulo in a different setup than ever before and I am not getting a connect.
I am having to ssh tunnel in. I have 2181 open (forwarded) and I can connect to Zookeeper using the Zk client from my laptop. When I try to connect to Accumulo using an Accumulo client from my laptop I get:
DEBUG org.apache.accumulo.core.clientImpl.MasterClient(78) - Failed to connect to master=haz11:9999, will retry...
(haz11 is one of my nodes.) I do have passwordless ssh working between all nodes. Netstat shows that 0.0.0.0:9999<https://urldefense.com/v3/__http:/0.0.0.0:9999__;!!May37g!ZUmDBXH4QTS4EGb6irABAqtH8rZ5iY12zGUjHqIwMJOT3j5aMh0mcq_hZ_tAr_EPABqp$> … LISTEN
I have a small (three node installation). I have all three hosts listed in all files (gc, masters, tservers, workers, monitors, et. al.). It all starts up well enough. I see the proper processes running on all nodes. I can start a shell—locally—from one of the nodes and run shell commands, all very normal.
Geoffry Roberts
Lead Technologist
702.290.9098
roberts_geoffry@bah.com<ma...@bah.com>
Booz | Allen | Hamilton
BoozAllen.com
Re: [External] Re: Accumulo access question
Posted by Christopher <ct...@apache.org>.
The "Connection refused" message indicates that, from the client's
perspective, the port is closed. See:
https://serverfault.com/questions/725262/what-causes-the-connection-refused-message
My guess is firewall, security group, proxy, or similar issue.
On Tue, Dec 29, 2020 at 8:38 PM Roberts, Geoffry [USA] <
Roberts_Geoffry@bah.com> wrote:
> Thanks for the quick response.
>
>
>
> I don’t think it’s a DNS issue. I am using /etc/hosts. I’ll double check
> the entries. I did, however, use the hostnames when setting up the
> passwordless ssh.
>
>
>
> The stack trace I put below.
>
>
>
> 16:37:33.842 [main] DEBUG
> org.apache.accumulo.core.rpc.SaslConnectionParams(117) - Kerberos krb5
> configuration not found, setting default realm to empty
>
> 16:37:33.843 [main] TRACE org.apache.accumulo.core.rpc.ThriftUtil(365) -
> Opening normal transport
>
> 16:37:38.878 [main] DEBUG
> org.apache.accumulo.core.clientImpl.MasterClient(78) - Failed to connect to
> master=haz11:9999, will retry...
>
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection refused
>
> at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
> ~[libthrift-0.12.0.jar:0.12.0]
>
> at org.apache.accumulo.core.rpc.ThriftUtil.createClientTransport(
> ThriftUtil.java:368) ~[accumulo-core-2.0.0.jar:2.0.0]
>
> at
> org.apache.accumulo.core.clientImpl.ThriftTransportPool.createNewTransport(
> ThriftTransportPool.java:496) ~[accumulo-core-2.0.0.jar:2.0.0]
>
> at
> org.apache.accumulo.core.clientImpl.ThriftTransportPool.getTransport(
> ThriftTransportPool.java:432) ~[accumulo-core-2.0.0.jar:2.0.0]
>
> at
> org.apache.accumulo.core.clientImpl.ThriftTransportPool.getTransport(
> ThriftTransportPool.java:410) ~[accumulo-core-2.0.0.jar:2.0.0]
>
> at org.apache.accumulo.core.rpc.ThriftUtil.getClient(
> ThriftUtil.java:150) ~[accumulo-core-2.0.0.jar:2.0.0]
>
> at org.apache.accumulo.core.rpc.ThriftUtil.getClientNoTimeout(
> ThriftUtil.java:113) ~[accumulo-core-2.0.0.jar:2.0.0]
>
> at org.apache.accumulo.core.clientImpl.MasterClient.getConnection(
> MasterClient.java:70) [accumulo-core-2.0.0.jar:2.0.0]
>
> at
> org.apache.accumulo.core.clientImpl.MasterClient.getConnectionWithRetry(
> MasterClient.java:47) [accumulo-core-2.0.0.jar:2.0.0]
>
> at
> org.apache.accumulo.core.clientImpl.TableOperationsImpl.beginFateOperation(
> TableOperationsImpl.java:246) [accumulo-core-2.0.0.jar:2.0.0]
>
> at
> org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(
> TableOperationsImpl.java:352) [accumulo-core-2.0.0.jar:2.0.0]
>
> at
> org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(
> TableOperationsImpl.java:342) [accumulo-core-2.0.0.jar:2.0.0]
>
> at
> org.apache.accumulo.core.clientImpl.TableOperationsImpl.doTableFateOperation(
> TableOperationsImpl.java:1599) [accumulo-core-2.0.0.jar:2.0.0]
>
> at org.apache.accumulo.core.clientImpl.TableOperationsImpl.create(
> TableOperationsImpl.java:234) [accumulo-core-2.0.0.jar:2.0.0]
>
> at org.apache.accumulo.core.clientImpl.TableOperationsImpl.create(
> TableOperationsImpl.java:206) [accumulo-core-2.0.0.jar:2.0.0]
>
> at gov.cdc.dhcp.hehr.dcp.cda2d4m.AccumuloComponent.initAccumulo(
> AccumuloComponent.java:84) [classes/:?]
>
> at gov.cdc.dhcp.hehr.dcp.cda2d4m.CDA2D4M.run(CDA2D4M.java:79)
> [classes/:?]
>
> at gov.cdc.dhcp.hehr.dcp.cda2d4m.CDA2D4M.main(CDA2D4M.java:133)
> [classes/:?]
>
> Caused by: java.net.ConnectException: Connection refused
>
> at sun.nio.ch.Net.connect0(Native Method) ~[?:?]
>
> at sun.nio.ch.Net.connect(Net.java:493) ~[?:?]
>
> at sun.nio.ch.Net.connect(Net.java:482) ~[?:?]
>
> at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588) ~[?:?]
>
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:339)
> ~[?:?]
>
> at java.net.Socket.connect(Socket.java:585) ~[?:?]
>
> at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
> ~[libthrift-0.12.0.jar:0.12.0]
>
> ... 17 more
>
> 16:37:39.134 [main] TRACE
> org.apache.accumulo.core.clientImpl.ClientContext(366) - tid=1 Looking up
> master location in zookeeper.
>
> 16:37:39.134 [main] TRACE
> org.apache.accumulo.core.clientImpl.ClientContext(374) - tid=1 Found master
> at haz11:9999 in 0.000 secs
>
> 16:37:39.134 [main] TRACE org.apache.accumulo.core.rpc.ThriftUtil(365) -
> Opening normal transport
>
> 16:37:39.160 [main] DEBUG
> org.apache.accumulo.core.clientImpl.MasterClient(78) - Failed to connect to
> master=haz11:9999, will retry...
>
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection refused
>
> at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
> ~[libthrift-0.12.0.jar:0.12.0]
>
> at org.apache.accumulo.core.rpc.ThriftUtil.createClientTransport(
> ThriftUtil.java:368) ~[accumulo-core-2.0.0.jar:2.0.0]
>
> at
> org.apache.accumulo.core.clientImpl.ThriftTransportPool.createNewTransport(
> ThriftTransportPool.java:496) ~[accumulo-core-2.0.0.jar:2.0.0]
>
> at
> org.apache.accumulo.core.clientImpl.ThriftTransportPool.getTransport(
> ThriftTransportPool.java:432) ~[accumulo-core-2.0.0.jar:2.0.0]
>
> at
> org.apache.accumulo.core.clientImpl.ThriftTransportPool.getTransport(
> ThriftTransportPool.java:410) ~[accumulo-core-2.0.0.jar:2.0.0]
>
> at org.apache.accumulo.core.rpc.ThriftUtil.getClient(
> ThriftUtil.java:150) ~[accumulo-core-2.0.0.jar:2.0.0]
>
> at org.apache.accumulo.core.rpc.ThriftUtil.getClientNoTimeout(
> ThriftUtil.java:113) ~[accumulo-core-2.0.0.jar:2.0.0]
>
> at org.apache.accumulo.core.clientImpl.MasterClient.getConnection(
> MasterClient.java:70) [accumulo-core-2.0.0.jar:2.0.0]
>
> at
> org.apache.accumulo.core.clientImpl.MasterClient.getConnectionWithRetry(
> MasterClient.java:47) [accumulo-core-2.0.0.jar:2.0.0]
>
> at
> org.apache.accumulo.core.clientImpl.TableOperationsImpl.beginFateOperation(
> TableOperationsImpl.java:246) [accumulo-core-2.0.0.jar:2.0.0]
>
> at
> org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(
> TableOperationsImpl.java:352) [accumulo-core-2.0.0.jar:2.0.0]
>
> at
> org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(
> TableOperationsImpl.java:342) [accumulo-core-2.0.0.jar:2.0.0]
>
> at
> org.apache.accumulo.core.clientImpl.TableOperationsImpl.doTableFateOperation(
> TableOperationsImpl.java:1599) [accumulo-core-2.0.0.jar:2.0.0]
>
> at org.apache.accumulo.core.clientImpl.TableOperationsImpl.create(
> TableOperationsImpl.java:234) [accumulo-core-2.0.0.jar:2.0.0]
>
> at org.apache.accumulo.core.clientImpl.TableOperationsImpl.create(
> TableOperationsImpl.java:206) [accumulo-core-2.0.0.jar:2.0.0]
>
> at gov.cdc.dhcp.hehr.dcp.cda2d4m.AccumuloComponent.initAccumulo(
> AccumuloComponent.java:84) [classes/:?]
>
> at gov.cdc.dhcp.hehr.dcp.cda2d4m.CDA2D4M.run(CDA2D4M.java:79)
> [classes/:?]
>
> at gov.cdc.dhcp.hehr.dcp.cda2d4m.CDA2D4M.main(CDA2D4M.java:133)
> [classes/:?]
>
> Caused by: java.net.ConnectException: Connection refused
>
> at sun.nio.ch.Net.connect0(Native Method) ~[?:?]
>
> at sun.nio.ch.Net.connect(Net.java:493) ~[?:?]
>
> at sun.nio.ch.Net.connect(Net.java:482) ~[?:?]
>
> at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588) ~[?:?]
>
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:339)
> ~[?:?]
>
> at java.net.Socket.connect(Socket.java:585) ~[?:?]
>
> at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
> ~[libthrift-0.12.0.jar:0.12.0]
>
> ... 17 more
>
>
>
>
>
> Geoffry Roberts
>
> Lead Technologist
>
> 702.290.9098
>
> roberts_geoffry@bah.com
>
>
>
> Booz | Allen | Hamilton
>
> BoozAllen.com
>
>
>
> *From: *Christopher <ct...@apache.org>
> *Reply-To: *"user@accumulo.apache.org" <us...@accumulo.apache.org>
> *Date: *Tuesday, December 29, 2020 at 8:17 PM
> *To: *accumulo-user <us...@accumulo.apache.org>
> *Subject: *[External] Re: Accumulo access question
>
>
>
> Is there more to the debug message? A full stack trace perhaps? It might
> show the specific IOException causing the problem connecting.
>
>
>
> The two most likely causes are DNS/name service causing the server host
> name to not be resolved (very common), and the port (9999 in this case)
> being unreachable from the client (in EC2, this can happen with an
> inadequate security group configuration, for example).
>
> On Tue, Dec 29, 2020, 17:04 Roberts, Geoffry [USA] <
> Roberts_Geoffry@bah.com> wrote:
>
> All,
>
>
>
> I have am using Accumulo in a different setup than ever before and I am
> not getting a connect.
>
>
>
> I am having to ssh tunnel in. I have 2181 open (forwarded) and I can
> connect to Zookeeper using the Zk client from my laptop. When I try to
> connect to Accumulo using an Accumulo client from my laptop I get:
>
>
>
> DEBUG org.apache.accumulo.core.clientImpl.MasterClient(78) - Failed to
> connect to master=haz11:9999, will retry...
>
>
>
> (haz11 is one of my nodes.) I do have passwordless ssh working between all
> nodes. Netstat shows that 0.0.0.0:9999
> <https://urldefense.com/v3/__http:/0.0.0.0:9999__;!!May37g!ZUmDBXH4QTS4EGb6irABAqtH8rZ5iY12zGUjHqIwMJOT3j5aMh0mcq_hZ_tAr_EPABqp$>
> … LISTEN
>
>
>
> I have a small (three node installation). I have all three hosts listed
> in all files (gc, masters, tservers, workers, monitors, et. al.). It all
> starts up well enough. I see the proper processes running on all nodes. I
> can start a shell—locally—from one of the nodes and run shell commands, all
> very normal.
>
>
>
> Geoffry Roberts
>
> Lead Technologist
>
> 702.290.9098
>
> roberts_geoffry@bah.com
>
>
>
> Booz | Allen | Hamilton
>
> BoozAllen.com
>
>