You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Geoffry Roberts <th...@gmail.com> on 2018/01/31 21:45:48 UTC
Question on how Accumulo binds to Hadoop
I have a situation where Accumulo cannot find Hadoop.
Hadoop is running and I can access hdfs from the cli.
Zookeeper also says it is ok and I can log in using the client.
Accumulo init is failing with a connection refused for localhost:9000.
netstat shows nothing listening on 9000.
Now the plot thickens...
The Hadoop I am running is Google's Dataproc and the Hadoop installation is
not my own. I have already found a number of differences.
Here's my question: Does Accumulo use RPC to talk to Hadoop? I ask because
of things like this:
From hfs-site.xml
<property>
<name>dfs.namenode.rpc-address</name>
<value>haz0-m:8020</value>
<description>
RPC address that handles all clients requests. If empty then we'll get
thevalue from fs.default.name.The value of this property will take the
form of hdfs://nn-host1:rpc-port.
</description>
</property>
Or does it use something else?
Thanks
--
There are ways and there are ways,
Geoffry Roberts
Re: Question on how Accumulo binds to Hadoop
Posted by Christopher <ct...@apache.org>.
On Thu, Feb 1, 2018 at 2:00 PM Geoffry Roberts <th...@gmail.com>
wrote:
> >> It's better if you explicitly specify on the command line the ZK entry
> point.
>
> Can you give an example?
>
>
bin/accumulo shell -u root -zh zoohost1:2181,zoohost2:2181,zoohost3:2181
-zi myInstance
You can also put a client configuration file containing the following in
~/.accumulo/client.conf:
instance.zookeeper.host=zoohost1:2181,zoohost2:2181,zoohost3:2181
instance.name=myInstance
> On Thu, Feb 1, 2018 at 12:54 PM, Christopher <ct...@apache.org> wrote:
>
>> Normally, you'd set up Accumulo to use the HDFS volume in your
>> accumulo-site.xml file for your servers by setting the instance.volumes
>> field (in your case to the value of 'hdfs://haz0-m:8020/accumulo' or
>> similar).
>>
>> The shell typically connects to either ZooKeeper using client
>> configuration files or command-line options as its entry point. Run it with
>> the '--help' or '-?' options to see the available options.
>>
>> If it has read permission for your accumulo-site.xml file and the
>> Accumulo conf directory where this file is located is on its class path,
>> the shell may fall back to using your hdfs-site.xml or your
>> accumulo-site.xml to try to figure out things using HDFS... but that's
>> mostly a backwards-compatible/legacy mode. It's better if you explicitly
>> specify on the command line the ZK entry point.
>>
>> On Thu, Feb 1, 2018 at 10:59 AM Geoffry Roberts <th...@gmail.com>
>> wrote:
>>
>>> Thanks Adam, that worked. Accumulo starts but when I try the shell I
>>> get:
>>>
>>> ERROR: unable obtain instance id at file:/accumulo/instance_id
>>>
>>> $ hadoop fs -ls /
>>>
>>>
>>> Shows the id file and the Hadoop configuration directory is on the
>>> Accumulo class path according to accumulo-site.xml.
>>>
>>> Is the shell looking in the local file system or in hdfs? I never had
>>> this problem until I started up with Google.
>>>
>>> Thanks
>>>
>>> On Wed, Jan 31, 2018 at 5:06 PM, Adam J. Shook <ad...@gmail.com>
>>> wrote:
>>>
>>>> Yes, it does use RPC to talk to HDFS. You will need to update the
>>>> value of instance.volumes in accumulo-site.xml to reference this address,
>>>> haz0-m:8020, instead of the default localhost:9000.
>>>>
>>>> --Adam
>>>>
>>>> On Wed, Jan 31, 2018 at 4:45 PM, Geoffry Roberts <
>>>> threadedblue@gmail.com> wrote:
>>>>
>>>>> I have a situation where Accumulo cannot find Hadoop.
>>>>>
>>>>> Hadoop is running and I can access hdfs from the cli.
>>>>> Zookeeper also says it is ok and I can log in using the client.
>>>>> Accumulo init is failing with a connection refused for localhost:9000.
>>>>>
>>>>> netstat shows nothing listening on 9000.
>>>>>
>>>>> Now the plot thickens...
>>>>>
>>>>> The Hadoop I am running is Google's Dataproc and the Hadoop
>>>>> installation is not my own. I have already found a number of differences.
>>>>>
>>>>> Here's my question: Does Accumulo use RPC to talk to Hadoop? I ask
>>>>> because of things like this:
>>>>>
>>>>> From hfs-site.xml
>>>>>
>>>>> <property>
>>>>>
>>>>> <name>dfs.namenode.rpc-address</name>
>>>>>
>>>>> <value>haz0-m:8020</value>
>>>>>
>>>>> <description>
>>>>>
>>>>> RPC address that handles all clients requests. If empty then
>>>>> we'll get
>>>>>
>>>>> thevalue from fs.default.name.The value of this property will
>>>>> take the
>>>>>
>>>>> form of hdfs://nn-host1:rpc-port.
>>>>>
>>>>> </description>
>>>>>
>>>>> </property>
>>>>>
>>>>> Or does it use something else?
>>>>>
>>>>> Thanks
>>>>> --
>>>>> There are ways and there are ways,
>>>>>
>>>>> Geoffry Roberts
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> There are ways and there are ways,
>>>
>>> Geoffry Roberts
>>>
>>
>
>
> --
> There are ways and there are ways,
>
> Geoffry Roberts
>
Re: Question on how Accumulo binds to Hadoop
Posted by Geoffry Roberts <th...@gmail.com>.
>> It's better if you explicitly specify on the command line the ZK entry
point.
Can you give an example?
On Thu, Feb 1, 2018 at 12:54 PM, Christopher <ct...@apache.org> wrote:
> Normally, you'd set up Accumulo to use the HDFS volume in your
> accumulo-site.xml file for your servers by setting the instance.volumes
> field (in your case to the value of 'hdfs://haz0-m:8020/accumulo' or
> similar).
>
> The shell typically connects to either ZooKeeper using client
> configuration files or command-line options as its entry point. Run it with
> the '--help' or '-?' options to see the available options.
>
> If it has read permission for your accumulo-site.xml file and the Accumulo
> conf directory where this file is located is on its class path, the shell
> may fall back to using your hdfs-site.xml or your accumulo-site.xml to try
> to figure out things using HDFS... but that's mostly a
> backwards-compatible/legacy mode. It's better if you explicitly specify on
> the command line the ZK entry point.
>
> On Thu, Feb 1, 2018 at 10:59 AM Geoffry Roberts <th...@gmail.com>
> wrote:
>
>> Thanks Adam, that worked. Accumulo starts but when I try the shell I get:
>>
>> ERROR: unable obtain instance id at file:/accumulo/instance_id
>>
>> $ hadoop fs -ls /
>>
>>
>> Shows the id file and the Hadoop configuration directory is on the
>> Accumulo class path according to accumulo-site.xml.
>>
>> Is the shell looking in the local file system or in hdfs? I never had
>> this problem until I started up with Google.
>>
>> Thanks
>>
>> On Wed, Jan 31, 2018 at 5:06 PM, Adam J. Shook <ad...@gmail.com>
>> wrote:
>>
>>> Yes, it does use RPC to talk to HDFS. You will need to update the value
>>> of instance.volumes in accumulo-site.xml to reference this address,
>>> haz0-m:8020, instead of the default localhost:9000.
>>>
>>> --Adam
>>>
>>> On Wed, Jan 31, 2018 at 4:45 PM, Geoffry Roberts <threadedblue@gmail.com
>>> > wrote:
>>>
>>>> I have a situation where Accumulo cannot find Hadoop.
>>>>
>>>> Hadoop is running and I can access hdfs from the cli.
>>>> Zookeeper also says it is ok and I can log in using the client.
>>>> Accumulo init is failing with a connection refused for localhost:9000.
>>>>
>>>> netstat shows nothing listening on 9000.
>>>>
>>>> Now the plot thickens...
>>>>
>>>> The Hadoop I am running is Google's Dataproc and the Hadoop
>>>> installation is not my own. I have already found a number of differences.
>>>>
>>>> Here's my question: Does Accumulo use RPC to talk to Hadoop? I ask
>>>> because of things like this:
>>>>
>>>> From hfs-site.xml
>>>>
>>>> <property>
>>>>
>>>> <name>dfs.namenode.rpc-address</name>
>>>>
>>>> <value>haz0-m:8020</value>
>>>>
>>>> <description>
>>>>
>>>> RPC address that handles all clients requests. If empty then
>>>> we'll get
>>>>
>>>> thevalue from fs.default.name.The value of this property will
>>>> take the
>>>>
>>>> form of hdfs://nn-host1:rpc-port.
>>>>
>>>> </description>
>>>>
>>>> </property>
>>>>
>>>> Or does it use something else?
>>>>
>>>> Thanks
>>>> --
>>>> There are ways and there are ways,
>>>>
>>>> Geoffry Roberts
>>>>
>>>
>>>
>>
>>
>> --
>> There are ways and there are ways,
>>
>> Geoffry Roberts
>>
>
--
There are ways and there are ways,
Geoffry Roberts
Re: Question on how Accumulo binds to Hadoop
Posted by Christopher <ct...@apache.org>.
Normally, you'd set up Accumulo to use the HDFS volume in your
accumulo-site.xml file for your servers by setting the instance.volumes
field (in your case to the value of 'hdfs://haz0-m:8020/accumulo' or
similar).
The shell typically connects to either ZooKeeper using client configuration
files or command-line options as its entry point. Run it with the '--help'
or '-?' options to see the available options.
If it has read permission for your accumulo-site.xml file and the Accumulo
conf directory where this file is located is on its class path, the shell
may fall back to using your hdfs-site.xml or your accumulo-site.xml to try
to figure out things using HDFS... but that's mostly a
backwards-compatible/legacy mode. It's better if you explicitly specify on
the command line the ZK entry point.
On Thu, Feb 1, 2018 at 10:59 AM Geoffry Roberts <th...@gmail.com>
wrote:
> Thanks Adam, that worked. Accumulo starts but when I try the shell I get:
>
> ERROR: unable obtain instance id at file:/accumulo/instance_id
>
> $ hadoop fs -ls /
>
>
> Shows the id file and the Hadoop configuration directory is on the
> Accumulo class path according to accumulo-site.xml.
>
> Is the shell looking in the local file system or in hdfs? I never had
> this problem until I started up with Google.
>
> Thanks
>
> On Wed, Jan 31, 2018 at 5:06 PM, Adam J. Shook <ad...@gmail.com>
> wrote:
>
>> Yes, it does use RPC to talk to HDFS. You will need to update the value
>> of instance.volumes in accumulo-site.xml to reference this address,
>> haz0-m:8020, instead of the default localhost:9000.
>>
>> --Adam
>>
>> On Wed, Jan 31, 2018 at 4:45 PM, Geoffry Roberts <th...@gmail.com>
>> wrote:
>>
>>> I have a situation where Accumulo cannot find Hadoop.
>>>
>>> Hadoop is running and I can access hdfs from the cli.
>>> Zookeeper also says it is ok and I can log in using the client.
>>> Accumulo init is failing with a connection refused for localhost:9000.
>>>
>>> netstat shows nothing listening on 9000.
>>>
>>> Now the plot thickens...
>>>
>>> The Hadoop I am running is Google's Dataproc and the Hadoop installation
>>> is not my own. I have already found a number of differences.
>>>
>>> Here's my question: Does Accumulo use RPC to talk to Hadoop? I ask
>>> because of things like this:
>>>
>>> From hfs-site.xml
>>>
>>> <property>
>>>
>>> <name>dfs.namenode.rpc-address</name>
>>>
>>> <value>haz0-m:8020</value>
>>>
>>> <description>
>>>
>>> RPC address that handles all clients requests. If empty then we'll
>>> get
>>>
>>> thevalue from fs.default.name.The value of this property will take
>>> the
>>>
>>> form of hdfs://nn-host1:rpc-port.
>>>
>>> </description>
>>>
>>> </property>
>>>
>>> Or does it use something else?
>>>
>>> Thanks
>>> --
>>> There are ways and there are ways,
>>>
>>> Geoffry Roberts
>>>
>>
>>
>
>
> --
> There are ways and there are ways,
>
> Geoffry Roberts
>
Re: Question on how Accumulo binds to Hadoop
Posted by Geoffry Roberts <th...@gmail.com>.
Thanks Adam, that worked. Accumulo starts but when I try the shell I get:
ERROR: unable obtain instance id at file:/accumulo/instance_id
$ hadoop fs -ls /
Shows the id file and the Hadoop configuration directory is on the
Accumulo class path according to accumulo-site.xml.
Is the shell looking in the local file system or in hdfs? I never had this
problem until I started up with Google.
Thanks
On Wed, Jan 31, 2018 at 5:06 PM, Adam J. Shook <ad...@gmail.com> wrote:
> Yes, it does use RPC to talk to HDFS. You will need to update the value
> of instance.volumes in accumulo-site.xml to reference this address,
> haz0-m:8020, instead of the default localhost:9000.
>
> --Adam
>
> On Wed, Jan 31, 2018 at 4:45 PM, Geoffry Roberts <th...@gmail.com>
> wrote:
>
>> I have a situation where Accumulo cannot find Hadoop.
>>
>> Hadoop is running and I can access hdfs from the cli.
>> Zookeeper also says it is ok and I can log in using the client.
>> Accumulo init is failing with a connection refused for localhost:9000.
>>
>> netstat shows nothing listening on 9000.
>>
>> Now the plot thickens...
>>
>> The Hadoop I am running is Google's Dataproc and the Hadoop installation
>> is not my own. I have already found a number of differences.
>>
>> Here's my question: Does Accumulo use RPC to talk to Hadoop? I ask
>> because of things like this:
>>
>> From hfs-site.xml
>>
>> <property>
>>
>> <name>dfs.namenode.rpc-address</name>
>>
>> <value>haz0-m:8020</value>
>>
>> <description>
>>
>> RPC address that handles all clients requests. If empty then we'll
>> get
>>
>> thevalue from fs.default.name.The value of this property will take
>> the
>>
>> form of hdfs://nn-host1:rpc-port.
>>
>> </description>
>>
>> </property>
>>
>> Or does it use something else?
>>
>> Thanks
>> --
>> There are ways and there are ways,
>>
>> Geoffry Roberts
>>
>
>
--
There are ways and there are ways,
Geoffry Roberts
Re: Question on how Accumulo binds to Hadoop
Posted by "Adam J. Shook" <ad...@gmail.com>.
Yes, it does use RPC to talk to HDFS. You will need to update the value of
instance.volumes in accumulo-site.xml to reference this address,
haz0-m:8020, instead of the default localhost:9000.
--Adam
On Wed, Jan 31, 2018 at 4:45 PM, Geoffry Roberts <th...@gmail.com>
wrote:
> I have a situation where Accumulo cannot find Hadoop.
>
> Hadoop is running and I can access hdfs from the cli.
> Zookeeper also says it is ok and I can log in using the client.
> Accumulo init is failing with a connection refused for localhost:9000.
>
> netstat shows nothing listening on 9000.
>
> Now the plot thickens...
>
> The Hadoop I am running is Google's Dataproc and the Hadoop installation
> is not my own. I have already found a number of differences.
>
> Here's my question: Does Accumulo use RPC to talk to Hadoop? I ask
> because of things like this:
>
> From hfs-site.xml
>
> <property>
>
> <name>dfs.namenode.rpc-address</name>
>
> <value>haz0-m:8020</value>
>
> <description>
>
> RPC address that handles all clients requests. If empty then we'll
> get
>
> thevalue from fs.default.name.The value of this property will take
> the
>
> form of hdfs://nn-host1:rpc-port.
>
> </description>
>
> </property>
>
> Or does it use something else?
>
> Thanks
> --
> There are ways and there are ways,
>
> Geoffry Roberts
>