You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by "Terry P." <te...@gmail.com> on 2013/10/08 23:32:14 UTC
Accumulo init over existing instance
So reverse DNS wasn't working when I deployed my new cluster, thus all my
Tablet Servers were showing up in the Monitor as IP addresses (even though
all configuration files had hostnames only). Lesson learned: trust, but
verify (and ensure your hardened base servers still have nslookup and/or
dig on them).
Now that DNS is fixed, I want to wipe everything clean and re-init Accumulo
to ensure everything is legit using hostnames to ensure the cluster is not
tied to IP addresses.
I know I need to do a new 'accumulo init' -- I'll pass in the same instance
name, and my understanding is that will overwrite everything currently in
Zookeeper.
My question is: is there anything else I could/should do first to "clean
up" from this botched instance? E.g. should I delete all files in HDFS,
the write-ahead logs on the Tablet Servers, etc.? I'm running Accumulo
1.4.2.
Thanks,
Terry
Re: Accumulo init over existing instance
Posted by Jared Winick <ja...@gmail.com>.
In my experience, you need to remove the accumulo directory in HDFS (hadoop
fs -rmr /accumulo) before "accumulo init" will allow you to proceed. That
is all you should have to do.
Jared
On Tue, Oct 8, 2013 at 3:32 PM, Terry P. <te...@gmail.com> wrote:
> So reverse DNS wasn't working when I deployed my new cluster, thus all my
> Tablet Servers were showing up in the Monitor as IP addresses (even though
> all configuration files had hostnames only). Lesson learned: trust, but
> verify (and ensure your hardened base servers still have nslookup and/or
> dig on them).
>
> Now that DNS is fixed, I want to wipe everything clean and re-init
> Accumulo to ensure everything is legit using hostnames to ensure the
> cluster is not tied to IP addresses.
>
> I know I need to do a new 'accumulo init' -- I'll pass in the same
> instance name, and my understanding is that will overwrite everything
> currently in Zookeeper.
>
> My question is: is there anything else I could/should do first to "clean
> up" from this botched instance? E.g. should I delete all files in HDFS,
> the write-ahead logs on the Tablet Servers, etc.? I'm running Accumulo
> 1.4.2.
>
> Thanks,
> Terry
>
Re: Accumulo init over existing instance
Posted by "Terry P." <te...@gmail.com>.
Thanks Keith, great information. We're just entering formal test though so
1.5 isn't an option with this project. But great to know that the move to
HDFS with the walogs at least helped this issue significantly it looks like.
Thanks again.
On Tue, Oct 8, 2013 at 7:37 PM, Keith Turner <ke...@deenlo.com> wrote:
>
>
>
> On Tue, Oct 8, 2013 at 7:50 PM, Terry P. <te...@gmail.com> wrote:
>
>> Thanks Jared.
>>
>> John, thanks for the warning! I lost a dev cluster once when we had to
>> re-IP the Accumulo servers, but reverse DNS wasn't configured and I assumed
>> that was why. Guess that wasn't.
>>
>> Keith, I read through ACCUMULO-1585 but it wasn't completely clear if the
>> change proposed would also allow a server or servers in a cluster to have
>> its IP address changed. I hope it will, as while having to re-IP a server
>> or cluster is fairly rare, it certainly happens (as it did in our case).
>>
>
> I think moving from 1.4 to 1.5 will help. In 1.4 Accumulo has logger
> servers that store write ahead logs/edit logs. Data stored on these
> loggers are needed when a tablet server crashes. Accumulo stores pointers
> to loggers using IP addresses. So if the IP address of the machine running
> a logger changes, then Accumulo can no longer find the data need to recover
> from a fault.
>
> Starting w/ 1.5 Accumulo started storing write ahead logs in HDFS and the
> pointers to these WAlogs are now hdfs paths. The IP addrs that are still
> stored in 1.5 in zookeeper and the metadata table are more transient. For
> example locations of tablets are stored in the metadata table using IP
> addrs. If a tablet server dies and restarts w/ a different IP addr its
> probably ok, because the tablet will just be reassigned to a different
> tablet server. You may lose some locality because Accumulo prefers to
> assign a tablet to the last place it compacted data, but things should
> still work.
>
> I have not tried changing IP addrs w/ a 1.5 instance, so I do not know if
> there are other problems. But I do know that the walogs were a problem in
> 1.4 and that should no longer be a problem in 1.5.
>
>
>>
>> Thanks all,
>> Terry
>>
>>
>>
>> On Tue, Oct 8, 2013 at 5:14 PM, Keith Turner <ke...@deenlo.com> wrote:
>>
>>>
>>>
>>>
>>> On Tue, Oct 8, 2013 at 6:07 PM, John Vines <vi...@apache.org> wrote:
>>>
>>>> Like Jared said, wiping /accumulo out of hdfs is all you need to do.
>>>>
>>>> But Accumulo still uses IP addresses internally, so I'm not quite
>>>> certain you're going to achieve what you set out for.
>>>>
>>>
>>> Until 1.6.0 w/ ACCUMULO-1585
>>>
>>>
>>>>
>>>>
>>>> On Tue, Oct 8, 2013 at 5:32 PM, Terry P. <te...@gmail.com> wrote:
>>>>
>>>>> So reverse DNS wasn't working when I deployed my new cluster, thus all
>>>>> my Tablet Servers were showing up in the Monitor as IP addresses (even
>>>>> though all configuration files had hostnames only). Lesson learned: trust,
>>>>> but verify (and ensure your hardened base servers still have nslookup
>>>>> and/or dig on them).
>>>>>
>>>>> Now that DNS is fixed, I want to wipe everything clean and re-init
>>>>> Accumulo to ensure everything is legit using hostnames to ensure the
>>>>> cluster is not tied to IP addresses.
>>>>>
>>>>> I know I need to do a new 'accumulo init' -- I'll pass in the same
>>>>> instance name, and my understanding is that will overwrite everything
>>>>> currently in Zookeeper.
>>>>>
>>>>> My question is: is there anything else I could/should do first to
>>>>> "clean up" from this botched instance? E.g. should I delete all files in
>>>>> HDFS, the write-ahead logs on the Tablet Servers, etc.? I'm running
>>>>> Accumulo 1.4.2.
>>>>>
>>>>> Thanks,
>>>>> Terry
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Accumulo init over existing instance
Posted by Keith Turner <ke...@deenlo.com>.
On Tue, Oct 8, 2013 at 7:50 PM, Terry P. <te...@gmail.com> wrote:
> Thanks Jared.
>
> John, thanks for the warning! I lost a dev cluster once when we had to
> re-IP the Accumulo servers, but reverse DNS wasn't configured and I assumed
> that was why. Guess that wasn't.
>
> Keith, I read through ACCUMULO-1585 but it wasn't completely clear if the
> change proposed would also allow a server or servers in a cluster to have
> its IP address changed. I hope it will, as while having to re-IP a server
> or cluster is fairly rare, it certainly happens (as it did in our case).
>
I think moving from 1.4 to 1.5 will help. In 1.4 Accumulo has logger
servers that store write ahead logs/edit logs. Data stored on these
loggers are needed when a tablet server crashes. Accumulo stores pointers
to loggers using IP addresses. So if the IP address of the machine running
a logger changes, then Accumulo can no longer find the data need to recover
from a fault.
Starting w/ 1.5 Accumulo started storing write ahead logs in HDFS and the
pointers to these WAlogs are now hdfs paths. The IP addrs that are still
stored in 1.5 in zookeeper and the metadata table are more transient. For
example locations of tablets are stored in the metadata table using IP
addrs. If a tablet server dies and restarts w/ a different IP addr its
probably ok, because the tablet will just be reassigned to a different
tablet server. You may lose some locality because Accumulo prefers to
assign a tablet to the last place it compacted data, but things should
still work.
I have not tried changing IP addrs w/ a 1.5 instance, so I do not know if
there are other problems. But I do know that the walogs were a problem in
1.4 and that should no longer be a problem in 1.5.
>
> Thanks all,
> Terry
>
>
>
> On Tue, Oct 8, 2013 at 5:14 PM, Keith Turner <ke...@deenlo.com> wrote:
>
>>
>>
>>
>> On Tue, Oct 8, 2013 at 6:07 PM, John Vines <vi...@apache.org> wrote:
>>
>>> Like Jared said, wiping /accumulo out of hdfs is all you need to do.
>>>
>>> But Accumulo still uses IP addresses internally, so I'm not quite
>>> certain you're going to achieve what you set out for.
>>>
>>
>> Until 1.6.0 w/ ACCUMULO-1585
>>
>>
>>>
>>>
>>> On Tue, Oct 8, 2013 at 5:32 PM, Terry P. <te...@gmail.com> wrote:
>>>
>>>> So reverse DNS wasn't working when I deployed my new cluster, thus all
>>>> my Tablet Servers were showing up in the Monitor as IP addresses (even
>>>> though all configuration files had hostnames only). Lesson learned: trust,
>>>> but verify (and ensure your hardened base servers still have nslookup
>>>> and/or dig on them).
>>>>
>>>> Now that DNS is fixed, I want to wipe everything clean and re-init
>>>> Accumulo to ensure everything is legit using hostnames to ensure the
>>>> cluster is not tied to IP addresses.
>>>>
>>>> I know I need to do a new 'accumulo init' -- I'll pass in the same
>>>> instance name, and my understanding is that will overwrite everything
>>>> currently in Zookeeper.
>>>>
>>>> My question is: is there anything else I could/should do first to
>>>> "clean up" from this botched instance? E.g. should I delete all files in
>>>> HDFS, the write-ahead logs on the Tablet Servers, etc.? I'm running
>>>> Accumulo 1.4.2.
>>>>
>>>> Thanks,
>>>> Terry
>>>>
>>>
>>>
>>
>
Re: Accumulo init over existing instance
Posted by "Terry P." <te...@gmail.com>.
Thanks Jared.
John, thanks for the warning! I lost a dev cluster once when we had to
re-IP the Accumulo servers, but reverse DNS wasn't configured and I assumed
that was why. Guess that wasn't.
Keith, I read through ACCUMULO-1585 but it wasn't completely clear if the
change proposed would also allow a server or servers in a cluster to have
its IP address changed. I hope it will, as while having to re-IP a server
or cluster is fairly rare, it certainly happens (as it did in our case).
Thanks all,
Terry
On Tue, Oct 8, 2013 at 5:14 PM, Keith Turner <ke...@deenlo.com> wrote:
>
>
>
> On Tue, Oct 8, 2013 at 6:07 PM, John Vines <vi...@apache.org> wrote:
>
>> Like Jared said, wiping /accumulo out of hdfs is all you need to do.
>>
>> But Accumulo still uses IP addresses internally, so I'm not quite certain
>> you're going to achieve what you set out for.
>>
>
> Until 1.6.0 w/ ACCUMULO-1585
>
>
>>
>>
>> On Tue, Oct 8, 2013 at 5:32 PM, Terry P. <te...@gmail.com> wrote:
>>
>>> So reverse DNS wasn't working when I deployed my new cluster, thus all
>>> my Tablet Servers were showing up in the Monitor as IP addresses (even
>>> though all configuration files had hostnames only). Lesson learned: trust,
>>> but verify (and ensure your hardened base servers still have nslookup
>>> and/or dig on them).
>>>
>>> Now that DNS is fixed, I want to wipe everything clean and re-init
>>> Accumulo to ensure everything is legit using hostnames to ensure the
>>> cluster is not tied to IP addresses.
>>>
>>> I know I need to do a new 'accumulo init' -- I'll pass in the same
>>> instance name, and my understanding is that will overwrite everything
>>> currently in Zookeeper.
>>>
>>> My question is: is there anything else I could/should do first to "clean
>>> up" from this botched instance? E.g. should I delete all files in HDFS,
>>> the write-ahead logs on the Tablet Servers, etc.? I'm running Accumulo
>>> 1.4.2.
>>>
>>> Thanks,
>>> Terry
>>>
>>
>>
>
Re: Accumulo init over existing instance
Posted by Keith Turner <ke...@deenlo.com>.
On Tue, Oct 8, 2013 at 6:07 PM, John Vines <vi...@apache.org> wrote:
> Like Jared said, wiping /accumulo out of hdfs is all you need to do.
>
> But Accumulo still uses IP addresses internally, so I'm not quite certain
> you're going to achieve what you set out for.
>
Until 1.6.0 w/ ACCUMULO-1585
>
>
> On Tue, Oct 8, 2013 at 5:32 PM, Terry P. <te...@gmail.com> wrote:
>
>> So reverse DNS wasn't working when I deployed my new cluster, thus all my
>> Tablet Servers were showing up in the Monitor as IP addresses (even though
>> all configuration files had hostnames only). Lesson learned: trust, but
>> verify (and ensure your hardened base servers still have nslookup and/or
>> dig on them).
>>
>> Now that DNS is fixed, I want to wipe everything clean and re-init
>> Accumulo to ensure everything is legit using hostnames to ensure the
>> cluster is not tied to IP addresses.
>>
>> I know I need to do a new 'accumulo init' -- I'll pass in the same
>> instance name, and my understanding is that will overwrite everything
>> currently in Zookeeper.
>>
>> My question is: is there anything else I could/should do first to "clean
>> up" from this botched instance? E.g. should I delete all files in HDFS,
>> the write-ahead logs on the Tablet Servers, etc.? I'm running Accumulo
>> 1.4.2.
>>
>> Thanks,
>> Terry
>>
>
>
Re: Accumulo init over existing instance
Posted by John Vines <vi...@apache.org>.
Like Jared said, wiping /accumulo out of hdfs is all you need to do.
But Accumulo still uses IP addresses internally, so I'm not quite certain
you're going to achieve what you set out for.
On Tue, Oct 8, 2013 at 5:32 PM, Terry P. <te...@gmail.com> wrote:
> So reverse DNS wasn't working when I deployed my new cluster, thus all my
> Tablet Servers were showing up in the Monitor as IP addresses (even though
> all configuration files had hostnames only). Lesson learned: trust, but
> verify (and ensure your hardened base servers still have nslookup and/or
> dig on them).
>
> Now that DNS is fixed, I want to wipe everything clean and re-init
> Accumulo to ensure everything is legit using hostnames to ensure the
> cluster is not tied to IP addresses.
>
> I know I need to do a new 'accumulo init' -- I'll pass in the same
> instance name, and my understanding is that will overwrite everything
> currently in Zookeeper.
>
> My question is: is there anything else I could/should do first to "clean
> up" from this botched instance? E.g. should I delete all files in HDFS,
> the write-ahead logs on the Tablet Servers, etc.? I'm running Accumulo
> 1.4.2.
>
> Thanks,
> Terry
>