proper way to rebuild accumulo cluster

["Xu (Simon) Chen" <xc...@gmail.com>]

Hi all,

I am trying to set up an accumulo cluster with kerberos support. I
initially used password auth, therefore accumulo wasn't configured with a
privileged kerberos principal. After I turned on kerberos, root password
doesn't work anymore (obviously..)

I then wanted to re-bootstrap my cluster - a complete rebuild is fine,
since there isn't any valuable data. I removed /accumulo from HDFS, and
re-ran "accumulo init". This time, I got the chance to enter a valid
kerberos principal, which is good.

Unfortunately, all the tservers stopped working after this. Looking at the
log, it looks like the tserver is saying "ERROR: Got message from a service
with a mismatched configuration. Please ensure a compatible configuration."

Any ideas what is the proper way rebuild a cluster?

Thanks.
-Simon

Re: proper way to rebuild accumulo cluster

[Josh Elser <jo...@gmail.com>]

Hi Simon,

Like you found out, a full restart of everything should have cleared up 
problems you were seeing. The old servers were still using the old HDFS 
directories.

For the record, if you did have data you wanted to keep, you could have 
used `accumulo init --reset-security` which would have reset the entire 
user/permissions/authorizations database, but kept the Accumulo 
tables/data intact.

- Josh

Xu (Simon) Chen wrote:
> Never mind. "stop-all.sh" and "start-all.sh" worked.
>
> I had two master nodes, and I only rebooted one of them. Possibly the
> secondary took over with stale data.
>
> -Simon
>
> On Thu, Jun 4, 2015 at 10:49 PM, Xu (Simon) Chen <xchenum@gmail.com
> <ma...@gmail.com>> wrote:
>
>     Hi all,
>
>     I am trying to set up an accumulo cluster with kerberos support. I
>     initially used password auth, therefore accumulo wasn't configured
>     with a privileged kerberos principal. After I turned on kerberos,
>     root password doesn't work anymore (obviously..)
>
>     I then wanted to re-bootstrap my cluster - a complete rebuild is
>     fine, since there isn't any valuable data. I removed /accumulo from
>     HDFS, and re-ran "accumulo init". This time, I got the chance to
>     enter a valid kerberos principal, which is good.
>
>     Unfortunately, all the tservers stopped working after this. Looking
>     at the log, it looks like the tserver is saying "ERROR: Got message
>     from a service with a mismatched configuration. Please ensure a
>     compatible configuration."
>
>     Any ideas what is the proper way rebuild a cluster?
>
>     Thanks.
>     -Simon
>
>

Re: proper way to rebuild accumulo cluster

["Xu (Simon) Chen" <xc...@gmail.com>]

Never mind. "stop-all.sh" and "start-all.sh" worked.

I had two master nodes, and I only rebooted one of them. Possibly the
secondary took over with stale data.

-Simon

On Thu, Jun 4, 2015 at 10:49 PM, Xu (Simon) Chen <xc...@gmail.com> wrote:

> Hi all,
>
> I am trying to set up an accumulo cluster with kerberos support. I
> initially used password auth, therefore accumulo wasn't configured with a
> privileged kerberos principal. After I turned on kerberos, root password
> doesn't work anymore (obviously..)
>
> I then wanted to re-bootstrap my cluster - a complete rebuild is fine,
> since there isn't any valuable data. I removed /accumulo from HDFS, and
> re-ran "accumulo init". This time, I got the chance to enter a valid
> kerberos principal, which is good.
>
> Unfortunately, all the tservers stopped working after this. Looking at the
> log, it looks like the tserver is saying "ERROR: Got message from a service
> with a mismatched configuration. Please ensure a compatible configuration."
>
> Any ideas what is the proper way rebuild a cluster?
>
> Thanks.
> -Simon
>