You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Hyokwon Lee <hy...@gmail.com> on 2014/02/25 22:35:25 UTC
Accumulo with Kerberos Error
I am currently running into an issue and was hoping someone may have some
insight to the problem.
Running Accumulo 1.4.3 on top of a Kerberos enabled Hadoop. I followed the
following instructions in the README:
"If you are running on top of hdfs with kerberos enabled, then you need to do
some extra work. First, create an Accumulo principal
kadmin.local -q "addprinc -randkey accumulo/<host.domain.name>"
where <host.domain.name> is replaced by a fully qualified domain name. Export
the principals to a keytab file. It is safer to create a unique keytab
file for each
server, but you can also glob them if you wish.
kadmin.local -q "xst -k accumulo.keytab -glob accumulo*"
Place this file in $ACCUMULO_HOME/conf for every host. It should be owned by
the accumulo user and chmodded to 400. Add the following to the accumulo-env.sh
In the accumulo-site.xml file on each node, add settings for
general.kerberos.keytab
and general.kerberos.principal, where the keytab setting is the absolute path
to the keytab file ($ACCUMULO_HOME is valid to use) and principal is set to
accumulo/_HOST@<REALM>, where REALM is set to your kerberos realm. You may use
_HOST in lieu of your individual host names.
<property>
<name>general.kerberos.keytab</name>
<value>$ACCUMULO_HOME/conf/accumulo.keytab</value>
</property>
<property>
<name>general.kerberos.principal</name>
<value>accumulo/_HOST@MYREALM</value>
</property>
You can then start up Accumulo as you would with the accumulo user, and it will
automatically handle the kerberos keys needed to access hdfs.
Please Note: You may have issues initializing Accumulo while running
kerberos HDFS.
You can resolve this by temporarily granting the accumulo user write
access to the
hdfs root directory, running init, and then revoking write permission
in the root
directory (be sure to maintain access to the /accumulo directory)."
After doing so, got accumulo to come up and initially it states on
start up that i authenticated using
accumulo/hostname.test.local@TEST.LOCAL. For the next 24 hour it is
happy and everything works. However after the 24 hour marker which
is when the kerberos ticket expires, I start seeing the following
errors on all TServers:
[securty.UserGroupInformation] ERROR: PrivilegedActionException
as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS)
cause:javax.security.sasl.SaslException: GSS initiate failed [Caused
by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]
[ipc.Client] WARN : Exception encountered while connecting to the
server : javax.security.sasl.SasleEception: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism
level: Failed to find any Kerberos tgt)]
[securty.UserGroupInformation] ERROR: PrivilegedActionException
as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS)
cause:javax.security.sasl.SaslException: GSS initiate failed [Caused
by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]
And as far as I can tell this just retries and keeps failing. I
checked the accumulo.keytab file and it is a glob so it has the
entries for every server that Accumulo is on. Also if I manually do
a kinit -kt accumulo.keytab accumulo/tserver1.test.local@TEST.LOCAL it
works find and I get a valid ticket. I also made sure everything in
hdfs under "/accumulo" is owned by accumulo so that doesn't seem to be
the problem. Also made sure after kiniting I can access the directory
path and all sub directories.
So far the only thing that seems to fix my issue is if I bounce all
accumulo services and it is happy again. Also until I bounce the
accumulo services, I get error logs stating it cannot scan any of the
tables (unable to scan metadata, root_tablet, default_tablet, etc.)
Has anyone else seen this issue? Did I miss a configuration somewhere
possibly?
Thanks,
Hokie
RE: Accumulo with Kerberos Error
Posted by "Ott, Charlie H." <CH...@leidos.com>.
I was running 1.5.0.
I didn't see anything in the documentation about the Kerberos principals being used by accumulo to renew tickets, so I went with a different approach. But I'm sure its there somewhere and I missed it.
The default Kerberos settings will set your ticket's maximum renew time to the same time it was issued, thus by default, Kerberos will not issue renewable tickets.
What are the settings for your accumulo principal in the Kerberos database? Here is mine, notice the ticket life, renewable life:
$ kadmin.local
kadmin.local: getprinc hdfs@BI2R.LEIDOS.COM<ma...@BI2R.LEIDOS.COM>
Principal: hdfs@BI2R.LEIDOS.COM
Expiration date: [never]
Last password change: Sat Jan 25 21:33:34 EST 2014
Password expiration date: [none]
Maximum ticket life: 365 days 00:00:00
Maximum renewable life: 365 days 00:00:00
Last modified: Tue Feb 18 14:39:47 EST 2014 (root/admin@BI2R.LEIDOS.COM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 4
Key: vno 1, aes256-cts-hmac-sha1-96, no salt
Key: vno 1, aes128-cts-hmac-sha1-96, no salt
Key: vno 1, des3-cbc-sha1, no salt
Key: vno 1, arcfour-hmac, no salt
MKey: vno 1
Attributes:
Policy: [none]
Also any principals you create will use the same settings as the krbtgt principal.
From: user-return-3781-CHARLES.H.OTT=leidos.com@accumulo.apache.org [mailto:user-return-3781-CHARLES.H.OTT=leidos.com@accumulo.apache.org] On Behalf Of Hyokwon Lee
Sent: Wednesday, February 26, 2014 2:10 PM
To: user@accumulo.apache.org
Subject: Re: Accumulo with Kerberos Error
Hi Charlie,
So from my understanding as of Accumulo 1.4.1 the need to manually kinit before starting the process has been resolved. I am currently running 1.4.3 and within the logs it shows that a new ticket is issued at the end of expiration of the former ticket that was being held. What version are you running?
~Hokie
On Wed, Feb 26, 2014 at 1:57 PM, Ott, Charlie H. <CH...@leidos.com>> wrote:
Usually I manually issue the ticket to the accumulo process owners on all servers in the cluster. Then using a bash script, you can create a cronjob that will renew the tickets.
Run on master as accumulo user, ensuring passwordless ssh is enabled...
#!/bin/bash
ssh server2 "kinit -R"
ssh server3 "kinit -R"
ssh server3 "kinit -R"
So if your tablet servers are running as user "accumulo", then I usually log in to the 'accumulo' account, perform a kinit, then start-all.sh
I had a lot of issues with renewing the ticket before the expiration date, so after tweaking the principal properties, eventually I just set the ticket life for a year on that specific principal.
From: user-return-3779-CHARLES.H.OTT=leidos.com@accumulo.apache.org<ma...@accumulo.apache.org> [mailto:user-return-3779-CHARLES.H.OTT<ma...@accumulo.apache.org>] On Behalf Of Hyokwon Lee
Sent: Wednesday, February 26, 2014 1:40 PM
To: user@accumulo.apache.org<ma...@accumulo.apache.org>
Subject: Re: Accumulo with Kerberos Error
Hi All,
I have an update to my issue. Have turning on kerberos debug logging, I have found that a new kerberos ticket is initialized after the current ticket expires. However table scans fail after the expiration of the original ticket that was issued at startup of accumulo services. In the logs I see the following errors:
When trying to scan a table:
ERROR: java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server accumulo.test.local:9997
Numerous
!METADATA -> FILE_READ -> /accumulo/tables/!0/root_tablet/xxxx
Call to accumulo.test.local/127.0.0.1:8020<http://127.0.0.1:8020> failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
and the log also keeps spamming:
[securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/accumulo.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
Has anyone seen this occur before? I can simply recreate this issue by letting the accumulo services start up and expire and then just watching the logs and scanning the tables.
Thanks,
Hokie
On Tue, Feb 25, 2014 at 5:08 PM, Hyokwon Lee <hy...@gmail.com>> wrote:
Hi Sean,
The Kerberos Tickets that are being used are not renewable. Should they be? I assume even if they are after their renewable time expires I will run into the same issue?
Thanks,
Hokie
On Tue, Feb 25, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com>> wrote:
Hi Hokie!
Are the kerberos tickets you're getting renewable?
-Sean
On Tue, Feb 25, 2014 at 4:35 PM, Hyokwon Lee <hy...@gmail.com>> wrote:
I am currently running into an issue and was hoping someone may have some insight to the problem.
Running Accumulo 1.4.3 on top of a Kerberos enabled Hadoop. I followed the following instructions in the README:
"If you are running on top of hdfs with kerberos enabled, then you need to do
some extra work. First, create an Accumulo principal
kadmin.local -q "addprinc -randkey accumulo/<host.domain.name<http://host.domain.name>>"
where <host.domain.name<http://host.domain.name>> is replaced by a fully qualified domain name. Export
the principals to a keytab file. It is safer to create a unique keytab file for each
server, but you can also glob them if you wish.
kadmin.local -q "xst -k accumulo.keytab -glob accumulo*"
Place this file in $ACCUMULO_HOME/conf for every host. It should be owned by
the accumulo user and chmodded to 400. Add the following to the accumulo-env.sh
In the accumulo-site.xml file on each node, add settings for general.kerberos.keytab
and general.kerberos.principal, where the keytab setting is the absolute path
to the keytab file ($ACCUMULO_HOME is valid to use) and principal is set to
accumulo/_HOST@<REALM>, where REALM is set to your kerberos realm. You may use
_HOST in lieu of your individual host names.
<property>
<name>general.kerberos.keytab</name>
<value>$ACCUMULO_HOME/conf/accumulo.keytab</value>
</property>
<property>
<name>general.kerberos.principal</name>
<value>accumulo/_HOST@MYREALM</value>
</property>
You can then start up Accumulo as you would with the accumulo user, and it will
automatically handle the kerberos keys needed to access hdfs.
Please Note: You may have issues initializing Accumulo while running kerberos HDFS.
You can resolve this by temporarily granting the accumulo user write access to the
hdfs root directory, running init, and then revoking write permission in the root
directory (be sure to maintain access to the /accumulo directory)."
After doing so, got accumulo to come up and initially it states on start up that i authenticated using accumulo/hostname.test.local@TEST.LOCAL<ma...@TEST.LOCAL>. For the next 24 hour it is happy and everything works. However after the 24 hour marker which is when the kerberos ticket expires, I start seeing the following errors on all TServers:
[securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
[ipc.Client] WARN : Exception encountered while connecting to the server : javax.security.sasl.SasleEception: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
[securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
And as far as I can tell this just retries and keeps failing. I checked the accumulo.keytab file and it is a glob so it has the entries for every server that Accumulo is on. Also if I manually do a kinit -kt accumulo.keytab accumulo/tserver1.test.local@TEST.LOCAL<ma...@TEST.LOCAL> it works find and I get a valid ticket. I also made sure everything in hdfs under "/accumulo" is owned by accumulo so that doesn't seem to be the problem. Also made sure after kiniting I can access the directory path and all sub directories.
So far the only thing that seems to fix my issue is if I bounce all accumulo services and it is happy again. Also until I bounce the accumulo services, I get error logs stating it cannot scan any of the tables (unable to scan metadata, root_tablet, default_tablet, etc.) Has anyone else seen this issue? Did I miss a configuration somewhere possibly?
Thanks,
Hokie
--
__________________________________________
Hyokwon Lee
hyokwon.lee@gmail.com<ma...@gmail.com>
--
__________________________________________
Hyokwon Lee
hyokwon.lee@gmail.com<ma...@gmail.com>
--
__________________________________________
Hyokwon Lee
hyokwon.lee@gmail.com<ma...@gmail.com>
Re: Accumulo with Kerberos Error
Posted by Hyokwon Lee <hy...@gmail.com>.
Hi Charlie,
So from my understanding as of Accumulo 1.4.1 the need to manually kinit
before starting the process has been resolved. I am currently running
1.4.3 and within the logs it shows that a new ticket is issued at the end
of expiration of the former ticket that was being held. What version are
you running?
~Hokie
On Wed, Feb 26, 2014 at 1:57 PM, Ott, Charlie H.
<CH...@leidos.com>wrote:
> Usually I manually issue the ticket to the accumulo process owners on
> all servers in the cluster. Then using a bash script, you can create a
> cronjob that will renew the tickets.
>
>
>
> Run on master as accumulo user, ensuring passwordless ssh is enabled...
>
>
>
> #!/bin/bash
>
> ssh server2 "kinit -R"
>
> ssh server3 "kinit -R"
>
> ssh server3 "kinit -R"
>
>
>
>
>
> So if your tablet servers are running as user "accumulo", then I usually
> log in to the 'accumulo' account, perform a kinit, then start-all.sh
>
>
>
> I had a lot of issues with renewing the ticket before the expiration date,
> so after tweaking the principal properties, eventually I just set the
> ticket life for a year on that specific principal.
>
>
>
>
>
> *From:* user-return-3779-CHARLES.H.OTT=leidos.com@accumulo.apache.org[mailto:
> user-return-3779-CHARLES.H.OTT=leidos.com@accumulo.apache.org] *On Behalf
> Of *Hyokwon Lee
> *Sent:* Wednesday, February 26, 2014 1:40 PM
> *To:* user@accumulo.apache.org
> *Subject:* Re: Accumulo with Kerberos Error
>
>
>
> Hi All,
>
>
>
> I have an update to my issue. Have turning on kerberos debug logging, I
> have found that a new kerberos ticket is initialized after the current
> ticket expires. However table scans fail after the expiration of the
> original ticket that was issued at startup of accumulo services. In the
> logs I see the following errors:
>
>
>
> When trying to scan a table:
>
> ERROR: java.lang.RuntimeException:
> org.apache.accumulo.core.client.impl.AccumuloServerException: Error on
> server accumulo.test.local:9997
>
>
>
> Numerous
>
> !METADATA -> FILE_READ -> /accumulo/tables/!0/root_tablet/xxxx
>
> Call to accumulo.test.local/127.0.0.1:8020 failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)]
>
>
>
> and the log also keeps spamming:
>
>
>
> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/accumulo.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>
>
>
> Has anyone seen this occur before? I can simply recreate this issue by
> letting the accumulo services start up and expire and then just watching
> the logs and scanning the tables.
>
>
>
> Thanks,
>
>
>
> Hokie
>
>
>
> On Tue, Feb 25, 2014 at 5:08 PM, Hyokwon Lee <hy...@gmail.com>
> wrote:
>
> Hi Sean,
>
>
>
> The Kerberos Tickets that are being used are not renewable. Should they
> be? I assume even if they are after their renewable time expires I will
> run into the same issue?
>
>
>
> Thanks,
>
>
>
> Hokie
>
>
>
> On Tue, Feb 25, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com>
> wrote:
>
> Hi Hokie!
>
>
>
> Are the kerberos tickets you're getting renewable?
>
>
>
> -Sean
>
>
>
>
>
> On Tue, Feb 25, 2014 at 4:35 PM, Hyokwon Lee <hy...@gmail.com>
> wrote:
>
> I am currently running into an issue and was hoping someone may have some
> insight to the problem.
>
>
>
> Running Accumulo 1.4.3 on top of a Kerberos enabled Hadoop. I followed the
> following instructions in the README:
>
>
>
> "If you are running on top of hdfs with kerberos enabled, then you need to do
>
> some extra work. First, create an Accumulo principal
>
>
>
> kadmin.local -q "addprinc -randkey accumulo/<host.domain.name>"
>
>
>
> where <host.domain.name> is replaced by a fully qualified domain name. Export
>
> the principals to a keytab file. It is safer to create a unique keytab file for each
>
> server, but you can also glob them if you wish.
>
>
>
> kadmin.local -q "xst -k accumulo.keytab -glob accumulo*"
>
>
>
> Place this file in $ACCUMULO_HOME/conf for every host. It should be owned by
>
> the accumulo user and chmodded to 400. Add the following to the accumulo-env.sh
>
>
>
> In the accumulo-site.xml file on each node, add settings for general.kerberos.keytab
>
> and general.kerberos.principal, where the keytab setting is the absolute path
>
> to the keytab file ($ACCUMULO_HOME is valid to use) and principal is set to
>
> accumulo/_HOST@<REALM>, where REALM is set to your kerberos realm. You may use
>
> _HOST in lieu of your individual host names.
>
>
>
> <property>
>
> <name>general.kerberos.keytab</name>
>
> <value>$ACCUMULO_HOME/conf/accumulo.keytab</value>
>
> </property>
>
>
>
> <property>
>
> <name>general.kerberos.principal</name>
>
> <value>accumulo/_HOST@MYREALM</value>
>
> </property>
>
>
>
> You can then start up Accumulo as you would with the accumulo user, and it will
>
> automatically handle the kerberos keys needed to access hdfs.
>
>
>
> Please Note: You may have issues initializing Accumulo while running kerberos HDFS.
>
> You can resolve this by temporarily granting the accumulo user write access to the
>
> hdfs root directory, running init, and then revoking write permission in the root
>
> directory (be sure to maintain access to the /accumulo directory)."
>
>
>
>
>
>
>
> After doing so, got accumulo to come up and initially it states on start up that i authenticated using accumulo/hostname.test.local@TEST.LOCAL. For the next 24 hour it is happy and everything works. However after the 24 hour marker which is when the kerberos ticket expires, I start seeing the following errors on all TServers:
>
>
>
>
>
>
>
> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>
> [ipc.Client] WARN : Exception encountered while connecting to the server : javax.security.sasl.SasleEception: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>
> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> And as far as I can tell this just retries and keeps failing. I checked the accumulo.keytab file and it is a glob so it has the entries for every server that Accumulo is on. Also if I manually do a kinit -kt accumulo.keytab accumulo/tserver1.test.local@TEST.LOCAL it works find and I get a valid ticket. I also made sure everything in hdfs under "/accumulo" is owned by accumulo so that doesn't seem to be the problem. Also made sure after kiniting I can access the directory path and all sub directories.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> So far the only thing that seems to fix my issue is if I bounce all accumulo services and it is happy again. Also until I bounce the accumulo services, I get error logs stating it cannot scan any of the tables (unable to scan metadata, root_tablet, default_tablet, etc.) Has anyone else seen this issue? Did I miss a configuration somewhere possibly?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Thanks,
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Hokie
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> __________________________________________
> Hyokwon Lee
> hyokwon.lee@gmail.com
>
>
>
>
>
> --
>
> __________________________________________
> Hyokwon Lee
> hyokwon.lee@gmail.com
>
--
__________________________________________
Hyokwon Lee
hyokwon.lee@gmail.com
RE: Accumulo with Kerberos Error
Posted by "Ott, Charlie H." <CH...@leidos.com>.
Usually I manually issue the ticket to the accumulo process owners on all servers in the cluster. Then using a bash script, you can create a cronjob that will renew the tickets.
Run on master as accumulo user, ensuring passwordless ssh is enabled...
#!/bin/bash
ssh server2 "kinit -R"
ssh server3 "kinit -R"
ssh server3 "kinit -R"
So if your tablet servers are running as user "accumulo", then I usually log in to the 'accumulo' account, perform a kinit, then start-all.sh
I had a lot of issues with renewing the ticket before the expiration date, so after tweaking the principal properties, eventually I just set the ticket life for a year on that specific principal.
From: user-return-3779-CHARLES.H.OTT=leidos.com@accumulo.apache.org [mailto:user-return-3779-CHARLES.H.OTT=leidos.com@accumulo.apache.org] On Behalf Of Hyokwon Lee
Sent: Wednesday, February 26, 2014 1:40 PM
To: user@accumulo.apache.org
Subject: Re: Accumulo with Kerberos Error
Hi All,
I have an update to my issue. Have turning on kerberos debug logging, I have found that a new kerberos ticket is initialized after the current ticket expires. However table scans fail after the expiration of the original ticket that was issued at startup of accumulo services. In the logs I see the following errors:
When trying to scan a table:
ERROR: java.lang.RuntimeException: org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server accumulo.test.local:9997
Numerous
!METADATA -> FILE_READ -> /accumulo/tables/!0/root_tablet/xxxx
Call to accumulo.test.local/127.0.0.1:8020<http://127.0.0.1:8020> failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
and the log also keeps spamming:
[securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/accumulo.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
Has anyone seen this occur before? I can simply recreate this issue by letting the accumulo services start up and expire and then just watching the logs and scanning the tables.
Thanks,
Hokie
On Tue, Feb 25, 2014 at 5:08 PM, Hyokwon Lee <hy...@gmail.com>> wrote:
Hi Sean,
The Kerberos Tickets that are being used are not renewable. Should they be? I assume even if they are after their renewable time expires I will run into the same issue?
Thanks,
Hokie
On Tue, Feb 25, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com>> wrote:
Hi Hokie!
Are the kerberos tickets you're getting renewable?
-Sean
On Tue, Feb 25, 2014 at 4:35 PM, Hyokwon Lee <hy...@gmail.com>> wrote:
I am currently running into an issue and was hoping someone may have some insight to the problem.
Running Accumulo 1.4.3 on top of a Kerberos enabled Hadoop. I followed the following instructions in the README:
"If you are running on top of hdfs with kerberos enabled, then you need to do
some extra work. First, create an Accumulo principal
kadmin.local -q "addprinc -randkey accumulo/<host.domain.name<http://host.domain.name>>"
where <host.domain.name<http://host.domain.name>> is replaced by a fully qualified domain name. Export
the principals to a keytab file. It is safer to create a unique keytab file for each
server, but you can also glob them if you wish.
kadmin.local -q "xst -k accumulo.keytab -glob accumulo*"
Place this file in $ACCUMULO_HOME/conf for every host. It should be owned by
the accumulo user and chmodded to 400. Add the following to the accumulo-env.sh
In the accumulo-site.xml file on each node, add settings for general.kerberos.keytab
and general.kerberos.principal, where the keytab setting is the absolute path
to the keytab file ($ACCUMULO_HOME is valid to use) and principal is set to
accumulo/_HOST@<REALM>, where REALM is set to your kerberos realm. You may use
_HOST in lieu of your individual host names.
<property>
<name>general.kerberos.keytab</name>
<value>$ACCUMULO_HOME/conf/accumulo.keytab</value>
</property>
<property>
<name>general.kerberos.principal</name>
<value>accumulo/_HOST@MYREALM</value>
</property>
You can then start up Accumulo as you would with the accumulo user, and it will
automatically handle the kerberos keys needed to access hdfs.
Please Note: You may have issues initializing Accumulo while running kerberos HDFS.
You can resolve this by temporarily granting the accumulo user write access to the
hdfs root directory, running init, and then revoking write permission in the root
directory (be sure to maintain access to the /accumulo directory)."
After doing so, got accumulo to come up and initially it states on start up that i authenticated using accumulo/hostname.test.local@TEST.LOCAL<ma...@TEST.LOCAL>. For the next 24 hour it is happy and everything works. However after the 24 hour marker which is when the kerberos ticket expires, I start seeing the following errors on all TServers:
[securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
[ipc.Client] WARN : Exception encountered while connecting to the server : javax.security.sasl.SasleEception: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
[securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
And as far as I can tell this just retries and keeps failing. I checked the accumulo.keytab file and it is a glob so it has the entries for every server that Accumulo is on. Also if I manually do a kinit -kt accumulo.keytab accumulo/tserver1.test.local@TEST.LOCAL<ma...@TEST.LOCAL> it works find and I get a valid ticket. I also made sure everything in hdfs under "/accumulo" is owned by accumulo so that doesn't seem to be the problem. Also made sure after kiniting I can access the directory path and all sub directories.
So far the only thing that seems to fix my issue is if I bounce all accumulo services and it is happy again. Also until I bounce the accumulo services, I get error logs stating it cannot scan any of the tables (unable to scan metadata, root_tablet, default_tablet, etc.) Has anyone else seen this issue? Did I miss a configuration somewhere possibly?
Thanks,
Hokie
--
__________________________________________
Hyokwon Lee
hyokwon.lee@gmail.com<ma...@gmail.com>
--
__________________________________________
Hyokwon Lee
hyokwon.lee@gmail.com<ma...@gmail.com>
Re: Accumulo with Kerberos Error
Posted by Hyokwon Lee <hy...@gmail.com>.
Hi All,
I have an update to my issue. Have turning on kerberos debug logging, I
have found that a new kerberos ticket is initialized after the current
ticket expires. However table scans fail after the expiration of the
original ticket that was issued at startup of accumulo services. In the
logs I see the following errors:
When trying to scan a table:
ERROR: java.lang.RuntimeException:
org.apache.accumulo.core.client.impl.AccumuloServerException: Error on
server accumulo.test.local:9997
Numerous
!METADATA -> FILE_READ -> /accumulo/tables/!0/root_tablet/xxxx
Call to accumulo.test.local/127.0.0.1:8020 failed on local exception:
java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]
and the log also keeps spamming:
[securty.UserGroupInformation] ERROR: PrivilegedActionException
as:accumulo/accumulo.test.local@TEST.LOCAL (auth:KERBEROS)
cause:javax.security.sasl.SaslException: GSS initiate failed [Caused
by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]
Has anyone seen this occur before? I can simply recreate this issue by
letting the accumulo services start up and expire and then just watching
the logs and scanning the tables.
Thanks,
Hokie
On Tue, Feb 25, 2014 at 5:08 PM, Hyokwon Lee <hy...@gmail.com> wrote:
> Hi Sean,
>
> The Kerberos Tickets that are being used are not renewable. Should they
> be? I assume even if they are after their renewable time expires I will
> run into the same issue?
>
> Thanks,
>
> Hokie
>
>
> On Tue, Feb 25, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com>wrote:
>
>> Hi Hokie!
>>
>> Are the kerberos tickets you're getting renewable?
>>
>> -Sean
>>
>>
>>
>> On Tue, Feb 25, 2014 at 4:35 PM, Hyokwon Lee <hy...@gmail.com>wrote:
>>
>>> I am currently running into an issue and was hoping someone may have
>>> some insight to the problem.
>>>
>>> Running Accumulo 1.4.3 on top of a Kerberos enabled Hadoop. I followed
>>> the following instructions in the README:
>>>
>>> "If you are running on top of hdfs with kerberos enabled, then you need to do
>>> some extra work. First, create an Accumulo principal
>>>
>>> kadmin.local -q "addprinc -randkey accumulo/<host.domain.name>"
>>>
>>> where <host.domain.name> is replaced by a fully qualified domain name. Export
>>> the principals to a keytab file. It is safer to create a unique keytab file for each
>>> server, but you can also glob them if you wish.
>>>
>>> kadmin.local -q "xst -k accumulo.keytab -glob accumulo*"
>>>
>>> Place this file in $ACCUMULO_HOME/conf for every host. It should be owned by
>>> the accumulo user and chmodded to 400. Add the following to the accumulo-env.sh
>>>
>>> In the accumulo-site.xml file on each node, add settings for general.kerberos.keytab
>>> and general.kerberos.principal, where the keytab setting is the absolute path
>>> to the keytab file ($ACCUMULO_HOME is valid to use) and principal is set to
>>> accumulo/_HOST@<REALM>, where REALM is set to your kerberos realm. You may use
>>> _HOST in lieu of your individual host names.
>>>
>>> <property>
>>> <name>general.kerberos.keytab</name>
>>> <value>$ACCUMULO_HOME/conf/accumulo.keytab</value>
>>> </property>
>>>
>>> <property>
>>> <name>general.kerberos.principal</name>
>>> <value>accumulo/_HOST@MYREALM</value>
>>> </property>
>>>
>>> You can then start up Accumulo as you would with the accumulo user, and it will
>>> automatically handle the kerberos keys needed to access hdfs.
>>>
>>> Please Note: You may have issues initializing Accumulo while running kerberos HDFS.
>>> You can resolve this by temporarily granting the accumulo user write access to the
>>> hdfs root directory, running init, and then revoking write permission in the root
>>> directory (be sure to maintain access to the /accumulo directory)."
>>>
>>>
>>> After doing so, got accumulo to come up and initially it states on start up that i authenticated using accumulo/hostname.test.local@TEST.LOCAL. For the next 24 hour it is happy and everything works. However after the 24 hour marker which is when the kerberos ticket expires, I start seeing the following errors on all TServers:
>>>
>>>
>>> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>>>
>>> [ipc.Client] WARN : Exception encountered while connecting to the server : javax.security.sasl.SasleEception: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>>>
>>> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>>>
>>>
>>> And as far as I can tell this just retries and keeps failing. I checked the accumulo.keytab file and it is a glob so it has the entries for every server that Accumulo is on. Also if I manually do a kinit -kt accumulo.keytab accumulo/tserver1.test.local@TEST.LOCAL it works find and I get a valid ticket. I also made sure everything in hdfs under "/accumulo" is owned by accumulo so that doesn't seem to be the problem. Also made sure after kiniting I can access the directory path and all sub directories.
>>>
>>>
>>> So far the only thing that seems to fix my issue is if I bounce all accumulo services and it is happy again. Also until I bounce the accumulo services, I get error logs stating it cannot scan any of the tables (unable to scan metadata, root_tablet, default_tablet, etc.) Has anyone else seen this issue? Did I miss a configuration somewhere possibly?
>>>
>>>
>>> Thanks,
>>>
>>>
>>> Hokie
>>>
>>>
>>>
>>
>
>
> --
> __________________________________________
> Hyokwon Lee
> hyokwon.lee@gmail.com
>
--
__________________________________________
Hyokwon Lee
hyokwon.lee@gmail.com
Re: Accumulo with Kerberos Error
Posted by Hyokwon Lee <hy...@gmail.com>.
I am seeing a couple differences in my kerberos debug logs depending on if
it is creating the ticket with the keytab or myself manually kiniting using
the same keytab.
Kerberos Debug Log for `hadoop fs -ls /accumulo` after I manually `kinit
-kt /opt/accumulo/conf/accumulo.keytab accumulo/accumulo.test.local`:
Config name: /etc/krb5.conf
>>>KinitOptions cache name is /tmp/krb5cc_0
>>>DEBUG <CCacheInputStream> client principal is
accumulo/accumulo.test.local@TEST.LOCAL
>>>DEBUG <CCacheInputStream> server principal is
krbtgt/TEST.LOCAL@TEST.LOCAL
>>>DEBUG <CCacheInputStream> key type: 18
>>>DEBUG <CCacheInputStream> auth time: Thu Feb 27 12:40:33 PST 2014
>>>DEBUG <CCacheInputStream> start time: Thu Feb 27 12:40:33 PST 2014
>>>DEBUG <CCacheInputStream> end time: Thu Feb 27 12:45:33 PST 2014
>>>DEBUG <CCacheInputStream> renew_till time: Thu Feb 27 13:00:33 PST 2014
CCacheInputStream: readFlags() FORWARDABLE; RENEWABLE; INITIAL;
Accumulo Kerberos Debug Log:
Client Principal = accumulo/accumulo.test.local@TEST.LOCAL
Server Principal = hdfs/accumulo.test.local@TEST.LOCAL
Forwardable Ticket true
Forwarded Ticket false
Proxiable Ticket false
Proxy Ticket false
Postdated Ticket false
Renewable Ticket false
Initial Ticket false
Auth Time = Thu Feb 27 12:43:22 PST 2014
Start Time = Thu Feb 27 12:43:25 PST 2014
End Time = Thu Feb 27 12:48:22 PST 2014
Renew Till = null
Major differences to note are Server Principal, Renewable and Initial
Flags, and the Renew Till values. Could this be the cause of my current
issue? If so why would accumulo using the same keytab and principal get
these different values?
Thanks,
Hokie
RE: Accumulo with Kerberos Error
Posted by "Ott, Charlie H." <CH...@leidos.com>.
My only thoughts are that this is why I ended up setting my ticket life to a year for the accumulo principal.
I actually had tickets renewing daily for about a week or so, then it seemed like out of nowhere, they would occasionally be expired as if the renewal had not occurred. Hopefully someone can give you some better feedback than that.
Charles
From: user-return-3791-CHARLES.H.OTT=leidos.com@accumulo.apache.org [mailto:user-return-3791-CHARLES.H.OTT=leidos.com@accumulo.apache.org] On Behalf Of Hyokwon Lee
Sent: Thursday, February 27, 2014 10:30 AM
To: user@accumulo.apache.org
Subject: Re: Accumulo with Kerberos Error
Hi Charles and John,
So I made the tickets renewable and regenerated the keytabs for accumulo. The ticket life was set to 1 hour with the renew life set to 1 day. However after the hour is up, I get a different error:
Call to accumulo.test.local/127.0.0.1:8020<http://127.0.0.1:8020> failed on local exception: java.io.IOException: java.lang.IllegalStateException: This ticket is no longer valid
immediately followed by:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
I listed the ticket to make sure I had a valid ticket that was for 1 hour with a renew expiration of 1 day. Then I went in and made sure that the user running the service has a valid ticket, and just in case on a few test added a cron job that renews the ticket before it expires. Either way I get the same error.
You mentioned that the system automatically renews the ticket when it expires, and kerberos debug logging enabled I am seeing the following:
Found ticket for accumulo/accumulo.test.local@TEST.LOCAL<ma...@TEST.LOCAL> to go to krbtgt/TEST.LOCAL@TEST.LOCAL<ma...@TEST.LOCAL> expiring on Thu Feb 27 07:14:20 PST 2014
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for accumulo/accumulo.test.local@TEST.LOCAL<ma...@TEST.LOCAL> to go to krbtgt/TEST.LOCAL@TEST.LOCAL<ma...@TEST.LOCAL> expiring on Thu Feb 27 07:14:20 PST 2014
Found ticket for accumulo/accumulo.test.local@TEST.LOCAL<ma...@TEST.LOCAL> to go to hdfs/accumulo.test.local@TEST.LOCAL<ma...@TEST.LOCAL> expiring on Thu Feb 27 07:14:20 PST 2014
There is no errors logged for the kerberos ticket creation however the "This ticket is no longer valid " error leads me to believe that the current ticket it had been using was destroyed.
Any thoughts?
Thanks,
Hokie
On Wed, Feb 26, 2014 at 3:11 PM, Hyokwon Lee <hy...@gmail.com>> wrote:
Charles and John,
Thanks for the help. I am going to make the tickets renewable and give it a test. I will let you guys know if it works.
~Hokie
On Wed, Feb 26, 2014 at 2:43 PM, John Vines <vi...@apache.org>> wrote:
No, they need to be renewable. The system automatically renews them when they expire.
On Tue, Feb 25, 2014 at 5:08 PM, Hyokwon Lee <hy...@gmail.com>> wrote:
Hi Sean,
The Kerberos Tickets that are being used are not renewable. Should they be? I assume even if they are after their renewable time expires I will run into the same issue?
Thanks,
Hokie
On Tue, Feb 25, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com>> wrote:
Hi Hokie!
Are the kerberos tickets you're getting renewable?
-Sean
On Tue, Feb 25, 2014 at 4:35 PM, Hyokwon Lee <hy...@gmail.com>> wrote:
I am currently running into an issue and was hoping someone may have some insight to the problem.
Running Accumulo 1.4.3 on top of a Kerberos enabled Hadoop. I followed the following instructions in the README:
"If you are running on top of hdfs with kerberos enabled, then you need to do
some extra work. First, create an Accumulo principal
kadmin.local -q "addprinc -randkey accumulo/<host.domain.name<http://host.domain.name>>"
where <host.domain.name<http://host.domain.name>> is replaced by a fully qualified domain name. Export
the principals to a keytab file. It is safer to create a unique keytab file for each
server, but you can also glob them if you wish.
kadmin.local -q "xst -k accumulo.keytab -glob accumulo*"
Place this file in $ACCUMULO_HOME/conf for every host. It should be owned by
the accumulo user and chmodded to 400. Add the following to the accumulo-env.sh
In the accumulo-site.xml file on each node, add settings for general.kerberos.keytab
and general.kerberos.principal, where the keytab setting is the absolute path
to the keytab file ($ACCUMULO_HOME is valid to use) and principal is set to
accumulo/_HOST@<REALM>, where REALM is set to your kerberos realm. You may use
_HOST in lieu of your individual host names.
<property>
<name>general.kerberos.keytab</name>
<value>$ACCUMULO_HOME/conf/accumulo.keytab</value>
</property>
<property>
<name>general.kerberos.principal</name>
<value>accumulo/_HOST@MYREALM</value>
</property>
You can then start up Accumulo as you would with the accumulo user, and it will
automatically handle the kerberos keys needed to access hdfs.
Please Note: You may have issues initializing Accumulo while running kerberos HDFS.
You can resolve this by temporarily granting the accumulo user write access to the
hdfs root directory, running init, and then revoking write permission in the root
directory (be sure to maintain access to the /accumulo directory)."
After doing so, got accumulo to come up and initially it states on start up that i authenticated using accumulo/hostname.test.local@TEST.LOCAL<ma...@TEST.LOCAL>. For the next 24 hour it is happy and everything works. However after the 24 hour marker which is when the kerberos ticket expires, I start seeing the following errors on all TServers:
[securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
[ipc.Client] WARN : Exception encountered while connecting to the server : javax.security.sasl.SasleEception: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
[securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
And as far as I can tell this just retries and keeps failing. I checked the accumulo.keytab file and it is a glob so it has the entries for every server that Accumulo is on. Also if I manually do a kinit -kt accumulo.keytab accumulo/tserver1.test.local@TEST.LOCAL<ma...@TEST.LOCAL> it works find and I get a valid ticket. I also made sure everything in hdfs under "/accumulo" is owned by accumulo so that doesn't seem to be the problem. Also made sure after kiniting I can access the directory path and all sub directories.
So far the only thing that seems to fix my issue is if I bounce all accumulo services and it is happy again. Also until I bounce the accumulo services, I get error logs stating it cannot scan any of the tables (unable to scan metadata, root_tablet, default_tablet, etc.) Has anyone else seen this issue? Did I miss a configuration somewhere possibly?
Thanks,
Hokie
--
__________________________________________
Hyokwon Lee
hyokwon.lee@gmail.com<ma...@gmail.com>
--
__________________________________________
Hyokwon Lee
hyokwon.lee@gmail.com<ma...@gmail.com>
--
__________________________________________
Hyokwon Lee
hyokwon.lee@gmail.com<ma...@gmail.com>
Re: Accumulo with Kerberos Error
Posted by Hyokwon Lee <hy...@gmail.com>.
Hi Charles and John,
So I made the tickets renewable and regenerated the keytabs for accumulo.
The ticket life was set to 1 hour with the renew life set to 1 day.
However after the hour is up, I get a different error:
Call to accumulo.test.local/127.0.0.1:8020 failed on local exception:
java.io.IOException: java.lang.IllegalStateException: This ticket is no
longer valid
immediately followed by:
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)]
I listed the ticket to make sure I had a valid ticket that was for 1
hour with a renew expiration of 1 day. Then I went in and made sure
that the user running the service has a valid ticket, and just in case
on a few test added a cron job that renews the ticket before it
expires. Either way I get the same error.
You mentioned that the system automatically renews the ticket when it
expires, and kerberos debug logging enabled I am seeing the following:
Found ticket for accumulo/accumulo.test.local@TEST.LOCAL to go to
krbtgt/TEST.LOCAL@TEST.LOCAL expiring on Thu Feb 27 07:14:20 PST 2014
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for accumulo/accumulo.test.local@TEST.LOCAL to go to
krbtgt/TEST.LOCAL@TEST.LOCAL expiring on Thu Feb 27 07:14:20 PST 2014
Found ticket for accumulo/accumulo.test.local@TEST.LOCAL to go to
hdfs/accumulo.test.local@TEST.LOCAL expiring on Thu Feb 27 07:14:20
PST 2014
There is no errors logged for the kerberos ticket creation however the
"This ticket is no longer valid " error leads me to believe that the
current ticket it had been using was destroyed.
Any thoughts?
Thanks,
Hokie
On Wed, Feb 26, 2014 at 3:11 PM, Hyokwon Lee <hy...@gmail.com> wrote:
> Charles and John,
>
> Thanks for the help. I am going to make the tickets renewable and give
> it a test. I will let you guys know if it works.
>
> ~Hokie
>
>
> On Wed, Feb 26, 2014 at 2:43 PM, John Vines <vi...@apache.org> wrote:
>
>> No, they need to be renewable. The system automatically renews them when
>> they expire.
>>
>>
>> On Tue, Feb 25, 2014 at 5:08 PM, Hyokwon Lee <hy...@gmail.com>wrote:
>>
>>> Hi Sean,
>>>
>>> The Kerberos Tickets that are being used are not renewable. Should
>>> they be? I assume even if they are after their renewable time expires I
>>> will run into the same issue?
>>>
>>> Thanks,
>>>
>>> Hokie
>>>
>>>
>>> On Tue, Feb 25, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com>wrote:
>>>
>>>> Hi Hokie!
>>>>
>>>> Are the kerberos tickets you're getting renewable?
>>>>
>>>> -Sean
>>>>
>>>>
>>>>
>>>> On Tue, Feb 25, 2014 at 4:35 PM, Hyokwon Lee <hy...@gmail.com>wrote:
>>>>
>>>>> I am currently running into an issue and was hoping someone may have
>>>>> some insight to the problem.
>>>>>
>>>>> Running Accumulo 1.4.3 on top of a Kerberos enabled Hadoop. I followed
>>>>> the following instructions in the README:
>>>>>
>>>>> "If you are running on top of hdfs with kerberos enabled, then you need to do
>>>>> some extra work. First, create an Accumulo principal
>>>>>
>>>>> kadmin.local -q "addprinc -randkey accumulo/<host.domain.name>"
>>>>>
>>>>> where <host.domain.name> is replaced by a fully qualified domain name. Export
>>>>> the principals to a keytab file. It is safer to create a unique keytab file for each
>>>>> server, but you can also glob them if you wish.
>>>>>
>>>>> kadmin.local -q "xst -k accumulo.keytab -glob accumulo*"
>>>>>
>>>>> Place this file in $ACCUMULO_HOME/conf for every host. It should be owned by
>>>>> the accumulo user and chmodded to 400. Add the following to the accumulo-env.sh
>>>>>
>>>>> In the accumulo-site.xml file on each node, add settings for general.kerberos.keytab
>>>>> and general.kerberos.principal, where the keytab setting is the absolute path
>>>>> to the keytab file ($ACCUMULO_HOME is valid to use) and principal is set to
>>>>> accumulo/_HOST@<REALM>, where REALM is set to your kerberos realm. You may use
>>>>> _HOST in lieu of your individual host names.
>>>>>
>>>>> <property>
>>>>> <name>general.kerberos.keytab</name>
>>>>> <value>$ACCUMULO_HOME/conf/accumulo.keytab</value>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>general.kerberos.principal</name>
>>>>> <value>accumulo/_HOST@MYREALM</value>
>>>>> </property>
>>>>>
>>>>> You can then start up Accumulo as you would with the accumulo user, and it will
>>>>> automatically handle the kerberos keys needed to access hdfs.
>>>>>
>>>>> Please Note: You may have issues initializing Accumulo while running kerberos HDFS.
>>>>> You can resolve this by temporarily granting the accumulo user write access to the
>>>>> hdfs root directory, running init, and then revoking write permission in the root
>>>>> directory (be sure to maintain access to the /accumulo directory)."
>>>>>
>>>>>
>>>>> After doing so, got accumulo to come up and initially it states on start up that i authenticated using accumulo/hostname.test.local@TEST.LOCAL. For the next 24 hour it is happy and everything works. However after the 24 hour marker which is when the kerberos ticket expires, I start seeing the following errors on all TServers:
>>>>>
>>>>>
>>>>> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>>>>>
>>>>> [ipc.Client] WARN : Exception encountered while connecting to the server : javax.security.sasl.SasleEception: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>>>>>
>>>>> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>>>>>
>>>>>
>>>>> And as far as I can tell this just retries and keeps failing. I checked the accumulo.keytab file and it is a glob so it has the entries for every server that Accumulo is on. Also if I manually do a kinit -kt accumulo.keytab accumulo/tserver1.test.local@TEST.LOCAL it works find and I get a valid ticket. I also made sure everything in hdfs under "/accumulo" is owned by accumulo so that doesn't seem to be the problem. Also made sure after kiniting I can access the directory path and all sub directories.
>>>>>
>>>>>
>>>>> So far the only thing that seems to fix my issue is if I bounce all accumulo services and it is happy again. Also until I bounce the accumulo services, I get error logs stating it cannot scan any of the tables (unable to scan metadata, root_tablet, default_tablet, etc.) Has anyone else seen this issue? Did I miss a configuration somewhere possibly?
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>> Hokie
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> __________________________________________
>>> Hyokwon Lee
>>> hyokwon.lee@gmail.com
>>>
>>
>>
>
>
> --
> __________________________________________
> Hyokwon Lee
> hyokwon.lee@gmail.com
>
--
__________________________________________
Hyokwon Lee
hyokwon.lee@gmail.com
Re: Accumulo with Kerberos Error
Posted by Hyokwon Lee <hy...@gmail.com>.
Charles and John,
Thanks for the help. I am going to make the tickets renewable and give it
a test. I will let you guys know if it works.
~Hokie
On Wed, Feb 26, 2014 at 2:43 PM, John Vines <vi...@apache.org> wrote:
> No, they need to be renewable. The system automatically renews them when
> they expire.
>
>
> On Tue, Feb 25, 2014 at 5:08 PM, Hyokwon Lee <hy...@gmail.com>wrote:
>
>> Hi Sean,
>>
>> The Kerberos Tickets that are being used are not renewable. Should they
>> be? I assume even if they are after their renewable time expires I will
>> run into the same issue?
>>
>> Thanks,
>>
>> Hokie
>>
>>
>> On Tue, Feb 25, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com>wrote:
>>
>>> Hi Hokie!
>>>
>>> Are the kerberos tickets you're getting renewable?
>>>
>>> -Sean
>>>
>>>
>>>
>>> On Tue, Feb 25, 2014 at 4:35 PM, Hyokwon Lee <hy...@gmail.com>wrote:
>>>
>>>> I am currently running into an issue and was hoping someone may have
>>>> some insight to the problem.
>>>>
>>>> Running Accumulo 1.4.3 on top of a Kerberos enabled Hadoop. I followed
>>>> the following instructions in the README:
>>>>
>>>> "If you are running on top of hdfs with kerberos enabled, then you need to do
>>>> some extra work. First, create an Accumulo principal
>>>>
>>>> kadmin.local -q "addprinc -randkey accumulo/<host.domain.name>"
>>>>
>>>> where <host.domain.name> is replaced by a fully qualified domain name. Export
>>>> the principals to a keytab file. It is safer to create a unique keytab file for each
>>>> server, but you can also glob them if you wish.
>>>>
>>>> kadmin.local -q "xst -k accumulo.keytab -glob accumulo*"
>>>>
>>>> Place this file in $ACCUMULO_HOME/conf for every host. It should be owned by
>>>> the accumulo user and chmodded to 400. Add the following to the accumulo-env.sh
>>>>
>>>> In the accumulo-site.xml file on each node, add settings for general.kerberos.keytab
>>>> and general.kerberos.principal, where the keytab setting is the absolute path
>>>> to the keytab file ($ACCUMULO_HOME is valid to use) and principal is set to
>>>> accumulo/_HOST@<REALM>, where REALM is set to your kerberos realm. You may use
>>>> _HOST in lieu of your individual host names.
>>>>
>>>> <property>
>>>> <name>general.kerberos.keytab</name>
>>>> <value>$ACCUMULO_HOME/conf/accumulo.keytab</value>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>general.kerberos.principal</name>
>>>> <value>accumulo/_HOST@MYREALM</value>
>>>> </property>
>>>>
>>>> You can then start up Accumulo as you would with the accumulo user, and it will
>>>> automatically handle the kerberos keys needed to access hdfs.
>>>>
>>>> Please Note: You may have issues initializing Accumulo while running kerberos HDFS.
>>>> You can resolve this by temporarily granting the accumulo user write access to the
>>>> hdfs root directory, running init, and then revoking write permission in the root
>>>> directory (be sure to maintain access to the /accumulo directory)."
>>>>
>>>>
>>>> After doing so, got accumulo to come up and initially it states on start up that i authenticated using accumulo/hostname.test.local@TEST.LOCAL. For the next 24 hour it is happy and everything works. However after the 24 hour marker which is when the kerberos ticket expires, I start seeing the following errors on all TServers:
>>>>
>>>>
>>>> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>>>>
>>>> [ipc.Client] WARN : Exception encountered while connecting to the server : javax.security.sasl.SasleEception: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>>>>
>>>> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>>>>
>>>>
>>>> And as far as I can tell this just retries and keeps failing. I checked the accumulo.keytab file and it is a glob so it has the entries for every server that Accumulo is on. Also if I manually do a kinit -kt accumulo.keytab accumulo/tserver1.test.local@TEST.LOCAL it works find and I get a valid ticket. I also made sure everything in hdfs under "/accumulo" is owned by accumulo so that doesn't seem to be the problem. Also made sure after kiniting I can access the directory path and all sub directories.
>>>>
>>>>
>>>> So far the only thing that seems to fix my issue is if I bounce all accumulo services and it is happy again. Also until I bounce the accumulo services, I get error logs stating it cannot scan any of the tables (unable to scan metadata, root_tablet, default_tablet, etc.) Has anyone else seen this issue? Did I miss a configuration somewhere possibly?
>>>>
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> Hokie
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> __________________________________________
>> Hyokwon Lee
>> hyokwon.lee@gmail.com
>>
>
>
--
__________________________________________
Hyokwon Lee
hyokwon.lee@gmail.com
Re: Accumulo with Kerberos Error
Posted by John Vines <vi...@apache.org>.
No, they need to be renewable. The system automatically renews them when
they expire.
On Tue, Feb 25, 2014 at 5:08 PM, Hyokwon Lee <hy...@gmail.com> wrote:
> Hi Sean,
>
> The Kerberos Tickets that are being used are not renewable. Should they
> be? I assume even if they are after their renewable time expires I will
> run into the same issue?
>
> Thanks,
>
> Hokie
>
>
> On Tue, Feb 25, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com>wrote:
>
>> Hi Hokie!
>>
>> Are the kerberos tickets you're getting renewable?
>>
>> -Sean
>>
>>
>>
>> On Tue, Feb 25, 2014 at 4:35 PM, Hyokwon Lee <hy...@gmail.com>wrote:
>>
>>> I am currently running into an issue and was hoping someone may have
>>> some insight to the problem.
>>>
>>> Running Accumulo 1.4.3 on top of a Kerberos enabled Hadoop. I followed
>>> the following instructions in the README:
>>>
>>> "If you are running on top of hdfs with kerberos enabled, then you need to do
>>> some extra work. First, create an Accumulo principal
>>>
>>> kadmin.local -q "addprinc -randkey accumulo/<host.domain.name>"
>>>
>>> where <host.domain.name> is replaced by a fully qualified domain name. Export
>>> the principals to a keytab file. It is safer to create a unique keytab file for each
>>> server, but you can also glob them if you wish.
>>>
>>> kadmin.local -q "xst -k accumulo.keytab -glob accumulo*"
>>>
>>> Place this file in $ACCUMULO_HOME/conf for every host. It should be owned by
>>> the accumulo user and chmodded to 400. Add the following to the accumulo-env.sh
>>>
>>> In the accumulo-site.xml file on each node, add settings for general.kerberos.keytab
>>> and general.kerberos.principal, where the keytab setting is the absolute path
>>> to the keytab file ($ACCUMULO_HOME is valid to use) and principal is set to
>>> accumulo/_HOST@<REALM>, where REALM is set to your kerberos realm. You may use
>>> _HOST in lieu of your individual host names.
>>>
>>> <property>
>>> <name>general.kerberos.keytab</name>
>>> <value>$ACCUMULO_HOME/conf/accumulo.keytab</value>
>>> </property>
>>>
>>> <property>
>>> <name>general.kerberos.principal</name>
>>> <value>accumulo/_HOST@MYREALM</value>
>>> </property>
>>>
>>> You can then start up Accumulo as you would with the accumulo user, and it will
>>> automatically handle the kerberos keys needed to access hdfs.
>>>
>>> Please Note: You may have issues initializing Accumulo while running kerberos HDFS.
>>> You can resolve this by temporarily granting the accumulo user write access to the
>>> hdfs root directory, running init, and then revoking write permission in the root
>>> directory (be sure to maintain access to the /accumulo directory)."
>>>
>>>
>>> After doing so, got accumulo to come up and initially it states on start up that i authenticated using accumulo/hostname.test.local@TEST.LOCAL. For the next 24 hour it is happy and everything works. However after the 24 hour marker which is when the kerberos ticket expires, I start seeing the following errors on all TServers:
>>>
>>>
>>> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>>>
>>> [ipc.Client] WARN : Exception encountered while connecting to the server : javax.security.sasl.SasleEception: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>>>
>>> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>>>
>>>
>>> And as far as I can tell this just retries and keeps failing. I checked the accumulo.keytab file and it is a glob so it has the entries for every server that Accumulo is on. Also if I manually do a kinit -kt accumulo.keytab accumulo/tserver1.test.local@TEST.LOCAL it works find and I get a valid ticket. I also made sure everything in hdfs under "/accumulo" is owned by accumulo so that doesn't seem to be the problem. Also made sure after kiniting I can access the directory path and all sub directories.
>>>
>>>
>>> So far the only thing that seems to fix my issue is if I bounce all accumulo services and it is happy again. Also until I bounce the accumulo services, I get error logs stating it cannot scan any of the tables (unable to scan metadata, root_tablet, default_tablet, etc.) Has anyone else seen this issue? Did I miss a configuration somewhere possibly?
>>>
>>>
>>> Thanks,
>>>
>>>
>>> Hokie
>>>
>>>
>>>
>>
>
>
> --
> __________________________________________
> Hyokwon Lee
> hyokwon.lee@gmail.com
>
Re: Accumulo with Kerberos Error
Posted by Hyokwon Lee <hy...@gmail.com>.
Hi Sean,
The Kerberos Tickets that are being used are not renewable. Should they
be? I assume even if they are after their renewable time expires I will
run into the same issue?
Thanks,
Hokie
On Tue, Feb 25, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com>wrote:
> Hi Hokie!
>
> Are the kerberos tickets you're getting renewable?
>
> -Sean
>
>
>
> On Tue, Feb 25, 2014 at 4:35 PM, Hyokwon Lee <hy...@gmail.com>wrote:
>
>> I am currently running into an issue and was hoping someone may have some
>> insight to the problem.
>>
>> Running Accumulo 1.4.3 on top of a Kerberos enabled Hadoop. I followed
>> the following instructions in the README:
>>
>> "If you are running on top of hdfs with kerberos enabled, then you need to do
>> some extra work. First, create an Accumulo principal
>>
>> kadmin.local -q "addprinc -randkey accumulo/<host.domain.name>"
>>
>> where <host.domain.name> is replaced by a fully qualified domain name. Export
>> the principals to a keytab file. It is safer to create a unique keytab file for each
>> server, but you can also glob them if you wish.
>>
>> kadmin.local -q "xst -k accumulo.keytab -glob accumulo*"
>>
>> Place this file in $ACCUMULO_HOME/conf for every host. It should be owned by
>> the accumulo user and chmodded to 400. Add the following to the accumulo-env.sh
>>
>> In the accumulo-site.xml file on each node, add settings for general.kerberos.keytab
>> and general.kerberos.principal, where the keytab setting is the absolute path
>> to the keytab file ($ACCUMULO_HOME is valid to use) and principal is set to
>> accumulo/_HOST@<REALM>, where REALM is set to your kerberos realm. You may use
>> _HOST in lieu of your individual host names.
>>
>> <property>
>> <name>general.kerberos.keytab</name>
>> <value>$ACCUMULO_HOME/conf/accumulo.keytab</value>
>> </property>
>>
>> <property>
>> <name>general.kerberos.principal</name>
>> <value>accumulo/_HOST@MYREALM</value>
>> </property>
>>
>> You can then start up Accumulo as you would with the accumulo user, and it will
>> automatically handle the kerberos keys needed to access hdfs.
>>
>> Please Note: You may have issues initializing Accumulo while running kerberos HDFS.
>> You can resolve this by temporarily granting the accumulo user write access to the
>> hdfs root directory, running init, and then revoking write permission in the root
>> directory (be sure to maintain access to the /accumulo directory)."
>>
>>
>> After doing so, got accumulo to come up and initially it states on start up that i authenticated using accumulo/hostname.test.local@TEST.LOCAL. For the next 24 hour it is happy and everything works. However after the 24 hour marker which is when the kerberos ticket expires, I start seeing the following errors on all TServers:
>>
>>
>> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>>
>> [ipc.Client] WARN : Exception encountered while connecting to the server : javax.security.sasl.SasleEception: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>>
>> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>>
>>
>> And as far as I can tell this just retries and keeps failing. I checked the accumulo.keytab file and it is a glob so it has the entries for every server that Accumulo is on. Also if I manually do a kinit -kt accumulo.keytab accumulo/tserver1.test.local@TEST.LOCAL it works find and I get a valid ticket. I also made sure everything in hdfs under "/accumulo" is owned by accumulo so that doesn't seem to be the problem. Also made sure after kiniting I can access the directory path and all sub directories.
>>
>>
>> So far the only thing that seems to fix my issue is if I bounce all accumulo services and it is happy again. Also until I bounce the accumulo services, I get error logs stating it cannot scan any of the tables (unable to scan metadata, root_tablet, default_tablet, etc.) Has anyone else seen this issue? Did I miss a configuration somewhere possibly?
>>
>>
>> Thanks,
>>
>>
>> Hokie
>>
>>
>>
>
--
__________________________________________
Hyokwon Lee
hyokwon.lee@gmail.com
Re: Accumulo with Kerberos Error
Posted by Sean Busbey <bu...@cloudera.com>.
Hi Hokie!
Are the kerberos tickets you're getting renewable?
-Sean
On Tue, Feb 25, 2014 at 4:35 PM, Hyokwon Lee <hy...@gmail.com> wrote:
> I am currently running into an issue and was hoping someone may have some
> insight to the problem.
>
> Running Accumulo 1.4.3 on top of a Kerberos enabled Hadoop. I followed the
> following instructions in the README:
>
> "If you are running on top of hdfs with kerberos enabled, then you need to do
> some extra work. First, create an Accumulo principal
>
> kadmin.local -q "addprinc -randkey accumulo/<host.domain.name>"
>
> where <host.domain.name> is replaced by a fully qualified domain name. Export
> the principals to a keytab file. It is safer to create a unique keytab file for each
> server, but you can also glob them if you wish.
>
> kadmin.local -q "xst -k accumulo.keytab -glob accumulo*"
>
> Place this file in $ACCUMULO_HOME/conf for every host. It should be owned by
> the accumulo user and chmodded to 400. Add the following to the accumulo-env.sh
>
> In the accumulo-site.xml file on each node, add settings for general.kerberos.keytab
> and general.kerberos.principal, where the keytab setting is the absolute path
> to the keytab file ($ACCUMULO_HOME is valid to use) and principal is set to
> accumulo/_HOST@<REALM>, where REALM is set to your kerberos realm. You may use
> _HOST in lieu of your individual host names.
>
> <property>
> <name>general.kerberos.keytab</name>
> <value>$ACCUMULO_HOME/conf/accumulo.keytab</value>
> </property>
>
> <property>
> <name>general.kerberos.principal</name>
> <value>accumulo/_HOST@MYREALM</value>
> </property>
>
> You can then start up Accumulo as you would with the accumulo user, and it will
> automatically handle the kerberos keys needed to access hdfs.
>
> Please Note: You may have issues initializing Accumulo while running kerberos HDFS.
> You can resolve this by temporarily granting the accumulo user write access to the
> hdfs root directory, running init, and then revoking write permission in the root
> directory (be sure to maintain access to the /accumulo directory)."
>
>
> After doing so, got accumulo to come up and initially it states on start up that i authenticated using accumulo/hostname.test.local@TEST.LOCAL. For the next 24 hour it is happy and everything works. However after the 24 hour marker which is when the kerberos ticket expires, I start seeing the following errors on all TServers:
>
>
> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>
> [ipc.Client] WARN : Exception encountered while connecting to the server : javax.security.sasl.SasleEception: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>
> [securty.UserGroupInformation] ERROR: PrivilegedActionException as:accumulo/tserver1.test.local@TEST.LOCAL (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
>
>
> And as far as I can tell this just retries and keeps failing. I checked the accumulo.keytab file and it is a glob so it has the entries for every server that Accumulo is on. Also if I manually do a kinit -kt accumulo.keytab accumulo/tserver1.test.local@TEST.LOCAL it works find and I get a valid ticket. I also made sure everything in hdfs under "/accumulo" is owned by accumulo so that doesn't seem to be the problem. Also made sure after kiniting I can access the directory path and all sub directories.
>
>
> So far the only thing that seems to fix my issue is if I bounce all accumulo services and it is happy again. Also until I bounce the accumulo services, I get error logs stating it cannot scan any of the tables (unable to scan metadata, root_tablet, default_tablet, etc.) Has anyone else seen this issue? Did I miss a configuration somewhere possibly?
>
>
> Thanks,
>
>
> Hokie
>
>
>