You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by takeshi <ta...@gmail.com> on 2014/01/17 06:57:36 UTC
Suggest that turn the msg "Request is a replay (34) - PROCESS_TGS"
from logging level from ERROR to WARN
Hi All,
Recently we got the error msg "Request is a replay (34) - PROCESS_TGS"
while we are using the HBase client API to put data into HBase-0.94.16 with
krb5-1.6.1 enabled. The related msg as follows...
{code}
[2014-01-15
09:40:38,452][hbase-tablepool-1-thread-3][ERROR][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1124)):
PriviledgedActionException
as:takeshi_miao@LABcause:javax.security.sasl.SaslException: GSS
initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Request is a
replay (34) - PROCESS_TGS)]
[2014-01-15
09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.logPriviledgedAction(UserGroupInformation.java:1143)):
PriviledgedAction
as:takeshi_miao@LABfrom:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
[2014-01-15
09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.ipc.SecureClient](org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:213)):
Exception encountered while connecting to the server :
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Request is a
replay (34) - PROCESS_TGS)]
[2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:657)):
Initiating logout for takeshi_miao@LAB
[2014-01-15
09:40:38,454][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.logout(UserGroupInformation.java:154)):
hadoop logout
[2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:667)):
Initiating re-login for takeshi_miao@LAB
[2014-01-15
09:40:38,455][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.login(UserGroupInformation.java:146)):
hadoop login
[2014-01-15
09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:95)):
hadoop login commit
[2014-01-15
09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:100)):
using existing subject:[takeshi_miao@LAB, UnixPrincipal: takeshi_miao,
UnixNumericUserPrincipal: 501, UnixNumericGroupPrincipal [Primary Group]:
501, UnixNumericGroupPrincipal [Supplementary Group]: 502, takeshi_miao@LAB]
{code}
At the beginning, we were worry about the "data loss" occurring while we
found the "Request is a replay (34) - PROCESS_TGS" (especially it is the
ERROR level) in log, but after code study, this is basically *NOT* a data
loss issue due to HBase client API would try 5 times internally
(o.a.h.hbase.ipc.SecureClient, L#296, a one thread) and also 10 times of
retry externally (o.a.h.hbase.client.HConnectionManager, L#1661, for all
failed thread), The HTable API would also throw IOEcxeption to client code
if any thread still fail after these retries.
Based on HBase users' viewpoint as us, we think this is better to change
the logging level from 'ERROR' to 'WARN', due to the 'ERROR' level had been
confused us for a while...But this code change may need to change boht
HBase code and Hadoop code as well; so I am wondering how community think
about this small thing but may be important to the pure HBase users.
P.S. more details are in this article
http://dumbointaiwan.blogspot.tw/2014/01/hbase-and-keberos-request-is-replay-34.html
Tks
Best regards
takeshi
Re: Suggest that turn the msg "Request is a replay (34) -
PROCESS_TGS" from logging level from ERROR to WARN
Posted by takeshi <ta...@gmail.com>.
Hi Stack,
https://issues.apache.org/jira/browse/HBASE-10379 ticket created,
tks for your response~
Best regards
takeshi
2014/1/18 Stack <st...@duboce.net>
> Lets change it in both places. Please file issues. Lets try minimize the
> freakout incidents running your hbase/hadoop cluster.
> Thanks Takeshi,
> St.Ack
>
>
> On Thu, Jan 16, 2014 at 9:57 PM, takeshi <ta...@gmail.com> wrote:
>
> > Hi All,
> >
> > Recently we got the error msg "Request is a replay (34) - PROCESS_TGS"
> > while we are using the HBase client API to put data into HBase-0.94.16
> with
> > krb5-1.6.1 enabled. The related msg as follows...
> > {code}
> > [2014-01-15
> >
> >
> 09:40:38,452][hbase-tablepool-1-thread-3][ERROR][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1124)):
> > PriviledgedActionException
> > as:takeshi_miao@LABcause:javax.security.sasl.SaslException: GSS
> > initiate failed [Caused by
> > GSSException: No valid credentials provided (Mechanism level: Request is
> a
> > replay (34) - PROCESS_TGS)]
> > [2014-01-15
> >
> >
> 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.logPriviledgedAction(UserGroupInformation.java:1143)):
> > PriviledgedAction
> > as:takeshi_miao@LABfrom
> > :sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> >
> > [2014-01-15
> >
> >
> 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.ipc.SecureClient](org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:213)):
> > Exception encountered while connecting to the server :
> > javax.security.sasl.SaslException: GSS initiate failed [Caused by
> > GSSException: No valid credentials provided (Mechanism level: Request is
> a
> > replay (34) - PROCESS_TGS)]
> > [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
> >
> >
> ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:657)):
> > Initiating logout for takeshi_miao@LAB
> > [2014-01-15
> >
> >
> 09:40:38,454][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.logout(UserGroupInformation.java:154)):
> > hadoop logout
> > [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
> >
> >
> ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:667)):
> > Initiating re-login for takeshi_miao@LAB
> > [2014-01-15
> >
> >
> 09:40:38,455][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.login(UserGroupInformation.java:146)):
> > hadoop login
> > [2014-01-15
> >
> >
> 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:95)):
> > hadoop login commit
> > [2014-01-15
> >
> >
> 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:100)):
> > using existing subject:[takeshi_miao@LAB, UnixPrincipal: takeshi_miao,
> > UnixNumericUserPrincipal: 501, UnixNumericGroupPrincipal [Primary Group]:
> > 501, UnixNumericGroupPrincipal [Supplementary Group]: 502,
> takeshi_miao@LAB
> > ]
> > {code}
> >
> > At the beginning, we were worry about the "data loss" occurring while we
> > found the "Request is a replay (34) - PROCESS_TGS" (especially it is the
> > ERROR level) in log, but after code study, this is basically *NOT* a data
> > loss issue due to HBase client API would try 5 times internally
> > (o.a.h.hbase.ipc.SecureClient, L#296, a one thread) and also 10 times of
> > retry externally (o.a.h.hbase.client.HConnectionManager, L#1661, for all
> > failed thread), The HTable API would also throw IOEcxeption to client
> code
> > if any thread still fail after these retries.
> >
> > Based on HBase users' viewpoint as us, we think this is better to change
> > the logging level from 'ERROR' to 'WARN', due to the 'ERROR' level had
> been
> > confused us for a while...But this code change may need to change boht
> > HBase code and Hadoop code as well; so I am wondering how community think
> > about this small thing but may be important to the pure HBase users.
> >
> > P.S. more details are in this article
> >
> >
> http://dumbointaiwan.blogspot.tw/2014/01/hbase-and-keberos-request-is-replay-34.html
> >
> > Tks
> >
> > Best regards
> >
> > takeshi
> >
>
Re: Suggest that turn the msg "Request is a replay (34) -
PROCESS_TGS" from logging level from ERROR to WARN
Posted by Stack <st...@duboce.net>.
Lets change it in both places. Please file issues. Lets try minimize the
freakout incidents running your hbase/hadoop cluster.
Thanks Takeshi,
St.Ack
On Thu, Jan 16, 2014 at 9:57 PM, takeshi <ta...@gmail.com> wrote:
> Hi All,
>
> Recently we got the error msg "Request is a replay (34) - PROCESS_TGS"
> while we are using the HBase client API to put data into HBase-0.94.16 with
> krb5-1.6.1 enabled. The related msg as follows...
> {code}
> [2014-01-15
>
> 09:40:38,452][hbase-tablepool-1-thread-3][ERROR][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1124)):
> PriviledgedActionException
> as:takeshi_miao@LABcause:javax.security.sasl.SaslException: GSS
> initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Request is a
> replay (34) - PROCESS_TGS)]
> [2014-01-15
>
> 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.logPriviledgedAction(UserGroupInformation.java:1143)):
> PriviledgedAction
> as:takeshi_miao@LABfrom
> :sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> [2014-01-15
>
> 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.ipc.SecureClient](org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:213)):
> Exception encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Request is a
> replay (34) - PROCESS_TGS)]
> [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
>
> ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:657)):
> Initiating logout for takeshi_miao@LAB
> [2014-01-15
>
> 09:40:38,454][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.logout(UserGroupInformation.java:154)):
> hadoop logout
> [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
>
> ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:667)):
> Initiating re-login for takeshi_miao@LAB
> [2014-01-15
>
> 09:40:38,455][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.login(UserGroupInformation.java:146)):
> hadoop login
> [2014-01-15
>
> 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:95)):
> hadoop login commit
> [2014-01-15
>
> 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:100)):
> using existing subject:[takeshi_miao@LAB, UnixPrincipal: takeshi_miao,
> UnixNumericUserPrincipal: 501, UnixNumericGroupPrincipal [Primary Group]:
> 501, UnixNumericGroupPrincipal [Supplementary Group]: 502, takeshi_miao@LAB
> ]
> {code}
>
> At the beginning, we were worry about the "data loss" occurring while we
> found the "Request is a replay (34) - PROCESS_TGS" (especially it is the
> ERROR level) in log, but after code study, this is basically *NOT* a data
> loss issue due to HBase client API would try 5 times internally
> (o.a.h.hbase.ipc.SecureClient, L#296, a one thread) and also 10 times of
> retry externally (o.a.h.hbase.client.HConnectionManager, L#1661, for all
> failed thread), The HTable API would also throw IOEcxeption to client code
> if any thread still fail after these retries.
>
> Based on HBase users' viewpoint as us, we think this is better to change
> the logging level from 'ERROR' to 'WARN', due to the 'ERROR' level had been
> confused us for a while...But this code change may need to change boht
> HBase code and Hadoop code as well; so I am wondering how community think
> about this small thing but may be important to the pure HBase users.
>
> P.S. more details are in this article
>
> http://dumbointaiwan.blogspot.tw/2014/01/hbase-and-keberos-request-is-replay-34.html
>
> Tks
>
> Best regards
>
> takeshi
>