You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Eric Yang <er...@gmail.com> on 2011/09/11 20:04:20 UTC
Secure Hadoop and non-secure HBase
Hi all,
Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to work with HBase 0.90.x. However, secure Hadoop and HBase would work until kerberos token expires. There is currently no code that renews kerberos token in HBase. Hence, it is possible to add a cron job to periodically renew the HBase user token to keep the system running. What does the community think about having a setup script for cron job as part of HBase upcoming minor release, and fix the token renewal in HBase code for the next major version. On the other hand, would the community accept the token renewal code in HBase as part of the upcoming 0.90.5 release? If yes, what is the time line for 0.90.5?
regards,
Eric
Re: Secure Hadoop and non-secure HBase
Posted by Eric Yang <ey...@hortonworks.com>.
I verified the configuration are working with secure append patch for 0.20.205.0. The parameters are also documented in HBase Configuration book. Thank you.
regards,
Eric
On Sep 12, 2011, at 3:58 PM, Gary Helmling wrote:
> Hi Eric,
>
> Glad that worked for you. We need to get some information on the security
> configuration into the HBase online docs. Let us know if you run into
> further problems.
>
> Gary
>
>
> On Sun, Sep 11, 2011 at 6:00 PM, Eric Yang <ey...@hortonworks.com> wrote:
>
>> Hi Gary,
>>
>> This is exactly what I missed. Without those settings in my
>> hbase-site.xml, I saw these messages in regional server log file:
>>
>> 2011-09-09 17:28:14,172 WARN org.apache.hadoop.ipc.Client: Couldn't setup
>> connection for hbase@TEST.EXAMPLE.COM tonn/
>> hrt8n37.cc1.ygridcore.net@TEST.EXAMPLE.COM
>> 2011-09-09 17:28:14,172 WARN org.apache.hadoop.hdfs.DFSClient: Problem
>> renewing lease for DFSClient_hb_rs_hrt8n38.cc1.ygridcore.net
>> ,60020,1315520835373_1315520835752
>> java.io.IOException: Call to hrt8n37.cc1.ygridcore.net/98.137.233.217:8020failed on local exception: java.io.IOException: Couldn't setup connection
>> forhbase@TEST.EXAMPLE.COM to nn/hrt8n37.cc1.ygridcore.net@TEST.EXAMPLE.COM
>> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1104)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1072)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
>> at $Proxy9.renewLease(Unknown Source)
>> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> at $Proxy9.renewLease(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.renew(DFSClient.java:1176)
>> at
>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1188)
>> at java.lang.Thread.run(Thread.java:619)
>>
>> The region server can respond query but data is not committed to HDFS. I
>> manually run kinit on the command line, and the log shows:
>>
>> 2011-09-09 17:28:50,466 INFO
>> org.apache.hadoop.security.UserGroupInformation: Initiating logout for
>> hbase@TEST.EXAMPLE.COM
>> 2011-09-09 17:28:50,466 INFO
>> org.apache.hadoop.security.UserGroupInformation: Initiating re-login for
>> hbase@TEST.EXAMPLE.COM
>>
>> I will put the keytab and kerberos.principal in the configuration file to
>> verify token renewal in HBase. Thank you.
>>
>> regards,
>> Eric
>>
>> On Sep 11, 2011, at 3:57 PM, Gary Helmling wrote:
>>
>>> Hi Eric,
>>>
>>> If you configure
>>>
>>> hbase.master.keytab.file
>>> hbase.master.kerberos.principal
>>> hbase.regionserver.keytab.file
>>> hbase.regionserver.kerberos.principal
>>>
>>> in your hbase-site.xml, then the master and region server processes
>> should
>>> login from the keytab files on startup, as Todd mentions. It's also my
>>> understanding that they don't need a renewal thread in that case. The
>> RPC
>>> client just tries a relogin from the keytab in the case of a connection
>>> error.
>>>
>>> Can you describe a bit more what you're seeing so that we can understand
>> the
>>> context?
>>>
>>> Gary
>>>
>>>
>>> On Sun, Sep 11, 2011 at 3:13 PM, Todd Lipcon <to...@cloudera.com> wrote:
>>>
>>>> Hi Eric,
>>>>
>>>> Could you please explain more fully what you mean by this? The daemons
>>>> generally run using keytabs, not user credentials, and thus shouldn't
>>>> need the explicit TGT Renewer, right?
>>>>
>>>> -Todd
>>>>
>>>> On Sun, Sep 11, 2011 at 11:04 AM, Eric Yang <er...@gmail.com> wrote:
>>>>> Hi all,
>>>>>
>>>>> Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to
>>>> work with HBase 0.90.x. However, secure Hadoop and HBase would work
>> until
>>>> kerberos token expires. There is currently no code that renews kerberos
>>>> token in HBase. Hence, it is possible to add a cron job to periodically
>>>> renew the HBase user token to keep the system running. What does the
>>>> community think about having a setup script for cron job as part of
>> HBase
>>>> upcoming minor release, and fix the token renewal in HBase code for the
>> next
>>>> major version. On the other hand, would the community accept the token
>>>> renewal code in HBase as part of the upcoming 0.90.5 release? If yes,
>> what
>>>> is the time line for 0.90.5?
>>>>>
>>>>> regards,
>>>>> Eric
>>>>
>>>>
>>>>
>>>> --
>>>> Todd Lipcon
>>>> Software Engineer, Cloudera
>>>>
>>
>>
Re: Secure Hadoop and non-secure HBase
Posted by Gary Helmling <gh...@gmail.com>.
Hi Eric,
Glad that worked for you. We need to get some information on the security
configuration into the HBase online docs. Let us know if you run into
further problems.
Gary
On Sun, Sep 11, 2011 at 6:00 PM, Eric Yang <ey...@hortonworks.com> wrote:
> Hi Gary,
>
> This is exactly what I missed. Without those settings in my
> hbase-site.xml, I saw these messages in regional server log file:
>
> 2011-09-09 17:28:14,172 WARN org.apache.hadoop.ipc.Client: Couldn't setup
> connection for hbase@TEST.EXAMPLE.COM tonn/
> hrt8n37.cc1.ygridcore.net@TEST.EXAMPLE.COM
> 2011-09-09 17:28:14,172 WARN org.apache.hadoop.hdfs.DFSClient: Problem
> renewing lease for DFSClient_hb_rs_hrt8n38.cc1.ygridcore.net
> ,60020,1315520835373_1315520835752
> java.io.IOException: Call to hrt8n37.cc1.ygridcore.net/98.137.233.217:8020failed on local exception: java.io.IOException: Couldn't setup connection
> forhbase@TEST.EXAMPLE.COM to nn/hrt8n37.cc1.ygridcore.net@TEST.EXAMPLE.COM
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1104)
> at org.apache.hadoop.ipc.Client.call(Client.java:1072)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> at $Proxy9.renewLease(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy9.renewLease(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.renew(DFSClient.java:1176)
> at
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1188)
> at java.lang.Thread.run(Thread.java:619)
>
> The region server can respond query but data is not committed to HDFS. I
> manually run kinit on the command line, and the log shows:
>
> 2011-09-09 17:28:50,466 INFO
> org.apache.hadoop.security.UserGroupInformation: Initiating logout for
> hbase@TEST.EXAMPLE.COM
> 2011-09-09 17:28:50,466 INFO
> org.apache.hadoop.security.UserGroupInformation: Initiating re-login for
> hbase@TEST.EXAMPLE.COM
>
> I will put the keytab and kerberos.principal in the configuration file to
> verify token renewal in HBase. Thank you.
>
> regards,
> Eric
>
> On Sep 11, 2011, at 3:57 PM, Gary Helmling wrote:
>
> > Hi Eric,
> >
> > If you configure
> >
> > hbase.master.keytab.file
> > hbase.master.kerberos.principal
> > hbase.regionserver.keytab.file
> > hbase.regionserver.kerberos.principal
> >
> > in your hbase-site.xml, then the master and region server processes
> should
> > login from the keytab files on startup, as Todd mentions. It's also my
> > understanding that they don't need a renewal thread in that case. The
> RPC
> > client just tries a relogin from the keytab in the case of a connection
> > error.
> >
> > Can you describe a bit more what you're seeing so that we can understand
> the
> > context?
> >
> > Gary
> >
> >
> > On Sun, Sep 11, 2011 at 3:13 PM, Todd Lipcon <to...@cloudera.com> wrote:
> >
> >> Hi Eric,
> >>
> >> Could you please explain more fully what you mean by this? The daemons
> >> generally run using keytabs, not user credentials, and thus shouldn't
> >> need the explicit TGT Renewer, right?
> >>
> >> -Todd
> >>
> >> On Sun, Sep 11, 2011 at 11:04 AM, Eric Yang <er...@gmail.com> wrote:
> >>> Hi all,
> >>>
> >>> Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to
> >> work with HBase 0.90.x. However, secure Hadoop and HBase would work
> until
> >> kerberos token expires. There is currently no code that renews kerberos
> >> token in HBase. Hence, it is possible to add a cron job to periodically
> >> renew the HBase user token to keep the system running. What does the
> >> community think about having a setup script for cron job as part of
> HBase
> >> upcoming minor release, and fix the token renewal in HBase code for the
> next
> >> major version. On the other hand, would the community accept the token
> >> renewal code in HBase as part of the upcoming 0.90.5 release? If yes,
> what
> >> is the time line for 0.90.5?
> >>>
> >>> regards,
> >>> Eric
> >>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
>
>
Re: Secure Hadoop and non-secure HBase
Posted by Eric Yang <ey...@hortonworks.com>.
Hi Gary,
This is exactly what I missed. Without those settings in my hbase-site.xml, I saw these messages in regional server log file:
2011-09-09 17:28:14,172 WARN org.apache.hadoop.ipc.Client: Couldn't setup connection for hbase@TEST.EXAMPLE.COM tonn/hrt8n37.cc1.ygridcore.net@TEST.EXAMPLE.COM
2011-09-09 17:28:14,172 WARN org.apache.hadoop.hdfs.DFSClient: Problem renewing lease for DFSClient_hb_rs_hrt8n38.cc1.ygridcore.net,60020,1315520835373_1315520835752
java.io.IOException: Call to hrt8n37.cc1.ygridcore.net/98.137.233.217:8020 failed on local exception: java.io.IOException: Couldn't setup connection forhbase@TEST.EXAMPLE.COM to nn/hrt8n37.cc1.ygridcore.net@TEST.EXAMPLE.COM
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1104)
at org.apache.hadoop.ipc.Client.call(Client.java:1072)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at $Proxy9.renewLease(Unknown Source)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy9.renewLease(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.renew(DFSClient.java:1176)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1188)
at java.lang.Thread.run(Thread.java:619)
The region server can respond query but data is not committed to HDFS. I manually run kinit on the command line, and the log shows:
2011-09-09 17:28:50,466 INFO org.apache.hadoop.security.UserGroupInformation: Initiating logout for hbase@TEST.EXAMPLE.COM
2011-09-09 17:28:50,466 INFO org.apache.hadoop.security.UserGroupInformation: Initiating re-login for hbase@TEST.EXAMPLE.COM
I will put the keytab and kerberos.principal in the configuration file to verify token renewal in HBase. Thank you.
regards,
Eric
On Sep 11, 2011, at 3:57 PM, Gary Helmling wrote:
> Hi Eric,
>
> If you configure
>
> hbase.master.keytab.file
> hbase.master.kerberos.principal
> hbase.regionserver.keytab.file
> hbase.regionserver.kerberos.principal
>
> in your hbase-site.xml, then the master and region server processes should
> login from the keytab files on startup, as Todd mentions. It's also my
> understanding that they don't need a renewal thread in that case. The RPC
> client just tries a relogin from the keytab in the case of a connection
> error.
>
> Can you describe a bit more what you're seeing so that we can understand the
> context?
>
> Gary
>
>
> On Sun, Sep 11, 2011 at 3:13 PM, Todd Lipcon <to...@cloudera.com> wrote:
>
>> Hi Eric,
>>
>> Could you please explain more fully what you mean by this? The daemons
>> generally run using keytabs, not user credentials, and thus shouldn't
>> need the explicit TGT Renewer, right?
>>
>> -Todd
>>
>> On Sun, Sep 11, 2011 at 11:04 AM, Eric Yang <er...@gmail.com> wrote:
>>> Hi all,
>>>
>>> Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to
>> work with HBase 0.90.x. However, secure Hadoop and HBase would work until
>> kerberos token expires. There is currently no code that renews kerberos
>> token in HBase. Hence, it is possible to add a cron job to periodically
>> renew the HBase user token to keep the system running. What does the
>> community think about having a setup script for cron job as part of HBase
>> upcoming minor release, and fix the token renewal in HBase code for the next
>> major version. On the other hand, would the community accept the token
>> renewal code in HBase as part of the upcoming 0.90.5 release? If yes, what
>> is the time line for 0.90.5?
>>>
>>> regards,
>>> Eric
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
Re: Secure Hadoop and non-secure HBase
Posted by Gary Helmling <gh...@gmail.com>.
Hi Eric,
If you configure
hbase.master.keytab.file
hbase.master.kerberos.principal
hbase.regionserver.keytab.file
hbase.regionserver.kerberos.principal
in your hbase-site.xml, then the master and region server processes should
login from the keytab files on startup, as Todd mentions. It's also my
understanding that they don't need a renewal thread in that case. The RPC
client just tries a relogin from the keytab in the case of a connection
error.
Can you describe a bit more what you're seeing so that we can understand the
context?
Gary
On Sun, Sep 11, 2011 at 3:13 PM, Todd Lipcon <to...@cloudera.com> wrote:
> Hi Eric,
>
> Could you please explain more fully what you mean by this? The daemons
> generally run using keytabs, not user credentials, and thus shouldn't
> need the explicit TGT Renewer, right?
>
> -Todd
>
> On Sun, Sep 11, 2011 at 11:04 AM, Eric Yang <er...@gmail.com> wrote:
> > Hi all,
> >
> > Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to
> work with HBase 0.90.x. However, secure Hadoop and HBase would work until
> kerberos token expires. There is currently no code that renews kerberos
> token in HBase. Hence, it is possible to add a cron job to periodically
> renew the HBase user token to keep the system running. What does the
> community think about having a setup script for cron job as part of HBase
> upcoming minor release, and fix the token renewal in HBase code for the next
> major version. On the other hand, would the community accept the token
> renewal code in HBase as part of the upcoming 0.90.5 release? If yes, what
> is the time line for 0.90.5?
> >
> > regards,
> > Eric
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: Secure Hadoop and non-secure HBase
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Eric,
Could you please explain more fully what you mean by this? The daemons
generally run using keytabs, not user credentials, and thus shouldn't
need the explicit TGT Renewer, right?
-Todd
On Sun, Sep 11, 2011 at 11:04 AM, Eric Yang <er...@gmail.com> wrote:
> Hi all,
>
> Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to work with HBase 0.90.x. However, secure Hadoop and HBase would work until kerberos token expires. There is currently no code that renews kerberos token in HBase. Hence, it is possible to add a cron job to periodically renew the HBase user token to keep the system running. What does the community think about having a setup script for cron job as part of HBase upcoming minor release, and fix the token renewal in HBase code for the next major version. On the other hand, would the community accept the token renewal code in HBase as part of the upcoming 0.90.5 release? If yes, what is the time line for 0.90.5?
>
> regards,
> Eric
--
Todd Lipcon
Software Engineer, Cloudera
Re: Secure Hadoop and non-secure HBase
Posted by Todd Lipcon <to...@cloudera.com>.
On Sun, Sep 11, 2011 at 11:04 AM, Eric Yang <er...@gmail.com> wrote:
> Hi all,
>
> Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to work with HBase 0.90.x.
A slight correction: I think you mean to say: "Hortonworks employees
have revisited their decision to -1 the work that Cloudera, Facebook,
and Stumbleupon did 18 months ago to allow Hadoop 0.20 to work with
HBase. And thus it has been merged into 0.20.205." :)
-Todd
--
Todd Lipcon
Software Engineer, Cloudera