You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Claudio Martella <cl...@tis.bz.it> on 2011/07/18 15:01:43 UTC
HBase and Hadoop 0.20-security-append
Hello,
I'm trying to sort out some things about hadoop versions.
I'm currently trying to run both HBase 0.90.3 and Yahoo!'s Giraph on the
same cluster. Officially Giraph supports 0.20.203 and depends on the
security package.
HBase on its side requires append. I thought about going with Yahoo!'s
0.20-security-append to get both these features and allowing to deploy
both the systems on the same cluster.
I'm guessing how HBase behaves with 0.20-security-append. Can I run it
on this hadoop version?
Can anybody quickly report on that?
Thanks
Claudio
--
Claudio Martella
Free Software & Open Technologies
Analyst
TIS innovation park
Via Siemens 19 | Siemensstr. 19
39100 Bolzano | 39100 Bozen
Tel. +39 0471 068 123
Fax +39 0471 068 129
claudio.martella@tis.bz.it http://www.tis.bz.it
Short information regarding use of personal data. According to Section 13 of Italian Legislative Decree no. 196 of 30 June 2003, we inform you that we process your personal data in order to fulfil contractual and fiscal obligations and also to send you information regarding our services and events. Your personal data are processed with and without electronic means and by respecting data subjects' rights, fundamental freedoms and dignity, particularly with regard to confidentiality, personal identity and the right to personal data protection. At any time and without formalities you can write an e-mail to privacy@tis.bz.it in order to object the processing of your personal data for the purpose of sending advertising materials and also to exercise the right to access personal data and other rights referred to in Section 7 of Decree 196/2003. The data controller is TIS Techno Innovation Alto Adige, Siemens Street n. 19, Bolzano. You can find the complete information on the web site www.tis.bz.it.
Re: HBase and Hadoop 0.20-security-append
Posted by Claudio Martella <cl...@tis.bz.it>.
Ok, I could try my work on 0.20-security-append, i'm doing research so i
don't have problems of production systems. As a matter of fact i don't
need security enabled, i'm just fighting with Giraph's dependencies on
hadoop 0.20.203, not my requirement there.
I'll try and see if Giraph runs on CDH which would be the best option
for me (as a matter of fact Giraph might not run on 0.20-security-append
neither).
Thanks for your feedback, I'll report back.
On 7/18/11 7:23 PM, Gary Helmling wrote:
> Hi Claudio,
>
> The Hadoop 0.20-security-append branch is what we're using to develop HBase
> security features (since those need both Hadoop security and append).
>
> It's a mashup of two different Apache Hadoop branches -- 0.20.203 for
> security and 0.20-append for the append support. To my knowledge there is
> no Apache Hadoop 0.20 branch at the moment that incorporates both of these.
> And it's seeming less likely that there will ever be an official Apache
> Hadoop 0.20-based release that incorporates both. Instead, for Apache
> releases, it seems more likely that the best approach to running Hadoop
> security+append in the future will be to use Hadoop 0.22 or 0.23, assuming
> those keep progressing well towards actual release.
>
> So my big word of warning is that no one that I know of is running the
> 0.20-security-append branch in or near production! Of course I could be
> wrong and others could be using it, but at the moment I would approach it
> with appropriate caution. For testing and experimentation it should be
> fine, but no real guarantees beyond that. If you do decide to take it up
> and it works for you, that would of course be useful to know.
>
> For production purposes internally, we're running a custom Hadoop build
> based off of CDH3.
>
> If you run into any problems getting things setup, let us know and we'll try
> to help out.
>
> Gary
>
>
>
> On Mon, Jul 18, 2011 at 9:13 AM, Claudio Martella <
> claudio.martella@tis.bz.it> wrote:
>
>> On 7/18/11 5:50 PM, Stack wrote:
>>> On Mon, Jul 18, 2011 at 6:01 AM, Claudio Martella
>>> <cl...@tis.bz.it> wrote:
>>>> I'm guessing how HBase behaves with 0.20-security-append. Can I run it
>>>> on this hadoop version?
>>>>
>>> My guess is that it will work (where'd you find this branch?). Will
>>> Giraph not work on CDH?
>>> St.Ack
>>>
>> In this mailinglist archive.
>>
>> Here's the link: https://github.com/trendmicro/hadoop-common
>>
>> I wanted to avoid CDH as I'm having problems with other software (Nutch).
>>
>>
>> --
>> Claudio Martella
>> Free Software & Open Technologies
>> Analyst
>>
>> TIS innovation park
>> Via Siemens 19 | Siemensstr. 19
>> 39100 Bolzano | 39100 Bozen
>> Tel. +39 0471 068 123
>> Fax +39 0471 068 129
>> claudio.martella@tis.bz.it http://www.tis.bz.it
>>
>> Short information regarding use of personal data. According to Section 13
>> of Italian Legislative Decree no. 196 of 30 June 2003, we inform you that we
>> process your personal data in order to fulfil contractual and fiscal
>> obligations and also to send you information regarding our services and
>> events. Your personal data are processed with and without electronic means
>> and by respecting data subjects' rights, fundamental freedoms and dignity,
>> particularly with regard to confidentiality, personal identity and the right
>> to personal data protection. At any time and without formalities you can
>> write an e-mail to privacy@tis.bz.it in order to object the processing of
>> your personal data for the purpose of sending advertising materials and also
>> to exercise the right to access personal data and other rights referred to
>> in Section 7 of Decree 196/2003. The data controller is TIS Techno
>> Innovation Alto Adige, Siemens Street n. 19, Bolzano. You can find the
>> complete information on the web site www.tis.bz.it.
>>
>>
>>
>>
>>
--
Claudio Martella
Free Software & Open Technologies
Analyst
TIS innovation park
Via Siemens 19 | Siemensstr. 19
39100 Bolzano | 39100 Bozen
Tel. +39 0471 068 123
Fax +39 0471 068 129
claudio.martella@tis.bz.it http://www.tis.bz.it
Short information regarding use of personal data. According to Section 13 of Italian Legislative Decree no. 196 of 30 June 2003, we inform you that we process your personal data in order to fulfil contractual and fiscal obligations and also to send you information regarding our services and events. Your personal data are processed with and without electronic means and by respecting data subjects' rights, fundamental freedoms and dignity, particularly with regard to confidentiality, personal identity and the right to personal data protection. At any time and without formalities you can write an e-mail to privacy@tis.bz.it in order to object the processing of your personal data for the purpose of sending advertising materials and also to exercise the right to access personal data and other rights referred to in Section 7 of Decree 196/2003. The data controller is TIS Techno Innovation Alto Adige, Siemens Street n. 19, Bolzano. You can find the complete information on the web site www.tis.bz.it.
Re: HBase and Hadoop 0.20-security-append
Posted by Gary Helmling <gh...@gmail.com>.
Hi Claudio,
The Hadoop 0.20-security-append branch is what we're using to develop HBase
security features (since those need both Hadoop security and append).
It's a mashup of two different Apache Hadoop branches -- 0.20.203 for
security and 0.20-append for the append support. To my knowledge there is
no Apache Hadoop 0.20 branch at the moment that incorporates both of these.
And it's seeming less likely that there will ever be an official Apache
Hadoop 0.20-based release that incorporates both. Instead, for Apache
releases, it seems more likely that the best approach to running Hadoop
security+append in the future will be to use Hadoop 0.22 or 0.23, assuming
those keep progressing well towards actual release.
So my big word of warning is that no one that I know of is running the
0.20-security-append branch in or near production! Of course I could be
wrong and others could be using it, but at the moment I would approach it
with appropriate caution. For testing and experimentation it should be
fine, but no real guarantees beyond that. If you do decide to take it up
and it works for you, that would of course be useful to know.
For production purposes internally, we're running a custom Hadoop build
based off of CDH3.
If you run into any problems getting things setup, let us know and we'll try
to help out.
Gary
On Mon, Jul 18, 2011 at 9:13 AM, Claudio Martella <
claudio.martella@tis.bz.it> wrote:
> On 7/18/11 5:50 PM, Stack wrote:
> > On Mon, Jul 18, 2011 at 6:01 AM, Claudio Martella
> > <cl...@tis.bz.it> wrote:
> >> I'm guessing how HBase behaves with 0.20-security-append. Can I run it
> >> on this hadoop version?
> >>
> > My guess is that it will work (where'd you find this branch?). Will
> > Giraph not work on CDH?
> > St.Ack
> >
> In this mailinglist archive.
>
> Here's the link: https://github.com/trendmicro/hadoop-common
>
> I wanted to avoid CDH as I'm having problems with other software (Nutch).
>
>
> --
> Claudio Martella
> Free Software & Open Technologies
> Analyst
>
> TIS innovation park
> Via Siemens 19 | Siemensstr. 19
> 39100 Bolzano | 39100 Bozen
> Tel. +39 0471 068 123
> Fax +39 0471 068 129
> claudio.martella@tis.bz.it http://www.tis.bz.it
>
> Short information regarding use of personal data. According to Section 13
> of Italian Legislative Decree no. 196 of 30 June 2003, we inform you that we
> process your personal data in order to fulfil contractual and fiscal
> obligations and also to send you information regarding our services and
> events. Your personal data are processed with and without electronic means
> and by respecting data subjects' rights, fundamental freedoms and dignity,
> particularly with regard to confidentiality, personal identity and the right
> to personal data protection. At any time and without formalities you can
> write an e-mail to privacy@tis.bz.it in order to object the processing of
> your personal data for the purpose of sending advertising materials and also
> to exercise the right to access personal data and other rights referred to
> in Section 7 of Decree 196/2003. The data controller is TIS Techno
> Innovation Alto Adige, Siemens Street n. 19, Bolzano. You can find the
> complete information on the web site www.tis.bz.it.
>
>
>
>
>
Re: HBase and Hadoop 0.20-security-append
Posted by Claudio Martella <cl...@tis.bz.it>.
On 7/18/11 5:50 PM, Stack wrote:
> On Mon, Jul 18, 2011 at 6:01 AM, Claudio Martella
> <cl...@tis.bz.it> wrote:
>> I'm guessing how HBase behaves with 0.20-security-append. Can I run it
>> on this hadoop version?
>>
> My guess is that it will work (where'd you find this branch?). Will
> Giraph not work on CDH?
> St.Ack
>
In this mailinglist archive.
Here's the link: https://github.com/trendmicro/hadoop-common
I wanted to avoid CDH as I'm having problems with other software (Nutch).
--
Claudio Martella
Free Software & Open Technologies
Analyst
TIS innovation park
Via Siemens 19 | Siemensstr. 19
39100 Bolzano | 39100 Bozen
Tel. +39 0471 068 123
Fax +39 0471 068 129
claudio.martella@tis.bz.it http://www.tis.bz.it
Short information regarding use of personal data. According to Section 13 of Italian Legislative Decree no. 196 of 30 June 2003, we inform you that we process your personal data in order to fulfil contractual and fiscal obligations and also to send you information regarding our services and events. Your personal data are processed with and without electronic means and by respecting data subjects' rights, fundamental freedoms and dignity, particularly with regard to confidentiality, personal identity and the right to personal data protection. At any time and without formalities you can write an e-mail to privacy@tis.bz.it in order to object the processing of your personal data for the purpose of sending advertising materials and also to exercise the right to access personal data and other rights referred to in Section 7 of Decree 196/2003. The data controller is TIS Techno Innovation Alto Adige, Siemens Street n. 19, Bolzano. You can find the complete information on the web site www.tis.bz.it.
Re: HBase and Hadoop 0.20-security-append
Posted by Gary Helmling <gh...@gmail.com>.
On Mon, Jul 18, 2011 at 9:25 AM, David Capwell <dc...@yahoo-inc.com>wrote:
> HBase does work on Hadoop with Security, but you will need the following in
> your hbase-site.xml
>
> <property>
> <name>hbase.master.keytab.file</name>
> <value>/path/to/keytab/hbase.keytab</value>
> </property>
> <property>
> <name>hbase.master.kerberos.principal</name>
> <value>hbase/_HOST@EXAMPLE.COM</value>
> </property>
> <property>
> <name>hbase.regionserver.keytab.file</name>
> <value>/path/to/keytab/hbase.keytab</value>
> </property>
> <property>
> <name>hbase.regionserver.kerberos.principal</name>
> <value>hbase/_HOST@EXAMPLE.COM</value>
> </property>
>
> You need to set up a keytab for each box running hbase and you will need
> kerberos server and replace EXAMMPLE.COM<http://EXAMMPLE.COM> with the url
>
> Also with Yahoo!'s version, there is no append support so there is a risk
> of data loss. If this is just for testing/experimentation then that is
> fine, but if you want this to go into production you might want to look into
> Cloudera's latest release that has append + security.
>
>
Yes, if you have HDFS security enabled
(hadoop.security.authentication=kerberos), you will need these configuration
entries (and appropriate keytab files) for the HBase daemons to authenticate
with HDFS.
Sorry this part of the setup is pretty undocumented at the moment. We'll be
filling that in in the near future. In the meantime, please ask questions
if you run into any issues or get stuck.
Re: HBase and Hadoop 0.20-security-append
Posted by David Capwell <dc...@yahoo-inc.com>.
HBase does work on Hadoop with Security, but you will need the following in your hbase-site.xml
<property>
<name>hbase.master.keytab.file</name>
<value>/path/to/keytab/hbase.keytab</value>
</property>
<property>
<name>hbase.master.kerberos.principal</name>
<value>hbase/_HOST@EXAMPLE.COM</value>
</property>
<property>
<name>hbase.regionserver.keytab.file</name>
<value>/path/to/keytab/hbase.keytab</value>
</property>
<property>
<name>hbase.regionserver.kerberos.principal</name>
<value>hbase/_HOST@EXAMPLE.COM</value>
</property>
You need to set up a keytab for each box running hbase and you will need kerberos server and replace EXAMMPLE.COM<http://EXAMMPLE.COM> with the url
Also with Yahoo!'s version, there is no append support so there is a risk of data loss. If this is just for testing/experimentation then that is fine, but if you want this to go into production you might want to look into Cloudera's latest release that has append + security.
On Jul 18, 2011, at 8:50:28AM, Stack wrote:
On Mon, Jul 18, 2011 at 6:01 AM, Claudio Martella
<cl...@tis.bz.it>> wrote:
I'm guessing how HBase behaves with 0.20-security-append. Can I run it
on this hadoop version?
My guess is that it will work (where'd you find this branch?). Will
Giraph not work on CDH?
St.Ack
Re: HBase and Hadoop 0.20-security-append
Posted by Stack <st...@duboce.net>.
On Mon, Jul 18, 2011 at 6:01 AM, Claudio Martella
<cl...@tis.bz.it> wrote:
> I'm guessing how HBase behaves with 0.20-security-append. Can I run it
> on this hadoop version?
>
My guess is that it will work (where'd you find this branch?). Will
Giraph not work on CDH?
St.Ack