You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by n keywal <nk...@gmail.com> on 2012/09/13 16:58:45 UTC
read short circuit
Hi,
Is there any known counter indication for read short circuit?
I had a look at JD's presentation from January. It was tested vs. HBase as
a whole; and still recommended for nearly all cases.
We could make it at least the default when reading HLog files (even if it
seems it's a 'per client" config, there is may be something to look at as
well)?
Cheers,
Nicolas
Re: read short circuit
Posted by Andrew Purtell <ap...@apache.org>.
On Thu, Sep 13, 2012 at 9:51 AM, Jean-Daniel Cryans <jd...@apache.org> wrote:
> One issue with the current implementation is that you can only have 1
> user that short circuit reads, all the others who have the config
> enabled will get a fat Exception. At SU we don't have special config
> files for every MR job so unless you disable it job by job then you'll
> get those exceptions everywhere. We live with it but it did take
> people by surprise when we rolled it out :)
Eli just fixed that in HDFS-3907. It's a three line change that
substitutes a list of strings for a single string, plus a unit test.
Should be a low risk change to get into HDFSes everywhere.
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)
Re: read short circuit
Posted by Jean-Daniel Cryans <jd...@apache.org>.
On Thu, Sep 13, 2012 at 10:28 AM, Stack <st...@duboce.net> wrote:
> Write a short paragraph and I'll make an HDFS configuration sections
> like this HBase configurations section on manual and stick it in
> there: http://hbase.apache.org/book.html#perf.configurations
Here's a first stab:
Leveraging local data
Since Hadoop 1.0.0 (also 0.22.1, 0.23.1, CDH3u3 and HDP 1.0) via
HDFS-2246[1], it is possible for the DFSClient to take a shortcut and
read directly from disk instead of going through the DataNode when the
data is local. What this means for HBase is that the RegionServers can
read directly off their machine's disks instead of having to open a
socket to talk to the DataNode, the former being generally much
faster[2]. In order to enable it, first hdfs-site.xml needs to be
amended with:
dfs.block.local-path-access.user = the _only_ user that can use the
shortcut. This has to be the user that started HBase.
And in hbase-site.xml:
dfs.client.read.shortcircuit = true
The DataNodes need to be restarted in order to pick up the new
configuration. Be aware that if a process started under another
username than the one configured here also has the shortcircuit
enabled, it will get an Exception regarding an unauthorized access but
the data will still be read.
1. https://issues.apache.org/jira/browse/HDFS-2246
2. http://files.meetup.com/1350427/hug_ebay_jdcryans.pdf
Re: read short circuit
Posted by Stack <st...@duboce.net>.
On Thu, Sep 13, 2012 at 9:51 AM, Jean-Daniel Cryans <jd...@apache.org> wrote:
> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <nk...@gmail.com> wrote:
>> It's all in the "pretty much" :-)
>> Should we make it the default (i.e. recommending it in the doc, emitting a
>> warning if it's not set) for all installation ?
>
> I'd recommend it in the doc as an important config to consider.
>
> One issue with the current implementation is that you can only have 1
> user that short circuit reads, all the others who have the config
> enabled will get a fat Exception. At SU we don't have special config
> files for every MR job so unless you disable it job by job then you'll
> get those exceptions everywhere. We live with it but it did take
> people by surprise when we rolled it out :)
>
>> Or would it make sense to separate the store and wal cases, meaning that we
>> need a change in hdfs to make this possible?
>
> Don't we only have one FS instance per region server? If so, that's a
> moot point.
>
Write a short paragraph and I'll make an HDFS configuration sections
like this HBase configurations section on manual and stick it in
there: http://hbase.apache.org/book.html#perf.configurations
St.Ack
Re: read short circuit
Posted by Jean-Daniel Cryans <jd...@apache.org>.
On Thu, Sep 13, 2012 at 9:42 AM, n keywal <nk...@gmail.com> wrote:
> It's all in the "pretty much" :-)
> Should we make it the default (i.e. recommending it in the doc, emitting a
> warning if it's not set) for all installation ?
I'd recommend it in the doc as an important config to consider.
One issue with the current implementation is that you can only have 1
user that short circuit reads, all the others who have the config
enabled will get a fat Exception. At SU we don't have special config
files for every MR job so unless you disable it job by job then you'll
get those exceptions everywhere. We live with it but it did take
people by surprise when we rolled it out :)
> Or would it make sense to separate the store and wal cases, meaning that we
> need a change in hdfs to make this possible?
Don't we only have one FS instance per region server? If so, that's a
moot point.
J-D
Re: read short circuit
Posted by n keywal <nk...@gmail.com>.
Done for trunk on HBASE-6783. I will need to modify the tests script to
unactivate it on HBase-0.94-security and HBase-TRUNK-on-Hadoop-2.0.0.
As a side note, there is a difference in behavior between the two modes: we
don't see anymore a datanode error. I've seen it mentioned by JD somewhere,
and I remember that the conclusion was "it's ok". Can't find it again,
though.
Conclusion; the TestFSErrorsExposed#testFullSystemBubblesFSErrors is not
executed if short circuit is on.
On Thu, Sep 13, 2012 at 11:07 PM, Andrew Purtell <ap...@apache.org>wrote:
> Yeah, on HBase-0.94-security
>
> On Thu, Sep 13, 2012 at 1:34 PM, n keywal <nk...@gmail.com> wrote:
> > We've got two security builds on 0.94: HBase-0.94-security and
> > HBase-0.94-security-on-Hadoop-23
> > We can change the setting on one of them?
> >
> > On Thu, Sep 13, 2012 at 10:20 PM, Andrew Purtell <apurtell@apache.org
> >wrote:
> >
> >> Hey N,
> >>
> >> It makes sense to have a 0.94 build with -Psecurity enabled and local
> >> shortcut disabled. What do you think? For trunk builds your points sound
> >> good, and I have no opinion.
> >>
> >> On Thursday, September 13, 2012, n keywal wrote:
> >>
> >> > If I sum-up :
> >> >
> >> > We make it the default:
> >> > 1) Documented as such (what JD wrote already + a reference to
> HDFS-3907)
> >> > 2) Set in the config we use for the unit tests.
> >> >
> >> > Andrew, you said that we should keep some test with it off ? May be we
> >> > could do that in one of the existing build on the trunk, for example
> >> > HBase-TRUNK-on-Hadoop-2.0.0? i.e.: We run the tests on trunk with
> >> > short-circuit on; we run the test on hadoop-2 with short circuit off.
> >> >
> >> > I propose it this way because may be the tests will be faster with
> >> > short-circuit on, and as well if it's the default production config it
> >> > should be the one in the "main" test path.
> >> >
> >> > If that's ok, I can do test it on our test suite and do the jira for
> the
> >> > integration.
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Sep 13, 2012 at 8:31 PM, Andrew Purtell <apurtell@apache.org
> >> <javascript:;>>
> >> > wrote:
> >> >
> >> > > That's https://issues.apache.org/jira/browse/HBASE-4867
> >> > >
> >> > > On Thu, Sep 13, 2012 at 10:57 AM, Jesse Yates <
> jesse.k.yates@gmail.com
> >> <javascript:;>
> >> > >
> >> > > wrote:
> >> > > > There was a jira a while back (can't remember the number) that I
> >> think
> >> > > came
> >> > > > from the Facebook fellas to have multiple hbase-site.xmls, with a
> >> > naming
> >> > > > scheme such that you can do layering of configurations.
> >> > > >
> >> > > > What ever happened to that?
> >> > > > -------------------
> >> > > > Jesse Yates
> >> > > > @jesse_yates
> >> > > > jyates.github.com
> >> > > >
> >> > > >
> >> > > > On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <
> >> apurtell@apache.org<javascript:;>
> >> > > >wrote:
> >> > > >
> >> > > >> Making it the default seems reasonable.
> >> > > >>
> >> > > >> For <= 0.94, the security profile has a separate hbase-site.xml
> file
> >> > > >> under test resources where there it could be defaulted as off.
> >> Reason
> >> > > >> is the local block sharing requires some shared group membership
> >> > > >> between HBase and HDFS service accounts that might be
> objectionable
> >> in
> >> > > >> a secure setting. So we should have test coverage with it off.
> >> > > >>
> >> > > >> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <nkeywal@gmail.com
> >> <javascript:;>>
> >> > wrote:
> >> > > >> > It's all in the "pretty much" :-)
> >> > > >> > Should we make it the default (i.e. recommending it in the doc,
> >> > > emitting
> >> > > >> a
> >> > > >> > warning if it's not set) for all installation ?
> >> > > >> > Or would it make sense to separate the store and wal cases,
> >> meaning
> >> > > that
> >> > > >> we
> >> > > >> > need a change in hdfs to make this possible?
> >> > > >> >
> >> > > >> >
> >> > > >> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <
> >> > > jdcryans@apache.org <javascript:;>
> >> > > >> >wrote:
> >> > > >> >
> >> > > >> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <nkeywal@gmail.com
> >> <javascript:;>>
> >> > wrote:
> >> > > >> >> > Hi,
> >> > > >> >> >
> >> > > >> >> > Is there any known counter indication for read short
> circuit?
> >> > > >> >> > I had a look at JD's presentation from January. It was
> tested
> >> vs.
> >> > > >> HBase
> >> > > >> >> as
> >> > > >> >> > a whole; and still recommended for nearly all cases.
> >> > > >> >>
> >> > > >> >> Pretty much.
> >> > > >> >>
> >> > > >> >> >
> >> > > >> >> > We could make it at least the default when reading HLog
> files
> >> > (even
> >> > > >> if it
> >> > > >> >> > seems it's a 'per client" config, there is may be something
> to
> >> > look
> >> > > >> at as
> >> > > >> >> > well)?
> >> > > >> >>
> >> > > >> >> You have to add 2 configs in hdfs-site.xml, which we can't do
> >> from
> >> > > >> HBase.
> >> > > >> >>
> >> > > >> >> J-D
> >> > > >> >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >> --
> >> > > >> Best regards,
> >> > > >>
> >> > > >> - Andy
> >> > > >>
> >> > > >> Problems worthy of attack prove their worth by hitting back. -
> Piet
> >> > > >> Hein (via Tom White)
> >> > > >>
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Best regards,
> >> > >
> >> > > - Andy
> >> > >
> >> > > Problems worthy of attack prove their worth by hitting back. - Piet
> >> > > Hein (via Tom White)
> >> > >
> >> >
> >>
> >>
> >> --
> >> Best regards,
> >>
> >> - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein (via Tom White)
>
Re: read short circuit
Posted by Andrew Purtell <ap...@apache.org>.
Yeah, on HBase-0.94-security
On Thu, Sep 13, 2012 at 1:34 PM, n keywal <nk...@gmail.com> wrote:
> We've got two security builds on 0.94: HBase-0.94-security and
> HBase-0.94-security-on-Hadoop-23
> We can change the setting on one of them?
>
> On Thu, Sep 13, 2012 at 10:20 PM, Andrew Purtell <ap...@apache.org>wrote:
>
>> Hey N,
>>
>> It makes sense to have a 0.94 build with -Psecurity enabled and local
>> shortcut disabled. What do you think? For trunk builds your points sound
>> good, and I have no opinion.
>>
>> On Thursday, September 13, 2012, n keywal wrote:
>>
>> > If I sum-up :
>> >
>> > We make it the default:
>> > 1) Documented as such (what JD wrote already + a reference to HDFS-3907)
>> > 2) Set in the config we use for the unit tests.
>> >
>> > Andrew, you said that we should keep some test with it off ? May be we
>> > could do that in one of the existing build on the trunk, for example
>> > HBase-TRUNK-on-Hadoop-2.0.0? i.e.: We run the tests on trunk with
>> > short-circuit on; we run the test on hadoop-2 with short circuit off.
>> >
>> > I propose it this way because may be the tests will be faster with
>> > short-circuit on, and as well if it's the default production config it
>> > should be the one in the "main" test path.
>> >
>> > If that's ok, I can do test it on our test suite and do the jira for the
>> > integration.
>> >
>> >
>> >
>> >
>> > On Thu, Sep 13, 2012 at 8:31 PM, Andrew Purtell <apurtell@apache.org
>> <javascript:;>>
>> > wrote:
>> >
>> > > That's https://issues.apache.org/jira/browse/HBASE-4867
>> > >
>> > > On Thu, Sep 13, 2012 at 10:57 AM, Jesse Yates <jesse.k.yates@gmail.com
>> <javascript:;>
>> > >
>> > > wrote:
>> > > > There was a jira a while back (can't remember the number) that I
>> think
>> > > came
>> > > > from the Facebook fellas to have multiple hbase-site.xmls, with a
>> > naming
>> > > > scheme such that you can do layering of configurations.
>> > > >
>> > > > What ever happened to that?
>> > > > -------------------
>> > > > Jesse Yates
>> > > > @jesse_yates
>> > > > jyates.github.com
>> > > >
>> > > >
>> > > > On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <
>> apurtell@apache.org<javascript:;>
>> > > >wrote:
>> > > >
>> > > >> Making it the default seems reasonable.
>> > > >>
>> > > >> For <= 0.94, the security profile has a separate hbase-site.xml file
>> > > >> under test resources where there it could be defaulted as off.
>> Reason
>> > > >> is the local block sharing requires some shared group membership
>> > > >> between HBase and HDFS service accounts that might be objectionable
>> in
>> > > >> a secure setting. So we should have test coverage with it off.
>> > > >>
>> > > >> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <nkeywal@gmail.com
>> <javascript:;>>
>> > wrote:
>> > > >> > It's all in the "pretty much" :-)
>> > > >> > Should we make it the default (i.e. recommending it in the doc,
>> > > emitting
>> > > >> a
>> > > >> > warning if it's not set) for all installation ?
>> > > >> > Or would it make sense to separate the store and wal cases,
>> meaning
>> > > that
>> > > >> we
>> > > >> > need a change in hdfs to make this possible?
>> > > >> >
>> > > >> >
>> > > >> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <
>> > > jdcryans@apache.org <javascript:;>
>> > > >> >wrote:
>> > > >> >
>> > > >> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <nkeywal@gmail.com
>> <javascript:;>>
>> > wrote:
>> > > >> >> > Hi,
>> > > >> >> >
>> > > >> >> > Is there any known counter indication for read short circuit?
>> > > >> >> > I had a look at JD's presentation from January. It was tested
>> vs.
>> > > >> HBase
>> > > >> >> as
>> > > >> >> > a whole; and still recommended for nearly all cases.
>> > > >> >>
>> > > >> >> Pretty much.
>> > > >> >>
>> > > >> >> >
>> > > >> >> > We could make it at least the default when reading HLog files
>> > (even
>> > > >> if it
>> > > >> >> > seems it's a 'per client" config, there is may be something to
>> > look
>> > > >> at as
>> > > >> >> > well)?
>> > > >> >>
>> > > >> >> You have to add 2 configs in hdfs-site.xml, which we can't do
>> from
>> > > >> HBase.
>> > > >> >>
>> > > >> >> J-D
>> > > >> >>
>> > > >>
>> > > >>
>> > > >>
>> > > >> --
>> > > >> Best regards,
>> > > >>
>> > > >> - Andy
>> > > >>
>> > > >> Problems worthy of attack prove their worth by hitting back. - Piet
>> > > >> Hein (via Tom White)
>> > > >>
>> > >
>> > >
>> > >
>> > > --
>> > > Best regards,
>> > >
>> > > - Andy
>> > >
>> > > Problems worthy of attack prove their worth by hitting back. - Piet
>> > > Hein (via Tom White)
>> > >
>> >
>>
>>
>> --
>> Best regards,
>>
>> - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)
Re: read short circuit
Posted by n keywal <nk...@gmail.com>.
We've got two security builds on 0.94: HBase-0.94-security and
HBase-0.94-security-on-Hadoop-23
We can change the setting on one of them?
On Thu, Sep 13, 2012 at 10:20 PM, Andrew Purtell <ap...@apache.org>wrote:
> Hey N,
>
> It makes sense to have a 0.94 build with -Psecurity enabled and local
> shortcut disabled. What do you think? For trunk builds your points sound
> good, and I have no opinion.
>
> On Thursday, September 13, 2012, n keywal wrote:
>
> > If I sum-up :
> >
> > We make it the default:
> > 1) Documented as such (what JD wrote already + a reference to HDFS-3907)
> > 2) Set in the config we use for the unit tests.
> >
> > Andrew, you said that we should keep some test with it off ? May be we
> > could do that in one of the existing build on the trunk, for example
> > HBase-TRUNK-on-Hadoop-2.0.0? i.e.: We run the tests on trunk with
> > short-circuit on; we run the test on hadoop-2 with short circuit off.
> >
> > I propose it this way because may be the tests will be faster with
> > short-circuit on, and as well if it's the default production config it
> > should be the one in the "main" test path.
> >
> > If that's ok, I can do test it on our test suite and do the jira for the
> > integration.
> >
> >
> >
> >
> > On Thu, Sep 13, 2012 at 8:31 PM, Andrew Purtell <apurtell@apache.org
> <javascript:;>>
> > wrote:
> >
> > > That's https://issues.apache.org/jira/browse/HBASE-4867
> > >
> > > On Thu, Sep 13, 2012 at 10:57 AM, Jesse Yates <jesse.k.yates@gmail.com
> <javascript:;>
> > >
> > > wrote:
> > > > There was a jira a while back (can't remember the number) that I
> think
> > > came
> > > > from the Facebook fellas to have multiple hbase-site.xmls, with a
> > naming
> > > > scheme such that you can do layering of configurations.
> > > >
> > > > What ever happened to that?
> > > > -------------------
> > > > Jesse Yates
> > > > @jesse_yates
> > > > jyates.github.com
> > > >
> > > >
> > > > On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <
> apurtell@apache.org<javascript:;>
> > > >wrote:
> > > >
> > > >> Making it the default seems reasonable.
> > > >>
> > > >> For <= 0.94, the security profile has a separate hbase-site.xml file
> > > >> under test resources where there it could be defaulted as off.
> Reason
> > > >> is the local block sharing requires some shared group membership
> > > >> between HBase and HDFS service accounts that might be objectionable
> in
> > > >> a secure setting. So we should have test coverage with it off.
> > > >>
> > > >> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <nkeywal@gmail.com
> <javascript:;>>
> > wrote:
> > > >> > It's all in the "pretty much" :-)
> > > >> > Should we make it the default (i.e. recommending it in the doc,
> > > emitting
> > > >> a
> > > >> > warning if it's not set) for all installation ?
> > > >> > Or would it make sense to separate the store and wal cases,
> meaning
> > > that
> > > >> we
> > > >> > need a change in hdfs to make this possible?
> > > >> >
> > > >> >
> > > >> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <
> > > jdcryans@apache.org <javascript:;>
> > > >> >wrote:
> > > >> >
> > > >> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <nkeywal@gmail.com
> <javascript:;>>
> > wrote:
> > > >> >> > Hi,
> > > >> >> >
> > > >> >> > Is there any known counter indication for read short circuit?
> > > >> >> > I had a look at JD's presentation from January. It was tested
> vs.
> > > >> HBase
> > > >> >> as
> > > >> >> > a whole; and still recommended for nearly all cases.
> > > >> >>
> > > >> >> Pretty much.
> > > >> >>
> > > >> >> >
> > > >> >> > We could make it at least the default when reading HLog files
> > (even
> > > >> if it
> > > >> >> > seems it's a 'per client" config, there is may be something to
> > look
> > > >> at as
> > > >> >> > well)?
> > > >> >>
> > > >> >> You have to add 2 configs in hdfs-site.xml, which we can't do
> from
> > > >> HBase.
> > > >> >>
> > > >> >> J-D
> > > >> >>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Best regards,
> > > >>
> > > >> - Andy
> > > >>
> > > >> Problems worthy of attack prove their worth by hitting back. - Piet
> > > >> Hein (via Tom White)
> > > >>
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> > > Hein (via Tom White)
> > >
> >
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: read short circuit
Posted by Andrew Purtell <ap...@apache.org>.
Hey N,
It makes sense to have a 0.94 build with -Psecurity enabled and local
shortcut disabled. What do you think? For trunk builds your points sound
good, and I have no opinion.
On Thursday, September 13, 2012, n keywal wrote:
> If I sum-up :
>
> We make it the default:
> 1) Documented as such (what JD wrote already + a reference to HDFS-3907)
> 2) Set in the config we use for the unit tests.
>
> Andrew, you said that we should keep some test with it off ? May be we
> could do that in one of the existing build on the trunk, for example
> HBase-TRUNK-on-Hadoop-2.0.0? i.e.: We run the tests on trunk with
> short-circuit on; we run the test on hadoop-2 with short circuit off.
>
> I propose it this way because may be the tests will be faster with
> short-circuit on, and as well if it's the default production config it
> should be the one in the "main" test path.
>
> If that's ok, I can do test it on our test suite and do the jira for the
> integration.
>
>
>
>
> On Thu, Sep 13, 2012 at 8:31 PM, Andrew Purtell <apurtell@apache.org<javascript:;>>
> wrote:
>
> > That's https://issues.apache.org/jira/browse/HBASE-4867
> >
> > On Thu, Sep 13, 2012 at 10:57 AM, Jesse Yates <jesse.k.yates@gmail.com<javascript:;>
> >
> > wrote:
> > > There was a jira a while back (can't remember the number) that I think
> > came
> > > from the Facebook fellas to have multiple hbase-site.xmls, with a
> naming
> > > scheme such that you can do layering of configurations.
> > >
> > > What ever happened to that?
> > > -------------------
> > > Jesse Yates
> > > @jesse_yates
> > > jyates.github.com
> > >
> > >
> > > On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <apurtell@apache.org<javascript:;>
> > >wrote:
> > >
> > >> Making it the default seems reasonable.
> > >>
> > >> For <= 0.94, the security profile has a separate hbase-site.xml file
> > >> under test resources where there it could be defaulted as off. Reason
> > >> is the local block sharing requires some shared group membership
> > >> between HBase and HDFS service accounts that might be objectionable in
> > >> a secure setting. So we should have test coverage with it off.
> > >>
> > >> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <nkeywal@gmail.com<javascript:;>>
> wrote:
> > >> > It's all in the "pretty much" :-)
> > >> > Should we make it the default (i.e. recommending it in the doc,
> > emitting
> > >> a
> > >> > warning if it's not set) for all installation ?
> > >> > Or would it make sense to separate the store and wal cases, meaning
> > that
> > >> we
> > >> > need a change in hdfs to make this possible?
> > >> >
> > >> >
> > >> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <
> > jdcryans@apache.org <javascript:;>
> > >> >wrote:
> > >> >
> > >> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <nkeywal@gmail.com<javascript:;>>
> wrote:
> > >> >> > Hi,
> > >> >> >
> > >> >> > Is there any known counter indication for read short circuit?
> > >> >> > I had a look at JD's presentation from January. It was tested vs.
> > >> HBase
> > >> >> as
> > >> >> > a whole; and still recommended for nearly all cases.
> > >> >>
> > >> >> Pretty much.
> > >> >>
> > >> >> >
> > >> >> > We could make it at least the default when reading HLog files
> (even
> > >> if it
> > >> >> > seems it's a 'per client" config, there is may be something to
> look
> > >> at as
> > >> >> > well)?
> > >> >>
> > >> >> You have to add 2 configs in hdfs-site.xml, which we can't do from
> > >> HBase.
> > >> >>
> > >> >> J-D
> > >> >>
> > >>
> > >>
> > >>
> > >> --
> > >> Best regards,
> > >>
> > >> - Andy
> > >>
> > >> Problems worthy of attack prove their worth by hitting back. - Piet
> > >> Hein (via Tom White)
> > >>
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein (via Tom White)
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: read short circuit
Posted by n keywal <nk...@gmail.com>.
If I sum-up :
We make it the default:
1) Documented as such (what JD wrote already + a reference to HDFS-3907)
2) Set in the config we use for the unit tests.
Andrew, you said that we should keep some test with it off ? May be we
could do that in one of the existing build on the trunk, for example
HBase-TRUNK-on-Hadoop-2.0.0? i.e.: We run the tests on trunk with
short-circuit on; we run the test on hadoop-2 with short circuit off.
I propose it this way because may be the tests will be faster with
short-circuit on, and as well if it's the default production config it
should be the one in the "main" test path.
If that's ok, I can do test it on our test suite and do the jira for the
integration.
On Thu, Sep 13, 2012 at 8:31 PM, Andrew Purtell <ap...@apache.org> wrote:
> That's https://issues.apache.org/jira/browse/HBASE-4867
>
> On Thu, Sep 13, 2012 at 10:57 AM, Jesse Yates <je...@gmail.com>
> wrote:
> > There was a jira a while back (can't remember the number) that I think
> came
> > from the Facebook fellas to have multiple hbase-site.xmls, with a naming
> > scheme such that you can do layering of configurations.
> >
> > What ever happened to that?
> > -------------------
> > Jesse Yates
> > @jesse_yates
> > jyates.github.com
> >
> >
> > On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <apurtell@apache.org
> >wrote:
> >
> >> Making it the default seems reasonable.
> >>
> >> For <= 0.94, the security profile has a separate hbase-site.xml file
> >> under test resources where there it could be defaulted as off. Reason
> >> is the local block sharing requires some shared group membership
> >> between HBase and HDFS service accounts that might be objectionable in
> >> a secure setting. So we should have test coverage with it off.
> >>
> >> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <nk...@gmail.com> wrote:
> >> > It's all in the "pretty much" :-)
> >> > Should we make it the default (i.e. recommending it in the doc,
> emitting
> >> a
> >> > warning if it's not set) for all installation ?
> >> > Or would it make sense to separate the store and wal cases, meaning
> that
> >> we
> >> > need a change in hdfs to make this possible?
> >> >
> >> >
> >> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <
> jdcryans@apache.org
> >> >wrote:
> >> >
> >> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <nk...@gmail.com> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > Is there any known counter indication for read short circuit?
> >> >> > I had a look at JD's presentation from January. It was tested vs.
> >> HBase
> >> >> as
> >> >> > a whole; and still recommended for nearly all cases.
> >> >>
> >> >> Pretty much.
> >> >>
> >> >> >
> >> >> > We could make it at least the default when reading HLog files (even
> >> if it
> >> >> > seems it's a 'per client" config, there is may be something to look
> >> at as
> >> >> > well)?
> >> >>
> >> >> You have to add 2 configs in hdfs-site.xml, which we can't do from
> >> HBase.
> >> >>
> >> >> J-D
> >> >>
> >>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >> - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet
> >> Hein (via Tom White)
> >>
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein (via Tom White)
>
Re: read short circuit
Posted by Andrew Purtell <ap...@apache.org>.
That's https://issues.apache.org/jira/browse/HBASE-4867
On Thu, Sep 13, 2012 at 10:57 AM, Jesse Yates <je...@gmail.com> wrote:
> There was a jira a while back (can't remember the number) that I think came
> from the Facebook fellas to have multiple hbase-site.xmls, with a naming
> scheme such that you can do layering of configurations.
>
> What ever happened to that?
> -------------------
> Jesse Yates
> @jesse_yates
> jyates.github.com
>
>
> On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <ap...@apache.org>wrote:
>
>> Making it the default seems reasonable.
>>
>> For <= 0.94, the security profile has a separate hbase-site.xml file
>> under test resources where there it could be defaulted as off. Reason
>> is the local block sharing requires some shared group membership
>> between HBase and HDFS service accounts that might be objectionable in
>> a secure setting. So we should have test coverage with it off.
>>
>> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <nk...@gmail.com> wrote:
>> > It's all in the "pretty much" :-)
>> > Should we make it the default (i.e. recommending it in the doc, emitting
>> a
>> > warning if it's not set) for all installation ?
>> > Or would it make sense to separate the store and wal cases, meaning that
>> we
>> > need a change in hdfs to make this possible?
>> >
>> >
>> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <jdcryans@apache.org
>> >wrote:
>> >
>> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <nk...@gmail.com> wrote:
>> >> > Hi,
>> >> >
>> >> > Is there any known counter indication for read short circuit?
>> >> > I had a look at JD's presentation from January. It was tested vs.
>> HBase
>> >> as
>> >> > a whole; and still recommended for nearly all cases.
>> >>
>> >> Pretty much.
>> >>
>> >> >
>> >> > We could make it at least the default when reading HLog files (even
>> if it
>> >> > seems it's a 'per client" config, there is may be something to look
>> at as
>> >> > well)?
>> >>
>> >> You have to add 2 configs in hdfs-site.xml, which we can't do from
>> HBase.
>> >>
>> >> J-D
>> >>
>>
>>
>>
>> --
>> Best regards,
>>
>> - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet
>> Hein (via Tom White)
>>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)
Re: read short circuit
Posted by Jesse Yates <je...@gmail.com>.
There was a jira a while back (can't remember the number) that I think came
from the Facebook fellas to have multiple hbase-site.xmls, with a naming
scheme such that you can do layering of configurations.
What ever happened to that?
-------------------
Jesse Yates
@jesse_yates
jyates.github.com
On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <ap...@apache.org>wrote:
> Making it the default seems reasonable.
>
> For <= 0.94, the security profile has a separate hbase-site.xml file
> under test resources where there it could be defaulted as off. Reason
> is the local block sharing requires some shared group membership
> between HBase and HDFS service accounts that might be objectionable in
> a secure setting. So we should have test coverage with it off.
>
> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <nk...@gmail.com> wrote:
> > It's all in the "pretty much" :-)
> > Should we make it the default (i.e. recommending it in the doc, emitting
> a
> > warning if it's not set) for all installation ?
> > Or would it make sense to separate the store and wal cases, meaning that
> we
> > need a change in hdfs to make this possible?
> >
> >
> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
> >
> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <nk...@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > Is there any known counter indication for read short circuit?
> >> > I had a look at JD's presentation from January. It was tested vs.
> HBase
> >> as
> >> > a whole; and still recommended for nearly all cases.
> >>
> >> Pretty much.
> >>
> >> >
> >> > We could make it at least the default when reading HLog files (even
> if it
> >> > seems it's a 'per client" config, there is may be something to look
> at as
> >> > well)?
> >>
> >> You have to add 2 configs in hdfs-site.xml, which we can't do from
> HBase.
> >>
> >> J-D
> >>
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein (via Tom White)
>
Re: read short circuit
Posted by Andrew Purtell <ap...@apache.org>.
Making it the default seems reasonable.
For <= 0.94, the security profile has a separate hbase-site.xml file
under test resources where there it could be defaulted as off. Reason
is the local block sharing requires some shared group membership
between HBase and HDFS service accounts that might be objectionable in
a secure setting. So we should have test coverage with it off.
On Thu, Sep 13, 2012 at 9:42 AM, n keywal <nk...@gmail.com> wrote:
> It's all in the "pretty much" :-)
> Should we make it the default (i.e. recommending it in the doc, emitting a
> warning if it's not set) for all installation ?
> Or would it make sense to separate the store and wal cases, meaning that we
> need a change in hdfs to make this possible?
>
>
> On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <nk...@gmail.com> wrote:
>> > Hi,
>> >
>> > Is there any known counter indication for read short circuit?
>> > I had a look at JD's presentation from January. It was tested vs. HBase
>> as
>> > a whole; and still recommended for nearly all cases.
>>
>> Pretty much.
>>
>> >
>> > We could make it at least the default when reading HLog files (even if it
>> > seems it's a 'per client" config, there is may be something to look at as
>> > well)?
>>
>> You have to add 2 configs in hdfs-site.xml, which we can't do from HBase.
>>
>> J-D
>>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)
Re: read short circuit
Posted by n keywal <nk...@gmail.com>.
It's all in the "pretty much" :-)
Should we make it the default (i.e. recommending it in the doc, emitting a
warning if it's not set) for all installation ?
Or would it make sense to separate the store and wal cases, meaning that we
need a change in hdfs to make this possible?
On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <nk...@gmail.com> wrote:
> > Hi,
> >
> > Is there any known counter indication for read short circuit?
> > I had a look at JD's presentation from January. It was tested vs. HBase
> as
> > a whole; and still recommended for nearly all cases.
>
> Pretty much.
>
> >
> > We could make it at least the default when reading HLog files (even if it
> > seems it's a 'per client" config, there is may be something to look at as
> > well)?
>
> You have to add 2 configs in hdfs-site.xml, which we can't do from HBase.
>
> J-D
>
Re: read short circuit
Posted by Jean-Daniel Cryans <jd...@apache.org>.
On Thu, Sep 13, 2012 at 7:58 AM, n keywal <nk...@gmail.com> wrote:
> Hi,
>
> Is there any known counter indication for read short circuit?
> I had a look at JD's presentation from January. It was tested vs. HBase as
> a whole; and still recommended for nearly all cases.
Pretty much.
>
> We could make it at least the default when reading HLog files (even if it
> seems it's a 'per client" config, there is may be something to look at as
> well)?
You have to add 2 configs in hdfs-site.xml, which we can't do from HBase.
J-D