You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Henning Blohm <he...@zfabrik.de> on 2010/10/13 16:09:36 UTC
Bug when reusing org.apache.hadoop.conf.Configuration in
HbaseConfiguration
Hi,
I used
Configuration cfg = new Configuration();
cfg.addResource(InputStream)
on org.apache.hadoop.conf.Configuration to create hadoop configuration.
That configuration is
used to construct HbaseConfiguration objects several times later on:
HbaseConfiguration hbcfg = new HbaseConfiguration(cfg)
At the second this leads to an error as the passed on configuraiton
object
is asked to load its properties again, trying to read from the
inputstream once more
(as the stream is memorized!!).
The fact that the properties of the passed on configuration object have
been reset somewhat
surprising: When using the hbase configuration object constructed above
in HbaseAdmin, this
will eventually load JobConf which has a static initializer that calls
Configuration.addDefaultResource("mapred-default.xml");
which in turn goes through the Configuration.REGISTRY which holds on to
any previously
not yet collected Configuration objects to (hold your breath!) force
them to reload their configuration
by calling reloadConfiguration that (now we are there) sets
properties=null.
Did anybody follow that....
It seems there is somewhat surprising side effects in hadoop/hbase
configuration handling.
Wouldn't it be better to have the default resource (pragmatically)
defined once in Configuration
and not (even think about) touch already instantiated config objects
later on?
Thanks,
Henning
Ps.:
Here is one relevant stack trace:
Caused by: java.lang.RuntimeException: java.io.IOException: Ungültiger
Dateideskriptor
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1162)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1040)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:980)
at
org.apache.hadoop.conf.Configuration.iterator(Configuration.java:1016)
at
org.apache.hadoop.hbase.HBaseConfiguration.<init>(HBaseConfiguration.java:43)
<some removed>
... 56 more
Caused by: java.io.IOException: Ungültiger Dateideskriptor
at java.io.FileInputStream.read(Native Method)
at com.sun.org.apache.xerces.internal.impl.XMLEntityManager
$RewindableInputStream.read(XMLEntityManager.java:2933)
at
com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:704)
at
com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineDocVersion(XMLVersionDetector.java:186)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:107)
at
com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:225)
at
com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:283)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:124)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1099)
Re: Bug when reusing org.apache.hadoop.conf.Configuration in
HbaseConfiguration
Posted by Henning Blohm <he...@zfabrik.de>.
Hi St.Ack,
thinking about it again, I think that HBase cannot really do much
about it.
I checked in hadoop/trunk and Configuration still has the WeakHashmap
REGISTRY that is used upon calling addDefaultResource (which is called
from mapred classes at least) to reset all reachable Configuration
objects plus it has addResource(InputStream).
I sent a note to the hadoop mailing list on this.
Thanks,
Henning
On Wed, 2010-10-13 at 18:06 -0400, Stack wrote:
> On Wed, Oct 13, 2010 at 10:09 AM, Henning Blohm
> <he...@zfabrik.de> wrote:
> > Hi,
> >
> > I used
> >
> > Configuration cfg = new Configuration();
> > cfg.addResource(InputStream)
> >
> > on org.apache.hadoop.conf.Configuration to create hadoop configuration.
> > That configuration is
> > used to construct HbaseConfiguration objects several times later on:
> >
> > HbaseConfiguration hbcfg = new HbaseConfiguration(cfg)
> >
>
> In TRUNK HBaseConfiguration is deprecated. Do you see same phenomena
> when you do HBaseConfiguration.create(cfg)?
>
> See below...
>
> > At the second this leads to an error as the passed on configuraiton
> > object
> > is asked to load its properties again, trying to read from the
> > inputstream once more
> > (as the stream is memorized!!).
> >
> > The fact that the properties of the passed on configuration object have
> > been reset somewhat
> > surprising: When using the hbase configuration object constructed above
> > in HbaseAdmin, this
> > will eventually load JobConf which has a static initializer that calls
> >
> > Configuration.addDefaultResource("mapred-default.xml");
> >
> > which in turn goes through the Configuration.REGISTRY which holds on to
> > any previously
> > not yet collected Configuration objects to (hold your breath!) force
> > them to reload their configuration
> > by calling reloadConfiguration that (now we are there) sets
> > properties=null.
> >
> > Did anybody follow that....
> >
> > It seems there is somewhat surprising side effects in hadoop/hbase
> > configuration handling.
> >
> > Wouldn't it be better to have the default resource (pragmatically)
> > defined once in Configuration
> > and not (even think about) touch already instantiated config objects
> > later on?
> >
>
> Yes. That sounds completely reasonable.
>
> See if HBaseConfiguration.create(cfg) gives you the same issue. If
> so, then its the way Hadoop Configuration works currently. I haven't
> spent time on it in a while but I remember getting into interesting
> scenarios loading properties. If you can confine the issue some, lets
> file an issue up in hadoop common?
>
> Thanks Henning,
> St.Ack
Re: Bug when reusing org.apache.hadoop.conf.Configuration in HbaseConfiguration
Posted by Stack <st...@duboce.net>.
On Wed, Oct 13, 2010 at 10:09 AM, Henning Blohm
<he...@zfabrik.de> wrote:
> Hi,
>
> I used
>
> Configuration cfg = new Configuration();
> cfg.addResource(InputStream)
>
> on org.apache.hadoop.conf.Configuration to create hadoop configuration.
> That configuration is
> used to construct HbaseConfiguration objects several times later on:
>
> HbaseConfiguration hbcfg = new HbaseConfiguration(cfg)
>
In TRUNK HBaseConfiguration is deprecated. Do you see same phenomena
when you do HBaseConfiguration.create(cfg)?
See below...
> At the second this leads to an error as the passed on configuraiton
> object
> is asked to load its properties again, trying to read from the
> inputstream once more
> (as the stream is memorized!!).
>
> The fact that the properties of the passed on configuration object have
> been reset somewhat
> surprising: When using the hbase configuration object constructed above
> in HbaseAdmin, this
> will eventually load JobConf which has a static initializer that calls
>
> Configuration.addDefaultResource("mapred-default.xml");
>
> which in turn goes through the Configuration.REGISTRY which holds on to
> any previously
> not yet collected Configuration objects to (hold your breath!) force
> them to reload their configuration
> by calling reloadConfiguration that (now we are there) sets
> properties=null.
>
> Did anybody follow that....
>
> It seems there is somewhat surprising side effects in hadoop/hbase
> configuration handling.
>
> Wouldn't it be better to have the default resource (pragmatically)
> defined once in Configuration
> and not (even think about) touch already instantiated config objects
> later on?
>
Yes. That sounds completely reasonable.
See if HBaseConfiguration.create(cfg) gives you the same issue. If
so, then its the way Hadoop Configuration works currently. I haven't
spent time on it in a while but I remember getting into interesting
scenarios loading properties. If you can confine the issue some, lets
file an issue up in hadoop common?
Thanks Henning,
St.Ack
Re: hbase and realtime
Posted by Barney Frank <ba...@gmail.com>.
Just to verify, so you are expecting, at peak load, 30 thousand requests to
hbase per second? Out of curiousity, what would be the size/configuration
of the hbase cluster for something like that? My application is similar to
yours, but nowhere near the volume...yet.
On Wed, Oct 13, 2010 at 5:14 PM, Stack <st...@duboce.net> wrote:
> On Wed, Oct 13, 2010 at 6:10 PM, Andre Reiter <a....@web.de> wrote:
> > hi Stack,
> >
> > thanks for your reply
> >
> >
> > Am 10/13/10 9:52 PM schrieb "Stack" unter <st...@duboce.net>:
> >
> >> How big will the records grow?
> > assuming 30.000 requests per second, a request appends 100 bytes to the
> user
> > record, one record should not be bigger than 10 kBytes, old "requests"
> can
> > actually be dropped from the record to keep it small
> >
>
> This should be fine.
> St.Ack
>
Re: hbase and realtime
Posted by Stack <st...@duboce.net>.
On Wed, Oct 13, 2010 at 6:10 PM, Andre Reiter <a....@web.de> wrote:
> hi Stack,
>
> thanks for your reply
>
>
> Am 10/13/10 9:52 PM schrieb "Stack" unter <st...@duboce.net>:
>
>> How big will the records grow?
> assuming 30.000 requests per second, a request appends 100 bytes to the user
> record, one record should not be bigger than 10 kBytes, old "requests" can
> actually be dropped from the record to keep it small
>
This should be fine.
St.Ack
Re: hbase and realtime
Posted by Andre Reiter <a....@web.de>.
hi Stack,
thanks for your reply
Am 10/13/10 9:52 PM schrieb "Stack" unter <st...@duboce.net>:
> How big will the records grow?
assuming 30.000 requests per second, a request appends 100 bytes to the user
record, one record should not be bigger than 10 kBytes, old "requests" can
actually be dropped from the record to keep it small
>
> 50ms should be enough to cover 95th percentile if not more of all
> requests, at least going by our experience at stumbleupon (servings
> that come out of cache will be well under 50ms).
>
> Its all random lookups? Any affinity between requests; i.e. will the
> caching layer in hbase help?
caching is for sure an option. if a user makes requests to our webservers,
its very likely he makes more than only one
>
> Have you tried it? Put up a little cluster and redirect some subset
> of your traffic and see what kinda rates you can sustain per server.
you are right, the best way is actually to make test runs... :-)
thanks again and best regards
andre
>
> St.Ack
>
> On Wed, Oct 13, 2010 at 10:19 AM, Andre Reiter <a....@web.de> wrote:
>> hi every body
>>
>> i'm evaluating hbase for a new platform.
>> the application is a web based application, and the challange is to handle
>> requests up to 1.000.000 times per second. this can actually be solved by
>> load ballanced webservers.
>> The problem now is to persist user data at real time. Let's say every
>> request appends some data to the user record, i.e. 100 bytes are appended.
>> The record would have an identifier placed in the cookie.
>>
>> The problem now is to look up the user record for every request at runtime.
>> Is it possible to get the user record with the information about all former
>> requests in a short time i.e. 50 ms ???
>>
>> i hove no experience with hadoop/hbase yet
>>
>> every help would be very appreciated
>>
>> thanks in advance
>> andre
>>
>>
>>
Re: hbase and realtime
Posted by Stack <st...@duboce.net>.
How big will the records grow?
50ms should be enough to cover 95th percentile if not more of all
requests, at least going by our experience at stumbleupon (servings
that come out of cache will be well under 50ms).
Its all random lookups? Any affinity between requests; i.e. will the
caching layer in hbase help?
Have you tried it? Put up a little cluster and redirect some subset
of your traffic and see what kinda rates you can sustain per server.
St.Ack
On Wed, Oct 13, 2010 at 10:19 AM, Andre Reiter <a....@web.de> wrote:
> hi every body
>
> i'm evaluating hbase for a new platform.
> the application is a web based application, and the challange is to handle
> requests up to 1.000.000 times per second. this can actually be solved by
> load ballanced webservers.
> The problem now is to persist user data at real time. Let's say every
> request appends some data to the user record, i.e. 100 bytes are appended.
> The record would have an identifier placed in the cookie.
>
> The problem now is to look up the user record for every request at runtime.
> Is it possible to get the user record with the information about all former
> requests in a short time i.e. 50 ms ???
>
> i hove no experience with hadoop/hbase yet
>
> every help would be very appreciated
>
> thanks in advance
> andre
>
>
>
hbase and realtime
Posted by Andre Reiter <a....@web.de>.
hi every body
i'm evaluating hbase for a new platform.
the application is a web based application, and the challange is to handle
requests up to 1.000.000 times per second. this can actually be solved by
load ballanced webservers.
The problem now is to persist user data at real time. Let's say every
request appends some data to the user record, i.e. 100 bytes are appended.
The record would have an identifier placed in the cookie.
The problem now is to look up the user record for every request at runtime.
Is it possible to get the user record with the information about all former
requests in a short time i.e. 50 ms ???
i hove no experience with hadoop/hbase yet
every help would be very appreciated
thanks in advance
andre