You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by Fuad Efendi <fu...@efendi.ca> on 2011/04/04 18:30:38 UTC
Strange Classloader Problem
Hi,
I am developing output connector to HBase.
After many hours it seems to me ManifoldCF class loader doesn't see any
config files (*.XML) inside connector-lib/*.jar, and I can't even manage it
via explicit -classpath variable. is that true?
Thanks,
Fuad
Re: Strange Classloader Problem
Posted by Karl Wright <da...@gmail.com>.
Hi Fuad,
At first glance this really does seem to be an HBase bug. Instead of
using the thread classloader, it should be starting with whatever
classloader was used to load the HBase class. Otherwise, it can't
reliably get at the configuration data in the jar (or anything else,
for that matter, that's located in the jar).
Perhaps you could file an HBase ticket. For now, though, your
workaround doesn't seem to be too horrible. Depending on what is
actually in the configuration xml, you may eventually want to include
the ability to set the configuration in the connector anyhow, since
ManifoldCF discourages external configuration files for connectors.
Karl
On Mon, Apr 4, 2011 at 2:12 PM, Fuad Efendi <fu...@efendi.ca> wrote:
> Ok, I located the problem (is it really a problem?) and found workaround.
>
> Scenario
> 1. Output connector has dependency on hbase.jar
> 2. hbase.jar has a file hbase-default.xml in a root (with default
> configuration)
> 3. hbase can't load this
>
> HBase uses this classloader:
>
> ClassLoader classLoader = Thread.currentThread().getContextClassLoader()
>
>
> Then, it tries this:
> classLoader.getResource("hbase-default.xml")
>
> and returns null (not found) if it runs under ManifoldCF... (or may by any
> other web container?)
>
>
> Workaround is explicit configuration for HBase
> Configuration config = new Configuration();
> File f = new
> File("C:\\java\\tools\\ManifoldCF\\example\\conf\\hbase-default.xml");
> FileInputStream is = new FileInputStream(f);
> config.addResource(is);
>
>
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: Fuad Efendi [mailto:fuad@efendi.ca]
> Sent: April-04-11 1:42 PM
> To: connectors-user@incubator.apache.org
> Subject: RE: Strange Classloader Problem
>
>
> Ok, I found workaround... HBase configuration should be explicit,
> Configuration config = new Configuration();
> File f = new
> File("C:\\java\\tools\\ManifoldCF\\example\\conf\\hbase-default.xml");
> FileInputStream is = new FileInputStream(f);
> config.addResource(is);
>
>
> Default code (referred in many guides for beginners) does not work:
>
> config = HBaseConfiguration.create();
>
>
> It will call implicitly this method of HBase:
> /**
> * Add a configuration resource.
> *
> * The properties of this resource will override properties of previously
> * added resources, unless they were marked <a href="#Final">final</a>.
> *
> * @param name resource to be added, the classpath is examined for a file
> * with that name.
> */
> public void addResource(String name) {
> addResourceObject(name);
> }
>
>
>
> And... somehow HBase run under ManifoldCF/connector-lib can't find this
> resource (in a classpath)... I tried all imaginable methods...
>
>
>
>
>
>
>
> -----Original Message-----
> From: Fuad Efendi [mailto:fuad@efendi.ca]
> Sent: April-04-11 12:52 PM
> To: connectors-user@incubator.apache.org
> Subject: RE: Strange Classloader Problem
>
> Hi Karl,
>
> I have hbase.jar file in connector-lib folder, and it seems it is loaded
> except hbase-default.xml file inside this hbase.jar file;
>
> 1. java -jar start.jar
> Caused by: java.lang.RuntimeException: hbase-default.xml file seems to be
> for and old version of HBase (null), this version is 0.90.1
> at
> org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfigu
> ration.java:66)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfigurat
> ion.java:76)
>
> 2. Explicit "test' command such as this one works just fine (so that nothing
> wrong with hbase.jar/hbase-default.xml):
> java -cp
> mcf-hbaseoutput-connector.jar;junit-4.7.jar;hbase-0.90.1.jar;hadoop-core-0.2
> 0-append-r1056497.jar;commons-logging.jar;zookeeper-3.3.2.jar;log4j-1.2.jar;
> mcf-hbaseoutput-connector.jar
> org.apache.manifoldcf.agents.output.hbaseconnector.HBaseTest
>
>
> Putting hbase.jar into "lib" doesn't help; it should be inside
> "connector-lib"; and ManifoldCF doesn't see this hbase-default.xml file...
>
>
>
>
>
> -----Original Message-----
> From: Karl Wright [mailto:daddywri@gmail.com]
> Sent: April-04-11 12:36 PM
> To: connectors-user@incubator.apache.org
> Subject: Re: Strange Classloader Problem
>
> The ManifoldCF class loader is used for loading classes, not for loading XML
> files. The only XML files ManifoldCF uses are its properties.xml file and
> its connectors.xml. There should be no reason to use a special class loader
> to get these files, since they are needed to bootstrap the whole system.
>
> Where are you trying to load XML files, and for what purpose? The place you
> access a file is where you would need to add support for loading it via the
> class loader.
>
> Karl
>
> On Mon, Apr 4, 2011 at 12:30 PM, Fuad Efendi <fu...@efendi.ca> wrote:
>> Hi,
>>
>>
>>
>> I am developing output connector to HBase.
>>
>>
>>
>> After many hours it seems to me ManifoldCF class loader doesn't see
>> any config files (*.XML) inside connector-lib/*.jar, and I can't even
>> manage it via explicit -classpath variable. is that true?
>>
>> Thanks,
>>
>> Fuad
>>
>>
>
>
RE: Strange Classloader Problem
Posted by Fuad Efendi <fu...@efendi.ca>.
Ok, I located the problem (is it really a problem?) and found workaround.
Scenario
1. Output connector has dependency on hbase.jar
2. hbase.jar has a file hbase-default.xml in a root (with default
configuration)
3. hbase can't load this
HBase uses this classloader:
ClassLoader classLoader = Thread.currentThread().getContextClassLoader()
Then, it tries this:
classLoader.getResource("hbase-default.xml")
and returns null (not found) if it runs under ManifoldCF... (or may by any
other web container?)
Workaround is explicit configuration for HBase
Configuration config = new Configuration();
File f = new
File("C:\\java\\tools\\ManifoldCF\\example\\conf\\hbase-default.xml");
FileInputStream is = new FileInputStream(f);
config.addResource(is);
-----Original Message-----
From: Fuad Efendi [mailto:fuad@efendi.ca]
Sent: April-04-11 1:42 PM
To: connectors-user@incubator.apache.org
Subject: RE: Strange Classloader Problem
Ok, I found workaround... HBase configuration should be explicit,
Configuration config = new Configuration();
File f = new
File("C:\\java\\tools\\ManifoldCF\\example\\conf\\hbase-default.xml");
FileInputStream is = new FileInputStream(f);
config.addResource(is);
Default code (referred in many guides for beginners) does not work:
config = HBaseConfiguration.create();
It will call implicitly this method of HBase:
/**
* Add a configuration resource.
*
* The properties of this resource will override properties of previously
* added resources, unless they were marked <a href="#Final">final</a>.
*
* @param name resource to be added, the classpath is examined for a file
* with that name.
*/
public void addResource(String name) {
addResourceObject(name);
}
And... somehow HBase run under ManifoldCF/connector-lib can't find this
resource (in a classpath)... I tried all imaginable methods...
-----Original Message-----
From: Fuad Efendi [mailto:fuad@efendi.ca]
Sent: April-04-11 12:52 PM
To: connectors-user@incubator.apache.org
Subject: RE: Strange Classloader Problem
Hi Karl,
I have hbase.jar file in connector-lib folder, and it seems it is loaded
except hbase-default.xml file inside this hbase.jar file;
1. java -jar start.jar
Caused by: java.lang.RuntimeException: hbase-default.xml file seems to be
for and old version of HBase (null), this version is 0.90.1
at
org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfigu
ration.java:66)
at
org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfigurat
ion.java:76)
2. Explicit "test' command such as this one works just fine (so that nothing
wrong with hbase.jar/hbase-default.xml):
java -cp
mcf-hbaseoutput-connector.jar;junit-4.7.jar;hbase-0.90.1.jar;hadoop-core-0.2
0-append-r1056497.jar;commons-logging.jar;zookeeper-3.3.2.jar;log4j-1.2.jar;
mcf-hbaseoutput-connector.jar
org.apache.manifoldcf.agents.output.hbaseconnector.HBaseTest
Putting hbase.jar into "lib" doesn't help; it should be inside
"connector-lib"; and ManifoldCF doesn't see this hbase-default.xml file...
-----Original Message-----
From: Karl Wright [mailto:daddywri@gmail.com]
Sent: April-04-11 12:36 PM
To: connectors-user@incubator.apache.org
Subject: Re: Strange Classloader Problem
The ManifoldCF class loader is used for loading classes, not for loading XML
files. The only XML files ManifoldCF uses are its properties.xml file and
its connectors.xml. There should be no reason to use a special class loader
to get these files, since they are needed to bootstrap the whole system.
Where are you trying to load XML files, and for what purpose? The place you
access a file is where you would need to add support for loading it via the
class loader.
Karl
On Mon, Apr 4, 2011 at 12:30 PM, Fuad Efendi <fu...@efendi.ca> wrote:
> Hi,
>
>
>
> I am developing output connector to HBase.
>
>
>
> After many hours it seems to me ManifoldCF class loader doesn't see
> any config files (*.XML) inside connector-lib/*.jar, and I can't even
> manage it via explicit -classpath variable. is that true?
>
> Thanks,
>
> Fuad
>
>
RE: Strange Classloader Problem
Posted by Fuad Efendi <fu...@efendi.ca>.
Ok, I found workaround... HBase configuration should be explicit,
Configuration config = new Configuration();
File f = new
File("C:\\java\\tools\\ManifoldCF\\example\\conf\\hbase-default.xml");
FileInputStream is = new FileInputStream(f);
config.addResource(is);
Default code (referred in many guides for beginners) does not work:
config = HBaseConfiguration.create();
It will call implicitly this method of HBase:
/**
* Add a configuration resource.
*
* The properties of this resource will override properties of previously
* added resources, unless they were marked <a href="#Final">final</a>.
*
* @param name resource to be added, the classpath is examined for a file
* with that name.
*/
public void addResource(String name) {
addResourceObject(name);
}
And... somehow HBase run under ManifoldCF/connector-lib can't find this
resource (in a classpath)... I tried all imaginable methods...
-----Original Message-----
From: Fuad Efendi [mailto:fuad@efendi.ca]
Sent: April-04-11 12:52 PM
To: connectors-user@incubator.apache.org
Subject: RE: Strange Classloader Problem
Hi Karl,
I have hbase.jar file in connector-lib folder, and it seems it is loaded
except hbase-default.xml file inside this hbase.jar file;
1. java -jar start.jar
Caused by: java.lang.RuntimeException: hbase-default.xml file seems to be
for and old version of HBase (null), this version is 0.90.1
at
org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfigu
ration.java:66)
at
org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfigurat
ion.java:76)
2. Explicit "test' command such as this one works just fine (so that nothing
wrong with hbase.jar/hbase-default.xml):
java -cp
mcf-hbaseoutput-connector.jar;junit-4.7.jar;hbase-0.90.1.jar;hadoop-core-0.2
0-append-r1056497.jar;commons-logging.jar;zookeeper-3.3.2.jar;log4j-1.2.jar;
mcf-hbaseoutput-connector.jar
org.apache.manifoldcf.agents.output.hbaseconnector.HBaseTest
Putting hbase.jar into "lib" doesn't help; it should be inside
"connector-lib"; and ManifoldCF doesn't see this hbase-default.xml file...
-----Original Message-----
From: Karl Wright [mailto:daddywri@gmail.com]
Sent: April-04-11 12:36 PM
To: connectors-user@incubator.apache.org
Subject: Re: Strange Classloader Problem
The ManifoldCF class loader is used for loading classes, not for loading XML
files. The only XML files ManifoldCF uses are its properties.xml file and
its connectors.xml. There should be no reason to use a special class loader
to get these files, since they are needed to bootstrap the whole system.
Where are you trying to load XML files, and for what purpose? The place you
access a file is where you would need to add support for loading it via the
class loader.
Karl
On Mon, Apr 4, 2011 at 12:30 PM, Fuad Efendi <fu...@efendi.ca> wrote:
> Hi,
>
>
>
> I am developing output connector to HBase.
>
>
>
> After many hours it seems to me ManifoldCF class loader doesn't see
> any config files (*.XML) inside connector-lib/*.jar, and I can't even
> manage it via explicit -classpath variable. is that true?
>
> Thanks,
>
> Fuad
>
>
RE: Strange Classloader Problem
Posted by Fuad Efendi <fu...@efendi.ca>.
Hi Karl,
I have hbase.jar file in connector-lib folder, and it seems it is loaded
except hbase-default.xml file inside this hbase.jar file;
1. java -jar start.jar
Caused by: java.lang.RuntimeException: hbase-default.xml file seems to be
for and old version of HBase (null), this version is 0.90.1
at
org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfigu
ration.java:66)
at
org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfigurat
ion.java:76)
2. Explicit "test' command such as this one works just fine (so that nothing
wrong with hbase.jar/hbase-default.xml):
java -cp
mcf-hbaseoutput-connector.jar;junit-4.7.jar;hbase-0.90.1.jar;hadoop-core-0.2
0-append-r1056497.jar;commons-logging.jar;zookeeper-3.3.2.jar;log4j-1.2.jar;
mcf-hbaseoutput-connector.jar
org.apache.manifoldcf.agents.output.hbaseconnector.HBaseTest
Putting hbase.jar into "lib" doesn't help; it should be inside
"connector-lib"; and ManifoldCF doesn't see this hbase-default.xml file...
-----Original Message-----
From: Karl Wright [mailto:daddywri@gmail.com]
Sent: April-04-11 12:36 PM
To: connectors-user@incubator.apache.org
Subject: Re: Strange Classloader Problem
The ManifoldCF class loader is used for loading classes, not for loading XML
files. The only XML files ManifoldCF uses are its properties.xml file and
its connectors.xml. There should be no reason to use a special class loader
to get these files, since they are needed to bootstrap the whole system.
Where are you trying to load XML files, and for what purpose? The place you
access a file is where you would need to add support for loading it via the
class loader.
Karl
On Mon, Apr 4, 2011 at 12:30 PM, Fuad Efendi <fu...@efendi.ca> wrote:
> Hi,
>
>
>
> I am developing output connector to HBase.
>
>
>
> After many hours it seems to me ManifoldCF class loader doesn't see
> any config files (*.XML) inside connector-lib/*.jar, and I can't even
> manage it via explicit -classpath variable. is that true?
>
> Thanks,
>
> Fuad
>
>
Re: Strange Classloader Problem
Posted by Karl Wright <da...@gmail.com>.
The ManifoldCF class loader is used for loading classes, not for
loading XML files. The only XML files ManifoldCF uses are its
properties.xml file and its connectors.xml. There should be no reason
to use a special class loader to get these files, since they are
needed to bootstrap the whole system.
Where are you trying to load XML files, and for what purpose? The
place you access a file is where you would need to add support for
loading it via the class loader.
Karl
On Mon, Apr 4, 2011 at 12:30 PM, Fuad Efendi <fu...@efendi.ca> wrote:
> Hi,
>
>
>
> I am developing output connector to HBase…
>
>
>
> After many hours it seems to me ManifoldCF class loader doesn’t see any
> config files (*.XML) inside connector-lib/*.jar, and I can’t even manage it
> via explicit –classpath variable… is that true?
>
> Thanks,
>
> Fuad
>
>