You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Jay Vyas <ja...@gmail.com> on 2013/01/16 19:05:52 UTC
Configuration object not loading parameters in unit tests
Hi guys:
I'm trying load some cluster parameters in a unit test, which uses a custom
filesystem.
However, my configuration object doesnt seem to be loading:
For example: conf.get("customparameter") (where "customparameter" is in
conf/core-site.xml) is returning null.
To investigate: I tried to print the Configuration object. Oddly, it
showed 4 paths, even though none of these files exist locally:
Configuration : Configuration: core-default.xml, core-site.xml,
mapred-default.xml, mapred-site.xml, conf/core-site.xml
How can I force my Configuration object to preferentially load
conf/core-site.xml ? I've added it using
Configuration.addDefaultResource(...), but it seems that the parameters are
not getting loaded into the runtime Configuration object.
Ultimately, the bug I have is that the default filesystem is being created,
rather than my custom filesystem.
File system: org.apache.hadoop.fs.LocalFileSystem@4ce2cb55
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Configuration object not loading parameters in unit tests
Posted by Jay Vyas <ja...@gmail.com>.
Is there a way to have the Configuration report wether or not it was able /
unable to find the default configuration resources? It looks at the moment
that it simply prints out all the resources it *wants* to find, but it
doesnt actually report the files which it *did* find on the classpath.
Re: Configuration object not loading parameters in unit tests
Posted by Jay Vyas <ja...@gmail.com>.
Is there a way to have the Configuration report wether or not it was able /
unable to find the default configuration resources? It looks at the moment
that it simply prints out all the resources it *wants* to find, but it
doesnt actually report the files which it *did* find on the classpath.
Re: Configuration object not loading parameters in unit tests
Posted by Jay Vyas <ja...@gmail.com>.
Is there a way to have the Configuration report wether or not it was able /
unable to find the default configuration resources? It looks at the moment
that it simply prints out all the resources it *wants* to find, but it
doesnt actually report the files which it *did* find on the classpath.
Re: Configuration object not loading parameters in unit tests
Posted by Jay Vyas <ja...@gmail.com>.
Is there a way to have the Configuration report wether or not it was able /
unable to find the default configuration resources? It looks at the moment
that it simply prints out all the resources it *wants* to find, but it
doesnt actually report the files which it *did* find on the classpath.
Re: Configuration object not loading parameters in unit tests
Posted by Harsh J <ha...@cloudera.com>.
The "known" config files (various *-site.xml) are loaded if they are
discoverable as system resources, i.e. they are present in a directory on
the classpath or in the root of a jar that is on the classpath.
You can perhaps add it to a directory under your root test folder that your
build system adds to the classpath. For instance, in Apache Hadoop which
uses Maven, we add it to the test/resources/ directory of a module and it
is made available in the classpath for test cases automatically.
On Wed, Jan 16, 2013 at 11:35 PM, Jay Vyas <ja...@gmail.com> wrote:
> Hi guys:
>
> I'm trying load some cluster parameters in a unit test, which uses a
> custom filesystem.
>
> However, my configuration object doesnt seem to be loading:
>
> For example: conf.get("customparameter") (where "customparameter" is in
> conf/core-site.xml) is returning null.
>
> To investigate: I tried to print the Configuration object. Oddly, it
> showed 4 paths, even though none of these files exist locally:
>
> Configuration : Configuration: core-default.xml, core-site.xml,
> mapred-default.xml, mapred-site.xml, conf/core-site.xml
>
> How can I force my Configuration object to preferentially load
> conf/core-site.xml ? I've added it using
> Configuration.addDefaultResource(...), but it seems that the parameters are
> not getting loaded into the runtime Configuration object.
>
> Ultimately, the bug I have is that the default filesystem is being
> created, rather than my custom filesystem.
>
> File system: org.apache.hadoop.fs.LocalFileSystem@4ce2cb55
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
--
Harsh J
Re: Configuration object not loading parameters in unit tests
Posted by Harsh J <ha...@cloudera.com>.
The "known" config files (various *-site.xml) are loaded if they are
discoverable as system resources, i.e. they are present in a directory on
the classpath or in the root of a jar that is on the classpath.
You can perhaps add it to a directory under your root test folder that your
build system adds to the classpath. For instance, in Apache Hadoop which
uses Maven, we add it to the test/resources/ directory of a module and it
is made available in the classpath for test cases automatically.
On Wed, Jan 16, 2013 at 11:35 PM, Jay Vyas <ja...@gmail.com> wrote:
> Hi guys:
>
> I'm trying load some cluster parameters in a unit test, which uses a
> custom filesystem.
>
> However, my configuration object doesnt seem to be loading:
>
> For example: conf.get("customparameter") (where "customparameter" is in
> conf/core-site.xml) is returning null.
>
> To investigate: I tried to print the Configuration object. Oddly, it
> showed 4 paths, even though none of these files exist locally:
>
> Configuration : Configuration: core-default.xml, core-site.xml,
> mapred-default.xml, mapred-site.xml, conf/core-site.xml
>
> How can I force my Configuration object to preferentially load
> conf/core-site.xml ? I've added it using
> Configuration.addDefaultResource(...), but it seems that the parameters are
> not getting loaded into the runtime Configuration object.
>
> Ultimately, the bug I have is that the default filesystem is being
> created, rather than my custom filesystem.
>
> File system: org.apache.hadoop.fs.LocalFileSystem@4ce2cb55
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
--
Harsh J
Re: Configuration object not loading parameters in unit tests
Posted by Harsh J <ha...@cloudera.com>.
The "known" config files (various *-site.xml) are loaded if they are
discoverable as system resources, i.e. they are present in a directory on
the classpath or in the root of a jar that is on the classpath.
You can perhaps add it to a directory under your root test folder that your
build system adds to the classpath. For instance, in Apache Hadoop which
uses Maven, we add it to the test/resources/ directory of a module and it
is made available in the classpath for test cases automatically.
On Wed, Jan 16, 2013 at 11:35 PM, Jay Vyas <ja...@gmail.com> wrote:
> Hi guys:
>
> I'm trying load some cluster parameters in a unit test, which uses a
> custom filesystem.
>
> However, my configuration object doesnt seem to be loading:
>
> For example: conf.get("customparameter") (where "customparameter" is in
> conf/core-site.xml) is returning null.
>
> To investigate: I tried to print the Configuration object. Oddly, it
> showed 4 paths, even though none of these files exist locally:
>
> Configuration : Configuration: core-default.xml, core-site.xml,
> mapred-default.xml, mapred-site.xml, conf/core-site.xml
>
> How can I force my Configuration object to preferentially load
> conf/core-site.xml ? I've added it using
> Configuration.addDefaultResource(...), but it seems that the parameters are
> not getting loaded into the runtime Configuration object.
>
> Ultimately, the bug I have is that the default filesystem is being
> created, rather than my custom filesystem.
>
> File system: org.apache.hadoop.fs.LocalFileSystem@4ce2cb55
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
--
Harsh J
Re: Configuration object not loading parameters in unit tests
Posted by Harsh J <ha...@cloudera.com>.
The "known" config files (various *-site.xml) are loaded if they are
discoverable as system resources, i.e. they are present in a directory on
the classpath or in the root of a jar that is on the classpath.
You can perhaps add it to a directory under your root test folder that your
build system adds to the classpath. For instance, in Apache Hadoop which
uses Maven, we add it to the test/resources/ directory of a module and it
is made available in the classpath for test cases automatically.
On Wed, Jan 16, 2013 at 11:35 PM, Jay Vyas <ja...@gmail.com> wrote:
> Hi guys:
>
> I'm trying load some cluster parameters in a unit test, which uses a
> custom filesystem.
>
> However, my configuration object doesnt seem to be loading:
>
> For example: conf.get("customparameter") (where "customparameter" is in
> conf/core-site.xml) is returning null.
>
> To investigate: I tried to print the Configuration object. Oddly, it
> showed 4 paths, even though none of these files exist locally:
>
> Configuration : Configuration: core-default.xml, core-site.xml,
> mapred-default.xml, mapred-site.xml, conf/core-site.xml
>
> How can I force my Configuration object to preferentially load
> conf/core-site.xml ? I've added it using
> Configuration.addDefaultResource(...), but it seems that the parameters are
> not getting loaded into the runtime Configuration object.
>
> Ultimately, the bug I have is that the default filesystem is being
> created, rather than my custom filesystem.
>
> File system: org.apache.hadoop.fs.LocalFileSystem@4ce2cb55
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
--
Harsh J