You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Enrico Triolo <en...@gmail.com> on 2006/06/28 16:13:52 UTC
Dynamic configuration properties
Hi all.
I embedded nutch in my application and implemented an indexing plugin
to add custom index fields. I need to pass some runtime parameters to
the plugin, so I was wondering if it is possible to set additional
configuration properties at runtime through the Configuration.setXXX
methods.
In my experiments I couldn't get this to work, so at the moment I'm
just creating a configuration file and adding it with
addDefaultResource each time I run nutch in my application.
Bye,
Enrico
Re: Dynamic configuration properties
Posted by TDLN <di...@gmail.com>.
My proposed solution will not help you and I don't know how else to
accomplish this.
Anybody else?
Rgrds, Thomas
On 6/29/06, Enrico Triolo <en...@gmail.com> wrote:
> Thank you Thomas, but (if I don't miss something in your suggestion)
> my question is somewhat different. In particular, I can't know in
> advance what values can assume my properties.
>
> Let's say I have a method in my class that starts up nutch and needs
> to pass runtime parameters to my indexing plugin; at the moment, the
> only way I found to accomplish this is something like:
>
> class NutchTest {
>
> private Configuration conf;
>
> public NutchTest() {
> conf = NutchConfiguration.create();
> }
>
> public myMethod(int propertyValue) {
> //set runtime value
> this.conf.setInt("propertyName", propertyValue);
> //write an alternate config file with the property just set
> this.conf.write(new FileOutputStream("myConfig.xml"));
> //instantiate JobConf that will be passed to nutch
> JobConf job = new JobConf(this.conf);
> //add my config file to the resources handled by the JobConf
> job.addDefaultResource(new Path("myConfig.xml"));
>
> //start up nutch
> this.startNutch(job);
> }
>
> private void startNutch(JobConf job) {
> //Let's say this acts as Crawl.main method, with the exception
> that 'job' is used...
> }
>
> }
>
> This way, in my plugin I can get the property value, and everything
> works as expected.
> I don't really like this solution, especially because the 'write'
> method creates a file with all the properties defined in the
> hadoop-default.xml and hadoop-site.xml along with the ones defined in
> my class.
> It could be a better solution if there could be a way of adding only
> new properties, but the Configuration constructor automatically adds
> hadoop-default and hadoop-site to the default resources. So, I patched
> it and added an alternate constructor that doesn't load default
> resources.
>
> I was just wondering if there is a best-practice for this kind of
> problem or if I missed something in the hadoop API that can simplify
> this task...
>
> Enrico
>
> On 6/28/06, TDLN <di...@gmail.com> wrote:
> > On 0.8-dev the following works for me.
> >
> > 1) add the config property to nutch-site.xml using this format:
> >
> > <property>
> > <name>propertyName</name>
> > <value>propertyValue</value>
> > <description></description>
> > </property>
> >
> > 2) implement org.apache.hadoop.conf.Configurable Interface -> all
> > Filters IndexingFilter, ParseFilter extend this Interface already.
> >
> > private Configuration configuration;
> >
> > public Configuration getConf() {
> > return configuration;
> > }
> >
> > public void setConf(Configuration configuration) {
> > this.configuration = configuration;
> > }
> >
> >
> > 3) now get your property:
> >
> > String propertyVal = configuration.get("propertyName");
> >
> > HTH Thomas
> >
> >
> >
> >
> > On 6/28/06, Enrico Triolo <en...@gmail.com> wrote:
> > > Hi all.
> > > I embedded nutch in my application and implemented an indexing plugin
> > > to add custom index fields. I need to pass some runtime parameters to
> > > the plugin, so I was wondering if it is possible to set additional
> > > configuration properties at runtime through the Configuration.setXXX
> > > methods.
> > >
> > > In my experiments I couldn't get this to work, so at the moment I'm
> > > just creating a configuration file and adding it with
> > > addDefaultResource each time I run nutch in my application.
> > >
> > > Bye,
> > > Enrico
> > >
> >
>
Re: Dynamic configuration properties
Posted by Enrico Triolo <en...@gmail.com>.
Thank you Thomas, but (if I don't miss something in your suggestion)
my question is somewhat different. In particular, I can't know in
advance what values can assume my properties.
Let's say I have a method in my class that starts up nutch and needs
to pass runtime parameters to my indexing plugin; at the moment, the
only way I found to accomplish this is something like:
class NutchTest {
private Configuration conf;
public NutchTest() {
conf = NutchConfiguration.create();
}
public myMethod(int propertyValue) {
//set runtime value
this.conf.setInt("propertyName", propertyValue);
//write an alternate config file with the property just set
this.conf.write(new FileOutputStream("myConfig.xml"));
//instantiate JobConf that will be passed to nutch
JobConf job = new JobConf(this.conf);
//add my config file to the resources handled by the JobConf
job.addDefaultResource(new Path("myConfig.xml"));
//start up nutch
this.startNutch(job);
}
private void startNutch(JobConf job) {
//Let's say this acts as Crawl.main method, with the exception
that 'job' is used...
}
}
This way, in my plugin I can get the property value, and everything
works as expected.
I don't really like this solution, especially because the 'write'
method creates a file with all the properties defined in the
hadoop-default.xml and hadoop-site.xml along with the ones defined in
my class.
It could be a better solution if there could be a way of adding only
new properties, but the Configuration constructor automatically adds
hadoop-default and hadoop-site to the default resources. So, I patched
it and added an alternate constructor that doesn't load default
resources.
I was just wondering if there is a best-practice for this kind of
problem or if I missed something in the hadoop API that can simplify
this task...
Enrico
On 6/28/06, TDLN <di...@gmail.com> wrote:
> On 0.8-dev the following works for me.
>
> 1) add the config property to nutch-site.xml using this format:
>
> <property>
> <name>propertyName</name>
> <value>propertyValue</value>
> <description></description>
> </property>
>
> 2) implement org.apache.hadoop.conf.Configurable Interface -> all
> Filters IndexingFilter, ParseFilter extend this Interface already.
>
> private Configuration configuration;
>
> public Configuration getConf() {
> return configuration;
> }
>
> public void setConf(Configuration configuration) {
> this.configuration = configuration;
> }
>
>
> 3) now get your property:
>
> String propertyVal = configuration.get("propertyName");
>
> HTH Thomas
>
>
>
>
> On 6/28/06, Enrico Triolo <en...@gmail.com> wrote:
> > Hi all.
> > I embedded nutch in my application and implemented an indexing plugin
> > to add custom index fields. I need to pass some runtime parameters to
> > the plugin, so I was wondering if it is possible to set additional
> > configuration properties at runtime through the Configuration.setXXX
> > methods.
> >
> > In my experiments I couldn't get this to work, so at the moment I'm
> > just creating a configuration file and adding it with
> > addDefaultResource each time I run nutch in my application.
> >
> > Bye,
> > Enrico
> >
>
Re: Dynamic configuration properties
Posted by TDLN <di...@gmail.com>.
On 0.8-dev the following works for me.
1) add the config property to nutch-site.xml using this format:
<property>
<name>propertyName</name>
<value>propertyValue</value>
<description></description>
</property>
2) implement org.apache.hadoop.conf.Configurable Interface -> all
Filters IndexingFilter, ParseFilter extend this Interface already.
private Configuration configuration;
public Configuration getConf() {
return configuration;
}
public void setConf(Configuration configuration) {
this.configuration = configuration;
}
3) now get your property:
String propertyVal = configuration.get("propertyName");
HTH Thomas
On 6/28/06, Enrico Triolo <en...@gmail.com> wrote:
> Hi all.
> I embedded nutch in my application and implemented an indexing plugin
> to add custom index fields. I need to pass some runtime parameters to
> the plugin, so I was wondering if it is possible to set additional
> configuration properties at runtime through the Configuration.setXXX
> methods.
>
> In my experiments I couldn't get this to work, so at the moment I'm
> just creating a configuration file and adding it with
> addDefaultResource each time I run nutch in my application.
>
> Bye,
> Enrico
>