How to re-read the config files

[Arvind Sharma <ar...@yahoo.com>]

Hi,

I was wondering if there is way to let Hadoop re-read the config file (hadoop-site.xml) after  making some changes in it.

I don't want to restart the whole cluster for that.

I am using Hadoop 0.19.2

Thanks!
Arvind



      

Re: How to re-read the config files

[Ted Dunning <te...@gmail.com>]

You can do a rolling restart of the nodes.  The customer won't notice and
running programs will still complete in good order.  If you have rack
awareness configured, you can restart as many datanodes in a single rack as
you like since that won't compromise replication.  Restarting task trackers
should be similarly painless since any tasks on that node will just be
re-run.

A few jobs may notice small delays in completion due to re-running tasks,
but the effect should be minimal, especially if you have speculative
execution running.

To change configuration on the namenode or jobtracker, you will need to
schedule a few seconds of cluster downtime to restart those processes.  That
should not be a problem since Hadoop should not generally be used in a
product situation requiring high availability.

On Thu, Aug 13, 2009 at 3:08 PM, Arvind Sharma <ar...@yahoo.com> wrote:

> Sorry,I should have mentioned that - this I want to do without the code
> change.
>
> Something like - I have the cluster up and running and suddenly I realize
> that forgot to add some properties in the hadoop-site.xml file.  Now I can
> add these new properties - but how do these take into effect ? Without
> re-starting the cluster (which is in production and customer wouldn't like
> that either :-) )
>



-- 
Ted Dunning, CTO
DeepDyve

Re: How to re-read the config files

[Arvind Sharma <ar...@yahoo.com>]

Sorry,I should have mentioned that - this I want to do without the code change.

Something like - I have the cluster up and running and suddenly I realize that forgot to add some properties in the hadoop-site.xml file.  Now I can add these new properties - but how do these take into effect ? Without re-starting the cluster (which is in production and customer wouldn't like that either :-) )

Thanks!
Arvind




________________________________
From: Jakob Homan <jh...@yahoo-inc.com>
To: common-user@hadoop.apache.org
Sent: Thursday, August 13, 2009 2:04:43 PM
Subject: Re: How to re-read the config files

Hey Arvind-
   You'll probably want to look at the Configuration.reload() method, as demonstrated:

public class TestReloadConfig {
  public static void main(String[] args) throws IOException {
    Configuration conf = new Configuration();
    FileSystem fs = FileSystem.get(conf); // pull in dfs settings

    System.out.println("Replication = " + conf.get("dfs.replication"));
    System.out.println("Update file and press <enter>");
    new Scanner(System.in).nextLine();

    conf.reloadConfiguration();
    System.out.println("Now replication = " + conf.get("dfs.replication"));
  }
}

Note from the Javadoc: Values that are added via set methods will overlay values read from the resources.

Hope this helps.  Write back if you have more questions.

Thanks,
Jakob Homan
Hadoop at Yahoo!


Arvind Sharma wrote:
> Hi,
> 
> I was wondering if there is way to let Hadoop re-read the config file (hadoop-site.xml) after  making some changes in it.
> 
> I don't want to restart the whole cluster for that.
> 
> I am using Hadoop 0.19.2
> 
> Thanks!
> Arvind
> 
> 
> 
>      


      

Re: How to re-read the config files

[Jakob Homan <jh...@yahoo-inc.com>]

Hey Arvind-
    You'll probably want to look at the Configuration.reload() method, 
as demonstrated:

public class TestReloadConfig {
   public static void main(String[] args) throws IOException {
     Configuration conf = new Configuration();
     FileSystem fs = FileSystem.get(conf); // pull in dfs settings

     System.out.println("Replication = " + conf.get("dfs.replication"));
     System.out.println("Update file and press <enter>");
     new Scanner(System.in).nextLine();

     conf.reloadConfiguration();
     System.out.println("Now replication = " + conf.get("dfs.replication"));
   }
}

Note from the Javadoc: Values that are added via set methods will 
overlay values read from the resources.

Hope this helps.  Write back if you have more questions.

Thanks,
Jakob Homan
Hadoop at Yahoo!


Arvind Sharma wrote:
> Hi,
> 
> I was wondering if there is way to let Hadoop re-read the config file (hadoop-site.xml) after  making some changes in it.
> 
> I don't want to restart the whole cluster for that.
> 
> I am using Hadoop 0.19.2
> 
> Thanks!
> Arvind
> 
> 
> 
>