Is it possible to change parameters using org.apache.hadoop.conf.Configuration API?

[Jinyeon Lee <ij...@gmail.com>]

I have a running Hadoop/HBase cluster.
When I want to change hadoop parameters without stopping the cluster, can I
use org.apache.hadoop.conf.Configuration API?

I wrote following java source, but it didn't do anything.
-------------------------
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
public class ChangeConf
{
    public static void main(String[] args) throws Exception
    {
        Configuration conf = new Configuration();
        conf.addResource(new
Path("/hadoop-0.18.1/conf/hadoop-default.xml"));
        conf.addResource(new Path("/hadoop-0.18.1/conf/hadoop-site.xml"));
        conf.setStrings("dfs.data.dir",
"/hadoop-data/data.dir","/data4/hadoop-data");
        return;
    }
}
----------------------------

Actually, I know I can use "decommission" to add more directory without
stopping whole cluster.
But I'm very confused "org.apache.hadoop.conf.Configuration" API.

Could anyone please let me know clearly?

Re: Is it possible to change parameters using org.apache.hadoop.conf.Configuration API?

[Jinyeon Lee <ij...@gmail.com>]

Thank you!!!

Re: Is it possible to change parameters using org.apache.hadoop.conf.Configuration API?

[Steve Loughran <st...@apache.org>]

Alex Loddengaard wrote:
> Just to be clear, you want to persist a configuration change to your entire
> cluster without bringing it down, and you're hoping to use the Configuration
> API to do so.  Did I get your question correct?
> 
> I don't know of a way to do this without restarting the cluster, because I'm
> pretty sure Configuration changes will only affect the current job.  Does
> anyone else have a suggestion?
> 
> Alex
> 

Its not doable with the current architecture
  -confs never get reread
  -they aren't persisted

If you did persist them, it would get into a mess if every node in the 
cluster was reading the same shared file from an NFS mount; the last 
node to start up and set a value would set it for everything else. 
Better to have a persistent central configuration management ("CM") 
infrastructure and drive hadoop that way.

Even with CM tooling, the idea of persisting all changes back from the 
apps is generally considered a bad thing. What is preferred is to have a 
gui for managing system state that you can view, rollback, compare 
configuration changes too, as having the app do it means you don't know 
what's going on -and your app can leave your system in a mess, which is 
exactly what CM tools try to prevent.

-steve

Re: Is it possible to change parameters using org.apache.hadoop.conf.Configuration API?

[Alex Loddengaard <al...@cloudera.com>]

Just to be clear, you want to persist a configuration change to your entire
cluster without bringing it down, and you're hoping to use the Configuration
API to do so.  Did I get your question correct?

I don't know of a way to do this without restarting the cluster, because I'm
pretty sure Configuration changes will only affect the current job.  Does
anyone else have a suggestion?

Alex

On Wed, Oct 22, 2008 at 12:44 AM, Jinyeon Lee <ij...@gmail.com> wrote:

> I have a running Hadoop/HBase cluster.
> When I want to change hadoop parameters without stopping the cluster, can I
> use org.apache.hadoop.conf.Configuration API?
>
> I wrote following java source, but it didn't do anything.
> -------------------------
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.Path;
> public class ChangeConf
> {
>    public static void main(String[] args) throws Exception
>    {
>        Configuration conf = new Configuration();
>        conf.addResource(new
> Path("/hadoop-0.18.1/conf/hadoop-default.xml"));
>        conf.addResource(new Path("/hadoop-0.18.1/conf/hadoop-site.xml"));
>        conf.setStrings("dfs.data.dir",
> "/hadoop-data/data.dir","/data4/hadoop-data");
>        return;
>    }
> }
> ----------------------------
>
> Actually, I know I can use "decommission" to add more directory without
> stopping whole cluster.
> But I'm very confused "org.apache.hadoop.conf.Configuration" API.
>
> Could anyone please let me know clearly?
>