[jira] Resolved: (HADOOP-88) Configuration: separate client config from server config (and from other-server config)

["Owen O'Malley (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-88?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley resolved HADOOP-88.
---------------------------------

    Resolution: Duplicate
      Assignee: Arun C Murthy  (was: Doug Cutting)

This was fixed by HADOOP-785.

> Configuration: separate client config from server config (and from other-server config)
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-88
>                 URL: https://issues.apache.org/jira/browse/HADOOP-88
>             Project: Hadoop
>          Issue Type: Wish
>          Components: conf
>            Reporter: Michel Tourn
>            Assignee: Arun C Murthy
>            Priority: Minor
>
> servers = JobTracker, NameNode, TaskTracker, DataNode
> clients =  runs JobClient (to submit MapReduce jobs), or runs DFSShell (to browse )
> Server machines are administered together.
> So it is OK to have all server config together (esp file paths and network ports).
> This is stored in hadoop-default.xml or hadoop-mycluster.xml
> Client machines:
> there may be as many client machines as there are MapRed developers.
> the temp space for DFS needs to be writable by the active user.
> So it should be possible to select the client temp space directory for the machine and for the user.
> (The global /tmp is not an option as discussed elsewhere: partition may be full)
> Current situation: 
> Both the server and the clients have a copy of the server config: hadoop-default.xml
> But the XML property  "dfs.data.dir" is being used as a LOCAL directory path 
> on both the server machines (Data nodes) and the client machines.
> Effect:
> Exception in thread "main" java.io.IOException: No valid local directories in property: dfs.data.dir
>  at org.apache.hadoop.conf.Configuration.getFile(Configuration.java:286)
>  at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.newBackupFile(DFSClient.java:560)
>  ...
>  at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:267)
> Current Workaround:
> On the client use hadoop-site.xml to override dfs.data.dir
> One proposed solution:
> For the purpose of JobClient operations, use a different property in place of dfs.data.dir.
> (Ex: dfs.client.data.dir) 
> On the client, set this property in hadoop-site.xml so that it will override hadoop-default.xml 
> Another proposed solution:
> Handle the fact that the world is made of a federation of independant Hadoop systems.
> They can talk to each other (as peers) but they are administered separately.
> Each Hadoop system should have its own separate XML config file.
> Clients should be able to specify the Hadoop system they want to talk to.
> An advantage is that clients can then easily sync their local copy of a given Hadoop system config:
>  just pull its config file
> In this view of the world, a Job client is also a kind of independant (serverless) Hadoop system
> In this case the client config file may have its own dfs.data.dir, which is 
> separate from the dfs.data.dir in the server config file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.