You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Suman Sehgal (JIRA)" <ji...@apache.org> on 2009/01/15 09:25:59 UTC
[jira] Created: (HADOOP-5047) Hod should be modified to generate
core-site.xml, mapred-site.xml and hdfs-site.xml
Hod should be modified to generate core-site.xml, mapred-site.xml and hdfs-site.xml
-----------------------------------------------------------------------------------
Key: HADOOP-5047
URL: https://issues.apache.org/jira/browse/HADOOP-5047
Project: Hadoop Core
Issue Type: Bug
Components: contrib/hod
Affects Versions: 0.20.0
Reporter: Suman Sehgal
Priority: Critical
Hod is currently generating hadoop-site.xml but as per the modifications done in hadoop (ref: JIRA-4631), hod should also be modified to be in synch with this change. It should also generate core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml from 0.20.0 onwards.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-5047) Hod should be modified to generate
core-site.xml, mapred-site.xml and hdfs-site.xml
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala reassigned HADOOP-5047:
----------------------------------------
Assignee: Peeyush Bishnoi
> Hod should be modified to generate core-site.xml, mapred-site.xml and hdfs-site.xml
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-5047
> URL: https://issues.apache.org/jira/browse/HADOOP-5047
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.20.0
> Reporter: Suman Sehgal
> Assignee: Peeyush Bishnoi
>
> Hod is currently generating hadoop-site.xml but as per the modifications done in hadoop (ref: JIRA-4631), hod should also be modified to be in synch with this change. It should also generate core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml from 0.20.0 onwards.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5047) Hod should be modified to generate
core-site.xml, mapred-site.xml and hdfs-site.xml
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-5047:
-------------------------------------
Priority: Major (was: Critical)
Downgrading the priority. This is backwards compatible, save for a warning message. Fixing this is also complicated.
> Hod should be modified to generate core-site.xml, mapred-site.xml and hdfs-site.xml
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-5047
> URL: https://issues.apache.org/jira/browse/HADOOP-5047
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.20.0
> Reporter: Suman Sehgal
>
> Hod is currently generating hadoop-site.xml but as per the modifications done in hadoop (ref: JIRA-4631), hod should also be modified to be in synch with this change. It should also generate core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml from 0.20.0 onwards.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5047) Hod should be modified to generate
core-site.xml, mapred-site.xml and hdfs-site.xml
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667688#action_12667688 ]
Hemanth Yamijala commented on HADOOP-5047:
------------------------------------------
HOD generates hadoop-site.xml on the client, and on the cluster nodes where it brings up the daemons. There are two types of configuration items it writes into the hadoop-site.xml:
- Entries it generates values for. E.g. the jobtracker URL
- Entries it is given via the hod configuration file. These are user specified entries.
It is easy to split the first category. Since the list is a hardcoded set of configuration items, we can hardcode the decision of which file (core, mapred, or dfs) this item belongs to and generate the entries into the appropriate file.
For the user specified entries, the decision cannot be taken by HOD in a maintainable fashion. Further, it causes more of a problem because of interface and usability issues.
Currently, HOD has a very convenient command line syntax to specify the list of configuration items, as follows:
-Cname=value - client params
-Mname=value - Mapred server params
-Fname=value - Final Mapred server params
-Hname=value - HDFS server params
-Sname=value - Final HDFS server params.
One approach to splitting the configuration is to split each of the categories into 3 as core, mapred and hdfs. However, I feel this might be too cumbersome for users to express.
So, a proposal is to introduce this syntax:
{code}
-<Option Letter>[category:]<name>=<value>, where category is 'core' or 'mapred' or 'hdfs'.
{code}
For e.g.
{code}
-Mmapred:mapred.reduce.tasks=4 or -Ccore:dfs.client.buffer.size=1000000
{code}
Based on the category, the configuration item can be generated to the appropriate configuration file.
For backwards compatibility, we can say that if category is not specified, the configuration item would be generated in the hadoop-site.xml file. When hadoop-site.xml is completely removed from hadoop, HOD can also stop supporting options where the category is not specified.
Comments ?
> Hod should be modified to generate core-site.xml, mapred-site.xml and hdfs-site.xml
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-5047
> URL: https://issues.apache.org/jira/browse/HADOOP-5047
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.20.0
> Reporter: Suman Sehgal
> Assignee: Peeyush Bishnoi
>
> Hod is currently generating hadoop-site.xml but as per the modifications done in hadoop (ref: JIRA-4631), hod should also be modified to be in synch with this change. It should also generate core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml from 0.20.0 onwards.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5047) Hod should be modified to generate
core-site.xml, mapred-site.xml and hdfs-site.xml
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664046#action_12664046 ]
Vinod K V commented on HADOOP-5047:
-----------------------------------
HADOOP-4631 is committed to and supports the old hadoop-site.xml in 0.20, so HOD in 0.20 continues to work. We actually need the fix only for trunk and above, where hadoop-site.xml will no longer be supported.
> Hod should be modified to generate core-site.xml, mapred-site.xml and hdfs-site.xml
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-5047
> URL: https://issues.apache.org/jira/browse/HADOOP-5047
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hod
> Affects Versions: 0.20.0
> Reporter: Suman Sehgal
> Priority: Critical
>
> Hod is currently generating hadoop-site.xml but as per the modifications done in hadoop (ref: JIRA-4631), hod should also be modified to be in synch with this change. It should also generate core-site.xml, mapred-site.xml and hdfs-site.xml instead of hadoop-site.xml from 0.20.0 onwards.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.