You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Venu Gopala Rao (JIRA)" <ji...@apache.org> on 2011/04/22 20:01:06 UTC
[jira] [Created] (HADOOP-7239) The variables in the configuration
files are not replaced with values, when the job is submitted by Job client
The variables in the configuration files are not replaced with values, when the job is submitted by Job client
--------------------------------------------------------------------------------------------------------------
Key: HADOOP-7239
URL: https://issues.apache.org/jira/browse/HADOOP-7239
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 0.21.0, 0.20.2
Environment: Linux + Sun JDK 1.6
Reporter: Venu Gopala Rao
We have a case where we wanted to create the Job names dynamically at run time.Since JobConf is an extension for the Configuration object, we thought we can make use of the variable substitution concept in configuration like below
Job job = new Job(conf, "${mapred.user.name}" + "-job" + new Random().nextInt());
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
job.submit();
We set the required run time variables(in this case mapred.user.name) before calling the job submit. But on the Job tracker side the variables are not replaced correctly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7239) The variables in the configuration
files are not replaced with values, when the job is submitted by Job client
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13023309#comment-13023309 ]
Todd Lipcon commented on HADOOP-7239:
-------------------------------------
Is there some documentation that suggests this would be possible? This is not a bug in my opinion.
> The variables in the configuration files are not replaced with values, when the job is submitted by Job client
> --------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7239
> URL: https://issues.apache.org/jira/browse/HADOOP-7239
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.20.2, 0.21.0
> Environment: Linux + Sun JDK 1.6
> Reporter: Venu Gopala Rao
>
> We have a case where we wanted to create the Job names dynamically at run time.Since JobConf is an extension for the Configuration object, we thought we can make use of the variable substitution concept in configuration like below
> Job job = new Job(conf, "${mapred.user.name}" + "-job" + new Random().nextInt());
> job.setJarByClass(WordCount.class);
> job.setMapperClass(TokenizerMapper.class);
> job.setCombinerClass(IntSumReducer.class);
> job.setReducerClass(IntSumReducer.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
> FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
> FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
> job.submit();
> We set the required run time variables(in this case mapred.user.name) before calling the job submit. But on the Job tracker side the variables are not replaced correctly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7239) The variables in the configuration
files are not replaced with values, when the job is submitted by Job client
Posted by "Venu Gopala Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13023313#comment-13023313 ]
Venu Gopala Rao commented on HADOOP-7239:
-----------------------------------------
I looked at code for the root cause. It is as follows
While writing the job.xml, it is using the Configuration.writeXML(), which is as follows
{code:title=Configuration.java|borderStyle=solid}
Properties properties = getProps();
try {
......
for (Enumeration e = properties.keys(); e.hasMoreElements();) {
String name = (String)e.nextElement();
Object object = properties.get(name);
String value = null;
if (object instanceof String) {
value = (String) object;
}else {
continue;
}
{code}
So here all properties in configuration are retrieved using getProps() which returns a ConcurrentHashMap and written to XML from that. So the variables are not substituted. instead of using *properties.get(name);*
in above snippet we should use *get(name)*, so that all variable substitution happen correctly.
> The variables in the configuration files are not replaced with values, when the job is submitted by Job client
> --------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7239
> URL: https://issues.apache.org/jira/browse/HADOOP-7239
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.20.2, 0.21.0
> Environment: Linux + Sun JDK 1.6
> Reporter: Venu Gopala Rao
>
> We have a case where we wanted to create the Job names dynamically at run time.Since JobConf is an extension for the Configuration object, we thought we can make use of the variable substitution concept in configuration like below
> Job job = new Job(conf, "${mapred.user.name}" + "-job" + new Random().nextInt());
> job.setJarByClass(WordCount.class);
> job.setMapperClass(TokenizerMapper.class);
> job.setCombinerClass(IntSumReducer.class);
> job.setReducerClass(IntSumReducer.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
> FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
> FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
> job.submit();
> We set the required run time variables(in this case mapred.user.name) before calling the job submit. But on the Job tracker side the variables are not replaced correctly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7239) The variables in the configuration
files are not replaced with values, when the job is submitted by Job client
Posted by "Venu Gopala Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13023316#comment-13023316 ]
Venu Gopala Rao commented on HADOOP-7239:
-----------------------------------------
Hi Todd,
But the point here is, after creating the Job in my client application
{noformat} Job job = new Job(conf, "${mapred.user.name}" + "-job" + new Random().nextInt());{noformat}
If I call job.getJobName() it returns me correct name. But if we get the Job Profile from the Job Tracker and check the name/ check the Job tracker UI, it contains the expression.
> The variables in the configuration files are not replaced with values, when the job is submitted by Job client
> --------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7239
> URL: https://issues.apache.org/jira/browse/HADOOP-7239
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.20.2, 0.21.0
> Environment: Linux + Sun JDK 1.6
> Reporter: Venu Gopala Rao
>
> We have a case where we wanted to create the Job names dynamically at run time.Since JobConf is an extension for the Configuration object, we thought we can make use of the variable substitution concept in configuration like below
> Job job = new Job(conf, "${mapred.user.name}" + "-job" + new Random().nextInt());
> job.setJarByClass(WordCount.class);
> job.setMapperClass(TokenizerMapper.class);
> job.setCombinerClass(IntSumReducer.class);
> job.setReducerClass(IntSumReducer.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
> FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
> FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
> job.submit();
> We set the required run time variables(in this case mapred.user.name) before calling the job submit. But on the Job tracker side the variables are not replaced correctly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira