You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Peng, Wei" <We...@xerox.com> on 2010/12/17 07:58:14 UTC
Please help with hadoop configuration parameter set and get
Hi,
I am a newbie of hadoop.
Today I was struggling with a hadoop problem for several hours.
I initialize a parameter by setting job configuration in main.
E.g. Configuration con = new Configuration();
con.set("test", "1");
Job job = new Job(con);
Then in the mapper class, I want to set "test" to "2". I did it by
context.getConfiguration().set("test","2");
Finally in the main method, after the job is finished, I check the
"test" again by
job.getConfiguration().get("test");
However, the value of "test" is still "1".
The reason why I want to change the parameter inside Mapper class is
that I want to determine when to stop an iteration in the main method.
For example, for doing breadth-first search, when there is no new nodes
are added for further expansion, the searching iteration should stop.
Your help will be deeply appreciated. Thank you
Wei
Re: Please help with hadoop configuration parameter set and get
Posted by James Seigel <ja...@tynt.com>.
If you're wondering where to get a counter from, I'll point you to "context"
Sent from my mobile. Please excuse the typos.
On 2010-12-17, at 7:39 AM, Ted Yu <yu...@gmail.com> wrote:
> You can use hadoop counter to pass this information.
> This way, you see the counters in job report.
>
> On Thu, Dec 16, 2010 at 10:58 PM, Peng, Wei <We...@xerox.com> wrote:
>
>> Hi,
>>
>>
>>
>> I am a newbie of hadoop.
>>
>> Today I was struggling with a hadoop problem for several hours.
>>
>>
>>
>> I initialize a parameter by setting job configuration in main.
>>
>> E.g. Configuration con = new Configuration();
>>
>> con.set("test", "1");
>>
>> Job job = new Job(con);
>>
>>
>>
>> Then in the mapper class, I want to set "test" to "2". I did it by
>>
>> context.getConfiguration().set("test","2");
>>
>>
>>
>> Finally in the main method, after the job is finished, I check the
>> "test" again by
>>
>> job.getConfiguration().get("test");
>>
>>
>>
>> However, the value of "test" is still "1".
>>
>>
>>
>> The reason why I want to change the parameter inside Mapper class is
>> that I want to determine when to stop an iteration in the main method.
>> For example, for doing breadth-first search, when there is no new nodes
>> are added for further expansion, the searching iteration should stop.
>>
>>
>>
>> Your help will be deeply appreciated. Thank you
>>
>>
>>
>> Wei
>>
>>
Re: Please help with hadoop configuration parameter set and get
Posted by Ted Yu <yu...@gmail.com>.
You can use hadoop counter to pass this information.
This way, you see the counters in job report.
On Thu, Dec 16, 2010 at 10:58 PM, Peng, Wei <We...@xerox.com> wrote:
> Hi,
>
>
>
> I am a newbie of hadoop.
>
> Today I was struggling with a hadoop problem for several hours.
>
>
>
> I initialize a parameter by setting job configuration in main.
>
> E.g. Configuration con = new Configuration();
>
> con.set("test", "1");
>
> Job job = new Job(con);
>
>
>
> Then in the mapper class, I want to set "test" to "2". I did it by
>
> context.getConfiguration().set("test","2");
>
>
>
> Finally in the main method, after the job is finished, I check the
> "test" again by
>
> job.getConfiguration().get("test");
>
>
>
> However, the value of "test" is still "1".
>
>
>
> The reason why I want to change the parameter inside Mapper class is
> that I want to determine when to stop an iteration in the main method.
> For example, for doing breadth-first search, when there is no new nodes
> are added for further expansion, the searching iteration should stop.
>
>
>
> Your help will be deeply appreciated. Thank you
>
>
>
> Wei
>
>
Re: Please help with hadoop configuration parameter set and get
Posted by Aman <am...@hotmail.com>.
Pei
You cannot change configuration in Mapper method and see it in main method.
There are 2 ways to do this
1. Have your mapper store data in HDFS
2. Use counters
I recommend using the 2nd approach. What you can do is set a counter in the
Mapper, something like
context.getCounter("MyCounter").increment(1);
and then extract this counter in the main method. I had this class that
emits out all counters which I then load into a DB. You can tweak this class
to get the value of the counter you want.
public class JobMetadata {
Job job = null;
public JobMetadata(Job job) {
this.job = job;
}
public String getJobMetadata() throws IOException {
/**
* Get Job Name, Id and URL
*/
String jobName = job.getJobName();
String trackingUrl = job.getTrackingURL();
/**
* Get Job Counter
*/
Counters jobCounters = job.getCounters();
int totalCounters = jobCounters.countCounters();
CounterGroup counterGroup = null;
Counter counter = null;
String counterGroupName = null;
String counterName = null;
long counterValue = 0;
Iterator itGroups = jobCounters.iterator();
StringBuffer sb = new StringBuffer();
while (itGroups.hasNext()) {
counterGroup = (CounterGroup) itGroups.next();
counterGroupName = counterGroup.getDisplayName();
Iterator itCounter = counterGroup.iterator();
while (itCounter.hasNext()) {
counter = (Counter) itCounter.next();
counterName = counter.getDisplayName();
counterValue = counter.getValue();
sb.append(jobName + "\t" + trackingUrl + "\t" +
counterGroupName
+ "\t" + counterName + "\t" + counterValue + "\n");
}
}
return sb.toString();
}
}
Peng, Wei wrote:
>
> Hi,
>
>
>
> I am a newbie of hadoop.
>
> Today I was struggling with a hadoop problem for several hours.
>
>
>
> I initialize a parameter by setting job configuration in main.
>
> E.g. Configuration con = new Configuration();
>
> con.set("test", "1");
>
> Job job = new Job(con);
>
>
>
> Then in the mapper class, I want to set "test" to "2". I did it by
>
> context.getConfiguration().set("test","2");
>
>
>
> Finally in the main method, after the job is finished, I check the
> "test" again by
>
> job.getConfiguration().get("test");
>
>
>
> However, the value of "test" is still "1".
>
>
>
> The reason why I want to change the parameter inside Mapper class is
> that I want to determine when to stop an iteration in the main method.
> For example, for doing breadth-first search, when there is no new nodes
> are added for further expansion, the searching iteration should stop.
>
>
>
> Your help will be deeply appreciated. Thank you
>
>
>
> Wei
>
>
>
--
View this message in context: http://lucene.472066.n3.nabble.com/Please-help-with-hadoop-configuration-parameter-set-and-get-tp2103303p2107316.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
RE: Please help with hadoop configuration parameter set and get
Posted by "Peng, Wei" <We...@xerox.com>.
Thank you for your kind reply.
I am running hadoop in the distributed mode.
It might be out of the topic.
For a lot of algorithms (especially data mining/machine learning
algorithms), we do need many iterations. However, hadoop needs to save
all the intermediate step output as the input of the next iteration. I
doubt it would be efficient.
Is hadoop the right choice for data mining/machine learning algorithm?
Can hadoop go with gpu computing?
Wei
-----Original Message-----
From: Arindam Khaled [mailto:akhaled@utdallas.edu]
Sent: Friday, December 17, 2010 1:14 PM
To: common-user@hadoop.apache.org
Subject: Re: Please help with hadoop configuration parameter set and get
There's no guarantee that it might work for a distributed environment
as a couple of developers suggested. I didn't have the time to play
with it in the distributed mode, rather, executed it in a standalone
environment.
Arindam
On Dec 17, 2010, at 12:07 PM, Arindam Khaled wrote:
> Wei,
>
> I implemented one of the algorithms, BFIDA*, included the paper "Out-
> of-Core Parallel Frontier Search with MapReduce" by Alexander
> Reinefeld and Thorsten Sch{\"u}tt. FYI, they implemented BFS for 15-
> puzzle using map-reduce and MPI, not Hadoop.
>
> For brevity, I am not including the whole source code. Here is my
> pseudo-code:
>
> public class standAloneIDA {
>
> static boolean solved = false;
> public static class BFIDAMapClass extends Mapper<Object, Text,
> Text, IntWritable>
> {
> public void map(Object key, Text value, Context context)
> throws IOException, InterruptedException
> {
> String line = value.toString();
> while(there can be more new moves)
> {
> emit("the new board", "move"); //key,
value pair
> }
> }
>
> }
>
> public static class BFIDAReducer
> extends Reducer<Text,IntWritable,Text,IntWritable> {
> private IntWritable result = new IntWritable();
>
> public void reduce(Text key, Iterable<IntWritable>
values,
> Context context
> ) throws IOException, InterruptedException {
> String line = "";
>
> ArrayList<Integer> previousMoves = new
ArrayList<Integer> ();
>
> for(IntWritable val: values)
> {
> String valInt = val.toString();
> int[] moves = stringToArrayInt(valInt);
>
> for(int bit: moves)
> {
> if(!previousMoves.contains(bit))
> previousMoves.add(bit);
> }
> }
>
> for(int val: previousMoves)
> {
> line = line + Integer.toString(val);
> }
>
> if (isSolved(key.toString(), size))
> {
> solved = true;
> }
>
> result.set(Integer.parseInt(line));
> context.write(key, result);
> }
> }
>
> public static void main(String[] args) throws Exception {
> // logger.isDebugEnabled();
> Configuration conf = new Configuration();
> String[] otherArgs = new GenericOptionsParser(conf,
> args).getRemainingArgs();
> if (otherArgs.length != 2) {
> System.err.println("Usage: testUnit2 <in>
<out>");
> System.exit(2);
> }
>
>
>
> while(!solved)
> {
> set up the file system
> include the input and output filenames
> Job job = new Job(conf, "some name");
> job.setJarByClass(standAloneIDA.class);
> job.setMapperClass(BFIDAMapClass.class);
> job.setReducerClass(BFIDAReducer.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(IntWritable.class);
>
> job.waitForCompletion(true);
> }
> }
> }
>
> Please excuse me if there are missing braces. There might be more
> efficient ways to setup the jobs and file system. I didn't have much
> time -- so, I ended up with something that worked for me then. Let
> me know if you have more questions.
>
> Kind regards,
>
> Arindam Khaled
>
>
>
>
>
> On Dec 17, 2010, at 10:31 AM, Peng, Wei wrote:
>
>> Arindam, how to set this global static Boolean variable?
>> I have tried to do something similarly yesterday in the following:
>> Public class BFSearch
>> {
>> Private static boolean expansion;
>> Public static class MapperClass {if no nodes expansion = false;}
>> Public static class ReducerClass
>> Public static void main {expansion = true; run job;
>> print(expansion)}
>> }
>> In this case, expansion is still true.
>> I will look at hadoop counter and report back here later.
>>
>> Thank you for all your help
>> Wei
>>
>> -----Original Message-----
>> From: Arindam Khaled [mailto:akhaled@utdallas.edu]
>> Sent: Friday, December 17, 2010 10:35 AM
>> To: common-user@hadoop.apache.org
>> Subject: Re: Please help with hadoop configuration parameter set
>> and get
>>
>> I did something like this using a global static boolean variable
>> (flag) while I was implementing breadth first IDA*. In my case, I set
>> the flag to something else if a solution was found, which was
>> examined
>> in the reducer.
>>
>> I guess in your case, since you know that if the mappers don't
>> produce
>> anything the reducers won't have anything as input, if I am not
>> wrong.
>>
>> And I had chaining map-reduce jobs (
>> http://developer.yahoo.com/hadoop/tutorial/module4.html
>> ) running until a solution was found.
>>
>>
>> Kind regards,
>>
>> Arindam Khaled
>>
>>
>>
>>
>>
>> On Dec 17, 2010, at 12:58 AM, Peng, Wei wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>> I am a newbie of hadoop.
>>>
>>> Today I was struggling with a hadoop problem for several hours.
>>>
>>>
>>>
>>> I initialize a parameter by setting job configuration in main.
>>>
>>> E.g. Configuration con = new Configuration();
>>>
>>> con.set("test", "1");
>>>
>>> Job job = new Job(con);
>>>
>>>
>>>
>>> Then in the mapper class, I want to set "test" to "2". I did it by
>>>
>>> context.getConfiguration().set("test","2");
>>>
>>>
>>>
>>> Finally in the main method, after the job is finished, I check the
>>> "test" again by
>>>
>>> job.getConfiguration().get("test");
>>>
>>>
>>>
>>> However, the value of "test" is still "1".
>>>
>>>
>>>
>>> The reason why I want to change the parameter inside Mapper class is
>>> that I want to determine when to stop an iteration in the main
>>> method.
>>> For example, for doing breadth-first search, when there is no new
>>> nodes
>>> are added for further expansion, the searching iteration should
>>> stop.
>>>
>>>
>>>
>>> Your help will be deeply appreciated. Thank you
>>>
>>>
>>>
>>> Wei
>>>
>>
>
RE: Please help with hadoop configuration parameter set and get
Posted by "Peng, Wei" <We...@xerox.com>.
If I wrote "return" in reducer class like
public class reducerClass{
public void reduce{
if(counter <= 0)
return;
}
}
The program has to go through all the mapper keys in order to return.
I even want to stop all mapreduce process before the mapper class does the sorting and shuffling.
Is there anything that I can call to exit mapreduce and return back to the main method?
Thanks
Wei
-----Original Message-----
From: Peng, Wei [mailto:Wei.Peng@xerox.com]
Sent: Fri 12/17/2010 2:14 PM
To: common-user@hadoop.apache.org
Subject: RE: Please help with hadoop configuration parameter set and get
My question is how to exit hadoop reducer.
Can I just write "return;" in reducer?
Wei
-----Original Message-----
From: Aman [mailto:aman_doon@hotmail.com]
Sent: Fri 12/17/2010 1:24 PM
To: hadoop-user@lucene.apache.org
Subject: RE: Please help with hadoop configuration parameter set and get
Set the counter in the mapper and in the reducer code, check for this
counter. If counter is set then run the reducer if not then exit without
doing anything.
Once there is no new node is found, I would like to exit mapper or reducer
process.
For example, if no new nodes are found in the mapper class, the setup of
reducer will stop reducer work (or the cleanup of mapper will stop the
reducer work).
--
View this message in context: http://lucene.472066.n3.nabble.com/Please-help-with-hadoop-configuration-parameter-set-and-get-tp2103303p2107388.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
RE: Please help with hadoop configuration parameter set and get
Posted by "Peng, Wei" <We...@xerox.com>.
My question is how to exit hadoop reducer.
Can I just write "return;" in reducer?
Wei
-----Original Message-----
From: Aman [mailto:aman_doon@hotmail.com]
Sent: Fri 12/17/2010 1:24 PM
To: hadoop-user@lucene.apache.org
Subject: RE: Please help with hadoop configuration parameter set and get
Set the counter in the mapper and in the reducer code, check for this
counter. If counter is set then run the reducer if not then exit without
doing anything.
Once there is no new node is found, I would like to exit mapper or reducer
process.
For example, if no new nodes are found in the mapper class, the setup of
reducer will stop reducer work (or the cleanup of mapper will stop the
reducer work).
--
View this message in context: http://lucene.472066.n3.nabble.com/Please-help-with-hadoop-configuration-parameter-set-and-get-tp2103303p2107388.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
RE: Please help with hadoop configuration parameter set and get
Posted by Aman <am...@hotmail.com>.
Set the counter in the mapper and in the reducer code, check for this
counter. If counter is set then run the reducer if not then exit without
doing anything.
Once there is no new node is found, I would like to exit mapper or reducer
process.
For example, if no new nodes are found in the mapper class, the setup of
reducer will stop reducer work (or the cleanup of mapper will stop the
reducer work).
--
View this message in context: http://lucene.472066.n3.nabble.com/Please-help-with-hadoop-configuration-parameter-set-and-get-tp2103303p2107388.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
RE: Please help with hadoop configuration parameter set and get
Posted by "Peng, Wei" <We...@xerox.com>.
I just tried hadoop counters. It works great now!!
I have one more question.
Once there is no new node is found, I would like to exit mapper or reducer process.
For example, if no new nodes are found in the mapper class, the setup of reducer will stop reducer work (or the cleanup of mapper will stop the reducer work).
Wei
-----Original Message-----
From: Arindam Khaled [mailto:akhaled@utdallas.edu]
Sent: Fri 12/17/2010 10:13 AM
To: common-user@hadoop.apache.org
Subject: Re: Please help with hadoop configuration parameter set and get
There's no guarantee that it might work for a distributed environment
as a couple of developers suggested. I didn't have the time to play
with it in the distributed mode, rather, executed it in a standalone
environment.
Arindam
On Dec 17, 2010, at 12:07 PM, Arindam Khaled wrote:
> Wei,
>
> I implemented one of the algorithms, BFIDA*, included the paper "Out-
> of-Core Parallel Frontier Search with MapReduce" by Alexander
> Reinefeld and Thorsten Sch{\"u}tt. FYI, they implemented BFS for 15-
> puzzle using map-reduce and MPI, not Hadoop.
>
> For brevity, I am not including the whole source code. Here is my
> pseudo-code:
>
> public class standAloneIDA {
>
> static boolean solved = false;
> public static class BFIDAMapClass extends Mapper<Object, Text,
> Text, IntWritable>
> {
> public void map(Object key, Text value, Context context)
> throws IOException, InterruptedException
> {
> String line = value.toString();
> while(there can be more new moves)
> {
> emit("the new board", "move"); //key, value pair
> }
> }
>
> }
>
> public static class BFIDAReducer
> extends Reducer<Text,IntWritable,Text,IntWritable> {
> private IntWritable result = new IntWritable();
>
> public void reduce(Text key, Iterable<IntWritable> values,
> Context context
> ) throws IOException, InterruptedException {
> String line = "";
>
> ArrayList<Integer> previousMoves = new ArrayList<Integer> ();
>
> for(IntWritable val: values)
> {
> String valInt = val.toString();
> int[] moves = stringToArrayInt(valInt);
>
> for(int bit: moves)
> {
> if(!previousMoves.contains(bit))
> previousMoves.add(bit);
> }
> }
>
> for(int val: previousMoves)
> {
> line = line + Integer.toString(val);
> }
>
> if (isSolved(key.toString(), size))
> {
> solved = true;
> }
>
> result.set(Integer.parseInt(line));
> context.write(key, result);
> }
> }
>
> public static void main(String[] args) throws Exception {
> // logger.isDebugEnabled();
> Configuration conf = new Configuration();
> String[] otherArgs = new GenericOptionsParser(conf,
> args).getRemainingArgs();
> if (otherArgs.length != 2) {
> System.err.println("Usage: testUnit2 <in> <out>");
> System.exit(2);
> }
>
>
>
> while(!solved)
> {
> set up the file system
> include the input and output filenames
> Job job = new Job(conf, "some name");
> job.setJarByClass(standAloneIDA.class);
> job.setMapperClass(BFIDAMapClass.class);
> job.setReducerClass(BFIDAReducer.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(IntWritable.class);
>
> job.waitForCompletion(true);
> }
> }
> }
>
> Please excuse me if there are missing braces. There might be more
> efficient ways to setup the jobs and file system. I didn't have much
> time -- so, I ended up with something that worked for me then. Let
> me know if you have more questions.
>
> Kind regards,
>
> Arindam Khaled
>
>
>
>
>
> On Dec 17, 2010, at 10:31 AM, Peng, Wei wrote:
>
>> Arindam, how to set this global static Boolean variable?
>> I have tried to do something similarly yesterday in the following:
>> Public class BFSearch
>> {
>> Private static boolean expansion;
>> Public static class MapperClass {if no nodes expansion = false;}
>> Public static class ReducerClass
>> Public static void main {expansion = true; run job;
>> print(expansion)}
>> }
>> In this case, expansion is still true.
>> I will look at hadoop counter and report back here later.
>>
>> Thank you for all your help
>> Wei
>>
>> -----Original Message-----
>> From: Arindam Khaled [mailto:akhaled@utdallas.edu]
>> Sent: Friday, December 17, 2010 10:35 AM
>> To: common-user@hadoop.apache.org
>> Subject: Re: Please help with hadoop configuration parameter set
>> and get
>>
>> I did something like this using a global static boolean variable
>> (flag) while I was implementing breadth first IDA*. In my case, I set
>> the flag to something else if a solution was found, which was
>> examined
>> in the reducer.
>>
>> I guess in your case, since you know that if the mappers don't
>> produce
>> anything the reducers won't have anything as input, if I am not
>> wrong.
>>
>> And I had chaining map-reduce jobs (
>> http://developer.yahoo.com/hadoop/tutorial/module4.html
>> ) running until a solution was found.
>>
>>
>> Kind regards,
>>
>> Arindam Khaled
>>
>>
>>
>>
>>
>> On Dec 17, 2010, at 12:58 AM, Peng, Wei wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>> I am a newbie of hadoop.
>>>
>>> Today I was struggling with a hadoop problem for several hours.
>>>
>>>
>>>
>>> I initialize a parameter by setting job configuration in main.
>>>
>>> E.g. Configuration con = new Configuration();
>>>
>>> con.set("test", "1");
>>>
>>> Job job = new Job(con);
>>>
>>>
>>>
>>> Then in the mapper class, I want to set "test" to "2". I did it by
>>>
>>> context.getConfiguration().set("test","2");
>>>
>>>
>>>
>>> Finally in the main method, after the job is finished, I check the
>>> "test" again by
>>>
>>> job.getConfiguration().get("test");
>>>
>>>
>>>
>>> However, the value of "test" is still "1".
>>>
>>>
>>>
>>> The reason why I want to change the parameter inside Mapper class is
>>> that I want to determine when to stop an iteration in the main
>>> method.
>>> For example, for doing breadth-first search, when there is no new
>>> nodes
>>> are added for further expansion, the searching iteration should
>>> stop.
>>>
>>>
>>>
>>> Your help will be deeply appreciated. Thank you
>>>
>>>
>>>
>>> Wei
>>>
>>
>
Re: Please help with hadoop configuration parameter set and get
Posted by Arindam Khaled <ak...@utdallas.edu>.
There's no guarantee that it might work for a distributed environment
as a couple of developers suggested. I didn't have the time to play
with it in the distributed mode, rather, executed it in a standalone
environment.
Arindam
On Dec 17, 2010, at 12:07 PM, Arindam Khaled wrote:
> Wei,
>
> I implemented one of the algorithms, BFIDA*, included the paper "Out-
> of-Core Parallel Frontier Search with MapReduce" by Alexander
> Reinefeld and Thorsten Sch{\"u}tt. FYI, they implemented BFS for 15-
> puzzle using map-reduce and MPI, not Hadoop.
>
> For brevity, I am not including the whole source code. Here is my
> pseudo-code:
>
> public class standAloneIDA {
>
> static boolean solved = false;
> public static class BFIDAMapClass extends Mapper<Object, Text,
> Text, IntWritable>
> {
> public void map(Object key, Text value, Context context)
> throws IOException, InterruptedException
> {
> String line = value.toString();
> while(there can be more new moves)
> {
> emit("the new board", "move"); //key, value pair
> }
> }
>
> }
>
> public static class BFIDAReducer
> extends Reducer<Text,IntWritable,Text,IntWritable> {
> private IntWritable result = new IntWritable();
>
> public void reduce(Text key, Iterable<IntWritable> values,
> Context context
> ) throws IOException, InterruptedException {
> String line = "";
>
> ArrayList<Integer> previousMoves = new ArrayList<Integer> ();
>
> for(IntWritable val: values)
> {
> String valInt = val.toString();
> int[] moves = stringToArrayInt(valInt);
>
> for(int bit: moves)
> {
> if(!previousMoves.contains(bit))
> previousMoves.add(bit);
> }
> }
>
> for(int val: previousMoves)
> {
> line = line + Integer.toString(val);
> }
>
> if (isSolved(key.toString(), size))
> {
> solved = true;
> }
>
> result.set(Integer.parseInt(line));
> context.write(key, result);
> }
> }
>
> public static void main(String[] args) throws Exception {
> // logger.isDebugEnabled();
> Configuration conf = new Configuration();
> String[] otherArgs = new GenericOptionsParser(conf,
> args).getRemainingArgs();
> if (otherArgs.length != 2) {
> System.err.println("Usage: testUnit2 <in> <out>");
> System.exit(2);
> }
>
>
>
> while(!solved)
> {
> set up the file system
> include the input and output filenames
> Job job = new Job(conf, "some name");
> job.setJarByClass(standAloneIDA.class);
> job.setMapperClass(BFIDAMapClass.class);
> job.setReducerClass(BFIDAReducer.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(IntWritable.class);
>
> job.waitForCompletion(true);
> }
> }
> }
>
> Please excuse me if there are missing braces. There might be more
> efficient ways to setup the jobs and file system. I didn't have much
> time -- so, I ended up with something that worked for me then. Let
> me know if you have more questions.
>
> Kind regards,
>
> Arindam Khaled
>
>
>
>
>
> On Dec 17, 2010, at 10:31 AM, Peng, Wei wrote:
>
>> Arindam, how to set this global static Boolean variable?
>> I have tried to do something similarly yesterday in the following:
>> Public class BFSearch
>> {
>> Private static boolean expansion;
>> Public static class MapperClass {if no nodes expansion = false;}
>> Public static class ReducerClass
>> Public static void main {expansion = true; run job;
>> print(expansion)}
>> }
>> In this case, expansion is still true.
>> I will look at hadoop counter and report back here later.
>>
>> Thank you for all your help
>> Wei
>>
>> -----Original Message-----
>> From: Arindam Khaled [mailto:akhaled@utdallas.edu]
>> Sent: Friday, December 17, 2010 10:35 AM
>> To: common-user@hadoop.apache.org
>> Subject: Re: Please help with hadoop configuration parameter set
>> and get
>>
>> I did something like this using a global static boolean variable
>> (flag) while I was implementing breadth first IDA*. In my case, I set
>> the flag to something else if a solution was found, which was
>> examined
>> in the reducer.
>>
>> I guess in your case, since you know that if the mappers don't
>> produce
>> anything the reducers won't have anything as input, if I am not
>> wrong.
>>
>> And I had chaining map-reduce jobs (
>> http://developer.yahoo.com/hadoop/tutorial/module4.html
>> ) running until a solution was found.
>>
>>
>> Kind regards,
>>
>> Arindam Khaled
>>
>>
>>
>>
>>
>> On Dec 17, 2010, at 12:58 AM, Peng, Wei wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>> I am a newbie of hadoop.
>>>
>>> Today I was struggling with a hadoop problem for several hours.
>>>
>>>
>>>
>>> I initialize a parameter by setting job configuration in main.
>>>
>>> E.g. Configuration con = new Configuration();
>>>
>>> con.set("test", "1");
>>>
>>> Job job = new Job(con);
>>>
>>>
>>>
>>> Then in the mapper class, I want to set "test" to "2". I did it by
>>>
>>> context.getConfiguration().set("test","2");
>>>
>>>
>>>
>>> Finally in the main method, after the job is finished, I check the
>>> "test" again by
>>>
>>> job.getConfiguration().get("test");
>>>
>>>
>>>
>>> However, the value of "test" is still "1".
>>>
>>>
>>>
>>> The reason why I want to change the parameter inside Mapper class is
>>> that I want to determine when to stop an iteration in the main
>>> method.
>>> For example, for doing breadth-first search, when there is no new
>>> nodes
>>> are added for further expansion, the searching iteration should
>>> stop.
>>>
>>>
>>>
>>> Your help will be deeply appreciated. Thank you
>>>
>>>
>>>
>>> Wei
>>>
>>
>
Re: Please help with hadoop configuration parameter set and get
Posted by Arindam Khaled <ak...@utdallas.edu>.
Wei,
I implemented one of the algorithms, BFIDA*, included the paper "Out-
of-Core Parallel Frontier Search with MapReduce" by Alexander
Reinefeld and Thorsten Sch{\"u}tt. FYI, they implemented BFS for 15-
puzzle using map-reduce and MPI, not Hadoop.
For brevity, I am not including the whole source code. Here is my
pseudo-code:
public class standAloneIDA {
static boolean solved = false;
public static class BFIDAMapClass extends Mapper<Object, Text, Text,
IntWritable>
{
public void map(Object key, Text value, Context context)
throws IOException, InterruptedException
{
String line = value.toString();
while(there can be more new moves)
{
emit("the new board", "move"); //key, value pair
}
}
}
public static class BFIDAReducer
extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();
public void reduce(Text key, Iterable<IntWritable> values,
Context context
) throws IOException, InterruptedException {
String line = "";
ArrayList<Integer> previousMoves = new ArrayList<Integer> ();
for(IntWritable val: values)
{
String valInt = val.toString();
int[] moves = stringToArrayInt(valInt);
for(int bit: moves)
{
if(!previousMoves.contains(bit))
previousMoves.add(bit);
}
}
for(int val: previousMoves)
{
line = line + Integer.toString(val);
}
if (isSolved(key.toString(), size))
{
solved = true;
}
result.set(Integer.parseInt(line));
context.write(key, result);
}
}
public static void main(String[] args) throws Exception {
// logger.isDebugEnabled();
Configuration conf = new Configuration();
String[] otherArgs = new GenericOptionsParser(conf,
args).getRemainingArgs();
if (otherArgs.length != 2) {
System.err.println("Usage: testUnit2 <in> <out>");
System.exit(2);
}
while(!solved)
{
set up the file system
include the input and output filenames
Job job = new Job(conf, "some name");
job.setJarByClass(standAloneIDA.class);
job.setMapperClass(BFIDAMapClass.class);
job.setReducerClass(BFIDAReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.waitForCompletion(true);
}
}
}
Please excuse me if there are missing braces. There might be more
efficient ways to setup the jobs and file system. I didn't have much
time -- so, I ended up with something that worked for me then. Let me
know if you have more questions.
Kind regards,
Arindam Khaled
On Dec 17, 2010, at 10:31 AM, Peng, Wei wrote:
> Arindam, how to set this global static Boolean variable?
> I have tried to do something similarly yesterday in the following:
> Public class BFSearch
> {
> Private static boolean expansion;
> Public static class MapperClass {if no nodes expansion = false;}
> Public static class ReducerClass
> Public static void main {expansion = true; run job;
> print(expansion)}
> }
> In this case, expansion is still true.
> I will look at hadoop counter and report back here later.
>
> Thank you for all your help
> Wei
>
> -----Original Message-----
> From: Arindam Khaled [mailto:akhaled@utdallas.edu]
> Sent: Friday, December 17, 2010 10:35 AM
> To: common-user@hadoop.apache.org
> Subject: Re: Please help with hadoop configuration parameter set and
> get
>
> I did something like this using a global static boolean variable
> (flag) while I was implementing breadth first IDA*. In my case, I set
> the flag to something else if a solution was found, which was examined
> in the reducer.
>
> I guess in your case, since you know that if the mappers don't produce
> anything the reducers won't have anything as input, if I am not wrong.
>
> And I had chaining map-reduce jobs (
> http://developer.yahoo.com/hadoop/tutorial/module4.html
> ) running until a solution was found.
>
>
> Kind regards,
>
> Arindam Khaled
>
>
>
>
>
> On Dec 17, 2010, at 12:58 AM, Peng, Wei wrote:
>
>> Hi,
>>
>>
>>
>> I am a newbie of hadoop.
>>
>> Today I was struggling with a hadoop problem for several hours.
>>
>>
>>
>> I initialize a parameter by setting job configuration in main.
>>
>> E.g. Configuration con = new Configuration();
>>
>> con.set("test", "1");
>>
>> Job job = new Job(con);
>>
>>
>>
>> Then in the mapper class, I want to set "test" to "2". I did it by
>>
>> context.getConfiguration().set("test","2");
>>
>>
>>
>> Finally in the main method, after the job is finished, I check the
>> "test" again by
>>
>> job.getConfiguration().get("test");
>>
>>
>>
>> However, the value of "test" is still "1".
>>
>>
>>
>> The reason why I want to change the parameter inside Mapper class is
>> that I want to determine when to stop an iteration in the main
>> method.
>> For example, for doing breadth-first search, when there is no new
>> nodes
>> are added for further expansion, the searching iteration should stop.
>>
>>
>>
>> Your help will be deeply appreciated. Thank you
>>
>>
>>
>> Wei
>>
>
Re: Please help with hadoop configuration parameter set and get
Posted by James Seigel <ja...@tynt.com>.
Exactly. It is hard to assume anything about order or coordination between maps or reducers. You should try and design with “sloppy” coordination strategies.
James.
On 2010-12-17, at 10:32 AM, Ted Dunning wrote:
> Statics won't work the way you might think because different mappers and
> different reducers are all running in different JVM's. It might work in
> local mode, but don't kid yourself about it working in a distributed mode.
> It won't.
>
> On Fri, Dec 17, 2010 at 8:31 AM, Peng, Wei <We...@xerox.com> wrote:
>
>> Arindam, how to set this global static Boolean variable?
>> I have tried to do something similarly yesterday in the following:
>> Public class BFSearch
>> {
>> Private static boolean expansion;
>> Public static class MapperClass {if no nodes expansion = false;}
>> Public static class ReducerClass
>> Public static void main {expansion = true; run job;
>> print(expansion)}
>> }
>> In this case, expansion is still true.
>> I will look at hadoop counter and report back here later.
>>
>> Thank you for all your help
>> Wei
>>
>> -----Original Message-----
>> From: Arindam Khaled [mailto:akhaled@utdallas.edu]
>> Sent: Friday, December 17, 2010 10:35 AM
>> To: common-user@hadoop.apache.org
>> Subject: Re: Please help with hadoop configuration parameter set and get
>>
>> I did something like this using a global static boolean variable
>> (flag) while I was implementing breadth first IDA*. In my case, I set
>> the flag to something else if a solution was found, which was examined
>> in the reducer.
>>
>> I guess in your case, since you know that if the mappers don't produce
>> anything the reducers won't have anything as input, if I am not wrong.
>>
>> And I had chaining map-reduce jobs (
>> http://developer.yahoo.com/hadoop/tutorial/module4.html
>> ) running until a solution was found.
>>
>>
>> Kind regards,
>>
>> Arindam Khaled
>>
>>
>>
>>
>>
>> On Dec 17, 2010, at 12:58 AM, Peng, Wei wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>> I am a newbie of hadoop.
>>>
>>> Today I was struggling with a hadoop problem for several hours.
>>>
>>>
>>>
>>> I initialize a parameter by setting job configuration in main.
>>>
>>> E.g. Configuration con = new Configuration();
>>>
>>> con.set("test", "1");
>>>
>>> Job job = new Job(con);
>>>
>>>
>>>
>>> Then in the mapper class, I want to set "test" to "2". I did it by
>>>
>>> context.getConfiguration().set("test","2");
>>>
>>>
>>>
>>> Finally in the main method, after the job is finished, I check the
>>> "test" again by
>>>
>>> job.getConfiguration().get("test");
>>>
>>>
>>>
>>> However, the value of "test" is still "1".
>>>
>>>
>>>
>>> The reason why I want to change the parameter inside Mapper class is
>>> that I want to determine when to stop an iteration in the main method.
>>> For example, for doing breadth-first search, when there is no new
>>> nodes
>>> are added for further expansion, the searching iteration should stop.
>>>
>>>
>>>
>>> Your help will be deeply appreciated. Thank you
>>>
>>>
>>>
>>> Wei
>>>
>>
>>
Re: Please help with hadoop configuration parameter set and get
Posted by Ted Dunning <td...@maprtech.com>.
Statics won't work the way you might think because different mappers and
different reducers are all running in different JVM's. It might work in
local mode, but don't kid yourself about it working in a distributed mode.
It won't.
On Fri, Dec 17, 2010 at 8:31 AM, Peng, Wei <We...@xerox.com> wrote:
> Arindam, how to set this global static Boolean variable?
> I have tried to do something similarly yesterday in the following:
> Public class BFSearch
> {
> Private static boolean expansion;
> Public static class MapperClass {if no nodes expansion = false;}
> Public static class ReducerClass
> Public static void main {expansion = true; run job;
> print(expansion)}
> }
> In this case, expansion is still true.
> I will look at hadoop counter and report back here later.
>
> Thank you for all your help
> Wei
>
> -----Original Message-----
> From: Arindam Khaled [mailto:akhaled@utdallas.edu]
> Sent: Friday, December 17, 2010 10:35 AM
> To: common-user@hadoop.apache.org
> Subject: Re: Please help with hadoop configuration parameter set and get
>
> I did something like this using a global static boolean variable
> (flag) while I was implementing breadth first IDA*. In my case, I set
> the flag to something else if a solution was found, which was examined
> in the reducer.
>
> I guess in your case, since you know that if the mappers don't produce
> anything the reducers won't have anything as input, if I am not wrong.
>
> And I had chaining map-reduce jobs (
> http://developer.yahoo.com/hadoop/tutorial/module4.html
> ) running until a solution was found.
>
>
> Kind regards,
>
> Arindam Khaled
>
>
>
>
>
> On Dec 17, 2010, at 12:58 AM, Peng, Wei wrote:
>
> > Hi,
> >
> >
> >
> > I am a newbie of hadoop.
> >
> > Today I was struggling with a hadoop problem for several hours.
> >
> >
> >
> > I initialize a parameter by setting job configuration in main.
> >
> > E.g. Configuration con = new Configuration();
> >
> > con.set("test", "1");
> >
> > Job job = new Job(con);
> >
> >
> >
> > Then in the mapper class, I want to set "test" to "2". I did it by
> >
> > context.getConfiguration().set("test","2");
> >
> >
> >
> > Finally in the main method, after the job is finished, I check the
> > "test" again by
> >
> > job.getConfiguration().get("test");
> >
> >
> >
> > However, the value of "test" is still "1".
> >
> >
> >
> > The reason why I want to change the parameter inside Mapper class is
> > that I want to determine when to stop an iteration in the main method.
> > For example, for doing breadth-first search, when there is no new
> > nodes
> > are added for further expansion, the searching iteration should stop.
> >
> >
> >
> > Your help will be deeply appreciated. Thank you
> >
> >
> >
> > Wei
> >
>
>
RE: Please help with hadoop configuration parameter set and get
Posted by "Peng, Wei" <We...@xerox.com>.
Arindam, how to set this global static Boolean variable?
I have tried to do something similarly yesterday in the following:
Public class BFSearch
{
Private static boolean expansion;
Public static class MapperClass {if no nodes expansion = false;}
Public static class ReducerClass
Public static void main {expansion = true; run job;
print(expansion)}
}
In this case, expansion is still true.
I will look at hadoop counter and report back here later.
Thank you for all your help
Wei
-----Original Message-----
From: Arindam Khaled [mailto:akhaled@utdallas.edu]
Sent: Friday, December 17, 2010 10:35 AM
To: common-user@hadoop.apache.org
Subject: Re: Please help with hadoop configuration parameter set and get
I did something like this using a global static boolean variable
(flag) while I was implementing breadth first IDA*. In my case, I set
the flag to something else if a solution was found, which was examined
in the reducer.
I guess in your case, since you know that if the mappers don't produce
anything the reducers won't have anything as input, if I am not wrong.
And I had chaining map-reduce jobs (
http://developer.yahoo.com/hadoop/tutorial/module4.html
) running until a solution was found.
Kind regards,
Arindam Khaled
On Dec 17, 2010, at 12:58 AM, Peng, Wei wrote:
> Hi,
>
>
>
> I am a newbie of hadoop.
>
> Today I was struggling with a hadoop problem for several hours.
>
>
>
> I initialize a parameter by setting job configuration in main.
>
> E.g. Configuration con = new Configuration();
>
> con.set("test", "1");
>
> Job job = new Job(con);
>
>
>
> Then in the mapper class, I want to set "test" to "2". I did it by
>
> context.getConfiguration().set("test","2");
>
>
>
> Finally in the main method, after the job is finished, I check the
> "test" again by
>
> job.getConfiguration().get("test");
>
>
>
> However, the value of "test" is still "1".
>
>
>
> The reason why I want to change the parameter inside Mapper class is
> that I want to determine when to stop an iteration in the main method.
> For example, for doing breadth-first search, when there is no new
> nodes
> are added for further expansion, the searching iteration should stop.
>
>
>
> Your help will be deeply appreciated. Thank you
>
>
>
> Wei
>
Re: Please help with hadoop configuration parameter set and get
Posted by Arindam Khaled <ak...@utdallas.edu>.
I did something like this using a global static boolean variable
(flag) while I was implementing breadth first IDA*. In my case, I set
the flag to something else if a solution was found, which was examined
in the reducer.
I guess in your case, since you know that if the mappers don't produce
anything the reducers won't have anything as input, if I am not wrong.
And I had chaining map-reduce jobs ( http://developer.yahoo.com/hadoop/tutorial/module4.html
) running until a solution was found.
Kind regards,
Arindam Khaled
On Dec 17, 2010, at 12:58 AM, Peng, Wei wrote:
> Hi,
>
>
>
> I am a newbie of hadoop.
>
> Today I was struggling with a hadoop problem for several hours.
>
>
>
> I initialize a parameter by setting job configuration in main.
>
> E.g. Configuration con = new Configuration();
>
> con.set("test", "1");
>
> Job job = new Job(con);
>
>
>
> Then in the mapper class, I want to set "test" to "2". I did it by
>
> context.getConfiguration().set("test","2");
>
>
>
> Finally in the main method, after the job is finished, I check the
> "test" again by
>
> job.getConfiguration().get("test");
>
>
>
> However, the value of "test" is still "1".
>
>
>
> The reason why I want to change the parameter inside Mapper class is
> that I want to determine when to stop an iteration in the main method.
> For example, for doing breadth-first search, when there is no new
> nodes
> are added for further expansion, the searching iteration should stop.
>
>
>
> Your help will be deeply appreciated. Thank you
>
>
>
> Wei
>