You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by unmesha sreeveni <un...@gmail.com> on 2013/12/18 11:02:43 UTC
Get dynamic values in a user defined class from reducer.
Can any one pls suggest a good way
My scenario:
i hav
1.Driver class
2.Mapper class
3.reducer class
4.Mid class
After completing mapper it goes to reduce. From reducer it will be going
to Driver and from driver to Mid class
so i need to get a data from reducer class to Mid class
So for that i declared a static variable in reduce and tried to access
that variable in Mid class.But while executing i am only getting the 0
value. How to overcome this.
Reducer.java
--------------------
public class Reduce extends MapReduceBase
implements Reducer<Text, IntWritable, Text, IntWritable> {
int ig;
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output,
Reporter reporter) throws IOException {
ig = 9;
}
}
Driver.java
gainObj.getcount();
Mid.java
public class Mid{
public void getcount(String args) throws IOException
{
//WANT TO GET THE IG VALUE IE "9" HERE So i did like this
Reduce r = new Reduce();
System.out.println("red "+ r.ig);
}
}
But my result is "0". Not 9.
How to solve this?
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
Re: Get dynamic values in a user defined class from reducer.
Posted by unmesha sreeveni <un...@gmail.com>.
Thanks for ur reply Robert Dyer.
Thanks for spending ur valuable time for clearing my doubts.
I need to pass some other values too
Reducer
------------
public class Reduce extends MapReduceBase
implements Reducer<Text, IntWritable, Text, IntWritable> {
static int cnt =0;
ArrayList<String> ar = new ArrayList<String>();
String data = null;
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output,
Reporter reporter) throws IOException {
System.out.println("In reducer");
int sum = 0;
String line = key.toString();
StringTokenizer itr = new StringTokenizer(line);
while (values.hasNext()) {
sum += values.next().get();
}
output.collect(key, new IntWritable(sum));
String data = key+" "+sum;
ar.add(data);
writeToFile(ar);
ar.add("\n");
int index=Integer.parseInt(itr.nextToken());
String value=itr.nextToken();
String classLabel=itr.nextToken();
int count=sum;
}
public static void writeToFile(ArrayList<String> text) {
System.out.println("In reduce write to file ");
try {
cnt++;
C45 id=new C45();
System.out.println("count "+cnt);
Path input = new Path("C45/intermediate"+id.current_index+".txt");
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
BufferedWriter bw = new BufferedWriter(
new OutputStreamWriter(fs.create(input, true)));
System.out.println("Text from Reducer: "+
"C45/intermediate"+id.current_index+".txt"+ text);
System.out.println("file
exists:C45/intermediate"+id.current_index+".txt"+fs.exists(input));
for(String str: text) {
bw.write(str);
}
bw.newLine();
bw.close();
} catch (Exception e) {
System.out.println("File is not creating in reduce");
}
}
}
Driver
--------
GainRatio gainObj = new GainRatio();
gainObj.getcount();
GainRatio
--------------
public void getcount()
{
System.out.println("In gainratio --- getcount ");
C45 id=new C45();
FileInputStream fstream;
try {
Configuration config = new Configuration();
FileSystem dfs = FileSystem.get(config);
Path path = new Path("C45/intermediate"+id.current_index+".txt");
System.out.println("file
exists:C45/intermediate"+id.current_index+".txt"+dfs.exists(path));
System.out.println(path);
BufferedReader br=new BufferedReader(new InputStreamReader(dfs.open(path)));
String line;
StringTokenizer itr;
while ((line = br.readLine()) != null) {
System.out.println("In gain Ratio : "+line);
itr= new StringTokenizer(line);
count[linenumber][0]=itr.nextToken();
count[linenumber][1]=itr.nextToken();
count[linenumber][2]=itr.nextToken();
count[linenumber][3]=itr.nextToken();
int i=linenumber;
linenumber++;
}
System.out.println("outsidec while");
count[linenumber][0]=null;
count[linenumber][1]=null;
count[linenumber][2]=null;
count[linenumber][3]=null;
br.close();
emptyfile(id.current_index);
} catch (Exception e) {
e.printStackTrace();
}
}
Here from reducer many files are created
eg: intermediate0.txt
.....upto intermediate7.txt according to my input dataset.
But here only intermediate0.txt is created all other creation of file get
failed.
Is that the same reason you explained before.
For this also should i go for couters?
On Wed, Dec 18, 2013 at 11:09 PM, Robert Dyer <ps...@gmail.com> wrote:
> Generally speaking, static fields are not useful in Hadoop.
>
> The issue you are seeing is that the reducer is running in a separate VM
> (possibly on a different node!) and thus the static value you are reading
> inside of Mid is actually a separate instantiation of that class and field.
>
> If you have an integer you need to set and read from the driver/mid, you
> can use a counter. E.g.:
>
>
> http://www.philippeadjiman.com/blog/2010/01/07/hadoop-tutorial-series-issue-3-counters-in-action/
>
> - Robert
>
> On Wed, Dec 18, 2013 at 4:02 AM, unmesha sreeveni <un...@gmail.com>wrote:
>
>> Can any one pls suggest a good way
>> My scenario:
>> i hav
>> 1.Driver class
>> 2.Mapper class
>> 3.reducer class
>> 4.Mid class
>> After completing mapper it goes to reduce. From reducer it will be going
>> to Driver and from driver to Mid class
>>
>> so i need to get a data from reducer class to Mid class
>> So for that i declared a static variable in reduce and tried to access
>> that variable in Mid class.But while executing i am only getting the 0
>> value. How to overcome this.
>>
>>
>> Reducer.java
>> --------------------
>> public class Reduce extends MapReduceBase
>> implements Reducer<Text, IntWritable, Text, IntWritable> {
>> int ig;
>> public void reduce(Text key, Iterator<IntWritable> values,
>> OutputCollector<Text, IntWritable> output,
>> Reporter reporter) throws IOException {
>> ig = 9;
>> }
>> }
>> Driver.java
>> gainObj.getcount();
>>
>>
>> Mid.java
>> public class Mid{
>> public void getcount(String args) throws IOException
>> {
>> //WANT TO GET THE IG VALUE IE "9" HERE So i did like
>> this
>> Reduce r = new Reduce();
>> System.out.println("red "+ r.ig);
>> }
>> }
>> But my result is "0". Not 9.
>> How to solve this?
>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>>
>>
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
Re: Get dynamic values in a user defined class from reducer.
Posted by unmesha sreeveni <un...@gmail.com>.
Thanks for ur reply Robert Dyer.
Thanks for spending ur valuable time for clearing my doubts.
I need to pass some other values too
Reducer
------------
public class Reduce extends MapReduceBase
implements Reducer<Text, IntWritable, Text, IntWritable> {
static int cnt =0;
ArrayList<String> ar = new ArrayList<String>();
String data = null;
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output,
Reporter reporter) throws IOException {
System.out.println("In reducer");
int sum = 0;
String line = key.toString();
StringTokenizer itr = new StringTokenizer(line);
while (values.hasNext()) {
sum += values.next().get();
}
output.collect(key, new IntWritable(sum));
String data = key+" "+sum;
ar.add(data);
writeToFile(ar);
ar.add("\n");
int index=Integer.parseInt(itr.nextToken());
String value=itr.nextToken();
String classLabel=itr.nextToken();
int count=sum;
}
public static void writeToFile(ArrayList<String> text) {
System.out.println("In reduce write to file ");
try {
cnt++;
C45 id=new C45();
System.out.println("count "+cnt);
Path input = new Path("C45/intermediate"+id.current_index+".txt");
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
BufferedWriter bw = new BufferedWriter(
new OutputStreamWriter(fs.create(input, true)));
System.out.println("Text from Reducer: "+
"C45/intermediate"+id.current_index+".txt"+ text);
System.out.println("file
exists:C45/intermediate"+id.current_index+".txt"+fs.exists(input));
for(String str: text) {
bw.write(str);
}
bw.newLine();
bw.close();
} catch (Exception e) {
System.out.println("File is not creating in reduce");
}
}
}
Driver
--------
GainRatio gainObj = new GainRatio();
gainObj.getcount();
GainRatio
--------------
public void getcount()
{
System.out.println("In gainratio --- getcount ");
C45 id=new C45();
FileInputStream fstream;
try {
Configuration config = new Configuration();
FileSystem dfs = FileSystem.get(config);
Path path = new Path("C45/intermediate"+id.current_index+".txt");
System.out.println("file
exists:C45/intermediate"+id.current_index+".txt"+dfs.exists(path));
System.out.println(path);
BufferedReader br=new BufferedReader(new InputStreamReader(dfs.open(path)));
String line;
StringTokenizer itr;
while ((line = br.readLine()) != null) {
System.out.println("In gain Ratio : "+line);
itr= new StringTokenizer(line);
count[linenumber][0]=itr.nextToken();
count[linenumber][1]=itr.nextToken();
count[linenumber][2]=itr.nextToken();
count[linenumber][3]=itr.nextToken();
int i=linenumber;
linenumber++;
}
System.out.println("outsidec while");
count[linenumber][0]=null;
count[linenumber][1]=null;
count[linenumber][2]=null;
count[linenumber][3]=null;
br.close();
emptyfile(id.current_index);
} catch (Exception e) {
e.printStackTrace();
}
}
Here from reducer many files are created
eg: intermediate0.txt
.....upto intermediate7.txt according to my input dataset.
But here only intermediate0.txt is created all other creation of file get
failed.
Is that the same reason you explained before.
For this also should i go for couters?
On Wed, Dec 18, 2013 at 11:09 PM, Robert Dyer <ps...@gmail.com> wrote:
> Generally speaking, static fields are not useful in Hadoop.
>
> The issue you are seeing is that the reducer is running in a separate VM
> (possibly on a different node!) and thus the static value you are reading
> inside of Mid is actually a separate instantiation of that class and field.
>
> If you have an integer you need to set and read from the driver/mid, you
> can use a counter. E.g.:
>
>
> http://www.philippeadjiman.com/blog/2010/01/07/hadoop-tutorial-series-issue-3-counters-in-action/
>
> - Robert
>
> On Wed, Dec 18, 2013 at 4:02 AM, unmesha sreeveni <un...@gmail.com>wrote:
>
>> Can any one pls suggest a good way
>> My scenario:
>> i hav
>> 1.Driver class
>> 2.Mapper class
>> 3.reducer class
>> 4.Mid class
>> After completing mapper it goes to reduce. From reducer it will be going
>> to Driver and from driver to Mid class
>>
>> so i need to get a data from reducer class to Mid class
>> So for that i declared a static variable in reduce and tried to access
>> that variable in Mid class.But while executing i am only getting the 0
>> value. How to overcome this.
>>
>>
>> Reducer.java
>> --------------------
>> public class Reduce extends MapReduceBase
>> implements Reducer<Text, IntWritable, Text, IntWritable> {
>> int ig;
>> public void reduce(Text key, Iterator<IntWritable> values,
>> OutputCollector<Text, IntWritable> output,
>> Reporter reporter) throws IOException {
>> ig = 9;
>> }
>> }
>> Driver.java
>> gainObj.getcount();
>>
>>
>> Mid.java
>> public class Mid{
>> public void getcount(String args) throws IOException
>> {
>> //WANT TO GET THE IG VALUE IE "9" HERE So i did like
>> this
>> Reduce r = new Reduce();
>> System.out.println("red "+ r.ig);
>> }
>> }
>> But my result is "0". Not 9.
>> How to solve this?
>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>>
>>
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
Re: Get dynamic values in a user defined class from reducer.
Posted by unmesha sreeveni <un...@gmail.com>.
Thanks for ur reply Robert Dyer.
Thanks for spending ur valuable time for clearing my doubts.
I need to pass some other values too
Reducer
------------
public class Reduce extends MapReduceBase
implements Reducer<Text, IntWritable, Text, IntWritable> {
static int cnt =0;
ArrayList<String> ar = new ArrayList<String>();
String data = null;
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output,
Reporter reporter) throws IOException {
System.out.println("In reducer");
int sum = 0;
String line = key.toString();
StringTokenizer itr = new StringTokenizer(line);
while (values.hasNext()) {
sum += values.next().get();
}
output.collect(key, new IntWritable(sum));
String data = key+" "+sum;
ar.add(data);
writeToFile(ar);
ar.add("\n");
int index=Integer.parseInt(itr.nextToken());
String value=itr.nextToken();
String classLabel=itr.nextToken();
int count=sum;
}
public static void writeToFile(ArrayList<String> text) {
System.out.println("In reduce write to file ");
try {
cnt++;
C45 id=new C45();
System.out.println("count "+cnt);
Path input = new Path("C45/intermediate"+id.current_index+".txt");
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
BufferedWriter bw = new BufferedWriter(
new OutputStreamWriter(fs.create(input, true)));
System.out.println("Text from Reducer: "+
"C45/intermediate"+id.current_index+".txt"+ text);
System.out.println("file
exists:C45/intermediate"+id.current_index+".txt"+fs.exists(input));
for(String str: text) {
bw.write(str);
}
bw.newLine();
bw.close();
} catch (Exception e) {
System.out.println("File is not creating in reduce");
}
}
}
Driver
--------
GainRatio gainObj = new GainRatio();
gainObj.getcount();
GainRatio
--------------
public void getcount()
{
System.out.println("In gainratio --- getcount ");
C45 id=new C45();
FileInputStream fstream;
try {
Configuration config = new Configuration();
FileSystem dfs = FileSystem.get(config);
Path path = new Path("C45/intermediate"+id.current_index+".txt");
System.out.println("file
exists:C45/intermediate"+id.current_index+".txt"+dfs.exists(path));
System.out.println(path);
BufferedReader br=new BufferedReader(new InputStreamReader(dfs.open(path)));
String line;
StringTokenizer itr;
while ((line = br.readLine()) != null) {
System.out.println("In gain Ratio : "+line);
itr= new StringTokenizer(line);
count[linenumber][0]=itr.nextToken();
count[linenumber][1]=itr.nextToken();
count[linenumber][2]=itr.nextToken();
count[linenumber][3]=itr.nextToken();
int i=linenumber;
linenumber++;
}
System.out.println("outsidec while");
count[linenumber][0]=null;
count[linenumber][1]=null;
count[linenumber][2]=null;
count[linenumber][3]=null;
br.close();
emptyfile(id.current_index);
} catch (Exception e) {
e.printStackTrace();
}
}
Here from reducer many files are created
eg: intermediate0.txt
.....upto intermediate7.txt according to my input dataset.
But here only intermediate0.txt is created all other creation of file get
failed.
Is that the same reason you explained before.
For this also should i go for couters?
On Wed, Dec 18, 2013 at 11:09 PM, Robert Dyer <ps...@gmail.com> wrote:
> Generally speaking, static fields are not useful in Hadoop.
>
> The issue you are seeing is that the reducer is running in a separate VM
> (possibly on a different node!) and thus the static value you are reading
> inside of Mid is actually a separate instantiation of that class and field.
>
> If you have an integer you need to set and read from the driver/mid, you
> can use a counter. E.g.:
>
>
> http://www.philippeadjiman.com/blog/2010/01/07/hadoop-tutorial-series-issue-3-counters-in-action/
>
> - Robert
>
> On Wed, Dec 18, 2013 at 4:02 AM, unmesha sreeveni <un...@gmail.com>wrote:
>
>> Can any one pls suggest a good way
>> My scenario:
>> i hav
>> 1.Driver class
>> 2.Mapper class
>> 3.reducer class
>> 4.Mid class
>> After completing mapper it goes to reduce. From reducer it will be going
>> to Driver and from driver to Mid class
>>
>> so i need to get a data from reducer class to Mid class
>> So for that i declared a static variable in reduce and tried to access
>> that variable in Mid class.But while executing i am only getting the 0
>> value. How to overcome this.
>>
>>
>> Reducer.java
>> --------------------
>> public class Reduce extends MapReduceBase
>> implements Reducer<Text, IntWritable, Text, IntWritable> {
>> int ig;
>> public void reduce(Text key, Iterator<IntWritable> values,
>> OutputCollector<Text, IntWritable> output,
>> Reporter reporter) throws IOException {
>> ig = 9;
>> }
>> }
>> Driver.java
>> gainObj.getcount();
>>
>>
>> Mid.java
>> public class Mid{
>> public void getcount(String args) throws IOException
>> {
>> //WANT TO GET THE IG VALUE IE "9" HERE So i did like
>> this
>> Reduce r = new Reduce();
>> System.out.println("red "+ r.ig);
>> }
>> }
>> But my result is "0". Not 9.
>> How to solve this?
>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>>
>>
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
Re: Get dynamic values in a user defined class from reducer.
Posted by unmesha sreeveni <un...@gmail.com>.
Thanks for ur reply Robert Dyer.
Thanks for spending ur valuable time for clearing my doubts.
I need to pass some other values too
Reducer
------------
public class Reduce extends MapReduceBase
implements Reducer<Text, IntWritable, Text, IntWritable> {
static int cnt =0;
ArrayList<String> ar = new ArrayList<String>();
String data = null;
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output,
Reporter reporter) throws IOException {
System.out.println("In reducer");
int sum = 0;
String line = key.toString();
StringTokenizer itr = new StringTokenizer(line);
while (values.hasNext()) {
sum += values.next().get();
}
output.collect(key, new IntWritable(sum));
String data = key+" "+sum;
ar.add(data);
writeToFile(ar);
ar.add("\n");
int index=Integer.parseInt(itr.nextToken());
String value=itr.nextToken();
String classLabel=itr.nextToken();
int count=sum;
}
public static void writeToFile(ArrayList<String> text) {
System.out.println("In reduce write to file ");
try {
cnt++;
C45 id=new C45();
System.out.println("count "+cnt);
Path input = new Path("C45/intermediate"+id.current_index+".txt");
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
BufferedWriter bw = new BufferedWriter(
new OutputStreamWriter(fs.create(input, true)));
System.out.println("Text from Reducer: "+
"C45/intermediate"+id.current_index+".txt"+ text);
System.out.println("file
exists:C45/intermediate"+id.current_index+".txt"+fs.exists(input));
for(String str: text) {
bw.write(str);
}
bw.newLine();
bw.close();
} catch (Exception e) {
System.out.println("File is not creating in reduce");
}
}
}
Driver
--------
GainRatio gainObj = new GainRatio();
gainObj.getcount();
GainRatio
--------------
public void getcount()
{
System.out.println("In gainratio --- getcount ");
C45 id=new C45();
FileInputStream fstream;
try {
Configuration config = new Configuration();
FileSystem dfs = FileSystem.get(config);
Path path = new Path("C45/intermediate"+id.current_index+".txt");
System.out.println("file
exists:C45/intermediate"+id.current_index+".txt"+dfs.exists(path));
System.out.println(path);
BufferedReader br=new BufferedReader(new InputStreamReader(dfs.open(path)));
String line;
StringTokenizer itr;
while ((line = br.readLine()) != null) {
System.out.println("In gain Ratio : "+line);
itr= new StringTokenizer(line);
count[linenumber][0]=itr.nextToken();
count[linenumber][1]=itr.nextToken();
count[linenumber][2]=itr.nextToken();
count[linenumber][3]=itr.nextToken();
int i=linenumber;
linenumber++;
}
System.out.println("outsidec while");
count[linenumber][0]=null;
count[linenumber][1]=null;
count[linenumber][2]=null;
count[linenumber][3]=null;
br.close();
emptyfile(id.current_index);
} catch (Exception e) {
e.printStackTrace();
}
}
Here from reducer many files are created
eg: intermediate0.txt
.....upto intermediate7.txt according to my input dataset.
But here only intermediate0.txt is created all other creation of file get
failed.
Is that the same reason you explained before.
For this also should i go for couters?
On Wed, Dec 18, 2013 at 11:09 PM, Robert Dyer <ps...@gmail.com> wrote:
> Generally speaking, static fields are not useful in Hadoop.
>
> The issue you are seeing is that the reducer is running in a separate VM
> (possibly on a different node!) and thus the static value you are reading
> inside of Mid is actually a separate instantiation of that class and field.
>
> If you have an integer you need to set and read from the driver/mid, you
> can use a counter. E.g.:
>
>
> http://www.philippeadjiman.com/blog/2010/01/07/hadoop-tutorial-series-issue-3-counters-in-action/
>
> - Robert
>
> On Wed, Dec 18, 2013 at 4:02 AM, unmesha sreeveni <un...@gmail.com>wrote:
>
>> Can any one pls suggest a good way
>> My scenario:
>> i hav
>> 1.Driver class
>> 2.Mapper class
>> 3.reducer class
>> 4.Mid class
>> After completing mapper it goes to reduce. From reducer it will be going
>> to Driver and from driver to Mid class
>>
>> so i need to get a data from reducer class to Mid class
>> So for that i declared a static variable in reduce and tried to access
>> that variable in Mid class.But while executing i am only getting the 0
>> value. How to overcome this.
>>
>>
>> Reducer.java
>> --------------------
>> public class Reduce extends MapReduceBase
>> implements Reducer<Text, IntWritable, Text, IntWritable> {
>> int ig;
>> public void reduce(Text key, Iterator<IntWritable> values,
>> OutputCollector<Text, IntWritable> output,
>> Reporter reporter) throws IOException {
>> ig = 9;
>> }
>> }
>> Driver.java
>> gainObj.getcount();
>>
>>
>> Mid.java
>> public class Mid{
>> public void getcount(String args) throws IOException
>> {
>> //WANT TO GET THE IG VALUE IE "9" HERE So i did like
>> this
>> Reduce r = new Reduce();
>> System.out.println("red "+ r.ig);
>> }
>> }
>> But my result is "0". Not 9.
>> How to solve this?
>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>>
>>
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
Re: Get dynamic values in a user defined class from reducer.
Posted by Robert Dyer <ps...@gmail.com>.
Generally speaking, static fields are not useful in Hadoop.
The issue you are seeing is that the reducer is running in a separate VM
(possibly on a different node!) and thus the static value you are reading
inside of Mid is actually a separate instantiation of that class and field.
If you have an integer you need to set and read from the driver/mid, you
can use a counter. E.g.:
http://www.philippeadjiman.com/blog/2010/01/07/hadoop-tutorial-series-issue-3-counters-in-action/
- Robert
On Wed, Dec 18, 2013 at 4:02 AM, unmesha sreeveni <un...@gmail.com>wrote:
> Can any one pls suggest a good way
> My scenario:
> i hav
> 1.Driver class
> 2.Mapper class
> 3.reducer class
> 4.Mid class
> After completing mapper it goes to reduce. From reducer it will be going
> to Driver and from driver to Mid class
>
> so i need to get a data from reducer class to Mid class
> So for that i declared a static variable in reduce and tried to access
> that variable in Mid class.But while executing i am only getting the 0
> value. How to overcome this.
>
>
> Reducer.java
> --------------------
> public class Reduce extends MapReduceBase
> implements Reducer<Text, IntWritable, Text, IntWritable> {
> int ig;
> public void reduce(Text key, Iterator<IntWritable> values,
> OutputCollector<Text, IntWritable> output,
> Reporter reporter) throws IOException {
> ig = 9;
> }
> }
> Driver.java
> gainObj.getcount();
>
>
> Mid.java
> public class Mid{
> public void getcount(String args) throws IOException
> {
> //WANT TO GET THE IG VALUE IE "9" HERE So i did like
> this
> Reduce r = new Reduce();
> System.out.println("red "+ r.ig);
> }
> }
> But my result is "0". Not 9.
> How to solve this?
>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
>
>
Re: Get dynamic values in a user defined class from reducer.
Posted by Robert Dyer <ps...@gmail.com>.
Generally speaking, static fields are not useful in Hadoop.
The issue you are seeing is that the reducer is running in a separate VM
(possibly on a different node!) and thus the static value you are reading
inside of Mid is actually a separate instantiation of that class and field.
If you have an integer you need to set and read from the driver/mid, you
can use a counter. E.g.:
http://www.philippeadjiman.com/blog/2010/01/07/hadoop-tutorial-series-issue-3-counters-in-action/
- Robert
On Wed, Dec 18, 2013 at 4:02 AM, unmesha sreeveni <un...@gmail.com>wrote:
> Can any one pls suggest a good way
> My scenario:
> i hav
> 1.Driver class
> 2.Mapper class
> 3.reducer class
> 4.Mid class
> After completing mapper it goes to reduce. From reducer it will be going
> to Driver and from driver to Mid class
>
> so i need to get a data from reducer class to Mid class
> So for that i declared a static variable in reduce and tried to access
> that variable in Mid class.But while executing i am only getting the 0
> value. How to overcome this.
>
>
> Reducer.java
> --------------------
> public class Reduce extends MapReduceBase
> implements Reducer<Text, IntWritable, Text, IntWritable> {
> int ig;
> public void reduce(Text key, Iterator<IntWritable> values,
> OutputCollector<Text, IntWritable> output,
> Reporter reporter) throws IOException {
> ig = 9;
> }
> }
> Driver.java
> gainObj.getcount();
>
>
> Mid.java
> public class Mid{
> public void getcount(String args) throws IOException
> {
> //WANT TO GET THE IG VALUE IE "9" HERE So i did like
> this
> Reduce r = new Reduce();
> System.out.println("red "+ r.ig);
> }
> }
> But my result is "0". Not 9.
> How to solve this?
>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
>
>
Re: Get dynamic values in a user defined class from reducer.
Posted by Robert Dyer <ps...@gmail.com>.
Generally speaking, static fields are not useful in Hadoop.
The issue you are seeing is that the reducer is running in a separate VM
(possibly on a different node!) and thus the static value you are reading
inside of Mid is actually a separate instantiation of that class and field.
If you have an integer you need to set and read from the driver/mid, you
can use a counter. E.g.:
http://www.philippeadjiman.com/blog/2010/01/07/hadoop-tutorial-series-issue-3-counters-in-action/
- Robert
On Wed, Dec 18, 2013 at 4:02 AM, unmesha sreeveni <un...@gmail.com>wrote:
> Can any one pls suggest a good way
> My scenario:
> i hav
> 1.Driver class
> 2.Mapper class
> 3.reducer class
> 4.Mid class
> After completing mapper it goes to reduce. From reducer it will be going
> to Driver and from driver to Mid class
>
> so i need to get a data from reducer class to Mid class
> So for that i declared a static variable in reduce and tried to access
> that variable in Mid class.But while executing i am only getting the 0
> value. How to overcome this.
>
>
> Reducer.java
> --------------------
> public class Reduce extends MapReduceBase
> implements Reducer<Text, IntWritable, Text, IntWritable> {
> int ig;
> public void reduce(Text key, Iterator<IntWritable> values,
> OutputCollector<Text, IntWritable> output,
> Reporter reporter) throws IOException {
> ig = 9;
> }
> }
> Driver.java
> gainObj.getcount();
>
>
> Mid.java
> public class Mid{
> public void getcount(String args) throws IOException
> {
> //WANT TO GET THE IG VALUE IE "9" HERE So i did like
> this
> Reduce r = new Reduce();
> System.out.println("red "+ r.ig);
> }
> }
> But my result is "0". Not 9.
> How to solve this?
>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
>
>
Re: Get dynamic values in a user defined class from reducer.
Posted by Robert Dyer <ps...@gmail.com>.
Generally speaking, static fields are not useful in Hadoop.
The issue you are seeing is that the reducer is running in a separate VM
(possibly on a different node!) and thus the static value you are reading
inside of Mid is actually a separate instantiation of that class and field.
If you have an integer you need to set and read from the driver/mid, you
can use a counter. E.g.:
http://www.philippeadjiman.com/blog/2010/01/07/hadoop-tutorial-series-issue-3-counters-in-action/
- Robert
On Wed, Dec 18, 2013 at 4:02 AM, unmesha sreeveni <un...@gmail.com>wrote:
> Can any one pls suggest a good way
> My scenario:
> i hav
> 1.Driver class
> 2.Mapper class
> 3.reducer class
> 4.Mid class
> After completing mapper it goes to reduce. From reducer it will be going
> to Driver and from driver to Mid class
>
> so i need to get a data from reducer class to Mid class
> So for that i declared a static variable in reduce and tried to access
> that variable in Mid class.But while executing i am only getting the 0
> value. How to overcome this.
>
>
> Reducer.java
> --------------------
> public class Reduce extends MapReduceBase
> implements Reducer<Text, IntWritable, Text, IntWritable> {
> int ig;
> public void reduce(Text key, Iterator<IntWritable> values,
> OutputCollector<Text, IntWritable> output,
> Reporter reporter) throws IOException {
> ig = 9;
> }
> }
> Driver.java
> gainObj.getcount();
>
>
> Mid.java
> public class Mid{
> public void getcount(String args) throws IOException
> {
> //WANT TO GET THE IG VALUE IE "9" HERE So i did like
> this
> Reduce r = new Reduce();
> System.out.println("red "+ r.ig);
> }
> }
> But my result is "0". Not 9.
> How to solve this?
>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
>
>