You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Answer Agrawal <yr...@gmail.com> on 2015/05/12 18:55:17 UTC
How to access value of variable in Driver class which has been
declared and modified inside Mapper class?
Hi,
I declared a variable and incremented/modified it inside Mapper class. Now
I need to use the modified value of that variable in Driver class. I
declared a static variable inside Mapper class and its modified value works
in Driver class when I run the code in Eclipse IDE. But after creating that
code as a runable jar from Eclipse and run jar file as “$ hadoop jar
filename.jar input output” modified value does not reflect (value is 0) in
Driver class.
Thanks,
Re: How to access value of variable in Driver class which has been
declared and modified inside Mapper class?
Posted by Shahab Yunus <sh...@gmail.com>.
Here are some examples of how to use custom counters:
http://www.ashishpaliwal.com/blog/2012/05/hadoop-recipe-using-custom-java-counters/
Regards,
Shahab
On May 12, 2015 1:29 PM, "Shahab Yunus" <sh...@gmail.com> wrote:
> Better options than using static variable are, imo:
>
> One option it use Counters. Check that API. We are using that for values
> that are numeric and we need those in the driver once the job finishes. You
> can create your custom counters too.
>
> Other option is (if you need more than just one value or your info is
> complicated) then we output data from each mapper or reducer as files in
> each task (e.g. in the cleanup method) and then read those once the job
> finishes.
>
> In both of these, you will have to collate and merge, according to the
> logic all the copies of the value/info that you end up with. The number of
> copies would be equal to the number of mappers or reducers (if you are
> keeping the value there) used in the job.
>
> You can also set the value in the job configuration and read that once the
> is over.
>
> Regards,
> Shahab
> On May 12, 2015 12:56 PM, "Answer Agrawal" <yr...@gmail.com> wrote:
>
>> Hi,
>>
>> I declared a variable and incremented/modified it inside Mapper class.
>> Now I need to use the modified value of that variable in Driver class. I
>> declared a static variable inside Mapper class and its modified value works
>> in Driver class when I run the code in Eclipse IDE. But after creating that
>> code as a runable jar from Eclipse and run jar file as “$ hadoop jar
>> filename.jar input output” modified value does not reflect (value is 0) in
>> Driver class.
>>
>> Thanks,
>>
>
Re: How to access value of variable in Driver class which has been
declared and modified inside Mapper class?
Posted by Shahab Yunus <sh...@gmail.com>.
Here are some examples of how to use custom counters:
http://www.ashishpaliwal.com/blog/2012/05/hadoop-recipe-using-custom-java-counters/
Regards,
Shahab
On May 12, 2015 1:29 PM, "Shahab Yunus" <sh...@gmail.com> wrote:
> Better options than using static variable are, imo:
>
> One option it use Counters. Check that API. We are using that for values
> that are numeric and we need those in the driver once the job finishes. You
> can create your custom counters too.
>
> Other option is (if you need more than just one value or your info is
> complicated) then we output data from each mapper or reducer as files in
> each task (e.g. in the cleanup method) and then read those once the job
> finishes.
>
> In both of these, you will have to collate and merge, according to the
> logic all the copies of the value/info that you end up with. The number of
> copies would be equal to the number of mappers or reducers (if you are
> keeping the value there) used in the job.
>
> You can also set the value in the job configuration and read that once the
> is over.
>
> Regards,
> Shahab
> On May 12, 2015 12:56 PM, "Answer Agrawal" <yr...@gmail.com> wrote:
>
>> Hi,
>>
>> I declared a variable and incremented/modified it inside Mapper class.
>> Now I need to use the modified value of that variable in Driver class. I
>> declared a static variable inside Mapper class and its modified value works
>> in Driver class when I run the code in Eclipse IDE. But after creating that
>> code as a runable jar from Eclipse and run jar file as “$ hadoop jar
>> filename.jar input output” modified value does not reflect (value is 0) in
>> Driver class.
>>
>> Thanks,
>>
>
Re: How to access value of variable in Driver class which has been
declared and modified inside Mapper class?
Posted by Shahab Yunus <sh...@gmail.com>.
Here are some examples of how to use custom counters:
http://www.ashishpaliwal.com/blog/2012/05/hadoop-recipe-using-custom-java-counters/
Regards,
Shahab
On May 12, 2015 1:29 PM, "Shahab Yunus" <sh...@gmail.com> wrote:
> Better options than using static variable are, imo:
>
> One option it use Counters. Check that API. We are using that for values
> that are numeric and we need those in the driver once the job finishes. You
> can create your custom counters too.
>
> Other option is (if you need more than just one value or your info is
> complicated) then we output data from each mapper or reducer as files in
> each task (e.g. in the cleanup method) and then read those once the job
> finishes.
>
> In both of these, you will have to collate and merge, according to the
> logic all the copies of the value/info that you end up with. The number of
> copies would be equal to the number of mappers or reducers (if you are
> keeping the value there) used in the job.
>
> You can also set the value in the job configuration and read that once the
> is over.
>
> Regards,
> Shahab
> On May 12, 2015 12:56 PM, "Answer Agrawal" <yr...@gmail.com> wrote:
>
>> Hi,
>>
>> I declared a variable and incremented/modified it inside Mapper class.
>> Now I need to use the modified value of that variable in Driver class. I
>> declared a static variable inside Mapper class and its modified value works
>> in Driver class when I run the code in Eclipse IDE. But after creating that
>> code as a runable jar from Eclipse and run jar file as “$ hadoop jar
>> filename.jar input output” modified value does not reflect (value is 0) in
>> Driver class.
>>
>> Thanks,
>>
>
Re: How to access value of variable in Driver class which has been
declared and modified inside Mapper class?
Posted by Shahab Yunus <sh...@gmail.com>.
Here are some examples of how to use custom counters:
http://www.ashishpaliwal.com/blog/2012/05/hadoop-recipe-using-custom-java-counters/
Regards,
Shahab
On May 12, 2015 1:29 PM, "Shahab Yunus" <sh...@gmail.com> wrote:
> Better options than using static variable are, imo:
>
> One option it use Counters. Check that API. We are using that for values
> that are numeric and we need those in the driver once the job finishes. You
> can create your custom counters too.
>
> Other option is (if you need more than just one value or your info is
> complicated) then we output data from each mapper or reducer as files in
> each task (e.g. in the cleanup method) and then read those once the job
> finishes.
>
> In both of these, you will have to collate and merge, according to the
> logic all the copies of the value/info that you end up with. The number of
> copies would be equal to the number of mappers or reducers (if you are
> keeping the value there) used in the job.
>
> You can also set the value in the job configuration and read that once the
> is over.
>
> Regards,
> Shahab
> On May 12, 2015 12:56 PM, "Answer Agrawal" <yr...@gmail.com> wrote:
>
>> Hi,
>>
>> I declared a variable and incremented/modified it inside Mapper class.
>> Now I need to use the modified value of that variable in Driver class. I
>> declared a static variable inside Mapper class and its modified value works
>> in Driver class when I run the code in Eclipse IDE. But after creating that
>> code as a runable jar from Eclipse and run jar file as “$ hadoop jar
>> filename.jar input output” modified value does not reflect (value is 0) in
>> Driver class.
>>
>> Thanks,
>>
>
Re: How to access value of variable in Driver class which has been
declared and modified inside Mapper class?
Posted by Shahab Yunus <sh...@gmail.com>.
Better options than using static variable are, imo:
One option it use Counters. Check that API. We are using that for values
that are numeric and we need those in the driver once the job finishes. You
can create your custom counters too.
Other option is (if you need more than just one value or your info is
complicated) then we output data from each mapper or reducer as files in
each task (e.g. in the cleanup method) and then read those once the job
finishes.
In both of these, you will have to collate and merge, according to the
logic all the copies of the value/info that you end up with. The number of
copies would be equal to the number of mappers or reducers (if you are
keeping the value there) used in the job.
You can also set the value in the job configuration and read that once the
is over.
Regards,
Shahab
On May 12, 2015 12:56 PM, "Answer Agrawal" <yr...@gmail.com> wrote:
> Hi,
>
> I declared a variable and incremented/modified it inside Mapper class. Now
> I need to use the modified value of that variable in Driver class. I
> declared a static variable inside Mapper class and its modified value works
> in Driver class when I run the code in Eclipse IDE. But after creating that
> code as a runable jar from Eclipse and run jar file as “$ hadoop jar
> filename.jar input output” modified value does not reflect (value is 0) in
> Driver class.
>
> Thanks,
>
Re: How to access value of variable in Driver class which has been
declared and modified inside Mapper class?
Posted by Shahab Yunus <sh...@gmail.com>.
Better options than using static variable are, imo:
One option it use Counters. Check that API. We are using that for values
that are numeric and we need those in the driver once the job finishes. You
can create your custom counters too.
Other option is (if you need more than just one value or your info is
complicated) then we output data from each mapper or reducer as files in
each task (e.g. in the cleanup method) and then read those once the job
finishes.
In both of these, you will have to collate and merge, according to the
logic all the copies of the value/info that you end up with. The number of
copies would be equal to the number of mappers or reducers (if you are
keeping the value there) used in the job.
You can also set the value in the job configuration and read that once the
is over.
Regards,
Shahab
On May 12, 2015 12:56 PM, "Answer Agrawal" <yr...@gmail.com> wrote:
> Hi,
>
> I declared a variable and incremented/modified it inside Mapper class. Now
> I need to use the modified value of that variable in Driver class. I
> declared a static variable inside Mapper class and its modified value works
> in Driver class when I run the code in Eclipse IDE. But after creating that
> code as a runable jar from Eclipse and run jar file as “$ hadoop jar
> filename.jar input output” modified value does not reflect (value is 0) in
> Driver class.
>
> Thanks,
>
Re: How to access value of variable in Driver class which has been
declared and modified inside Mapper class?
Posted by Shahab Yunus <sh...@gmail.com>.
Better options than using static variable are, imo:
One option it use Counters. Check that API. We are using that for values
that are numeric and we need those in the driver once the job finishes. You
can create your custom counters too.
Other option is (if you need more than just one value or your info is
complicated) then we output data from each mapper or reducer as files in
each task (e.g. in the cleanup method) and then read those once the job
finishes.
In both of these, you will have to collate and merge, according to the
logic all the copies of the value/info that you end up with. The number of
copies would be equal to the number of mappers or reducers (if you are
keeping the value there) used in the job.
You can also set the value in the job configuration and read that once the
is over.
Regards,
Shahab
On May 12, 2015 12:56 PM, "Answer Agrawal" <yr...@gmail.com> wrote:
> Hi,
>
> I declared a variable and incremented/modified it inside Mapper class. Now
> I need to use the modified value of that variable in Driver class. I
> declared a static variable inside Mapper class and its modified value works
> in Driver class when I run the code in Eclipse IDE. But after creating that
> code as a runable jar from Eclipse and run jar file as “$ hadoop jar
> filename.jar input output” modified value does not reflect (value is 0) in
> Driver class.
>
> Thanks,
>
Re: How to access value of variable in Driver class which has been
declared and modified inside Mapper class?
Posted by Shahab Yunus <sh...@gmail.com>.
Better options than using static variable are, imo:
One option it use Counters. Check that API. We are using that for values
that are numeric and we need those in the driver once the job finishes. You
can create your custom counters too.
Other option is (if you need more than just one value or your info is
complicated) then we output data from each mapper or reducer as files in
each task (e.g. in the cleanup method) and then read those once the job
finishes.
In both of these, you will have to collate and merge, according to the
logic all the copies of the value/info that you end up with. The number of
copies would be equal to the number of mappers or reducers (if you are
keeping the value there) used in the job.
You can also set the value in the job configuration and read that once the
is over.
Regards,
Shahab
On May 12, 2015 12:56 PM, "Answer Agrawal" <yr...@gmail.com> wrote:
> Hi,
>
> I declared a variable and incremented/modified it inside Mapper class. Now
> I need to use the modified value of that variable in Driver class. I
> declared a static variable inside Mapper class and its modified value works
> in Driver class when I run the code in Eclipse IDE. But after creating that
> code as a runable jar from Eclipse and run jar file as “$ hadoop jar
> filename.jar input output” modified value does not reflect (value is 0) in
> Driver class.
>
> Thanks,
>