You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by "Shrivastava, Himnshu (GE Global Research, Non-GE)" <Hi...@ge.com> on 2014/06/19 11:00:31 UTC
Map reduce Query
Can we take input in a mapper function and in some way pass it to reducer ?
Regards,
Re: Map reduce Query
Posted by Raj K Singh <ra...@gmail.com>.
you can get the input from some source(e.g. files) in the mapper setup()
method and emit it to the context.write() so that it can reach to the
reducer.
::::::::::::::::::::::::::::::::::::::::
Raj K Singh
http://in.linkedin.com/in/rajkrrsingh
http://www.rajkrrsingh.blogspot.com
Mobile Tel: +91 (0)9899821370
On Thu, Jun 19, 2014 at 2:30 PM, Shrivastava, Himnshu (GE Global Research,
Non-GE) <Hi...@ge.com> wrote:
> Can we take input in a mapper function and in some way pass it to
> reducer ?
>
>
>
> Regards,
>
Re: Map reduce Query
Posted by Mohammad Tariq <do...@gmail.com>.
This is what we do. I'm sorry I didn't quite this.
We read in the data through Mapper, do some operations on it, and pass the
Mapper output on to the Reducer. If you intend to just pass the data as it
is then just context.write it without doing anything else.
*Warm regards,*
*Mohammad Tariq*
*cloudfront.blogspot.com <http://cloudfront.blogspot.com>*
On Thu, Jun 19, 2014 at 2:30 PM, Shrivastava, Himnshu (GE Global Research,
Non-GE) <Hi...@ge.com> wrote:
> Can we take input in a mapper function and in some way pass it to
> reducer ?
>
>
>
> Regards,
>
Re: Map reduce Query
Posted by Mohammad Tariq <do...@gmail.com>.
This is what we do. I'm sorry I didn't quite this.
We read in the data through Mapper, do some operations on it, and pass the
Mapper output on to the Reducer. If you intend to just pass the data as it
is then just context.write it without doing anything else.
*Warm regards,*
*Mohammad Tariq*
*cloudfront.blogspot.com <http://cloudfront.blogspot.com>*
On Thu, Jun 19, 2014 at 2:30 PM, Shrivastava, Himnshu (GE Global Research,
Non-GE) <Hi...@ge.com> wrote:
> Can we take input in a mapper function and in some way pass it to
> reducer ?
>
>
>
> Regards,
>
Re: Map reduce Query
Posted by Kai Voigt <k...@123.org>.
That’s exactly what MapReduce does. The input is processed by the mapper function, and its output will be automatically sent into the reducer function. Between mappers and reducers we have the automatic shuffle phase which sends records with identical keys into one reducer call.
If you want to simply not modify the data through the map phase, just don’t specify the mapper class in your driver code. The identity function will be used for the map phase in this case.
Kai
Am 19.06.2014 um 11:00 schrieb Shrivastava, Himnshu (GE Global Research, Non-GE) <Hi...@ge.com>:
> Can we take input in a mapper function and in some way pass it to reducer ?
>
> Regards,
Kai Voigt Am Germaniahafen 1 k@123.org
24143 Kiel +49 160 96683050
Germany @KaiVoigt
Re: Map reduce Query
Posted by Raj K Singh <ra...@gmail.com>.
you can get the input from some source(e.g. files) in the mapper setup()
method and emit it to the context.write() so that it can reach to the
reducer.
::::::::::::::::::::::::::::::::::::::::
Raj K Singh
http://in.linkedin.com/in/rajkrrsingh
http://www.rajkrrsingh.blogspot.com
Mobile Tel: +91 (0)9899821370
On Thu, Jun 19, 2014 at 2:30 PM, Shrivastava, Himnshu (GE Global Research,
Non-GE) <Hi...@ge.com> wrote:
> Can we take input in a mapper function and in some way pass it to
> reducer ?
>
>
>
> Regards,
>
Re: Map reduce Query
Posted by Kai Voigt <k...@123.org>.
That’s exactly what MapReduce does. The input is processed by the mapper function, and its output will be automatically sent into the reducer function. Between mappers and reducers we have the automatic shuffle phase which sends records with identical keys into one reducer call.
If you want to simply not modify the data through the map phase, just don’t specify the mapper class in your driver code. The identity function will be used for the map phase in this case.
Kai
Am 19.06.2014 um 11:00 schrieb Shrivastava, Himnshu (GE Global Research, Non-GE) <Hi...@ge.com>:
> Can we take input in a mapper function and in some way pass it to reducer ?
>
> Regards,
Kai Voigt Am Germaniahafen 1 k@123.org
24143 Kiel +49 160 96683050
Germany @KaiVoigt
Re: Map reduce Query
Posted by Raj K Singh <ra...@gmail.com>.
you can get the input from some source(e.g. files) in the mapper setup()
method and emit it to the context.write() so that it can reach to the
reducer.
::::::::::::::::::::::::::::::::::::::::
Raj K Singh
http://in.linkedin.com/in/rajkrrsingh
http://www.rajkrrsingh.blogspot.com
Mobile Tel: +91 (0)9899821370
On Thu, Jun 19, 2014 at 2:30 PM, Shrivastava, Himnshu (GE Global Research,
Non-GE) <Hi...@ge.com> wrote:
> Can we take input in a mapper function and in some way pass it to
> reducer ?
>
>
>
> Regards,
>
Re: Map reduce Query
Posted by Mohammad Tariq <do...@gmail.com>.
This is what we do. I'm sorry I didn't quite this.
We read in the data through Mapper, do some operations on it, and pass the
Mapper output on to the Reducer. If you intend to just pass the data as it
is then just context.write it without doing anything else.
*Warm regards,*
*Mohammad Tariq*
*cloudfront.blogspot.com <http://cloudfront.blogspot.com>*
On Thu, Jun 19, 2014 at 2:30 PM, Shrivastava, Himnshu (GE Global Research,
Non-GE) <Hi...@ge.com> wrote:
> Can we take input in a mapper function and in some way pass it to
> reducer ?
>
>
>
> Regards,
>
Re: Map reduce Query
Posted by Kai Voigt <k...@123.org>.
That’s exactly what MapReduce does. The input is processed by the mapper function, and its output will be automatically sent into the reducer function. Between mappers and reducers we have the automatic shuffle phase which sends records with identical keys into one reducer call.
If you want to simply not modify the data through the map phase, just don’t specify the mapper class in your driver code. The identity function will be used for the map phase in this case.
Kai
Am 19.06.2014 um 11:00 schrieb Shrivastava, Himnshu (GE Global Research, Non-GE) <Hi...@ge.com>:
> Can we take input in a mapper function and in some way pass it to reducer ?
>
> Regards,
Kai Voigt Am Germaniahafen 1 k@123.org
24143 Kiel +49 160 96683050
Germany @KaiVoigt
Re: Map reduce Query
Posted by Raj K Singh <ra...@gmail.com>.
you can get the input from some source(e.g. files) in the mapper setup()
method and emit it to the context.write() so that it can reach to the
reducer.
::::::::::::::::::::::::::::::::::::::::
Raj K Singh
http://in.linkedin.com/in/rajkrrsingh
http://www.rajkrrsingh.blogspot.com
Mobile Tel: +91 (0)9899821370
On Thu, Jun 19, 2014 at 2:30 PM, Shrivastava, Himnshu (GE Global Research,
Non-GE) <Hi...@ge.com> wrote:
> Can we take input in a mapper function and in some way pass it to
> reducer ?
>
>
>
> Regards,
>
Re: Map reduce Query
Posted by Kai Voigt <k...@123.org>.
That’s exactly what MapReduce does. The input is processed by the mapper function, and its output will be automatically sent into the reducer function. Between mappers and reducers we have the automatic shuffle phase which sends records with identical keys into one reducer call.
If you want to simply not modify the data through the map phase, just don’t specify the mapper class in your driver code. The identity function will be used for the map phase in this case.
Kai
Am 19.06.2014 um 11:00 schrieb Shrivastava, Himnshu (GE Global Research, Non-GE) <Hi...@ge.com>:
> Can we take input in a mapper function and in some way pass it to reducer ?
>
> Regards,
Kai Voigt Am Germaniahafen 1 k@123.org
24143 Kiel +49 160 96683050
Germany @KaiVoigt
Re: Map reduce Query
Posted by Mohammad Tariq <do...@gmail.com>.
This is what we do. I'm sorry I didn't quite this.
We read in the data through Mapper, do some operations on it, and pass the
Mapper output on to the Reducer. If you intend to just pass the data as it
is then just context.write it without doing anything else.
*Warm regards,*
*Mohammad Tariq*
*cloudfront.blogspot.com <http://cloudfront.blogspot.com>*
On Thu, Jun 19, 2014 at 2:30 PM, Shrivastava, Himnshu (GE Global Research,
Non-GE) <Hi...@ge.com> wrote:
> Can we take input in a mapper function and in some way pass it to
> reducer ?
>
>
>
> Regards,
>