You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by HARIPRIYA AYYALASOMAYAJULA <ah...@gmail.com> on 2014/10/20 18:31:29 UTC
How to emit multiple keys for the same value?
Hello,
I am facing a problem with implementing this - My mapper should emit
multiple keys for the same value -> for every input (k, v) it should emit
(k, v), (k+1, v),(k+2,v).... (k+n,v).
In MapReduce, it was pretty straight forward - I used a for loop and
performed Context write within that.
This is the code I am using in the map function of MR job:
for (int k = 0; k < 8; k++)
{
key = date + ": " + k;
context.write(new Text(key), new DoubleWritable(value));
}
I tried using the map function, but i'm stuck in understanding how I can
use the loop within it.
It would be great if someone can suggest me what could be possible ways to
do it.
Thanks in advance.
--
Regards,
Haripriya Ayyalasomayajula
Re: How to emit multiple keys for the same value?
Posted by Boromir Widas <vc...@gmail.com>.
flatMap should help, it returns a Seq for every input.
On Mon, Oct 20, 2014 at 12:31 PM, HARIPRIYA AYYALASOMAYAJULA <
aharipriya92@gmail.com> wrote:
> Hello,
>
> I am facing a problem with implementing this - My mapper should emit
> multiple keys for the same value -> for every input (k, v) it should emit
> (k, v), (k+1, v),(k+2,v).... (k+n,v).
> In MapReduce, it was pretty straight forward - I used a for loop and
> performed Context write within that.
>
> This is the code I am using in the map function of MR job:
>
> for (int k = 0; k < 8; k++)
>
> {
>
> key = date + ": " + k;
>
> context.write(new Text(key), new DoubleWritable(value));
>
> }
>
> I tried using the map function, but i'm stuck in understanding how I can
> use the loop within it.
>
> It would be great if someone can suggest me what could be possible ways to
> do it.
>
> Thanks in advance.
>
> --
> Regards,
> Haripriya Ayyalasomayajula
>
>
Re: How to emit multiple keys for the same value?
Posted by DB Tsai <db...@dbtsai.com>.
You can do this using flatMap which return a Seq of (key, value) pairs.
Sincerely,
DB Tsai
-------------------------------------------------------
My Blog: https://www.dbtsai.com
LinkedIn: https://www.linkedin.com/in/dbtsai
On Mon, Oct 20, 2014 at 9:31 AM, HARIPRIYA AYYALASOMAYAJULA <
aharipriya92@gmail.com> wrote:
> Hello,
>
> I am facing a problem with implementing this - My mapper should emit
> multiple keys for the same value -> for every input (k, v) it should emit
> (k, v), (k+1, v),(k+2,v).... (k+n,v).
> In MapReduce, it was pretty straight forward - I used a for loop and
> performed Context write within that.
>
> This is the code I am using in the map function of MR job:
>
> for (int k = 0; k < 8; k++)
>
> {
>
> key = date + ": " + k;
>
> context.write(new Text(key), new DoubleWritable(value));
>
> }
>
> I tried using the map function, but i'm stuck in understanding how I can
> use the loop within it.
>
> It would be great if someone can suggest me what could be possible ways to
> do it.
>
> Thanks in advance.
>
> --
> Regards,
> Haripriya Ayyalasomayajula
>
>