You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by parnab kumar <pa...@gmail.com> on 2013/06/14 06:41:46 UTC
how to design the mapper and reducer for the below problem
Consider a following input file of format :
input File :
1 2
2 3
3 4
6 7
7 9
10 11
The output Should be as follows :
1 2 3 4
6 7 9
10 11
Re: how to design the mapper and reducer for the below problem
Posted by Harsh J <ha...@cloudera.com>.
Hey Parnab,
Please checkout Giraph (http://giraph.apache.org), which should help
you develop a program to solve this.
On Fri, Jun 14, 2013 at 10:11 AM, parnab kumar <pa...@gmail.com> wrote:
> Consider a following input file of format :
> input File :
> 1 2
> 2 3
> 3 4
> 6 7
> 7 9
> 10 11
>
> The output Should be as follows :
>
> 1 2 3 4
> 6 7 9
> 10 11
--
Harsh J
Re: how to design the mapper and reducer for the below problem
Posted by Harsh J <ha...@cloudera.com>.
Hey Parnab,
Please checkout Giraph (http://giraph.apache.org), which should help
you develop a program to solve this.
On Fri, Jun 14, 2013 at 10:11 AM, parnab kumar <pa...@gmail.com> wrote:
> Consider a following input file of format :
> input File :
> 1 2
> 2 3
> 3 4
> 6 7
> 7 9
> 10 11
>
> The output Should be as follows :
>
> 1 2 3 4
> 6 7 9
> 10 11
--
Harsh J
RE: how to design the mapper and reducer for the below problem
Posted by John Lilley <jo...@redpoint.net>.
Sorry this is the link I meant: http://hortonworks.com/blog/transitive-closure-in-apache-pig/
john
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Sunday, June 16, 2013 1:02 PM
To: user@hadoop.apache.org
Subject: RE: how to design the mapper and reducer for the below problem
I don't think can be done in a single map/reduce pass.
Here the author discusses an implementation in PIG: http://techblug.wordpress.com/2011/08/07/transitive-closure-in-pig/
john
From: parnab kumar [mailto:parnab.2007@gmail.com]
Sent: Thursday, June 13, 2013 10:42 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: how to design the mapper and reducer for the below problem
Consider a following input file of format :
input File :
1 2
2 3
3 4
6 7
7 9
10 11
The output Should be as follows :
1 2 3 4
6 7 9
10 11
RE: how to design the mapper and reducer for the below problem
Posted by John Lilley <jo...@redpoint.net>.
Sorry this is the link I meant: http://hortonworks.com/blog/transitive-closure-in-apache-pig/
john
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Sunday, June 16, 2013 1:02 PM
To: user@hadoop.apache.org
Subject: RE: how to design the mapper and reducer for the below problem
I don't think can be done in a single map/reduce pass.
Here the author discusses an implementation in PIG: http://techblug.wordpress.com/2011/08/07/transitive-closure-in-pig/
john
From: parnab kumar [mailto:parnab.2007@gmail.com]
Sent: Thursday, June 13, 2013 10:42 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: how to design the mapper and reducer for the below problem
Consider a following input file of format :
input File :
1 2
2 3
3 4
6 7
7 9
10 11
The output Should be as follows :
1 2 3 4
6 7 9
10 11
RE: how to design the mapper and reducer for the below problem
Posted by John Lilley <jo...@redpoint.net>.
Sorry this is the link I meant: http://hortonworks.com/blog/transitive-closure-in-apache-pig/
john
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Sunday, June 16, 2013 1:02 PM
To: user@hadoop.apache.org
Subject: RE: how to design the mapper and reducer for the below problem
I don't think can be done in a single map/reduce pass.
Here the author discusses an implementation in PIG: http://techblug.wordpress.com/2011/08/07/transitive-closure-in-pig/
john
From: parnab kumar [mailto:parnab.2007@gmail.com]
Sent: Thursday, June 13, 2013 10:42 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: how to design the mapper and reducer for the below problem
Consider a following input file of format :
input File :
1 2
2 3
3 4
6 7
7 9
10 11
The output Should be as follows :
1 2 3 4
6 7 9
10 11
RE: how to design the mapper and reducer for the below problem
Posted by John Lilley <jo...@redpoint.net>.
Sorry this is the link I meant: http://hortonworks.com/blog/transitive-closure-in-apache-pig/
john
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Sunday, June 16, 2013 1:02 PM
To: user@hadoop.apache.org
Subject: RE: how to design the mapper and reducer for the below problem
I don't think can be done in a single map/reduce pass.
Here the author discusses an implementation in PIG: http://techblug.wordpress.com/2011/08/07/transitive-closure-in-pig/
john
From: parnab kumar [mailto:parnab.2007@gmail.com]
Sent: Thursday, June 13, 2013 10:42 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: how to design the mapper and reducer for the below problem
Consider a following input file of format :
input File :
1 2
2 3
3 4
6 7
7 9
10 11
The output Should be as follows :
1 2 3 4
6 7 9
10 11
RE: how to design the mapper and reducer for the below problem
Posted by John Lilley <jo...@redpoint.net>.
I don't think can be done in a single map/reduce pass.
Here the author discusses an implementation in PIG: http://techblug.wordpress.com/2011/08/07/transitive-closure-in-pig/
john
From: parnab kumar [mailto:parnab.2007@gmail.com]
Sent: Thursday, June 13, 2013 10:42 PM
To: user@hadoop.apache.org
Subject: how to design the mapper and reducer for the below problem
Consider a following input file of format :
input File :
1 2
2 3
3 4
6 7
7 9
10 11
The output Should be as follows :
1 2 3 4
6 7 9
10 11
Re: how to design the mapper and reducer for the below problem
Posted by Bhasker Allene <al...@gmail.com>.
That looks like graph algorithms using MapReduce.
Sorry couldn't give you specific answer!
On 14/06/2013 05:41, parnab kumar wrote:
> Consider a following input file of format :
> input File :
> 1 2
> 2 3
> 3 4
> 6 7
> 7 9
> 10 11
>
> The output Should be as follows :
>
> 1 2 3 4
> 6 7 9
> 10 11
Re: how to design the mapper and reducer for the below problem
Posted by Bhasker Allene <al...@gmail.com>.
That looks like graph algorithms using MapReduce.
Sorry couldn't give you specific answer!
On 14/06/2013 05:41, parnab kumar wrote:
> Consider a following input file of format :
> input File :
> 1 2
> 2 3
> 3 4
> 6 7
> 7 9
> 10 11
>
> The output Should be as follows :
>
> 1 2 3 4
> 6 7 9
> 10 11
Re: how to design the mapper and reducer for the below problem
Posted by Bhasker Allene <al...@gmail.com>.
That looks like graph algorithms using MapReduce.
Sorry couldn't give you specific answer!
On 14/06/2013 05:41, parnab kumar wrote:
> Consider a following input file of format :
> input File :
> 1 2
> 2 3
> 3 4
> 6 7
> 7 9
> 10 11
>
> The output Should be as follows :
>
> 1 2 3 4
> 6 7 9
> 10 11
RE: how to design the mapper and reducer for the below problem
Posted by John Lilley <jo...@redpoint.net>.
I don't think can be done in a single map/reduce pass.
Here the author discusses an implementation in PIG: http://techblug.wordpress.com/2011/08/07/transitive-closure-in-pig/
john
From: parnab kumar [mailto:parnab.2007@gmail.com]
Sent: Thursday, June 13, 2013 10:42 PM
To: user@hadoop.apache.org
Subject: how to design the mapper and reducer for the below problem
Consider a following input file of format :
input File :
1 2
2 3
3 4
6 7
7 9
10 11
The output Should be as follows :
1 2 3 4
6 7 9
10 11
Re: how to design the mapper and reducer for the below problem
Posted by Harsh J <ha...@cloudera.com>.
Hey Parnab,
Please checkout Giraph (http://giraph.apache.org), which should help
you develop a program to solve this.
On Fri, Jun 14, 2013 at 10:11 AM, parnab kumar <pa...@gmail.com> wrote:
> Consider a following input file of format :
> input File :
> 1 2
> 2 3
> 3 4
> 6 7
> 7 9
> 10 11
>
> The output Should be as follows :
>
> 1 2 3 4
> 6 7 9
> 10 11
--
Harsh J
Re: how to design the mapper and reducer for the below problem
Posted by Azuryy Yu <az...@gmail.com>.
This is a Graph problem, you want to find all joined sub graphs, so i don't
think it's easy using Map/Reduce.
but you can try Yarn, it can be iterated easily, at least compared with
M/R.
On Fri, Jun 14, 2013 at 12:41 PM, parnab kumar <pa...@gmail.com>wrote:
> Consider a following input file of format :
> input File :
> 1 2
> 2 3
> 3 4
> 6 7
> 7 9
> 10 11
>
> The output Should be as follows :
>
> 1 2 3 4
> 6 7 9
> 10 11
Re: how to design the mapper and reducer for the below problem
Posted by Azuryy Yu <az...@gmail.com>.
This is a Graph problem, you want to find all joined sub graphs, so i don't
think it's easy using Map/Reduce.
but you can try Yarn, it can be iterated easily, at least compared with
M/R.
On Fri, Jun 14, 2013 at 12:41 PM, parnab kumar <pa...@gmail.com>wrote:
> Consider a following input file of format :
> input File :
> 1 2
> 2 3
> 3 4
> 6 7
> 7 9
> 10 11
>
> The output Should be as follows :
>
> 1 2 3 4
> 6 7 9
> 10 11
RE: how to design the mapper and reducer for the below problem
Posted by John Lilley <jo...@redpoint.net>.
I don't think can be done in a single map/reduce pass.
Here the author discusses an implementation in PIG: http://techblug.wordpress.com/2011/08/07/transitive-closure-in-pig/
john
From: parnab kumar [mailto:parnab.2007@gmail.com]
Sent: Thursday, June 13, 2013 10:42 PM
To: user@hadoop.apache.org
Subject: how to design the mapper and reducer for the below problem
Consider a following input file of format :
input File :
1 2
2 3
3 4
6 7
7 9
10 11
The output Should be as follows :
1 2 3 4
6 7 9
10 11
Re: how to design the mapper and reducer for the below problem
Posted by Azuryy Yu <az...@gmail.com>.
This is a Graph problem, you want to find all joined sub graphs, so i don't
think it's easy using Map/Reduce.
but you can try Yarn, it can be iterated easily, at least compared with
M/R.
On Fri, Jun 14, 2013 at 12:41 PM, parnab kumar <pa...@gmail.com>wrote:
> Consider a following input file of format :
> input File :
> 1 2
> 2 3
> 3 4
> 6 7
> 7 9
> 10 11
>
> The output Should be as follows :
>
> 1 2 3 4
> 6 7 9
> 10 11
Re: how to design the mapper and reducer for the below problem
Posted by Harsh J <ha...@cloudera.com>.
Hey Parnab,
Please checkout Giraph (http://giraph.apache.org), which should help
you develop a program to solve this.
On Fri, Jun 14, 2013 at 10:11 AM, parnab kumar <pa...@gmail.com> wrote:
> Consider a following input file of format :
> input File :
> 1 2
> 2 3
> 3 4
> 6 7
> 7 9
> 10 11
>
> The output Should be as follows :
>
> 1 2 3 4
> 6 7 9
> 10 11
--
Harsh J
Re: how to design the mapper and reducer for the below problem
Posted by Bhasker Allene <al...@gmail.com>.
That looks like graph algorithms using MapReduce.
Sorry couldn't give you specific answer!
On 14/06/2013 05:41, parnab kumar wrote:
> Consider a following input file of format :
> input File :
> 1 2
> 2 3
> 3 4
> 6 7
> 7 9
> 10 11
>
> The output Should be as follows :
>
> 1 2 3 4
> 6 7 9
> 10 11
RE: how to design the mapper and reducer for the below problem
Posted by John Lilley <jo...@redpoint.net>.
I don't think can be done in a single map/reduce pass.
Here the author discusses an implementation in PIG: http://techblug.wordpress.com/2011/08/07/transitive-closure-in-pig/
john
From: parnab kumar [mailto:parnab.2007@gmail.com]
Sent: Thursday, June 13, 2013 10:42 PM
To: user@hadoop.apache.org
Subject: how to design the mapper and reducer for the below problem
Consider a following input file of format :
input File :
1 2
2 3
3 4
6 7
7 9
10 11
The output Should be as follows :
1 2 3 4
6 7 9
10 11
Re: how to design the mapper and reducer for the below problem
Posted by Azuryy Yu <az...@gmail.com>.
This is a Graph problem, you want to find all joined sub graphs, so i don't
think it's easy using Map/Reduce.
but you can try Yarn, it can be iterated easily, at least compared with
M/R.
On Fri, Jun 14, 2013 at 12:41 PM, parnab kumar <pa...@gmail.com>wrote:
> Consider a following input file of format :
> input File :
> 1 2
> 2 3
> 3 4
> 6 7
> 7 9
> 10 11
>
> The output Should be as follows :
>
> 1 2 3 4
> 6 7 9
> 10 11