You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by HRoger <ha...@163.com> on 2009/06/15 15:30:56 UTC
how to transfer data from one reduce to another map
Hi !
I write a application which has two jobs: the second job use the input
datasource same as the first job's added the the output(some objects) of
first job.Can I transfer some objects from one job to another job or make
the job has two input source?
--
View this message in context: http://www.nabble.com/how-to-transfer-data-from-one-reduce-to-another-map-tp24034706p24034706.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: how to transfer data from one reduce to another map
Posted by Aaron Kimball <aa...@cloudera.com>.
You can add multiple paths to the same job -- FileInputFormat.addInputPath()
can be called multiple times.
- Aaron
On Mon, Jun 15, 2009 at 6:56 AM, bharath vissapragada <
bharathvissapragada1990@gmail.com> wrote:
> if your doubt is related to chaining of mapreduce jobs .. then this link
> might be useful ...
>
> http://developer.yahoo.com/hadoop/tutorial/module4.html#chaining
>
> On Mon, Jun 15, 2009 at 7:22 PM, HRoger <ha...@163.com> wrote:
>
> >
> > I'm sorry for my confusing description,It is job2 has to use two input
> > source
> > one from job1's output and another from anywhere.
> >
> > TimRobertson100 wrote:
> > >
> > > Hi
> > >
> > > I am not sure I understand the question correctly. If you mean you
> > > want to use the output of Job1 as the input of Job2, then you can set
> > > the input path to the second job as the output path (e.g. output
> > > directory) from the first job.
> > >
> > > Cheers
> > >
> > > Tim
> > >
> > >
> > > On Mon, Jun 15, 2009 at 3:30 PM, HRoger<ha...@163.com>
> wrote:
> > >>
> > >> Hi !
> > >> I write a application which has two jobs: the second job use the input
> > >> datasource same as the first job's added the the output(some objects)
> of
> > >> first job.Can I transfer some objects from one job to another job or
> > make
> > >> the job has two input source?
> > >> --
> > >> View this message in context:
> > >>
> >
> http://www.nabble.com/how-to-transfer-data-from-one-reduce-to-another-map-tp24034706p24034706.html
> > >> Sent from the Hadoop core-user mailing list archive at Nabble.com.
> > >>
> > >>
> > >
> > >
> >
> > --
> > View this message in context:
> >
> http://www.nabble.com/how-to-transfer-data-from-one-reduce-to-another-map-tp24034706p24035057.html
> > Sent from the Hadoop core-user mailing list archive at Nabble.com.
> >
> >
>
Re: how to transfer data from one reduce to another map
Posted by bharath vissapragada <bh...@gmail.com>.
if your doubt is related to chaining of mapreduce jobs .. then this link
might be useful ...
http://developer.yahoo.com/hadoop/tutorial/module4.html#chaining
On Mon, Jun 15, 2009 at 7:22 PM, HRoger <ha...@163.com> wrote:
>
> I'm sorry for my confusing description,It is job2 has to use two input
> source
> one from job1's output and another from anywhere.
>
> TimRobertson100 wrote:
> >
> > Hi
> >
> > I am not sure I understand the question correctly. If you mean you
> > want to use the output of Job1 as the input of Job2, then you can set
> > the input path to the second job as the output path (e.g. output
> > directory) from the first job.
> >
> > Cheers
> >
> > Tim
> >
> >
> > On Mon, Jun 15, 2009 at 3:30 PM, HRoger<ha...@163.com> wrote:
> >>
> >> Hi !
> >> I write a application which has two jobs: the second job use the input
> >> datasource same as the first job's added the the output(some objects) of
> >> first job.Can I transfer some objects from one job to another job or
> make
> >> the job has two input source?
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/how-to-transfer-data-from-one-reduce-to-another-map-tp24034706p24034706.html
> >> Sent from the Hadoop core-user mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/how-to-transfer-data-from-one-reduce-to-another-map-tp24034706p24035057.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>
Re: how to transfer data from one reduce to another map
Posted by Nick Cen <ce...@gmail.com>.
If all the file in different path has the same format, then FileInputFormat
is sufficient.
If files in different path has different format , then I guest you may take
a look at the MultipleInputs.
2009/6/15 HRoger <ha...@163.com>
>
> I'm sorry for my confusing description,It is job2 has to use two input
> source
> one from job1's output and another from anywhere.
>
> TimRobertson100 wrote:
> >
> > Hi
> >
> > I am not sure I understand the question correctly. If you mean you
> > want to use the output of Job1 as the input of Job2, then you can set
> > the input path to the second job as the output path (e.g. output
> > directory) from the first job.
> >
> > Cheers
> >
> > Tim
> >
> >
> > On Mon, Jun 15, 2009 at 3:30 PM, HRoger<ha...@163.com> wrote:
> >>
> >> Hi !
> >> I write a application which has two jobs: the second job use the input
> >> datasource same as the first job's added the the output(some objects) of
> >> first job.Can I transfer some objects from one job to another job or
> make
> >> the job has two input source?
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/how-to-transfer-data-from-one-reduce-to-another-map-tp24034706p24034706.html
> >> Sent from the Hadoop core-user mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/how-to-transfer-data-from-one-reduce-to-another-map-tp24034706p24035057.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>
--
http://daily.appspot.com/food/
Re: how to transfer data from one reduce to another map
Posted by HRoger <ha...@163.com>.
I'm sorry for my confusing description,It is job2 has to use two input source
one from job1's output and another from anywhere.
TimRobertson100 wrote:
>
> Hi
>
> I am not sure I understand the question correctly. If you mean you
> want to use the output of Job1 as the input of Job2, then you can set
> the input path to the second job as the output path (e.g. output
> directory) from the first job.
>
> Cheers
>
> Tim
>
>
> On Mon, Jun 15, 2009 at 3:30 PM, HRoger<ha...@163.com> wrote:
>>
>> Hi !
>> I write a application which has two jobs: the second job use the input
>> datasource same as the first job's added the the output(some objects) of
>> first job.Can I transfer some objects from one job to another job or make
>> the job has two input source?
>> --
>> View this message in context:
>> http://www.nabble.com/how-to-transfer-data-from-one-reduce-to-another-map-tp24034706p24034706.html
>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/how-to-transfer-data-from-one-reduce-to-another-map-tp24034706p24035057.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: how to transfer data from one reduce to another map
Posted by tim robertson <ti...@gmail.com>.
Hi
I am not sure I understand the question correctly. If you mean you
want to use the output of Job1 as the input of Job2, then you can set
the input path to the second job as the output path (e.g. output
directory) from the first job.
Cheers
Tim
On Mon, Jun 15, 2009 at 3:30 PM, HRoger<ha...@163.com> wrote:
>
> Hi !
> I write a application which has two jobs: the second job use the input
> datasource same as the first job's added the the output(some objects) of
> first job.Can I transfer some objects from one job to another job or make
> the job has two input source?
> --
> View this message in context: http://www.nabble.com/how-to-transfer-data-from-one-reduce-to-another-map-tp24034706p24034706.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>