You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by ZhiHong Fu <dd...@gmail.com> on 2008/10/27 07:27:49 UTC
help: InputFormat problem ?
Hello :
In hadoop InputFormat are always based on the InputFileFormat ,
But Now I will get data from a web service application. The data will be
wrapped as ResultSet type. Now I am wandering " should I write the
ResultSet to a file And then read out to do mapreduce job. Or How can I
process the ResultSet as Input of the MapReduce Job, and How should I define
the InputFormat" Any suggestions are welcome , thanks!
Re: help: InputFormat problem ?
Posted by ZhiHong Fu <dd...@gmail.com>.
I'm a little confused about the implemention of DBInputFormat. In my view ,
The method getSplits of DBInputFormat splits the resultset into serval
splits logically. so The DbRecordReader should process the DbSplit. But I
find in the real implement of DbRecordReader It process the resultset
instead of the DbSplit. I don't understand why. Thanks for any help.
2008/10/27 ZhiHong Fu <dd...@gmail.com>
>
>
> 2008/10/27 Owen O'Malley <om...@apache.org>
>
>> If your application that you are drawing from is doing some sort of web
>> crawl that connects to lots of random servers, you may want to use
>> MultiThreadedMapRunner and do the remote connections in the map. If you are
>> just connecting to a small set of servers for each map, you should put it in
>> the InputFormat.
>>
>> Using MultiThreadedMapRunner means that each map can have multiple threads
>> transferring data from the external sources.
>>
>> -- Owen
>>
>
>
Re: help: InputFormat problem ?
Posted by ZhiHong Fu <dd...@gmail.com>.
2008/10/27 Owen O'Malley <om...@apache.org>
> If your application that you are drawing from is doing some sort of web
> crawl that connects to lots of random servers, you may want to use
> MultiThreadedMapRunner and do the remote connections in the map. If you are
> just connecting to a small set of servers for each map, you should put it in
> the InputFormat.
>
> Using MultiThreadedMapRunner means that each map can have multiple threads
> transferring data from the external sources.
>
> -- Owen
>
Re: help: InputFormat problem ?
Posted by Owen O'Malley <om...@apache.org>.
If your application that you are drawing from is doing some sort of
web crawl that connects to lots of random servers, you may want to use
MultiThreadedMapRunner and do the remote connections in the map. If
you are just connecting to a small set of servers for each map, you
should put it in the InputFormat.
Using MultiThreadedMapRunner means that each map can have multiple
threads transferring data from the external sources.
-- Owen
Re: help: InputFormat problem ?
Posted by ZhiHong Fu <dd...@gmail.com>.
Thanks for your suggestion , I think writing a customized inputFormat like
DBInputFormat is what I need .
Regards
ddream
2008/10/27 Mice <mi...@gmail.com>
> I think both of your solution work.
> Saving the result into a file is trivial. In related to process the
> ResultSet as Input , you need to write your customized inputFormat.
>
> If your web service provides splitable results, it is similiar to read
> from a DB; you can check DBInputFormat in trunk
>
>
> http://svn.apache.org/viewvc/hadoop/core/trunk/src/mapred/org/apache/hadoop/mapred/lib/db/DBInputFormat.java?view=markup
>
> Regards
> Mice
>
> 2008/10/27 ZhiHong Fu <dd...@gmail.com>:
> > Hello :
> >
> > In hadoop InputFormat are always based on the InputFileFormat ,
> > But Now I will get data from a web service application. The data will be
> > wrapped as ResultSet type. Now I am wandering " should I write the
> > ResultSet to a file And then read out to do mapreduce job. Or How can I
> > process the ResultSet as Input of the MapReduce Job, and How should I
> define
> > the InputFormat" Any suggestions are welcome , thanks!
> >
>
Re: help: InputFormat problem ?
Posted by Mice <mi...@gmail.com>.
I think both of your solution work.
Saving the result into a file is trivial. In related to process the
ResultSet as Input , you need to write your customized inputFormat.
If your web service provides splitable results, it is similiar to read
from a DB; you can check DBInputFormat in trunk
http://svn.apache.org/viewvc/hadoop/core/trunk/src/mapred/org/apache/hadoop/mapred/lib/db/DBInputFormat.java?view=markup
Regards
Mice
2008/10/27 ZhiHong Fu <dd...@gmail.com>:
> Hello :
>
> In hadoop InputFormat are always based on the InputFileFormat ,
> But Now I will get data from a web service application. The data will be
> wrapped as ResultSet type. Now I am wandering " should I write the
> ResultSet to a file And then read out to do mapreduce job. Or How can I
> process the ResultSet as Input of the MapReduce Job, and How should I define
> the InputFormat" Any suggestions are welcome , thanks!
>