You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Rakhi Khatwani <rk...@gmail.com> on 2009/09/10 08:16:12 UTC

Reading a subset of records from hdfs

Hi,
       Suppose i have a hdfs file with 10,000 entries. and i want my job to
process 100 records at one time (to minimize loss of data during job
crashes/ network errors etc). so if a job can read a subset of records from
a fine in HDFS, i can combine with chaining to achieve my objective.  for
example i have job1 which reads 1-100 lines of input from hdfs, and job 2
which reads from 101-200 lines of input...etc.
 is there a way in which you can configure a job 2 read only a subset of
records from a file in HDFS.
Regards,
Raakhi

Re: Reading a subset of records from hdfs

Posted by Wasim Bari <wa...@msn.com>.
Hi,
    I also have similar kind of question.

Is it possible for a job to start reading a file ( start split ) from a 
specific position in file rather than beginning. Idea is I have some 
information in a file, first part of information can only be read in 
sequence not parallel, so I get this data with isSplitable= false. Now when 
I have this data next part of search can be made parallel. So its better to 
start searching from this point onward in parallel with multiple mappers ( 
isSiplitable = true ).

BR,
Bari

--------------------------------------------------
From: "Amandeep Khurana" <am...@gmail.com>
Sent: Thursday, September 10, 2009 9:49 AM
To: <co...@hadoop.apache.org>
Subject: Re: Reading a subset of records from hdfs

> Why not just have a higher number of mappers? Why split into multiple
> jobs? Any particular case that you think this will be useful in?
>
> On 9/9/09, Rakhi Khatwani <rk...@gmail.com> wrote:
>> Hi,
>>        Suppose i have a hdfs file with 10,000 entries. and i want my job 
>> to
>> process 100 records at one time (to minimize loss of data during job
>> crashes/ network errors etc). so if a job can read a subset of records 
>> from
>> a fine in HDFS, i can combine with chaining to achieve my objective.  for
>> example i have job1 which reads 1-100 lines of input from hdfs, and job 2
>> which reads from 101-200 lines of input...etc.
>>  is there a way in which you can configure a job 2 read only a subset of
>> records from a file in HDFS.
>> Regards,
>> Raakhi
>>
>
>
> -- 
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
> 

Re: Reading a subset of records from hdfs

Posted by Amandeep Khurana <am...@gmail.com>.
Why not just have a higher number of mappers? Why split into multiple
jobs? Any particular case that you think this will be useful in?

On 9/9/09, Rakhi Khatwani <rk...@gmail.com> wrote:
> Hi,
>        Suppose i have a hdfs file with 10,000 entries. and i want my job to
> process 100 records at one time (to minimize loss of data during job
> crashes/ network errors etc). so if a job can read a subset of records from
> a fine in HDFS, i can combine with chaining to achieve my objective.  for
> example i have job1 which reads 1-100 lines of input from hdfs, and job 2
> which reads from 101-200 lines of input...etc.
>  is there a way in which you can configure a job 2 read only a subset of
> records from a file in HDFS.
> Regards,
> Raakhi
>


-- 


Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz