You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Kayla Jay <ka...@yahoo.com> on 2008/04/18 14:42:27 UTC
Map Intermediate key/value pairs written to file system
Hi
I have no reduces. I would like to directly write my map results while they are produced after each map has completed to disk. I don't want to collect then write to output.
If I wanted to directly write my map output 1-by-1 (intermediate key/value pairs) after each map completes into individual files instead of collecting them until the end then writing them in 1 swoop into the composite results file (part-000X), is this possible and how do I do that?
Can I force a write within the map to write the map key/value pairs as an individual file for each results set instead of output.collect and having them all the key/value pairs written to the output?
I.e I would like the intermediate key/value pairs produced from the maps to write to disk immediatly than having it to collect in the end all of the key/value pairs and writing it out. I want individual files per key/value pair produced.
Thanks.
---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now.
Re: Map Intermediate key/value pairs written to file system
Posted by Kayla Jay <ka...@yahoo.com>.
Here's what I am trying to do.
I get a set of huge text files that has many XML docs contained within one file. Now, within each of those xml docs within that one huge text file, it has some junk in the xml, so I want to clean it up basically.
My custom input file format doesn't split the file. I don't know where you could know how to break up an xml document for the input splits. So 1 file = 1 mapper in my custom case.
I have a custom reader that breaks the file down into the individual xml records contained within that big file as the key/value pair for the mapper to handle. So, the key/value would be key (doesn't matter) and the value itself is the xml doc snippet.
Then, for each key/value pair that is an individual xml record passed to the map function, I would like to clean the xml up then produce a new xml document that has all the clean stuff I want.
So, I want to create individual xml documents from that one huge file that has them all in it only after I have cleaned out the junk in them.
I want to produce individual xml documents from that one huge file that contains many multiple xml docs within it.
Am I going about this wrong?
There is no common "key" between the individual xml documents. They are just a bunch of xml docs.
Ted Dunning <td...@veoh.com> wrote:
Having one file per key value has several bad consequences:
A) you are essentially doing a sort, but inefficiently. Better to have the
framework do it well even if you really want this.
B) having one file per key leads (often, not always) to having many files.
This means that subsequent steps are likely to be very inefficient since
they will not be able to read contiguous disk regions. This also affects
write speed.
On 4/18/08 11:32 AM, "Kayla Jay" wrote:
>
> I do need one file per key/value. I don't want to sit and collect then
> create the output file.
>
> Is this something bad based on this misguidance?
>
> I can explain what I am doing if you think I am way off in what I am trying to
> achieve.
---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now.
Re: Map Intermediate key/value pairs written to file system
Posted by Ted Dunning <td...@veoh.com>.
Having one file per key value has several bad consequences:
A) you are essentially doing a sort, but inefficiently. Better to have the
framework do it well even if you really want this.
B) having one file per key leads (often, not always) to having many files.
This means that subsequent steps are likely to be very inefficient since
they will not be able to read contiguous disk regions. This also affects
write speed.
On 4/18/08 11:32 AM, "Kayla Jay" <ka...@yahoo.com> wrote:
>
> I do need one file per key/value. I don't want to sit and collect then
> create the output file.
>
> Is this something bad based on this misguidance?
>
> I can explain what I am doing if you think I am way off in what I am trying to
> achieve.
RE: Map Intermediate key/value pairs written to file system
Posted by Kayla Jay <ka...@yahoo.com>.
Hi.
Thanks for all of your responses -- it has been very helpful indeed. I am working on the filename as suggested and will ask more questions if I get stuck.
I do need one file per key/value. I don't want to sit and collect then create the output file.
Is this something bad based on this misguidance?
I can explain what I am doing if you think I am way off in what I am trying to achieve.
Please advise.
Thanks.
Devaraj Das <dd...@yahoo-inc.com> wrote: Well.. Kayla specifically mentioned that he wants one file per key/value..
Kayla should clarify this..
> -----Original Message-----
> From: Ted Dunning [mailto:tdunning@veoh.com]
> Sent: Friday, April 18, 2008 11:54 PM
> To: core-user@hadoop.apache.org
> Subject: Re: Map Intermediate key/value pairs written to file system
>
>
> Yes, but Kayla is likely misguided in this respect.
>
> (my apologies for sounding doctrinaire)
>
>
> On 4/18/08 11:08 AM, "Devaraj Das"
wrote:
>
> > Ted, note that Kayla wants one file per output key/value.
> >
> >> -----Original Message-----
> >> From: Ted Dunning [mailto:tdunning@veoh.com]
> >> Sent: Friday, April 18, 2008 11:20 PM
> >> To: core-user@hadoop.apache.org
> >> Subject: Re: Map Intermediate key/value pairs written to
> file system
> >>
> >>
> >> Isn't this just what Hadoop does when you set numReduces = 0?
> >>
> >>
> >> On 4/18/08 10:45 AM, "Devaraj Das"
wrote:
> >>
> >>> Within a task you can get the taskId (which are unique). Define
> >>> "public void configure(JobConf job)" and in that get the
> taskId by
> >>> doing
> >>> job.get("mapred.task.id") ).
> >>>
> >>> Now create filenames starting with that as the prefix and maybe a
> >>> monotonically increasing integer as the suffix (defined
> as a static
> >>> field in the task)..
> >>>
> >>>> -----Original Message-----
> >>>> From: Kayla Jay [mailto:kaylais30@yahoo.com]
> >>>> Sent: Friday, April 18, 2008 10:46 PM
> >>>> To: core-user@hadoop.apache.org
> >>>> Subject: RE: Map Intermediate key/value pairs written to
> >> file system
> >>>>
> >>>> Hi.
> >>>>
> >>>> I don't know how to create unique individual file names for each
> >>>> mapper's key/value pairs. How do you create individual
> files per
> >>>> mappers key/value pairs so they don't overwrite one another?
> >>>>
> >>>> I.e how do you create a new file each time and use that
> >> code for all
> >>>> the mappers and not have each of the mappers trying to
> >> overwrite the
> >>>> other's output? If that makes any sense?
> >>>>
> >>>> For example, at end of map(), i create sequence file for
> output of
> >>>> the key/value pair. But, if another mapper is running
> and it does
> >>>> the same thing, the file gets overwritten b/c this other
> mapper is
> >>>> creating the exact file name and doing the exact same
> thing as the
> >>>> other mapper is doing.
> >>>>
> >>>> Devaraj Das
wrote: Will your requirement be
> >>>> addressed if, from within the map method, you create a
> >> sequence file
> >>>> using SequenceFile.createWriter api, write a
> >>>> key/value using the writer's append(key,value) API and then
> >>>> close the file
> >>>> ? You can do this for every key/value.
> >>>> Pls have a look at createWriter APIs and the Writer class in
> >>>> o.a.h.i.SequenceFile..
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Kayla Jay [mailto:kaylais30@yahoo.com]
> >>>>> Sent: Friday, April 18, 2008 6:12 PM
> >>>>> To: core-user@hadoop.apache.org
> >>>>> Subject: Map Intermediate key/value pairs written to file system
> >>>>>
> >>>>> Hi
> >>>>>
> >>>>> I have no reduces. I would like to directly write my map
> >>>> results while
> >>>>> they are produced after each map has completed to disk. I
> >>>> don't want
> >>>>> to collect then write to output.
> >>>>>
> >>>>> If I wanted to directly write my map output 1-by-1
> (intermediate
> >>>>> key/value pairs) after each map completes into individual files
> >>>>> instead of collecting them until the end then writing them
> >>>> in 1 swoop
> >>>>> into the composite results file (part-000X), is this
> >>>> possible and how
> >>>>> do I do that?
> >>>>>
> >>>>> Can I force a write within the map to write the map
> >>>> key/value pairs as
> >>>>> an individual file for each results set instead of
> >>>> output.collect and
> >>>>> having them all the key/value pairs written to the output?
> >>>>>
> >>>>> I.e I would like the intermediate key/value pairs
> >> produced from the
> >>>>> maps to write to disk immediatly than having it to collect
> >>>> in the end
> >>>>> all of the key/value pairs and writing it out. I want
> individual
> >>>>> files per key/value pair produced.
> >>>>>
> >>>>> Thanks.
> >>>>>
> >>>>>
> >>>>> ---------------------------------
> >>>>> Be a better friend, newshound, and know-it-all with Yahoo!
> >>>>> Mobile. Try it now.
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> ---------------------------------
> >>>> Be a better friend, newshound, and know-it-all with Yahoo!
> >>>> Mobile. Try it now.
> >>>>
> >>>
> >>
> >>
> >
>
>
---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now.
RE: Map Intermediate key/value pairs written to file system
Posted by Devaraj Das <dd...@yahoo-inc.com>.
Well.. Kayla specifically mentioned that he wants one file per key/value..
Kayla should clarify this..
> -----Original Message-----
> From: Ted Dunning [mailto:tdunning@veoh.com]
> Sent: Friday, April 18, 2008 11:54 PM
> To: core-user@hadoop.apache.org
> Subject: Re: Map Intermediate key/value pairs written to file system
>
>
> Yes, but Kayla is likely misguided in this respect.
>
> (my apologies for sounding doctrinaire)
>
>
> On 4/18/08 11:08 AM, "Devaraj Das" <dd...@yahoo-inc.com> wrote:
>
> > Ted, note that Kayla wants one file per output key/value.
> >
> >> -----Original Message-----
> >> From: Ted Dunning [mailto:tdunning@veoh.com]
> >> Sent: Friday, April 18, 2008 11:20 PM
> >> To: core-user@hadoop.apache.org
> >> Subject: Re: Map Intermediate key/value pairs written to
> file system
> >>
> >>
> >> Isn't this just what Hadoop does when you set numReduces = 0?
> >>
> >>
> >> On 4/18/08 10:45 AM, "Devaraj Das" <dd...@yahoo-inc.com> wrote:
> >>
> >>> Within a task you can get the taskId (which are unique). Define
> >>> "public void configure(JobConf job)" and in that get the
> taskId by
> >>> doing
> >>> job.get("mapred.task.id") ).
> >>>
> >>> Now create filenames starting with that as the prefix and maybe a
> >>> monotonically increasing integer as the suffix (defined
> as a static
> >>> field in the task)..
> >>>
> >>>> -----Original Message-----
> >>>> From: Kayla Jay [mailto:kaylais30@yahoo.com]
> >>>> Sent: Friday, April 18, 2008 10:46 PM
> >>>> To: core-user@hadoop.apache.org
> >>>> Subject: RE: Map Intermediate key/value pairs written to
> >> file system
> >>>>
> >>>> Hi.
> >>>>
> >>>> I don't know how to create unique individual file names for each
> >>>> mapper's key/value pairs. How do you create individual
> files per
> >>>> mappers key/value pairs so they don't overwrite one another?
> >>>>
> >>>> I.e how do you create a new file each time and use that
> >> code for all
> >>>> the mappers and not have each of the mappers trying to
> >> overwrite the
> >>>> other's output? If that makes any sense?
> >>>>
> >>>> For example, at end of map(), i create sequence file for
> output of
> >>>> the key/value pair. But, if another mapper is running
> and it does
> >>>> the same thing, the file gets overwritten b/c this other
> mapper is
> >>>> creating the exact file name and doing the exact same
> thing as the
> >>>> other mapper is doing.
> >>>>
> >>>> Devaraj Das <dd...@yahoo-inc.com> wrote: Will your requirement be
> >>>> addressed if, from within the map method, you create a
> >> sequence file
> >>>> using SequenceFile.createWriter api, write a
> >>>> key/value using the writer's append(key,value) API and then
> >>>> close the file
> >>>> ? You can do this for every key/value.
> >>>> Pls have a look at createWriter APIs and the Writer class in
> >>>> o.a.h.i.SequenceFile..
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Kayla Jay [mailto:kaylais30@yahoo.com]
> >>>>> Sent: Friday, April 18, 2008 6:12 PM
> >>>>> To: core-user@hadoop.apache.org
> >>>>> Subject: Map Intermediate key/value pairs written to file system
> >>>>>
> >>>>> Hi
> >>>>>
> >>>>> I have no reduces. I would like to directly write my map
> >>>> results while
> >>>>> they are produced after each map has completed to disk. I
> >>>> don't want
> >>>>> to collect then write to output.
> >>>>>
> >>>>> If I wanted to directly write my map output 1-by-1
> (intermediate
> >>>>> key/value pairs) after each map completes into individual files
> >>>>> instead of collecting them until the end then writing them
> >>>> in 1 swoop
> >>>>> into the composite results file (part-000X), is this
> >>>> possible and how
> >>>>> do I do that?
> >>>>>
> >>>>> Can I force a write within the map to write the map
> >>>> key/value pairs as
> >>>>> an individual file for each results set instead of
> >>>> output.collect and
> >>>>> having them all the key/value pairs written to the output?
> >>>>>
> >>>>> I.e I would like the intermediate key/value pairs
> >> produced from the
> >>>>> maps to write to disk immediatly than having it to collect
> >>>> in the end
> >>>>> all of the key/value pairs and writing it out. I want
> individual
> >>>>> files per key/value pair produced.
> >>>>>
> >>>>> Thanks.
> >>>>>
> >>>>>
> >>>>> ---------------------------------
> >>>>> Be a better friend, newshound, and know-it-all with Yahoo!
> >>>>> Mobile. Try it now.
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> ---------------------------------
> >>>> Be a better friend, newshound, and know-it-all with Yahoo!
> >>>> Mobile. Try it now.
> >>>>
> >>>
> >>
> >>
> >
>
>
Re: Map Intermediate key/value pairs written to file system
Posted by Ted Dunning <td...@veoh.com>.
Yes, but Kayla is likely misguided in this respect.
(my apologies for sounding doctrinaire)
On 4/18/08 11:08 AM, "Devaraj Das" <dd...@yahoo-inc.com> wrote:
> Ted, note that Kayla wants one file per output key/value.
>
>> -----Original Message-----
>> From: Ted Dunning [mailto:tdunning@veoh.com]
>> Sent: Friday, April 18, 2008 11:20 PM
>> To: core-user@hadoop.apache.org
>> Subject: Re: Map Intermediate key/value pairs written to file system
>>
>>
>> Isn't this just what Hadoop does when you set numReduces = 0?
>>
>>
>> On 4/18/08 10:45 AM, "Devaraj Das" <dd...@yahoo-inc.com> wrote:
>>
>>> Within a task you can get the taskId (which are unique). Define
>>> "public void configure(JobConf job)" and in that get the taskId by
>>> doing
>>> job.get("mapred.task.id") ).
>>>
>>> Now create filenames starting with that as the prefix and maybe a
>>> monotonically increasing integer as the suffix (defined as a static
>>> field in the task)..
>>>
>>>> -----Original Message-----
>>>> From: Kayla Jay [mailto:kaylais30@yahoo.com]
>>>> Sent: Friday, April 18, 2008 10:46 PM
>>>> To: core-user@hadoop.apache.org
>>>> Subject: RE: Map Intermediate key/value pairs written to
>> file system
>>>>
>>>> Hi.
>>>>
>>>> I don't know how to create unique individual file names for each
>>>> mapper's key/value pairs. How do you create individual files per
>>>> mappers key/value pairs so they don't overwrite one another?
>>>>
>>>> I.e how do you create a new file each time and use that
>> code for all
>>>> the mappers and not have each of the mappers trying to
>> overwrite the
>>>> other's output? If that makes any sense?
>>>>
>>>> For example, at end of map(), i create sequence file for output of
>>>> the key/value pair. But, if another mapper is running and it does
>>>> the same thing, the file gets overwritten b/c this other mapper is
>>>> creating the exact file name and doing the exact same thing as the
>>>> other mapper is doing.
>>>>
>>>> Devaraj Das <dd...@yahoo-inc.com> wrote: Will your requirement be
>>>> addressed if, from within the map method, you create a
>> sequence file
>>>> using SequenceFile.createWriter api, write a
>>>> key/value using the writer's append(key,value) API and then
>>>> close the file
>>>> ? You can do this for every key/value.
>>>> Pls have a look at createWriter APIs and the Writer class in
>>>> o.a.h.i.SequenceFile..
>>>>
>>>>> -----Original Message-----
>>>>> From: Kayla Jay [mailto:kaylais30@yahoo.com]
>>>>> Sent: Friday, April 18, 2008 6:12 PM
>>>>> To: core-user@hadoop.apache.org
>>>>> Subject: Map Intermediate key/value pairs written to file system
>>>>>
>>>>> Hi
>>>>>
>>>>> I have no reduces. I would like to directly write my map
>>>> results while
>>>>> they are produced after each map has completed to disk. I
>>>> don't want
>>>>> to collect then write to output.
>>>>>
>>>>> If I wanted to directly write my map output 1-by-1 (intermediate
>>>>> key/value pairs) after each map completes into individual files
>>>>> instead of collecting them until the end then writing them
>>>> in 1 swoop
>>>>> into the composite results file (part-000X), is this
>>>> possible and how
>>>>> do I do that?
>>>>>
>>>>> Can I force a write within the map to write the map
>>>> key/value pairs as
>>>>> an individual file for each results set instead of
>>>> output.collect and
>>>>> having them all the key/value pairs written to the output?
>>>>>
>>>>> I.e I would like the intermediate key/value pairs
>> produced from the
>>>>> maps to write to disk immediatly than having it to collect
>>>> in the end
>>>>> all of the key/value pairs and writing it out. I want individual
>>>>> files per key/value pair produced.
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> ---------------------------------
>>>>> Be a better friend, newshound, and know-it-all with Yahoo!
>>>>> Mobile. Try it now.
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------
>>>> Be a better friend, newshound, and know-it-all with Yahoo!
>>>> Mobile. Try it now.
>>>>
>>>
>>
>>
>
RE: Map Intermediate key/value pairs written to file system
Posted by Devaraj Das <dd...@yahoo-inc.com>.
Ted, note that Kayla wants one file per output key/value.
> -----Original Message-----
> From: Ted Dunning [mailto:tdunning@veoh.com]
> Sent: Friday, April 18, 2008 11:20 PM
> To: core-user@hadoop.apache.org
> Subject: Re: Map Intermediate key/value pairs written to file system
>
>
> Isn't this just what Hadoop does when you set numReduces = 0?
>
>
> On 4/18/08 10:45 AM, "Devaraj Das" <dd...@yahoo-inc.com> wrote:
>
> > Within a task you can get the taskId (which are unique). Define
> > "public void configure(JobConf job)" and in that get the taskId by
> > doing
> > job.get("mapred.task.id") ).
> >
> > Now create filenames starting with that as the prefix and maybe a
> > monotonically increasing integer as the suffix (defined as a static
> > field in the task)..
> >
> >> -----Original Message-----
> >> From: Kayla Jay [mailto:kaylais30@yahoo.com]
> >> Sent: Friday, April 18, 2008 10:46 PM
> >> To: core-user@hadoop.apache.org
> >> Subject: RE: Map Intermediate key/value pairs written to
> file system
> >>
> >> Hi.
> >>
> >> I don't know how to create unique individual file names for each
> >> mapper's key/value pairs. How do you create individual files per
> >> mappers key/value pairs so they don't overwrite one another?
> >>
> >> I.e how do you create a new file each time and use that
> code for all
> >> the mappers and not have each of the mappers trying to
> overwrite the
> >> other's output? If that makes any sense?
> >>
> >> For example, at end of map(), i create sequence file for output of
> >> the key/value pair. But, if another mapper is running and it does
> >> the same thing, the file gets overwritten b/c this other mapper is
> >> creating the exact file name and doing the exact same thing as the
> >> other mapper is doing.
> >>
> >> Devaraj Das <dd...@yahoo-inc.com> wrote: Will your requirement be
> >> addressed if, from within the map method, you create a
> sequence file
> >> using SequenceFile.createWriter api, write a
> >> key/value using the writer's append(key,value) API and then
> >> close the file
> >> ? You can do this for every key/value.
> >> Pls have a look at createWriter APIs and the Writer class in
> >> o.a.h.i.SequenceFile..
> >>
> >>> -----Original Message-----
> >>> From: Kayla Jay [mailto:kaylais30@yahoo.com]
> >>> Sent: Friday, April 18, 2008 6:12 PM
> >>> To: core-user@hadoop.apache.org
> >>> Subject: Map Intermediate key/value pairs written to file system
> >>>
> >>> Hi
> >>>
> >>> I have no reduces. I would like to directly write my map
> >> results while
> >>> they are produced after each map has completed to disk. I
> >> don't want
> >>> to collect then write to output.
> >>>
> >>> If I wanted to directly write my map output 1-by-1 (intermediate
> >>> key/value pairs) after each map completes into individual files
> >>> instead of collecting them until the end then writing them
> >> in 1 swoop
> >>> into the composite results file (part-000X), is this
> >> possible and how
> >>> do I do that?
> >>>
> >>> Can I force a write within the map to write the map
> >> key/value pairs as
> >>> an individual file for each results set instead of
> >> output.collect and
> >>> having them all the key/value pairs written to the output?
> >>>
> >>> I.e I would like the intermediate key/value pairs
> produced from the
> >>> maps to write to disk immediatly than having it to collect
> >> in the end
> >>> all of the key/value pairs and writing it out. I want individual
> >>> files per key/value pair produced.
> >>>
> >>> Thanks.
> >>>
> >>>
> >>> ---------------------------------
> >>> Be a better friend, newshound, and know-it-all with Yahoo!
> >>> Mobile. Try it now.
> >>>
> >>
> >>
> >>
> >>
> >> ---------------------------------
> >> Be a better friend, newshound, and know-it-all with Yahoo!
> >> Mobile. Try it now.
> >>
> >
>
>
Re: Map Intermediate key/value pairs written to file system
Posted by Ted Dunning <td...@veoh.com>.
Isn't this just what Hadoop does when you set numReduces = 0?
On 4/18/08 10:45 AM, "Devaraj Das" <dd...@yahoo-inc.com> wrote:
> Within a task you can get the taskId (which are unique). Define "public void
> configure(JobConf job)" and in that get the taskId by doing
> job.get("mapred.task.id") ).
>
> Now create filenames starting with that as the prefix and maybe a
> monotonically increasing integer as the suffix (defined as a static field in
> the task)..
>
>> -----Original Message-----
>> From: Kayla Jay [mailto:kaylais30@yahoo.com]
>> Sent: Friday, April 18, 2008 10:46 PM
>> To: core-user@hadoop.apache.org
>> Subject: RE: Map Intermediate key/value pairs written to file system
>>
>> Hi.
>>
>> I don't know how to create unique individual file names for
>> each mapper's key/value pairs. How do you create individual
>> files per mappers key/value pairs so they don't overwrite one another?
>>
>> I.e how do you create a new file each time and use that code
>> for all the mappers and not have each of the mappers trying
>> to overwrite the other's output? If that makes any sense?
>>
>> For example, at end of map(), i create sequence file for
>> output of the key/value pair. But, if another mapper is
>> running and it does the same thing, the file gets overwritten
>> b/c this other mapper is creating the exact file name and
>> doing the exact same thing as the other mapper is doing.
>>
>> Devaraj Das <dd...@yahoo-inc.com> wrote: Will your requirement
>> be addressed if, from within the map method, you create a
>> sequence file using SequenceFile.createWriter api, write a
>> key/value using the writer's append(key,value) API and then
>> close the file
>> ? You can do this for every key/value.
>> Pls have a look at createWriter APIs and the Writer class in
>> o.a.h.i.SequenceFile..
>>
>>> -----Original Message-----
>>> From: Kayla Jay [mailto:kaylais30@yahoo.com]
>>> Sent: Friday, April 18, 2008 6:12 PM
>>> To: core-user@hadoop.apache.org
>>> Subject: Map Intermediate key/value pairs written to file system
>>>
>>> Hi
>>>
>>> I have no reduces. I would like to directly write my map
>> results while
>>> they are produced after each map has completed to disk. I
>> don't want
>>> to collect then write to output.
>>>
>>> If I wanted to directly write my map output 1-by-1 (intermediate
>>> key/value pairs) after each map completes into individual files
>>> instead of collecting them until the end then writing them
>> in 1 swoop
>>> into the composite results file (part-000X), is this
>> possible and how
>>> do I do that?
>>>
>>> Can I force a write within the map to write the map
>> key/value pairs as
>>> an individual file for each results set instead of
>> output.collect and
>>> having them all the key/value pairs written to the output?
>>>
>>> I.e I would like the intermediate key/value pairs produced from the
>>> maps to write to disk immediatly than having it to collect
>> in the end
>>> all of the key/value pairs and writing it out. I want individual
>>> files per key/value pair produced.
>>>
>>> Thanks.
>>>
>>>
>>> ---------------------------------
>>> Be a better friend, newshound, and know-it-all with Yahoo!
>>> Mobile. Try it now.
>>>
>>
>>
>>
>>
>> ---------------------------------
>> Be a better friend, newshound, and know-it-all with Yahoo!
>> Mobile. Try it now.
>>
>
RE: Map Intermediate key/value pairs written to file system
Posted by Devaraj Das <dd...@yahoo-inc.com>.
Within a task you can get the taskId (which are unique). Define "public void
configure(JobConf job)" and in that get the taskId by doing
job.get("mapred.task.id") ).
Now create filenames starting with that as the prefix and maybe a
monotonically increasing integer as the suffix (defined as a static field in
the task)..
> -----Original Message-----
> From: Kayla Jay [mailto:kaylais30@yahoo.com]
> Sent: Friday, April 18, 2008 10:46 PM
> To: core-user@hadoop.apache.org
> Subject: RE: Map Intermediate key/value pairs written to file system
>
> Hi.
>
> I don't know how to create unique individual file names for
> each mapper's key/value pairs. How do you create individual
> files per mappers key/value pairs so they don't overwrite one another?
>
> I.e how do you create a new file each time and use that code
> for all the mappers and not have each of the mappers trying
> to overwrite the other's output? If that makes any sense?
>
> For example, at end of map(), i create sequence file for
> output of the key/value pair. But, if another mapper is
> running and it does the same thing, the file gets overwritten
> b/c this other mapper is creating the exact file name and
> doing the exact same thing as the other mapper is doing.
>
> Devaraj Das <dd...@yahoo-inc.com> wrote: Will your requirement
> be addressed if, from within the map method, you create a
> sequence file using SequenceFile.createWriter api, write a
> key/value using the writer's append(key,value) API and then
> close the file
> ? You can do this for every key/value.
> Pls have a look at createWriter APIs and the Writer class in
> o.a.h.i.SequenceFile..
>
> > -----Original Message-----
> > From: Kayla Jay [mailto:kaylais30@yahoo.com]
> > Sent: Friday, April 18, 2008 6:12 PM
> > To: core-user@hadoop.apache.org
> > Subject: Map Intermediate key/value pairs written to file system
> >
> > Hi
> >
> > I have no reduces. I would like to directly write my map
> results while
> > they are produced after each map has completed to disk. I
> don't want
> > to collect then write to output.
> >
> > If I wanted to directly write my map output 1-by-1 (intermediate
> > key/value pairs) after each map completes into individual files
> > instead of collecting them until the end then writing them
> in 1 swoop
> > into the composite results file (part-000X), is this
> possible and how
> > do I do that?
> >
> > Can I force a write within the map to write the map
> key/value pairs as
> > an individual file for each results set instead of
> output.collect and
> > having them all the key/value pairs written to the output?
> >
> > I.e I would like the intermediate key/value pairs produced from the
> > maps to write to disk immediatly than having it to collect
> in the end
> > all of the key/value pairs and writing it out. I want individual
> > files per key/value pair produced.
> >
> > Thanks.
> >
> >
> > ---------------------------------
> > Be a better friend, newshound, and know-it-all with Yahoo!
> > Mobile. Try it now.
> >
>
>
>
>
> ---------------------------------
> Be a better friend, newshound, and know-it-all with Yahoo!
> Mobile. Try it now.
>
RE: Map Intermediate key/value pairs written to file system
Posted by Kayla Jay <ka...@yahoo.com>.
Hi.
I don't know how to create unique individual file names for each mapper's key/value pairs. How do you create individual files per mappers key/value pairs so they don't overwrite one another?
I.e how do you create a new file each time and use that code for all the mappers and not have each of the mappers trying to overwrite the other's output? If that makes any sense?
For example, at end of map(), i create sequence file for output of the key/value pair. But, if another mapper is running and it does the same thing, the file gets overwritten b/c this other mapper is creating the exact file name and doing the exact same thing as the other mapper is doing.
Devaraj Das <dd...@yahoo-inc.com> wrote: Will your requirement be addressed if, from within the map method, you
create a sequence file using SequenceFile.createWriter api, write a
key/value using the writer's append(key,value) API and then close the file
? You can do this for every key/value.
Pls have a look at createWriter APIs and the Writer class in
o.a.h.i.SequenceFile..
> -----Original Message-----
> From: Kayla Jay [mailto:kaylais30@yahoo.com]
> Sent: Friday, April 18, 2008 6:12 PM
> To: core-user@hadoop.apache.org
> Subject: Map Intermediate key/value pairs written to file system
>
> Hi
>
> I have no reduces. I would like to directly write my map
> results while they are produced after each map has completed
> to disk. I don't want to collect then write to output.
>
> If I wanted to directly write my map output 1-by-1
> (intermediate key/value pairs) after each map completes into
> individual files instead of collecting them until the end
> then writing them in 1 swoop into the composite results file
> (part-000X), is this possible and how do I do that?
>
> Can I force a write within the map to write the map key/value
> pairs as an individual file for each results set instead of
> output.collect and having them all the key/value pairs
> written to the output?
>
> I.e I would like the intermediate key/value pairs produced
> from the maps to write to disk immediatly than having it to
> collect in the end all of the key/value pairs and writing it
> out. I want individual files per key/value pair produced.
>
> Thanks.
>
>
> ---------------------------------
> Be a better friend, newshound, and know-it-all with Yahoo!
> Mobile. Try it now.
>
---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now.
RE: Map Intermediate key/value pairs written to file system
Posted by Kayla Jay <ka...@yahoo.com>.
Thanks for the quick reply. I will look into that and post further questions.
I'm not familiar with the API, so I may have more questions.
I'm assuming I will have to take out the output.collect so that it doesn't try to create that end file in addition to the individual files I will force/create.
Also, is there a way to write to local file system if you are set up to use HDFS, but you want the output to go to a local file system output directory on the node itself?
I need to move all output from out of HDFS to local file system in order to perform additional stuff like move to database or something else. I hate to have to write a script once the job is done to call the HDFS shell script command to copytoLocal. Can I do any of this within Map itself? Write to another data source other than HDFS like to a database or at least for now I want to just write to local file system that's on the nodes.
Thanks.
Devaraj Das <dd...@yahoo-inc.com> wrote: Will your requirement be addressed if, from within the map method, you
create a sequence file using SequenceFile.createWriter api, write a
key/value using the writer's append(key,value) API and then close the file
? You can do this for every key/value.
Pls have a look at createWriter APIs and the Writer class in
o.a.h.i.SequenceFile..
> -----Original Message-----
> From: Kayla Jay [mailto:kaylais30@yahoo.com]
> Sent: Friday, April 18, 2008 6:12 PM
> To: core-user@hadoop.apache.org
> Subject: Map Intermediate key/value pairs written to file system
>
> Hi
>
> I have no reduces. I would like to directly write my map
> results while they are produced after each map has completed
> to disk. I don't want to collect then write to output.
>
> If I wanted to directly write my map output 1-by-1
> (intermediate key/value pairs) after each map completes into
> individual files instead of collecting them until the end
> then writing them in 1 swoop into the composite results file
> (part-000X), is this possible and how do I do that?
>
> Can I force a write within the map to write the map key/value
> pairs as an individual file for each results set instead of
> output.collect and having them all the key/value pairs
> written to the output?
>
> I.e I would like the intermediate key/value pairs produced
> from the maps to write to disk immediatly than having it to
> collect in the end all of the key/value pairs and writing it
> out. I want individual files per key/value pair produced.
>
> Thanks.
>
>
> ---------------------------------
> Be a better friend, newshound, and know-it-all with Yahoo!
> Mobile. Try it now.
>
---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now.
RE: Map Intermediate key/value pairs written to file system
Posted by Devaraj Das <dd...@yahoo-inc.com>.
Will your requirement be addressed if, from within the map method, you
create a sequence file using SequenceFile.createWriter api, write a
key/value using the writer's append(key,value) API and then close the file
? You can do this for every key/value.
Pls have a look at createWriter APIs and the Writer class in
o.a.h.i.SequenceFile..
> -----Original Message-----
> From: Kayla Jay [mailto:kaylais30@yahoo.com]
> Sent: Friday, April 18, 2008 6:12 PM
> To: core-user@hadoop.apache.org
> Subject: Map Intermediate key/value pairs written to file system
>
> Hi
>
> I have no reduces. I would like to directly write my map
> results while they are produced after each map has completed
> to disk. I don't want to collect then write to output.
>
> If I wanted to directly write my map output 1-by-1
> (intermediate key/value pairs) after each map completes into
> individual files instead of collecting them until the end
> then writing them in 1 swoop into the composite results file
> (part-000X), is this possible and how do I do that?
>
> Can I force a write within the map to write the map key/value
> pairs as an individual file for each results set instead of
> output.collect and having them all the key/value pairs
> written to the output?
>
> I.e I would like the intermediate key/value pairs produced
> from the maps to write to disk immediatly than having it to
> collect in the end all of the key/value pairs and writing it
> out. I want individual files per key/value pair produced.
>
> Thanks.
>
>
> ---------------------------------
> Be a better friend, newshound, and know-it-all with Yahoo!
> Mobile. Try it now.
>