You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Pratyush Chandra <ch...@gmail.com> on 2013/01/10 12:21:21 UTC
Not committing output in map reduce
Hi,
I am using s3n as file system. I do not wish to create output folders and
file if there is no output from RecordReader implementation. Currently it
creates empty part* files and _SUCCESS. Is there a way to do so ?
--
Pratyush Chandra
Re: Not committing output in map reduce
Posted by Pratyush Chandra <ch...@gmail.com>.
Yes, it worked. Thanks
Pratyush
On Thu, Jan 10, 2013 at 6:14 PM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:
> Is this the same as:
> http://stackoverflow.com/questions/6137139/how-to-save-only-non-empty-reducers-output-in-hdfs?
>
> i.e. LazyOutputFormat, etc. ?
>
>
> On Thu, Jan 10, 2013 at 4:51 PM, Pratyush Chandra <
> chandra.pratyush@gmail.com> wrote:
>
>> Hi,
>>
>> I am using s3n as file system. I do not wish to create output folders and
>> file if there is no output from RecordReader implementation. Currently it
>> creates empty part* files and _SUCCESS. Is there a way to do so ?
>>
>> --
>> Pratyush Chandra
>>
>
>
--
Pratyush Chandra
Re: Not committing output in map reduce
Posted by Pratyush Chandra <ch...@gmail.com>.
Yes, it worked. Thanks
Pratyush
On Thu, Jan 10, 2013 at 6:14 PM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:
> Is this the same as:
> http://stackoverflow.com/questions/6137139/how-to-save-only-non-empty-reducers-output-in-hdfs?
>
> i.e. LazyOutputFormat, etc. ?
>
>
> On Thu, Jan 10, 2013 at 4:51 PM, Pratyush Chandra <
> chandra.pratyush@gmail.com> wrote:
>
>> Hi,
>>
>> I am using s3n as file system. I do not wish to create output folders and
>> file if there is no output from RecordReader implementation. Currently it
>> creates empty part* files and _SUCCESS. Is there a way to do so ?
>>
>> --
>> Pratyush Chandra
>>
>
>
--
Pratyush Chandra
Re: Not committing output in map reduce
Posted by Pratyush Chandra <ch...@gmail.com>.
Yes, it worked. Thanks
Pratyush
On Thu, Jan 10, 2013 at 6:14 PM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:
> Is this the same as:
> http://stackoverflow.com/questions/6137139/how-to-save-only-non-empty-reducers-output-in-hdfs?
>
> i.e. LazyOutputFormat, etc. ?
>
>
> On Thu, Jan 10, 2013 at 4:51 PM, Pratyush Chandra <
> chandra.pratyush@gmail.com> wrote:
>
>> Hi,
>>
>> I am using s3n as file system. I do not wish to create output folders and
>> file if there is no output from RecordReader implementation. Currently it
>> creates empty part* files and _SUCCESS. Is there a way to do so ?
>>
>> --
>> Pratyush Chandra
>>
>
>
--
Pratyush Chandra
Re: Not committing output in map reduce
Posted by Pratyush Chandra <ch...@gmail.com>.
Yes, it worked. Thanks
Pratyush
On Thu, Jan 10, 2013 at 6:14 PM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:
> Is this the same as:
> http://stackoverflow.com/questions/6137139/how-to-save-only-non-empty-reducers-output-in-hdfs?
>
> i.e. LazyOutputFormat, etc. ?
>
>
> On Thu, Jan 10, 2013 at 4:51 PM, Pratyush Chandra <
> chandra.pratyush@gmail.com> wrote:
>
>> Hi,
>>
>> I am using s3n as file system. I do not wish to create output folders and
>> file if there is no output from RecordReader implementation. Currently it
>> creates empty part* files and _SUCCESS. Is there a way to do so ?
>>
>> --
>> Pratyush Chandra
>>
>
>
--
Pratyush Chandra
Re: Not committing output in map reduce
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Is this the same as:
http://stackoverflow.com/questions/6137139/how-to-save-only-non-empty-reducers-output-in-hdfs?
i.e. LazyOutputFormat, etc. ?
On Thu, Jan 10, 2013 at 4:51 PM, Pratyush Chandra <
chandra.pratyush@gmail.com> wrote:
> Hi,
>
> I am using s3n as file system. I do not wish to create output folders and
> file if there is no output from RecordReader implementation. Currently it
> creates empty part* files and _SUCCESS. Is there a way to do so ?
>
> --
> Pratyush Chandra
>
Re: Not committing output in map reduce
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Is this the same as:
http://stackoverflow.com/questions/6137139/how-to-save-only-non-empty-reducers-output-in-hdfs?
i.e. LazyOutputFormat, etc. ?
On Thu, Jan 10, 2013 at 4:51 PM, Pratyush Chandra <
chandra.pratyush@gmail.com> wrote:
> Hi,
>
> I am using s3n as file system. I do not wish to create output folders and
> file if there is no output from RecordReader implementation. Currently it
> creates empty part* files and _SUCCESS. Is there a way to do so ?
>
> --
> Pratyush Chandra
>
Re: Not committing output in map reduce
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Is this the same as:
http://stackoverflow.com/questions/6137139/how-to-save-only-non-empty-reducers-output-in-hdfs?
i.e. LazyOutputFormat, etc. ?
On Thu, Jan 10, 2013 at 4:51 PM, Pratyush Chandra <
chandra.pratyush@gmail.com> wrote:
> Hi,
>
> I am using s3n as file system. I do not wish to create output folders and
> file if there is no output from RecordReader implementation. Currently it
> creates empty part* files and _SUCCESS. Is there a way to do so ?
>
> --
> Pratyush Chandra
>
Re: Not committing output in map reduce
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Is this the same as:
http://stackoverflow.com/questions/6137139/how-to-save-only-non-empty-reducers-output-in-hdfs?
i.e. LazyOutputFormat, etc. ?
On Thu, Jan 10, 2013 at 4:51 PM, Pratyush Chandra <
chandra.pratyush@gmail.com> wrote:
> Hi,
>
> I am using s3n as file system. I do not wish to create output folders and
> file if there is no output from RecordReader implementation. Currently it
> creates empty part* files and _SUCCESS. Is there a way to do so ?
>
> --
> Pratyush Chandra
>