You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by "Balaraman, Anand" <An...@SYNTELINC.COM> on 2012/10/05 08:06:35 UTC
Avoid creation of _SUCCESS folder in reducer output folder
Hi
While using Map reduce programs, the output folder where reducer writes
out the result contains 2 auto-generated folders: _SUCCESS and _logs.
To avoid generation of _log folder, I can set the configuration
parameter "hadoop.job.history.user.location" with value as "none".
But, I don't know how to avoid the creation of _SUCCESS folder. Can
anyone please help.
Note: For now, as a work around, I am using FileSystem commands at the
end of job to delete the folder manually.
Anand B
Confidential: This electronic message and all contents contain information from Syntel, Inc. which may be privileged, confidential or otherwise protected from disclosure. The information is intended to be for the addressee only. If you are not the addressee, any disclosure, copy, distribution or use of the contents of this message is prohibited. If you have received this electronic message in error, please notify the sender immediately and destroy the original message and all copies.
RE: Avoid creation of _SUCCESS folder in reducer output folder
Posted by "Balaraman, Anand" <An...@SYNTELINC.COM>.
I tried setting "mapreduce.fileoutputcommitter.marksuccessfuljobs=false"
in configuration and it worked.
Found information about this in a JIRA ticket as well:
https://issues.apache.org/jira/browse/MAPREDUCE-947
Thank you Aniket.
From: Aniket Mokashi [mailto:aniket486@gmail.com]
Sent: 05 October 2012 12:04
To: user@hive.apache.org
Cc: hive-user@hadoop.apache.org; user@hadoop.apache.org
Subject: Re: Avoid creation of _SUCCESS folder in reducer output folder
mapreduce.fileoutputcommitter.marksuccessfuljobs=false;
MAPREDUCE-947, i guess..
~Aniket
On Thu, Oct 4, 2012 at 11:06 PM, Balaraman, Anand
<An...@syntelinc.com> wrote:
Hi
While using Map reduce programs, the output folder where reducer writes
out the result contains 2 auto-generated folders: _SUCCESS and _logs.
To avoid generation of _log folder, I can set the configuration
parameter "hadoop.job.history.user.location" with value as "none".
But, I don't know how to avoid the creation of _SUCCESS folder. Can
anyone please help.
Note: For now, as a work around, I am using FileSystem commands at the
end of job to delete the folder manually.
Anand B
Confidential: This electronic message and all contents contain
information from Syntel, Inc. which may be privileged, confidential or
otherwise protected from disclosure. The information is intended to be
for the addressee only. If you are not the addressee, any disclosure,
copy, distribution or use of the contents of this message is prohibited.
If you have received this electronic message in error, please notify the
sender immediately and destroy the original message and all copies.
--
"...:::Aniket:::... Quetzalco@tl"
Confidential: This electronic message and all contents contain information from Syntel, Inc. which may be privileged, confidential or otherwise protected from disclosure. The information is intended to be for the addressee only. If you are not the addressee, any disclosure, copy, distribution or use of the contents of this message is prohibited. If you have received this electronic message in error, please notify the sender immediately and destroy the original message and all copies.
Re: Avoid creation of _SUCCESS folder in reducer output folder
Posted by Aniket Mokashi <an...@gmail.com>.
mapreduce.fileoutputcommitter.marksuccessfuljobs=false;
MAPREDUCE-947, i guess..
~Aniket
On Thu, Oct 4, 2012 at 11:06 PM, Balaraman, Anand <
Anand_Balaraman@syntelinc.com> wrote:
> Hi****
>
> ** **
>
> While using Map reduce programs, the output folder where reducer writes
> out the result contains 2 auto-generated folders: *_SUCCESS* and *_logs*.*
> ***
>
> To avoid generation of *_log* folder, I can set the configuration
> parameter “*hadoop.job.history.user.location*” with value as “*none*”.****
>
> But, I don’t know how to avoid the creation of *_SUCCESS* folder. Can
> anyone please help.****
>
> ** **
>
> *Note*: For now, as a work around, I am using FileSystem commands at the
> end of job to delete the folder manually.****
>
> ** **
>
> Anand B****
> Confidential: This electronic message and all contents contain
> information from Syntel, Inc. which may be privileged, confidential or
> otherwise protected from disclosure. The information is intended to be for
> the addressee only. If you are not the addressee, any disclosure, copy,
> distribution or use of the contents of this message is prohibited. If you
> have received this electronic message in error, please notify the sender
> immediately and destroy the original message and all copies.
>
--
"...:::Aniket:::... Quetzalco@tl"