You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by st...@apache.org on 2025/10/14 10:44:55 UTC
(hadoop) branch branch-3.4 updated: MAPREDUCE-7448. Skipping cleanup with FileOutputCommitter V1 can corrupt output: warn and document (#6038)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new ecc32943804 MAPREDUCE-7448. Skipping cleanup with FileOutputCommitter V1 can corrupt output: warn and document (#6038)
ecc32943804 is described below
commit ecc32943804c3d7db8043683443a739851d482f6
Author: ConfX <11...@users.noreply.github.com>
AuthorDate: Tue Oct 14 05:33:45 2025 -0500
MAPREDUCE-7448. Skipping cleanup with FileOutputCommitter V1 can corrupt output: warn and document (#6038)
* update documentation for mapreduce committer
* add warning if the user attempts to use FileOutputCommiter V1 with skipping cleanup
Contributed by ConfX
---
.../org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java | 5 +++++
.../src/site/markdown/manifest_committer.md | 1 +
2 files changed, 6 insertions(+)
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
index 82b7fcb5046..5adc1cd14d4 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
@@ -158,6 +158,11 @@ public FileOutputCommitter(Path outputPath,
"output directory:" + skipCleanup + ", ignore cleanup failures: " +
ignoreCleanupFailures);
+ if (algorithmVersion == 1 && skipCleanup) {
+ LOG.warn("Skip cleaning up when using FileOutputCommitter V1 can lead to unexpected behaviors. " +
+ "For example, committing several times may be allowed falsely.");
+ }
+
if (outputPath != null) {
FileSystem fs = outputPath.getFileSystem(context.getConfiguration());
this.outputPath = fs.makeQualified(outputPath);
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/manifest_committer.md b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/manifest_committer.md
index 0ac03080195..3c2c5891bc2 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/manifest_committer.md
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/manifest_committer.md
@@ -202,6 +202,7 @@ Here are the main configuration options of the committer.
There are some more, as covered in the (Advanced)[#advanced] section.
+WARNING: setting `mapreduce.fileoutputcommitter.cleanup.skipped` to `true` is not compatible with version 1 of the committer and can cause unexpected behaviors.
## <a name="scaling"></a> Scaling jobs `mapreduce.manifest.committer.io.threads`
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org