You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by st...@apache.org on 2023/06/27 12:56:22 UTC
[hadoop] branch branch-3.3 updated: MAPREDUCE-7432. Make manifest committer default on abfs and gcs stores (#5378)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 48f1a3c5483 MAPREDUCE-7432. Make manifest committer default on abfs and gcs stores (#5378)
48f1a3c5483 is described below
commit 48f1a3c54836f7110723b170f91aa05cde95c46b
Author: Steve Loughran <st...@cloudera.com>
AuthorDate: Tue Jun 27 13:55:20 2023 +0100
MAPREDUCE-7432. Make manifest committer default on abfs and gcs stores (#5378)
By default, the mapreduce manifest committer is used for jobs working with abfs and gcs.
Hadoop mapreduce will pick this up automatically; for Spark it is a bit complicated: read the docs
to see the steps required.
---
.../src/main/resources/mapred-default.xml | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
index 06e9c03861f..6fe49400f0c 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
@@ -2242,23 +2242,23 @@
</description>
</property>
-<!-- not yet enabled by default.
-
+<!-- use manifest committer for abfs URLs -->
<property>
<name>mapreduce.outputcommitter.factory.scheme.abfs</name>
<value>org.apache.hadoop.fs.azurebfs.commit.AzureManifestCommitterFactory</value>
<description>
- The default committer factory for ABFS is for the manifest committer with
- abfs-specific tuning.
+ The default committer factory for ABFS is the manifest committer with
+ abfs-specific recovery.
</description>
</property>
+<!-- use manifest committer for gs URLs -->
<property>
<name>mapreduce.outputcommitter.factory.scheme.gs</name>
<value>org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterFactory</value>
<description>
- The default committer factory for google cloud storage is for the manifest committer.
+ The default committer factory for google cloud storage is the manifest committer.
</description>
</property>
--->
+
</configuration>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org