You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/09/29 16:59:06 UTC
[GitHub] [hadoop] steveloughran opened a new pull request #2349: MAPREDUCE-7282. Move away from V2 commit algorithm
steveloughran opened a new pull request #2349:
URL: https://github.com/apache/hadoop/pull/2349
* Default to v1 commit algorithm
* log at WARN on job set up when v2 is used
* use a special log for that so people who know what they are doing can
tell it to be quiet
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2349: MAPREDUCE-7282. Move away from V2 commit algorithm
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2349:
URL: https://github.com/apache/hadoop/pull/2349#issuecomment-702137474
@jbrennan333 what do you think we should say instead of deprecated? "not recommended".
I was thinking of adding a link to the JIRA and changing the issue text there to clarify
* safe if names and content of generated output files is consistent across all task attempts
* unsafe if different TAs generate bad files (biggest risk, as partial failure of 1st attempt may leave)
* unsafe if different TAs generate different content in same files (only an issue on a network partition and TA #1 generates output as/after TA #2 does its work.
cleanup of job will delete the whole job attempt dir so that's the maximum time that a partitioned TA may commit work. There's no risk of some VM pausing for 3 hours, restarting and an in progress TA completing its work and overwriting the final output. This is good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2349: MAPREDUCE-7282. Move away from V2 commit algorithm
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2349:
URL: https://github.com/apache/hadoop/pull/2349#discussion_r498491655
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
##########
@@ -348,6 +348,16 @@ public Path getWorkPath() throws IOException {
* @param context the job's context
*/
public void setupJob(JobContext context) throws IOException {
+ // Downgrade v2 to v1 with a warning.
+ if (algorithmVersion == 2) {
+ Logger log = LoggerFactory.getLogger(
+ "org.apache.hadoop.mapreduce.lib.output."
+ + "FileOutputCommitter.Algorithm");
+
+ log.warn("The v2 commit algorithm is deprecated;"
+ + " please switch to the v1 algorithm");
Review comment:
switching to your text
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jbrennan333 commented on a change in pull request #2349: MAPREDUCE-7282. Move away from V2 commit algorithm
Posted by GitBox <gi...@apache.org>.
jbrennan333 commented on a change in pull request #2349:
URL: https://github.com/apache/hadoop/pull/2349#discussion_r496954586
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
##########
@@ -348,6 +348,16 @@ public Path getWorkPath() throws IOException {
* @param context the job's context
*/
public void setupJob(JobContext context) throws IOException {
+ // Downgrade v2 to v1 with a warning.
Review comment:
This comment is no longer accurate.
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
##########
@@ -348,6 +348,16 @@ public Path getWorkPath() throws IOException {
* @param context the job's context
*/
public void setupJob(JobContext context) throws IOException {
+ // Downgrade v2 to v1 with a warning.
+ if (algorithmVersion == 2) {
+ Logger log = LoggerFactory.getLogger(
+ "org.apache.hadoop.mapreduce.lib.output."
+ + "FileOutputCommitter.Algorithm");
+
+ log.warn("The v2 commit algorithm is deprecated;"
+ + " please switch to the v1 algorithm");
Review comment:
I don't think we should use the word deprecated. That implies that this algorithm will be removed in a future release
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
##########
@@ -1562,10 +1562,35 @@
<property>
<name>mapreduce.fileoutputcommitter.algorithm.version</name>
- <value>2</value>
- <description>The file output committer algorithm version
- valid algorithm version number: 1 or 2
- default to 2, which is the original algorithm
+ <value>1</value>
+ <description>The file output committer algorithm version.
+
+ There are two algorithm versions in Hadoop, "1" and "2".
+
+ The version 2 algorithm is deprecated and no longer the default
+ as task commits were not atomic.
Review comment:
Similarly, remove "deprecated".
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
##########
@@ -1562,10 +1562,35 @@
<property>
<name>mapreduce.fileoutputcommitter.algorithm.version</name>
- <value>2</value>
- <description>The file output committer algorithm version
- valid algorithm version number: 1 or 2
- default to 2, which is the original algorithm
+ <value>1</value>
+ <description>The file output committer algorithm version.
+
+ There are two algorithm versions in Hadoop, "1" and "2".
+
+ The version 2 algorithm is deprecated and no longer the default
+ as task commits were not atomic.
+ If a first task attempt fails part-way
+ through its task commit, the output directory could end up
+ with data from that failed commit, alongside the data
+ from any subsequent attempts.
+
+ See https://issues.apache.org/jira/browse/MAPREDUCE-7282
+
+ Although no-longer the default, this algorithm is safe to use if
+ all task attempts for a single task meet the following requirements
+ -they generate exactly the same set of files
+ -the contents of each file are exactly the same in each task attempt
+
+ That is:
+ 1. If a second attempt commits work, there will be no leftover files from
+ a first attempt which failed during its task commit.
+ 2. If a network partition causes the first task attempt to overwrite
+ some/all of the output of a second attempt, the result will be
+ exactly the same as if it had not done so.
+
+ To avoid the warning message on job setup, set the log level of the log
+ org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.Algorithm
+ to ERROR.
Review comment:
I think this section should be moved to the end of the Algorithm 2 section below. You can add (see below for details) to the end of the line that says why algorithm v2 in no longer the default.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #2349: MAPREDUCE-7282. Move away from V2 commit algorithm
Posted by GitBox <gi...@apache.org>.
steveloughran closed pull request #2349:
URL: https://github.com/apache/hadoop/pull/2349
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #2349: MAPREDUCE-7282. Move away from V2 commit algorithm
Posted by GitBox <gi...@apache.org>.
steveloughran closed pull request #2349:
URL: https://github.com/apache/hadoop/pull/2349
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2349: MAPREDUCE-7282. Move away from V2 commit algorithm
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2349:
URL: https://github.com/apache/hadoop/pull/2349#issuecomment-702161929
(Yetus failure is from no new tests)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jbrennan333 commented on pull request #2349: MAPREDUCE-7282. Move away from V2 commit algorithm
Posted by GitBox <gi...@apache.org>.
jbrennan333 commented on pull request #2349:
URL: https://github.com/apache/hadoop/pull/2349#issuecomment-702176681
@steveloughran It's hard to think of a terse warning for this. I think your comment above gets close. Maybe something like "The v2 commit algorithm assumes that the content of generated output files is consistent across all task attempts - if this is not true for this job, the v1 commit algorithm is strongly recommended."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2349: MAPREDUCE-7282. Move away from V2 commit algorithm
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2349:
URL: https://github.com/apache/hadoop/pull/2349#discussion_r498242126
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
##########
@@ -348,6 +348,16 @@ public Path getWorkPath() throws IOException {
* @param context the job's context
*/
public void setupJob(JobContext context) throws IOException {
+ // Downgrade v2 to v1 with a warning.
+ if (algorithmVersion == 2) {
+ Logger log = LoggerFactory.getLogger(
+ "org.apache.hadoop.mapreduce.lib.output."
+ + "FileOutputCommitter.Algorithm");
+
+ log.warn("The v2 commit algorithm is deprecated;"
+ + " please switch to the v1 algorithm");
Review comment:
what do you suggest?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2349: MAPREDUCE-7282. Move away from V2 commit algorithm
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2349:
URL: https://github.com/apache/hadoop/pull/2349#discussion_r498242247
##########
File path: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
##########
@@ -1562,10 +1562,35 @@
<property>
<name>mapreduce.fileoutputcommitter.algorithm.version</name>
- <value>2</value>
- <description>The file output committer algorithm version
- valid algorithm version number: 1 or 2
- default to 2, which is the original algorithm
+ <value>1</value>
+ <description>The file output committer algorithm version.
+
+ There are two algorithm versions in Hadoop, "1" and "2".
+
+ The version 2 algorithm is deprecated and no longer the default
+ as task commits were not atomic.
+ If a first task attempt fails part-way
+ through its task commit, the output directory could end up
+ with data from that failed commit, alongside the data
+ from any subsequent attempts.
+
+ See https://issues.apache.org/jira/browse/MAPREDUCE-7282
+
+ Although no-longer the default, this algorithm is safe to use if
+ all task attempts for a single task meet the following requirements
+ -they generate exactly the same set of files
+ -the contents of each file are exactly the same in each task attempt
+
+ That is:
+ 1. If a second attempt commits work, there will be no leftover files from
+ a first attempt which failed during its task commit.
+ 2. If a network partition causes the first task attempt to overwrite
+ some/all of the output of a second attempt, the result will be
+ exactly the same as if it had not done so.
+
+ To avoid the warning message on job setup, set the log level of the log
+ org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.Algorithm
+ to ERROR.
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2349: MAPREDUCE-7282. Move away from V2 commit algorithm
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2349:
URL: https://github.com/apache/hadoop/pull/2349#issuecomment-859061589
Once the manifest committer #2971 is in, the abfs and gcs stores will get something faster than v1 but with its task failure semantics
* all treewalking for task attempt listing will be in task commit
* no dir renames in task commit, just saving of the manifest
* job commit: parallel load of manifests, merge of list of directories to create, parallel set of mkdirs and then the parallelized renames.
In this world, we can just leave people using v1/v2 alone, and for (spark) jobs in azure and google cloud say "use the manifest committer".
Which means I can just close this as a wontfix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org