You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean R. Owen (Jira)" <ji...@apache.org> on 2020/08/27 14:07:00 UTC
[jira] [Resolved] (SPARK-32701)
mapreduce.fileoutputcommitter.algorithm.version default depends on runtime
environment
[ https://issues.apache.org/jira/browse/SPARK-32701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean R. Owen resolved SPARK-32701.
----------------------------------
Fix Version/s: 3.1.0
3.0.1
Resolution: Fixed
Issue resolved by pull request 29541
[https://github.com/apache/spark/pull/29541]
> mapreduce.fileoutputcommitter.algorithm.version default depends on runtime environment
> --------------------------------------------------------------------------------------
>
> Key: SPARK-32701
> URL: https://issues.apache.org/jira/browse/SPARK-32701
> Project: Spark
> Issue Type: Improvement
> Components: docs, Documentation
> Affects Versions: 2.4.0, 3.0.0
> Reporter: Waleed Fateem
> Assignee: Waleed Fateem
> Priority: Minor
> Fix For: 3.0.1, 3.1.0
>
>
> When someone reads the documentation in its current state, the assumption is that the default value of spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version is 1 and that's not entirely accurate.
> Spark doesn't explicitly set this configuration and instead is inherited from Hadoop's FileOutputCommitter class. The default value is 1 until Hadoop 3.0 where this changed to 2.
> I'm proposing that we clarify that this value's default will depend on the Hadoop version in a user's runtime environment, where:
> 1 for < Hadoop 3.0
> 2 for >= Hadoop 3.0
> There are also plans to revert this default again to v1 so might also be useful to reference this JIRA:
> https://issues.apache.org/jira/browse/MAPREDUCE-7282
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org