You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by ae...@apache.org on 2017/09/01 20:01:18 UTC
[30/74] [abbrv] [partial] hadoop git commit: HADOOP-14364. refresh
changelog/release notes with newer Apache Yetus build
http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/0.22.0/RELEASENOTES.0.22.0.md
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.22.0/RELEASENOTES.0.22.0.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.22.0/RELEASENOTES.0.22.0.md
index 41ffd77..1678634 100644
--- a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.22.0/RELEASENOTES.0.22.0.md
+++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.22.0/RELEASENOTES.0.22.0.md
@@ -23,609 +23,609 @@ These release notes cover new developer and user-facing incompatibilities, impor
---
-* [HADOOP-7302](https://issues.apache.org/jira/browse/HADOOP-7302) | *Major* | **webinterface.private.actions should not be in common**
+* [MAPREDUCE-478](https://issues.apache.org/jira/browse/MAPREDUCE-478) | *Minor* | **separate jvm param for mapper and reducer**
-Option webinterface.private.actions has been renamed to mapreduce.jobtracker.webinterface.trusted and should be specified in mapred-site.xml instead of core-site.xml
+Allow map and reduce jvm parameters, environment variables and ulimit to be set separately.
+
+Configuration changes:
+ add mapred.map.child.java.opts
+ add mapred.reduce.child.java.opts
+ add mapred.map.child.env
+ add mapred.reduce.child.ulimit
+ add mapred.map.child.env
+ add mapred.reduce.child.ulimit
+ deprecated mapred.child.java.opts
+ deprecated mapred.child.env
+ deprecated mapred.child.ulimit
---
-* [HADOOP-7229](https://issues.apache.org/jira/browse/HADOOP-7229) | *Major* | **Absolute path to kinit in auto-renewal thread**
+* [HADOOP-6344](https://issues.apache.org/jira/browse/HADOOP-6344) | *Major* | **rm and rmr fail to correctly move the user's files to the trash prior to deleting when they are over quota.**
-When Hadoop's Kerberos integration is enabled, it is now required that either {{kinit}} be on the path for user accounts running the Hadoop client, or that the {{hadoop.kerberos.kinit.command}} configuration option be manually set to the absolute path to {{kinit}}.
+Trash feature notifies user of over-quota condition rather than silently deleting files/directories; deletion can be compelled with "rm -skiptrash".
---
-* [HADOOP-7193](https://issues.apache.org/jira/browse/HADOOP-7193) | *Minor* | **Help message is wrong for touchz command.**
+* [HADOOP-6599](https://issues.apache.org/jira/browse/HADOOP-6599) | *Major* | **Split RPC metrics into summary and detailed metrics**
-Updated the help for the touchz command.
+Split existing RpcMetrics into RpcMetrics and RpcDetailedMetrics. The new RpcDetailedMetrics has per method usage details and is available under context name "rpc" and record name "detailed-metrics"
---
-* [HADOOP-7192](https://issues.apache.org/jira/browse/HADOOP-7192) | *Trivial* | **fs -stat docs aren't updated to reflect the format features**
+* [MAPREDUCE-927](https://issues.apache.org/jira/browse/MAPREDUCE-927) | *Major* | **Cleanup of task-logs should happen in TaskTracker instead of the Child**
-Updated the web documentation to reflect the formatting abilities of 'fs -stat'.
+Moved Task log cleanup into a separate thread in TaskTracker.
+Added configuration "mapreduce.job.userlog.retain.hours" to specify the time(in hours) for which the user-logs are to be retained after the job completion.
---
-* [HADOOP-7156](https://issues.apache.org/jira/browse/HADOOP-7156) | *Critical* | **getpwuid\_r is not thread-safe on RHEL6**
+* [HADOOP-6730](https://issues.apache.org/jira/browse/HADOOP-6730) | *Major* | **Bug in FileContext#copy and provide base class for FileContext tests**
-Adds a new configuration hadoop.work.around.non.threadsafe.getpwuid which can be used to enable a mutex around this call to workaround thread-unsafe implementations of getpwuid\_r. Users should consult http://wiki.apache.org/hadoop/KnownBrokenPwuidImplementations for a list of such systems.
+**WARNING: No release note provided for this change.**
---
-* [HADOOP-7137](https://issues.apache.org/jira/browse/HADOOP-7137) | *Major* | **Remove hod contrib**
+* [MAPREDUCE-1707](https://issues.apache.org/jira/browse/MAPREDUCE-1707) | *Major* | **TaskRunner can get NPE in getting ugi from TaskTracker**
-Removed contrib related build targets.
+Fixed a bug that causes TaskRunner to get NPE in getting ugi from TaskTracker and subsequently crashes it resulting in a failing task after task-timeout period.
---
-* [HADOOP-7134](https://issues.apache.org/jira/browse/HADOOP-7134) | *Major* | **configure files that are generated as part of the released tarball need to have executable bit set**
+* [MAPREDUCE-1680](https://issues.apache.org/jira/browse/MAPREDUCE-1680) | *Major* | **Add a metrics to track the number of heartbeats processed**
-I have just committed this to trunk and branch-0.22. Thanks Roman!
+Added a metric to track number of heartbeats processed by the JobTracker.
---
-* [HADOOP-7117](https://issues.apache.org/jira/browse/HADOOP-7117) | *Major* | **Move secondary namenode checkpoint configs from core-default.xml to hdfs-default.xml**
+* [HDFS-1109](https://issues.apache.org/jira/browse/HDFS-1109) | *Major* | **HFTP and URL Encoding**
-Removed references to the older fs.checkpoint.\* properties that resided in core-site.xml
+**WARNING: No release note provided for this change.**
---
-* [HADOOP-7089](https://issues.apache.org/jira/browse/HADOOP-7089) | *Minor* | **Fix link resolution logic in hadoop-config.sh**
+* [HDFS-1061](https://issues.apache.org/jira/browse/HDFS-1061) | *Minor* | **Memory footprint optimization for INodeFile object.**
-Updates hadoop-config.sh to always resolve symlinks when determining HADOOP\_HOME. Bash built-ins or POSIX:2001 compliant cmds are now required.
+**WARNING: No release note provided for this change.**
---
-* [HADOOP-7013](https://issues.apache.org/jira/browse/HADOOP-7013) | *Major* | **Add boolean field isCorrupt to BlockLocation**
+* [HDFS-1079](https://issues.apache.org/jira/browse/HDFS-1079) | *Major* | **HDFS implementation should throw exceptions defined in AbstractFileSystem**
-This patch has changed the serialization format of BlockLocation.
+Specific exceptions are thrown from HDFS implementation and protocol per the interface defined in AbstractFileSystem. The compatibility is not affected as the applications catch IOException and will be able to handle specific exceptions that are subclasses of IOException.
---
-* [HADOOP-7005](https://issues.apache.org/jira/browse/HADOOP-7005) | *Major* | **Update test-patch.sh to remove callback to Hudson master**
+* [MAPREDUCE-1558](https://issues.apache.org/jira/browse/MAPREDUCE-1558) | *Major* | **specify correct server principal for RefreshAuthorizationPolicyProtocol and RefreshUserToGroupMappingsProtocol protocols in MRAdmin (for HADOOP-6612)**
-N/A
+new config:
+hadoop.security.service.user.name.key
+this setting points to the server principal for RefreshUserToGroupMappingsProtocol.
+The value should be either NN or JT principal depending if it is used in DFAdmin or MRAdmin. The value is set by the application. No need for default value.
---
-* [HADOOP-6949](https://issues.apache.org/jira/browse/HADOOP-6949) | *Major* | **Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting**
+* [HDFS-708](https://issues.apache.org/jira/browse/HDFS-708) | *Major* | **A stress-test tool for HDFS.**
-Increments the RPC protocol version in org.apache.hadoop.ipc.Server from 4 to 5.
-Introduces ArrayPrimitiveWritable for a much more efficient wire format to transmit arrays of primitives over RPC. ObjectWritable uses the new writable for array of primitives for RPC and continues to use existing format for on-disk data.
+Does not currently provide anything but uniform distribution.
+Uses some older depreciated class interfaces (for mapper and reducer)
+This was tested on 0.20 and 0.22 (locally) so it should be fairly backwards compatible.
---
-* [HADOOP-6922](https://issues.apache.org/jira/browse/HADOOP-6922) | *Major* | **COMMON part of MAPREDUCE-1664**
+* [MAPREDUCE-1354](https://issues.apache.org/jira/browse/MAPREDUCE-1354) | *Critical* | **Incremental enhancements to the JobTracker for better scalability**
-Makes AccessControlList a writable and updates documentation for Job ACLs.
+Incremental enhancements to the JobTracker include a no-lock version of JT.getTaskCompletion events, no lock on the JT while doing i/o during job-submission and several fixes to cut down configuration parsing during heartbeat-handling.
---
-* [HADOOP-6905](https://issues.apache.org/jira/browse/HADOOP-6905) | *Major* | **Better logging messages when a delegation token is invalid**
+* [MAPREDUCE-1683](https://issues.apache.org/jira/browse/MAPREDUCE-1683) | *Major* | **Remove JNI calls from ClusterStatus cstr**
-**WARNING: No release note provided for this incompatible change.**
+Removes JNI calls to get jvm current/max heap usage in ClusterStatus. Any instances of ClusterStatus serialized in a prior version will not be correctly deserialized using the updated class.
---
-* [HADOOP-6835](https://issues.apache.org/jira/browse/HADOOP-6835) | *Major* | **Support concatenated gzip files**
+* [MAPREDUCE-1773](https://issues.apache.org/jira/browse/MAPREDUCE-1773) | *Major* | **streaming doesn't support jobclient.output.filter**
-Processing of concatenated gzip files formerly stopped (quietly) at the end of the first substream/"member"; now processing will continue to the end of the concatenated stream, like gzip(1) does. (bzip2 support is unaffected by this patch.)
+Improved console messaging for streaming jobs by using the generic JobClient API itself instead of the existing streaming-specific code.
---
-* [HADOOP-6787](https://issues.apache.org/jira/browse/HADOOP-6787) | *Major* | **Factor out glob pattern code from FileContext and Filesystem**
+* [MAPREDUCE-1785](https://issues.apache.org/jira/browse/MAPREDUCE-1785) | *Minor* | **Add streaming config option for not emitting the key**
-**WARNING: No release note provided for this incompatible change.**
+Added a configuration property "stream.map.input.ignoreKey" to specify whether to ignore key or not while writing input for the mapper. This configuration parameter is valid only if stream.map.input.writer.class is org.apache.hadoop.streaming.io.TextInputWriter.class. For all other InputWriter's, key is always written.
---
-* [HADOOP-6730](https://issues.apache.org/jira/browse/HADOOP-6730) | *Major* | **Bug in FileContext#copy and provide base class for FileContext tests**
+* [MAPREDUCE-572](https://issues.apache.org/jira/browse/MAPREDUCE-572) | *Minor* | **If #link is missing from uri format of -cacheArchive then streaming does not throw error.**
-**WARNING: No release note provided for this incompatible change.**
+Improved streaming job failure when #link is missing from uri format of -cacheArchive. Earlier it used to fail when launching individual tasks, now it fails during job submission itself.
---
-* [HADOOP-6693](https://issues.apache.org/jira/browse/HADOOP-6693) | *Major* | **Add metrics to track kerberos login activity**
+* [HDFS-1096](https://issues.apache.org/jira/browse/HDFS-1096) | *Major* | **allow dfsadmin/mradmin refresh of superuser proxy group mappings**
-New metrics "login" of type MetricTimeVaryingRate is added under new metrics context name "ugi" and metrics record name "ugi".
+changed protocol name (may be used in hadoop-policy.xml)
+from security.refresh.usertogroups.mappings.protocol.acl to security.refresh.user.mappings.protocol.acl
---
-* [HADOOP-6683](https://issues.apache.org/jira/browse/HADOOP-6683) | *Minor* | **the first optimization: ZlibCompressor does not fully utilize the buffer**
+* [HADOOP-6787](https://issues.apache.org/jira/browse/HADOOP-6787) | *Major* | **Factor out glob pattern code from FileContext and Filesystem**
-Improve the buffer utilization of ZlibCompressor to avoid invoking a JNI per write request.
+**WARNING: No release note provided for this change.**
---
-* [HADOOP-6663](https://issues.apache.org/jira/browse/HADOOP-6663) | *Major* | **BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file**
+* [MAPREDUCE-1836](https://issues.apache.org/jira/browse/MAPREDUCE-1836) | *Major* | **Refresh for proxy superuser config (mr part for HDFS-1096)**
-Fix EOF exception in BlockDecompressorStream when decompressing previous compressed empty file
+changing name of the protocol (may be used in hadoop-policy.xml)
+from
+security.refresh.usertogroups.mappings.protocol.acl
+to
+security.refresh.user.mappings.protocol.acl
---
-* [HADOOP-6599](https://issues.apache.org/jira/browse/HADOOP-6599) | *Major* | **Split RPC metrics into summary and detailed metrics**
+* [MAPREDUCE-1505](https://issues.apache.org/jira/browse/MAPREDUCE-1505) | *Major* | **Cluster class should create the rpc client only when needed**
-Split existing RpcMetrics into RpcMetrics and RpcDetailedMetrics. The new RpcDetailedMetrics has per method usage details and is available under context name "rpc" and record name "detailed-metrics"
+Lazily construct a connection to the JobTracker from the job-submission client.
---
-* [HADOOP-6436](https://issues.apache.org/jira/browse/HADOOP-6436) | *Major* | **Remove auto-generated native build files**
+* [MAPREDUCE-1543](https://issues.apache.org/jira/browse/MAPREDUCE-1543) | *Major* | **Log messages of JobACLsManager should use security logging of HADOOP-6586**
-The native build run when from trunk now requires autotools, libtool and openssl dev libraries.
+Adds the audit logging facility to MapReduce. All authorization/authentication events are logged to audit log. Audit log entries are stored as key=value.
---
-* [HADOOP-6344](https://issues.apache.org/jira/browse/HADOOP-6344) | *Major* | **rm and rmr fail to correctly move the user's files to the trash prior to deleting when they are over quota.**
+* [MAPREDUCE-1533](https://issues.apache.org/jira/browse/MAPREDUCE-1533) | *Major* | **Reduce or remove usage of String.format() usage in CapacityTaskScheduler.updateQSIObjects and Counters.makeEscapedString()**
-Trash feature notifies user of over-quota condition rather than silently deleting files/directories; deletion can be compelled with "rm -skiptrash".
+Incremental enhancements to the JobTracker to optimize heartbeat handling.
---
-* [HADOOP-4675](https://issues.apache.org/jira/browse/HADOOP-4675) | *Major* | **Current Ganglia metrics implementation is incompatible with Ganglia 3.1**
+* [HDFS-1080](https://issues.apache.org/jira/browse/HDFS-1080) | *Major* | **SecondaryNameNode image transfer should use the defined http address rather than local ip address**
-Support for reporting metrics to Ganglia 3.1 servers
+**WARNING: No release note provided for this change.**
---
-* [HDFS-1948](https://issues.apache.org/jira/browse/HDFS-1948) | *Major* | **Forward port 'hdfs-1520 lightweight namenode operation to trigger lease reccovery'**
+* [MAPREDUCE-1813](https://issues.apache.org/jira/browse/MAPREDUCE-1813) | *Major* | **NPE in PipeMapred.MRErrorThread**
-Adds method to NameNode/ClientProtocol that allows for rude revoke of lease on current lease holder
+Fixed an NPE in streaming that occurs when there is no input to reduce and the streaming reducer sends status updates by writing "reporter:status: xxx" statements to stderr.
---
-* [HDFS-1825](https://issues.apache.org/jira/browse/HDFS-1825) | *Major* | **Remove thriftfs contrib**
+* [MAPREDUCE-1829](https://issues.apache.org/jira/browse/MAPREDUCE-1829) | *Major* | **JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()**
-Removed thriftfs contrib component.
+Improved performance of the method JobInProgress.findSpeculativeTask() which is in the critical heartbeat code path.
---
-* [HDFS-1596](https://issues.apache.org/jira/browse/HDFS-1596) | *Major* | **Move secondary namenode checkpoint configs from core-default.xml to hdfs-default.xml**
+* [MAPREDUCE-1887](https://issues.apache.org/jira/browse/MAPREDUCE-1887) | *Major* | **MRAsyncDiskService does not properly absolutize volume root paths**
-Removed references to the older fs.checkpoint.\* properties that resided in core-site.xml
+MAPREDUCE-1887. MRAsyncDiskService now properly absolutizes volume root paths. (Aaron Kimball via zshao)
---
-* [HDFS-1582](https://issues.apache.org/jira/browse/HDFS-1582) | *Major* | **Remove auto-generated native build files**
+* [HADOOP-6835](https://issues.apache.org/jira/browse/HADOOP-6835) | *Major* | **Support concatenated gzip files**
-The native build run when from trunk now requires autotools, libtool and openssl dev libraries.
+Processing of concatenated gzip files formerly stopped (quietly) at the end of the first substream/"member"; now processing will continue to the end of the concatenated stream, like gzip(1) does. (bzip2 support is unaffected by this patch.)
---
-* [HDFS-1560](https://issues.apache.org/jira/browse/HDFS-1560) | *Minor* | **dfs.data.dir permissions should default to 700**
+* [MAPREDUCE-1733](https://issues.apache.org/jira/browse/MAPREDUCE-1733) | *Major* | **Authentication between pipes processes and java counterparts.**
-The permissions on datanode data directories (configured by dfs.datanode.data.dir.perm) now default to 0700. Upon startup, the datanode will automatically change the permissions to match the configured value.
+This jira introduces backward incompatibility. Existing pipes applications MUST be recompiled with new hadoop pipes library once the changes in this jira are deployed.
---
-* [HDFS-1457](https://issues.apache.org/jira/browse/HDFS-1457) | *Major* | **Limit transmission rate when transfering image between primary and secondary NNs**
+* [HDFS-1315](https://issues.apache.org/jira/browse/HDFS-1315) | *Major* | **Add fsck event to audit log and remove other audit log events corresponding to FSCK listStatus and open calls**
-Add a configuration variable dfs.image.transfer.bandwidthPerSec to allow the user to specify the amount of bandwidth for transferring image and edits. Its default value is 0 indicating no throttling.
+When running fsck, audit log events are not logged for listStatus and open are not logged. A new event with cmd=fsck is logged with ugi field set to the user requesting fsck and src field set to the fsck path.
---
-* [HDFS-1435](https://issues.apache.org/jira/browse/HDFS-1435) | *Major* | **Provide an option to store fsimage compressed**
+* [MAPREDUCE-1866](https://issues.apache.org/jira/browse/MAPREDUCE-1866) | *Minor* | **Remove deprecated class org.apache.hadoop.streaming.UTF8ByteArrayUtils**
-This provides an option to store fsimage compressed. The layout version is bumped to -25. The user could configure if s/he wants the fsimage to be compressed or not and which codec to use. By default the fsimage is not compressed.
+Removed public deprecated class org.apache.hadoop.streaming.UTF8ByteArrayUtils.
---
-* [HDFS-1318](https://issues.apache.org/jira/browse/HDFS-1318) | *Major* | **HDFS Namenode and Datanode WebUI information needs to be accessible programmatically for scripts**
+* [HDFS-330](https://issues.apache.org/jira/browse/HDFS-330) | *Trivial* | **Datanode Web UIs should provide robots.txt**
-resubmit the patch for HDFS1318 as Hudson was down last week.
+A robots.txt is now in place which will prevent well behaved crawlers from perusing Hadoop web interfaces.
---
-* [HDFS-1315](https://issues.apache.org/jira/browse/HDFS-1315) | *Major* | **Add fsck event to audit log and remove other audit log events corresponding to FSCK listStatus and open calls**
+* [HDFS-202](https://issues.apache.org/jira/browse/HDFS-202) | *Major* | **Add a bulk FIleSystem.getFileBlockLocations**
-When running fsck, audit log events are not logged for listStatus and open are not logged. A new event with cmd=fsck is logged with ugi field set to the user requesting fsck and src field set to the fsck path.
+**WARNING: No release note provided for this change.**
---
-* [HDFS-1109](https://issues.apache.org/jira/browse/HDFS-1109) | *Major* | **HFTP and URL Encoding**
+* [MAPREDUCE-1780](https://issues.apache.org/jira/browse/MAPREDUCE-1780) | *Major* | **AccessControlList.toString() is used for serialization of ACL in JobStatus.java**
-**WARNING: No release note provided for this incompatible change.**
+Fixes serialization of job-acls in JobStatus to use AccessControlList.write() instead of AccessControlList.toString().
---
-* [HDFS-1096](https://issues.apache.org/jira/browse/HDFS-1096) | *Major* | **allow dfsadmin/mradmin refresh of superuser proxy group mappings**
+* [HADOOP-6905](https://issues.apache.org/jira/browse/HADOOP-6905) | *Major* | **Better logging messages when a delegation token is invalid**
-changed protocol name (may be used in hadoop-policy.xml)
-from security.refresh.usertogroups.mappings.protocol.acl to security.refresh.user.mappings.protocol.acl
+**WARNING: No release note provided for this change.**
---
-* [HDFS-1080](https://issues.apache.org/jira/browse/HDFS-1080) | *Major* | **SecondaryNameNode image transfer should use the defined http address rather than local ip address**
+* [HADOOP-6693](https://issues.apache.org/jira/browse/HADOOP-6693) | *Major* | **Add metrics to track kerberos login activity**
-**WARNING: No release note provided for this incompatible change.**
+New metrics "login" of type MetricTimeVaryingRate is added under new metrics context name "ugi" and metrics record name "ugi".
---
-* [HDFS-1079](https://issues.apache.org/jira/browse/HDFS-1079) | *Major* | **HDFS implementation should throw exceptions defined in AbstractFileSystem**
+* [HDFS-1318](https://issues.apache.org/jira/browse/HDFS-1318) | *Major* | **HDFS Namenode and Datanode WebUI information needs to be accessible programmatically for scripts**
-Specific exceptions are thrown from HDFS implementation and protocol per the interface defined in AbstractFileSystem. The compatibility is not affected as the applications catch IOException and will be able to handle specific exceptions that are subclasses of IOException.
+resubmit the patch for HDFS1318 as Hudson was down last week.
---
-* [HDFS-1061](https://issues.apache.org/jira/browse/HDFS-1061) | *Minor* | **Memory footprint optimization for INodeFile object.**
+* [MAPREDUCE-220](https://issues.apache.org/jira/browse/MAPREDUCE-220) | *Major* | **Collecting cpu and memory usage for MapReduce tasks**
-**WARNING: No release note provided for this incompatible change.**
+Collect cpu and memory statistics per task.
---
-* [HDFS-1035](https://issues.apache.org/jira/browse/HDFS-1035) | *Major* | **Generate Eclipse's .classpath file from Ivy config**
+* [HDFS-712](https://issues.apache.org/jira/browse/HDFS-712) | *Major* | **Move libhdfs from mr to hdfs**
-Added support to auto-generate the Eclipse .classpath file from ivy.
+Moved the libhdfs package to the HDFS subproject.
---
-* [HDFS-903](https://issues.apache.org/jira/browse/HDFS-903) | *Critical* | **NN should verify images and edit logs on startup**
+* [MAPREDUCE-2032](https://issues.apache.org/jira/browse/MAPREDUCE-2032) | *Major* | **TestJobOutputCommitter fails in ant test run**
-Store fsimage MD5 checksum in VERSION file. Validate checksum when loading a fsimage. Layout version bumped.
+Clears a problem that {{TestJobCleanup}} leaves behind files that cause {{TestJobOutputCommitter}} to error out.
---
-* [HDFS-712](https://issues.apache.org/jira/browse/HDFS-712) | *Major* | **Move libhdfs from mr to hdfs**
+* [HADOOP-6922](https://issues.apache.org/jira/browse/HADOOP-6922) | *Major* | **COMMON part of MAPREDUCE-1664**
-Moved the libhdfs package to the HDFS subproject.
+Makes AccessControlList a writable and updates documentation for Job ACLs.
---
-* [HDFS-708](https://issues.apache.org/jira/browse/HDFS-708) | *Major* | **A stress-test tool for HDFS.**
+* [MAPREDUCE-1664](https://issues.apache.org/jira/browse/MAPREDUCE-1664) | *Major* | **Job Acls affect Queue Acls**
-Does not currently provide anything but uniform distribution.
-Uses some older depreciated class interfaces (for mapper and reducer)
-This was tested on 0.20 and 0.22 (locally) so it should be fairly backwards compatible.
+<!-- markdown -->
+* Removed aclsEnabled flag from queues configuration files.
+* Removed the configuration property mapreduce.cluster.job-authorization-enabled.
+* Added mapreduce.cluster.acls.enabled as the single configuration property in mapred-default.xml that enables the authorization checks for all job level and queue level operations.
+* To enable authorization of users to do job level and queue level operations, mapreduce.cluster.acls.enabled is to be set to true in JobTracker's configuration and in all TaskTrackers' configurations.
+* To get access to a job, it is enough for a user to be part of one of the access lists i.e. either job-acl or queue-admins-acl(unlike before, when, one has to be part of both the lists).
+* Queue administrators(configured via acl-administer-jobs) of a queue can do all view-job and modify-job operations on all jobs submitted to that queue.
+* ClusterOwner(who started the mapreduce cluster) and cluster administrators(configured via mapreduce.cluster.permissions.supergroup) can do all job level operations and queue level operations on all jobs on all queues in that cluster irrespective of job-acls and queue-acls configured.
+* JobOwner(who submitted job to a queue) can do all view-job and modify-job operations on his/her job irrespective of job-acls and queue-acls.
+* Since aclsEnabled flag is removed from queues configuration files, "refresh of queues configuration" will not change mapreduce.cluster.acls.enabled on the fly. mapreduce.cluster.acls.enabled can be modified only when restarting the mapreduce cluster.
---
-* [HDFS-330](https://issues.apache.org/jira/browse/HDFS-330) | *Trivial* | **Datanode Web UIs should provide robots.txt**
+* [MAPREDUCE-1517](https://issues.apache.org/jira/browse/MAPREDUCE-1517) | *Major* | **streaming should support running on background**
-A robots.txt is now in place which will prevent well behaved crawlers from perusing Hadoop web interfaces.
+Adds -background option to run a streaming job in background.
---
-* [HDFS-202](https://issues.apache.org/jira/browse/HDFS-202) | *Major* | **Add a bulk FIleSystem.getFileBlockLocations**
+* [MAPREDUCE-2147](https://issues.apache.org/jira/browse/MAPREDUCE-2147) | *Trivial* | **JobInProgress has some redundant lines in its ctor**
-**WARNING: No release note provided for this incompatible change.**
+Remove some redundant lines from JobInProgress's constructor which was re-initializing things unnecessarily.
---
-* [MAPREDUCE-3151](https://issues.apache.org/jira/browse/MAPREDUCE-3151) | *Major* | **Contrib tests failing**
+* [HDFS-1435](https://issues.apache.org/jira/browse/HDFS-1435) | *Major* | **Provide an option to store fsimage compressed**
-Confirmed that problem of finding ivy file occurs w/o patch with ant 1.7, and not with patch (with either ant 1.7 or 1.8).
-Other unit tests are still failing the test steps themselves on my laptop, but that is not due not finding the ivy file.
+This provides an option to store fsimage compressed. The layout version is bumped to -25. The user could configure if s/he wants the fsimage to be compressed or not and which codec to use. By default the fsimage is not compressed.
---
-* [MAPREDUCE-2516](https://issues.apache.org/jira/browse/MAPREDUCE-2516) | *Minor* | **option to control sensitive web actions**
+* [HADOOP-7005](https://issues.apache.org/jira/browse/HADOOP-7005) | *Major* | **Update test-patch.sh to remove callback to Hudson master**
-Configuration option webinterface.private.actions has been renamed to mapreduce.jobtracker.webinterface.trusted
+N/A
---
-* [MAPREDUCE-2410](https://issues.apache.org/jira/browse/MAPREDUCE-2410) | *Minor* | **document multiple keys per reducer oddity in hadoop streaming FAQ**
+* [HADOOP-6663](https://issues.apache.org/jira/browse/HADOOP-6663) | *Major* | **BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file**
-Add an FAQ entry regarding the differences between Java API and Streaming development of MR programs.
+Fix EOF exception in BlockDecompressorStream when decompressing previous compressed empty file
---
-* [MAPREDUCE-2272](https://issues.apache.org/jira/browse/MAPREDUCE-2272) | *Trivial* | **Job ACL file should not be executable**
+* [HDFS-903](https://issues.apache.org/jira/browse/HDFS-903) | *Critical* | **NN should verify images and edit logs on startup**
-Job ACL files now have permissions set to 600 (previously 700).
+Store fsimage MD5 checksum in VERSION file. Validate checksum when loading a fsimage. Layout version bumped.
---
-* [MAPREDUCE-2260](https://issues.apache.org/jira/browse/MAPREDUCE-2260) | *Major* | **Remove auto-generated native build files**
+* [HDFS-1457](https://issues.apache.org/jira/browse/HDFS-1457) | *Major* | **Limit transmission rate when transfering image between primary and secondary NNs**
-The native build run when from trunk now requires autotools, libtool and openssl dev libraries.
+Add a configuration variable dfs.image.transfer.bandwidthPerSec to allow the user to specify the amount of bandwidth for transferring image and edits. Its default value is 0 indicating no throttling.
---
-* [MAPREDUCE-2251](https://issues.apache.org/jira/browse/MAPREDUCE-2251) | *Major* | **Remove mapreduce.job.userhistorylocation config**
+* [HDFS-1035](https://issues.apache.org/jira/browse/HDFS-1035) | *Major* | **Generate Eclipse's .classpath file from Ivy config**
-Remove the now defunct property `mapreduce.job.userhistorylocation`.
+Added support to auto-generate the Eclipse .classpath file from ivy.
---
-* [MAPREDUCE-2147](https://issues.apache.org/jira/browse/MAPREDUCE-2147) | *Trivial* | **JobInProgress has some redundant lines in its ctor**
+* [MAPREDUCE-1592](https://issues.apache.org/jira/browse/MAPREDUCE-1592) | *Major* | **Generate Eclipse's .classpath file from Ivy config**
-Remove some redundant lines from JobInProgress's constructor which was re-initializing things unnecessarily.
+Added support to auto-generate the Eclipse .classpath file from ivy.
---
-* [MAPREDUCE-2096](https://issues.apache.org/jira/browse/MAPREDUCE-2096) | *Blocker* | **Secure local filesystem IO from symlink vulnerabilities**
+* [HADOOP-4675](https://issues.apache.org/jira/browse/HADOOP-4675) | *Major* | **Current Ganglia metrics implementation is incompatible with Ganglia 3.1**
-The TaskTracker now uses the libhadoop JNI library to operate securely on local files when security is enabled. Secure clusters must ensure that libhadoop.so is available to the TaskTracker.
+Support for reporting metrics to Ganglia 3.1 servers
---
-* [MAPREDUCE-2054](https://issues.apache.org/jira/browse/MAPREDUCE-2054) | *Major* | **Hierarchical queue implementation broke dynamic queue addition in Dynamic Scheduler**
+* [MAPREDUCE-1905](https://issues.apache.org/jira/browse/MAPREDUCE-1905) | *Blocker* | **Context.setStatus() and progress() api are ignored**
-Fix Dynamic Priority Scheduler to work with hierarchical queue names
+Moved the api public Counter getCounter(Enum\<?\> counterName), public Counter getCounter(String groupName, String counterName) from org.apache.hadoop.mapreduce.TaskInputOutputContext to org.apache.hadoop.mapreduce.TaskAttemptContext
---
-* [MAPREDUCE-2032](https://issues.apache.org/jira/browse/MAPREDUCE-2032) | *Major* | **TestJobOutputCommitter fails in ant test run**
+* [HADOOP-7013](https://issues.apache.org/jira/browse/HADOOP-7013) | *Major* | **Add boolean field isCorrupt to BlockLocation**
-Clears a problem that {{TestJobCleanup}} leaves behind files that cause {{TestJobOutputCommitter}} to error out.
+This patch has changed the serialization format of BlockLocation.
---
-* [MAPREDUCE-1996](https://issues.apache.org/jira/browse/MAPREDUCE-1996) | *Trivial* | **API: Reducer.reduce() method detail misstatement**
+* [HADOOP-6683](https://issues.apache.org/jira/browse/HADOOP-6683) | *Minor* | **the first optimization: ZlibCompressor does not fully utilize the buffer**
-Fix a misleading documentation note about the usage of Reporter objects in Reducers.
+Improve the buffer utilization of ZlibCompressor to avoid invoking a JNI per write request.
---
-* [MAPREDUCE-1905](https://issues.apache.org/jira/browse/MAPREDUCE-1905) | *Blocker* | **Context.setStatus() and progress() api are ignored**
+* [HDFS-1560](https://issues.apache.org/jira/browse/HDFS-1560) | *Minor* | **dfs.data.dir permissions should default to 700**
-Moved the api public Counter getCounter(Enum\<?\> counterName), public Counter getCounter(String groupName, String counterName) from org.apache.hadoop.mapreduce.TaskInputOutputContext to org.apache.hadoop.mapreduce.TaskAttemptContext
+The permissions on datanode data directories (configured by dfs.datanode.data.dir.perm) now default to 0700. Upon startup, the datanode will automatically change the permissions to match the configured value.
---
-* [MAPREDUCE-1887](https://issues.apache.org/jira/browse/MAPREDUCE-1887) | *Major* | **MRAsyncDiskService does not properly absolutize volume root paths**
+* [MAPREDUCE-2096](https://issues.apache.org/jira/browse/MAPREDUCE-2096) | *Blocker* | **Secure local filesystem IO from symlink vulnerabilities**
-MAPREDUCE-1887. MRAsyncDiskService now properly absolutizes volume root paths. (Aaron Kimball via zshao)
+The TaskTracker now uses the libhadoop JNI library to operate securely on local files when security is enabled. Secure clusters must ensure that libhadoop.so is available to the TaskTracker.
---
-* [MAPREDUCE-1866](https://issues.apache.org/jira/browse/MAPREDUCE-1866) | *Minor* | **Remove deprecated class org.apache.hadoop.streaming.UTF8ByteArrayUtils**
+* [HADOOP-7089](https://issues.apache.org/jira/browse/HADOOP-7089) | *Minor* | **Fix link resolution logic in hadoop-config.sh**
-Removed public deprecated class org.apache.hadoop.streaming.UTF8ByteArrayUtils.
+Updates hadoop-config.sh to always resolve symlinks when determining HADOOP\_HOME. Bash built-ins or POSIX:2001 compliant cmds are now required.
---
-* [MAPREDUCE-1836](https://issues.apache.org/jira/browse/MAPREDUCE-1836) | *Major* | **Refresh for proxy superuser config (mr part for HDFS-1096)**
+* [HADOOP-6436](https://issues.apache.org/jira/browse/HADOOP-6436) | *Major* | **Remove auto-generated native build files**
-changing name of the protocol (may be used in hadoop-policy.xml)
-from
-security.refresh.usertogroups.mappings.protocol.acl
-to
-security.refresh.user.mappings.protocol.acl
+The native build run when from trunk now requires autotools, libtool and openssl dev libraries.
---
-* [MAPREDUCE-1829](https://issues.apache.org/jira/browse/MAPREDUCE-1829) | *Major* | **JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()**
+* [MAPREDUCE-2260](https://issues.apache.org/jira/browse/MAPREDUCE-2260) | *Major* | **Remove auto-generated native build files**
-Improved performance of the method JobInProgress.findSpeculativeTask() which is in the critical heartbeat code path.
+The native build run when from trunk now requires autotools, libtool and openssl dev libraries.
---
-* [MAPREDUCE-1813](https://issues.apache.org/jira/browse/MAPREDUCE-1813) | *Major* | **NPE in PipeMapred.MRErrorThread**
+* [HDFS-1582](https://issues.apache.org/jira/browse/HDFS-1582) | *Major* | **Remove auto-generated native build files**
-Fixed an NPE in streaming that occurs when there is no input to reduce and the streaming reducer sends status updates by writing "reporter:status: xxx" statements to stderr.
+The native build run when from trunk now requires autotools, libtool and openssl dev libraries.
---
-* [MAPREDUCE-1785](https://issues.apache.org/jira/browse/MAPREDUCE-1785) | *Minor* | **Add streaming config option for not emitting the key**
+* [HADOOP-7134](https://issues.apache.org/jira/browse/HADOOP-7134) | *Major* | **configure files that are generated as part of the released tarball need to have executable bit set**
-Added a configuration property "stream.map.input.ignoreKey" to specify whether to ignore key or not while writing input for the mapper. This configuration parameter is valid only if stream.map.input.writer.class is org.apache.hadoop.streaming.io.TextInputWriter.class. For all other InputWriter's, key is always written.
+I have just committed this to trunk and branch-0.22. Thanks Roman!
---
-* [MAPREDUCE-1780](https://issues.apache.org/jira/browse/MAPREDUCE-1780) | *Major* | **AccessControlList.toString() is used for serialization of ACL in JobStatus.java**
+* [MAPREDUCE-2054](https://issues.apache.org/jira/browse/MAPREDUCE-2054) | *Major* | **Hierarchical queue implementation broke dynamic queue addition in Dynamic Scheduler**
-Fixes serialization of job-acls in JobStatus to use AccessControlList.write() instead of AccessControlList.toString().
+Fix Dynamic Priority Scheduler to work with hierarchical queue names
---
-* [MAPREDUCE-1773](https://issues.apache.org/jira/browse/MAPREDUCE-1773) | *Major* | **streaming doesn't support jobclient.output.filter**
+* [MAPREDUCE-1996](https://issues.apache.org/jira/browse/MAPREDUCE-1996) | *Trivial* | **API: Reducer.reduce() method detail misstatement**
-Improved console messaging for streaming jobs by using the generic JobClient API itself instead of the existing streaming-specific code.
+Fix a misleading documentation note about the usage of Reporter objects in Reducers.
---
-* [MAPREDUCE-1733](https://issues.apache.org/jira/browse/MAPREDUCE-1733) | *Major* | **Authentication between pipes processes and java counterparts.**
+* [MAPREDUCE-1159](https://issues.apache.org/jira/browse/MAPREDUCE-1159) | *Trivial* | **Limit Job name on jobtracker.jsp to be 80 char long**
-This jira introduces backward incompatibility. Existing pipes applications MUST be recompiled with new hadoop pipes library once the changes in this jira are deployed.
+Job names on jobtracker.jsp should be 80 characters long at most.
---
-* [MAPREDUCE-1707](https://issues.apache.org/jira/browse/MAPREDUCE-1707) | *Major* | **TaskRunner can get NPE in getting ugi from TaskTracker**
+* [MAPREDUCE-2272](https://issues.apache.org/jira/browse/MAPREDUCE-2272) | *Trivial* | **Job ACL file should not be executable**
-Fixed a bug that causes TaskRunner to get NPE in getting ugi from TaskTracker and subsequently crashes it resulting in a failing task after task-timeout period.
+Job ACL files now have permissions set to 600 (previously 700).
---
-* [MAPREDUCE-1683](https://issues.apache.org/jira/browse/MAPREDUCE-1683) | *Major* | **Remove JNI calls from ClusterStatus cstr**
+* [MAPREDUCE-2251](https://issues.apache.org/jira/browse/MAPREDUCE-2251) | *Major* | **Remove mapreduce.job.userhistorylocation config**
-Removes JNI calls to get jvm current/max heap usage in ClusterStatus. Any instances of ClusterStatus serialized in a prior version will not be correctly deserialized using the updated class.
+Remove the now defunct property \`mapreduce.job.userhistorylocation\`.
---
-* [MAPREDUCE-1680](https://issues.apache.org/jira/browse/MAPREDUCE-1680) | *Major* | **Add a metrics to track the number of heartbeats processed**
+* [HADOOP-7156](https://issues.apache.org/jira/browse/HADOOP-7156) | *Critical* | **getpwuid\_r is not thread-safe on RHEL6**
-Added a metric to track number of heartbeats processed by the JobTracker.
+Adds a new configuration hadoop.work.around.non.threadsafe.getpwuid which can be used to enable a mutex around this call to workaround thread-unsafe implementations of getpwuid\_r. Users should consult http://wiki.apache.org/hadoop/KnownBrokenPwuidImplementations for a list of such systems.
---
-* [MAPREDUCE-1664](https://issues.apache.org/jira/browse/MAPREDUCE-1664) | *Major* | **Job Acls affect Queue Acls**
+* [HDFS-1596](https://issues.apache.org/jira/browse/HDFS-1596) | *Major* | **Move secondary namenode checkpoint configs from core-default.xml to hdfs-default.xml**
-<!-- markdown -->
-* Removed aclsEnabled flag from queues configuration files.
-* Removed the configuration property mapreduce.cluster.job-authorization-enabled.
-* Added mapreduce.cluster.acls.enabled as the single configuration property in mapred-default.xml that enables the authorization checks for all job level and queue level operations.
-* To enable authorization of users to do job level and queue level operations, mapreduce.cluster.acls.enabled is to be set to true in JobTracker's configuration and in all TaskTrackers' configurations.
-* To get access to a job, it is enough for a user to be part of one of the access lists i.e. either job-acl or queue-admins-acl(unlike before, when, one has to be part of both the lists).
-* Queue administrators(configured via acl-administer-jobs) of a queue can do all view-job and modify-job operations on all jobs submitted to that queue.
-* ClusterOwner(who started the mapreduce cluster) and cluster administrators(configured via mapreduce.cluster.permissions.supergroup) can do all job level operations and queue level operations on all jobs on all queues in that cluster irrespective of job-acls and queue-acls configured.
-* JobOwner(who submitted job to a queue) can do all view-job and modify-job operations on his/her job irrespective of job-acls and queue-acls.
-* Since aclsEnabled flag is removed from queues configuration files, "refresh of queues configuration" will not change mapreduce.cluster.acls.enabled on the fly. mapreduce.cluster.acls.enabled can be modified only when restarting the mapreduce cluster.
+Removed references to the older fs.checkpoint.\* properties that resided in core-site.xml
---
-* [MAPREDUCE-1592](https://issues.apache.org/jira/browse/MAPREDUCE-1592) | *Major* | **Generate Eclipse's .classpath file from Ivy config**
+* [HADOOP-7117](https://issues.apache.org/jira/browse/HADOOP-7117) | *Major* | **Move secondary namenode checkpoint configs from core-default.xml to hdfs-default.xml**
-Added support to auto-generate the Eclipse .classpath file from ivy.
+Removed references to the older fs.checkpoint.\* properties that resided in core-site.xml
---
-* [MAPREDUCE-1558](https://issues.apache.org/jira/browse/MAPREDUCE-1558) | *Major* | **specify correct server principal for RefreshAuthorizationPolicyProtocol and RefreshUserToGroupMappingsProtocol protocols in MRAdmin (for HADOOP-6612)**
+* [HADOOP-6949](https://issues.apache.org/jira/browse/HADOOP-6949) | *Major* | **Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting**
-new config:
-hadoop.security.service.user.name.key
-this setting points to the server principal for RefreshUserToGroupMappingsProtocol.
-The value should be either NN or JT principal depending if it is used in DFAdmin or MRAdmin. The value is set by the application. No need for default value.
+Increments the RPC protocol version in org.apache.hadoop.ipc.Server from 4 to 5.
+Introduces ArrayPrimitiveWritable for a much more efficient wire format to transmit arrays of primitives over RPC. ObjectWritable uses the new writable for array of primitives for RPC and continues to use existing format for on-disk data.
---
-* [MAPREDUCE-1543](https://issues.apache.org/jira/browse/MAPREDUCE-1543) | *Major* | **Log messages of JobACLsManager should use security logging of HADOOP-6586**
+* [HADOOP-7193](https://issues.apache.org/jira/browse/HADOOP-7193) | *Minor* | **Help message is wrong for touchz command.**
-Adds the audit logging facility to MapReduce. All authorization/authentication events are logged to audit log. Audit log entries are stored as key=value.
+Updated the help for the touchz command.
---
-* [MAPREDUCE-1533](https://issues.apache.org/jira/browse/MAPREDUCE-1533) | *Major* | **Reduce or remove usage of String.format() usage in CapacityTaskScheduler.updateQSIObjects and Counters.makeEscapedString()**
+* [HADOOP-7229](https://issues.apache.org/jira/browse/HADOOP-7229) | *Major* | **Absolute path to kinit in auto-renewal thread**
-Incremental enhancements to the JobTracker to optimize heartbeat handling.
+When Hadoop's Kerberos integration is enabled, it is now required that either {{kinit}} be on the path for user accounts running the Hadoop client, or that the {{hadoop.kerberos.kinit.command}} configuration option be manually set to the absolute path to {{kinit}}.
---
-* [MAPREDUCE-1517](https://issues.apache.org/jira/browse/MAPREDUCE-1517) | *Major* | **streaming should support running on background**
+* [MAPREDUCE-2410](https://issues.apache.org/jira/browse/MAPREDUCE-2410) | *Minor* | **document multiple keys per reducer oddity in hadoop streaming FAQ**
-Adds -background option to run a streaming job in background.
+Add an FAQ entry regarding the differences between Java API and Streaming development of MR programs.
---
-* [MAPREDUCE-1505](https://issues.apache.org/jira/browse/MAPREDUCE-1505) | *Major* | **Cluster class should create the rpc client only when needed**
+* [HDFS-1825](https://issues.apache.org/jira/browse/HDFS-1825) | *Major* | **Remove thriftfs contrib**
-Lazily construct a connection to the JobTracker from the job-submission client.
+Removed thriftfs contrib component.
---
-* [MAPREDUCE-1354](https://issues.apache.org/jira/browse/MAPREDUCE-1354) | *Critical* | **Incremental enhancements to the JobTracker for better scalability**
+* [HADOOP-7137](https://issues.apache.org/jira/browse/HADOOP-7137) | *Major* | **Remove hod contrib**
-Incremental enhancements to the JobTracker include a no-lock version of JT.getTaskCompletion events, no lock on the JT while doing i/o during job-submission and several fixes to cut down configuration parsing during heartbeat-handling.
+Removed contrib related build targets.
---
-* [MAPREDUCE-1159](https://issues.apache.org/jira/browse/MAPREDUCE-1159) | *Trivial* | **Limit Job name on jobtracker.jsp to be 80 char long**
+* [HADOOP-7192](https://issues.apache.org/jira/browse/HADOOP-7192) | *Trivial* | **fs -stat docs aren't updated to reflect the format features**
-Job names on jobtracker.jsp should be 80 characters long at most.
+Updated the web documentation to reflect the formatting abilities of 'fs -stat'.
---
-* [MAPREDUCE-1118](https://issues.apache.org/jira/browse/MAPREDUCE-1118) | *Major* | **Capacity Scheduler scheduling information is hard to read / should be tabular format**
+* [HADOOP-7302](https://issues.apache.org/jira/browse/HADOOP-7302) | *Major* | **webinterface.private.actions should not be in common**
-Add CapacityScheduler servlet to enhance web UI for queue information.
+Option webinterface.private.actions has been renamed to mapreduce.jobtracker.webinterface.trusted and should be specified in mapred-site.xml instead of core-site.xml
---
-* [MAPREDUCE-927](https://issues.apache.org/jira/browse/MAPREDUCE-927) | *Major* | **Cleanup of task-logs should happen in TaskTracker instead of the Child**
+* [MAPREDUCE-2516](https://issues.apache.org/jira/browse/MAPREDUCE-2516) | *Minor* | **option to control sensitive web actions**
-Moved Task log cleanup into a separate thread in TaskTracker.
-Added configuration "mapreduce.job.userlog.retain.hours" to specify the time(in hours) for which the user-logs are to be retained after the job completion.
+Configuration option webinterface.private.actions has been renamed to mapreduce.jobtracker.webinterface.trusted
---
-* [MAPREDUCE-572](https://issues.apache.org/jira/browse/MAPREDUCE-572) | *Minor* | **If #link is missing from uri format of -cacheArchive then streaming does not throw error.**
+* [HDFS-1948](https://issues.apache.org/jira/browse/HDFS-1948) | *Major* | **Forward port 'hdfs-1520 lightweight namenode operation to trigger lease reccovery'**
-Improved streaming job failure when #link is missing from uri format of -cacheArchive. Earlier it used to fail when launching individual tasks, now it fails during job submission itself.
+Adds method to NameNode/ClientProtocol that allows for rude revoke of lease on current lease holder
---
-* [MAPREDUCE-478](https://issues.apache.org/jira/browse/MAPREDUCE-478) | *Minor* | **separate jvm param for mapper and reducer**
-
-Allow map and reduce jvm parameters, environment variables and ulimit to be set separately.
+* [MAPREDUCE-3151](https://issues.apache.org/jira/browse/MAPREDUCE-3151) | *Major* | **Contrib tests failing**
-Configuration changes:
- add mapred.map.child.java.opts
- add mapred.reduce.child.java.opts
- add mapred.map.child.env
- add mapred.reduce.child.ulimit
- add mapred.map.child.env
- add mapred.reduce.child.ulimit
- deprecated mapred.child.java.opts
- deprecated mapred.child.env
- deprecated mapred.child.ulimit
+Confirmed that problem of finding ivy file occurs w/o patch with ant 1.7, and not with patch (with either ant 1.7 or 1.8).
+Other unit tests are still failing the test steps themselves on my laptop, but that is not due not finding the ivy file.
---
-* [MAPREDUCE-220](https://issues.apache.org/jira/browse/MAPREDUCE-220) | *Major* | **Collecting cpu and memory usage for MapReduce tasks**
+* [MAPREDUCE-1118](https://issues.apache.org/jira/browse/MAPREDUCE-1118) | *Major* | **Capacity Scheduler scheduling information is hard to read / should be tabular format**
-Collect cpu and memory statistics per task.
+Add CapacityScheduler servlet to enhance web UI for queue information.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/0.22.1/CHANGES.0.22.1.md
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.22.1/CHANGES.0.22.1.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.22.1/CHANGES.0.22.1.md
index ac195f0..2346cad 100644
--- a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.22.1/CHANGES.0.22.1.md
+++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.22.1/CHANGES.0.22.1.md
@@ -18,7 +18,7 @@
-->
# Apache Hadoop Changelog
-## Release 0.22.1 - Unreleased (as of 2016-03-04)
+## Release 0.22.1 - Unreleased (as of 2017-08-28)
### INCOMPATIBLE CHANGES:
@@ -27,18 +27,12 @@
| [HADOOP-6453](https://issues.apache.org/jira/browse/HADOOP-6453) | Hadoop wrapper script shouldn't ignore an existing JAVA\_LIBRARY\_PATH | Minor | scripts | Chad Metcalf | |
-### IMPORTANT ISSUES:
-
-| JIRA | Summary | Priority | Component | Reporter | Contributor |
-|:---- |:---- | :--- |:---- |:---- |:---- |
-
-
### NEW FEATURES:
| JIRA | Summary | Priority | Component | Reporter | Contributor |
|:---- |:---- | :--- |:---- |:---- |:---- |
-| [HADOOP-7937](https://issues.apache.org/jira/browse/HADOOP-7937) | Forward port SequenceFile#syncFs and friends from Hadoop 1.x | Major | io | Eli Collins | Tom White |
| [HADOOP-7119](https://issues.apache.org/jira/browse/HADOOP-7119) | add Kerberos HTTP SPNEGO authentication support to Hadoop JT/NN/DN/TT web-consoles | Major | security | Alejandro Abdelnur | Alejandro Abdelnur |
+| [HADOOP-7937](https://issues.apache.org/jira/browse/HADOOP-7937) | Forward port SequenceFile#syncFs and friends from Hadoop 1.x | Major | io | Eli Collins | Tom White |
| [MAPREDUCE-3837](https://issues.apache.org/jira/browse/MAPREDUCE-3837) | Job tracker is not able to recover job in case of crash and after that no user can submit job. | Major | . | Mayank Bansal | Mayank Bansal |
@@ -46,13 +40,13 @@
| JIRA | Summary | Priority | Component | Reporter | Contributor |
|:---- |:---- | :--- |:---- |:---- |:---- |
-| [HADOOP-7338](https://issues.apache.org/jira/browse/HADOOP-7338) | LocalDirAllocator improvements for MR-2178 | Major | . | Todd Lipcon | Benoy Antony |
-| [HADOOP-7272](https://issues.apache.org/jira/browse/HADOOP-7272) | Remove unnecessary security related info logs | Major | ipc, security | Suresh Srinivas | Suresh Srinivas |
| [HADOOP-6995](https://issues.apache.org/jira/browse/HADOOP-6995) | Allow wildcards to be used in ProxyUsers configurations | Minor | security | Todd Lipcon | Todd Lipcon |
-| [HDFS-2246](https://issues.apache.org/jira/browse/HDFS-2246) | Shortcut a local client reads to a Datanodes files directly | Major | . | Sanjay Radia | Jitendra Nath Pandey |
| [HDFS-1601](https://issues.apache.org/jira/browse/HDFS-1601) | Pipeline ACKs are sent as lots of tiny TCP packets | Major | datanode | Todd Lipcon | Todd Lipcon |
-| [MAPREDUCE-4405](https://issues.apache.org/jira/browse/MAPREDUCE-4405) | Adding test case for HierarchicalQueue in TestJobQueueClient | Minor | client | Mayank Bansal | Mayank Bansal |
+| [HADOOP-7272](https://issues.apache.org/jira/browse/HADOOP-7272) | Remove unnecessary security related info logs | Major | ipc, security | Suresh Srinivas | Suresh Srinivas |
+| [HDFS-2246](https://issues.apache.org/jira/browse/HDFS-2246) | Shortcut a local client reads to a Datanodes files directly | Major | . | Sanjay Radia | Jitendra Nath Pandey |
+| [HADOOP-7338](https://issues.apache.org/jira/browse/HADOOP-7338) | LocalDirAllocator improvements for MR-2178 | Major | . | Todd Lipcon | Benoy Antony |
| [MAPREDUCE-4403](https://issues.apache.org/jira/browse/MAPREDUCE-4403) | Adding test case for resubmission of jobs in TestRecoveryManager | Minor | jobtracker | Mayank Bansal | Mayank Bansal |
+| [MAPREDUCE-4405](https://issues.apache.org/jira/browse/MAPREDUCE-4405) | Adding test case for HierarchicalQueue in TestJobQueueClient | Minor | client | Mayank Bansal | Mayank Bansal |
| [MAPREDUCE-4349](https://issues.apache.org/jira/browse/MAPREDUCE-4349) | Distributed Cache gives inconsistent result if cache Archive files get deleted from task tracker | Minor | . | Mayank Bansal | Mayank Bansal |
| [MAPREDUCE-2353](https://issues.apache.org/jira/browse/MAPREDUCE-2353) | Make the MR changes to reflect the API changes in SecureIO library | Major | security, task, tasktracker | Devaraj Das | Benoy Antony |
| [MAPREDUCE-1521](https://issues.apache.org/jira/browse/MAPREDUCE-1521) | Protection against incorrectly configured reduces | Major | jobtracker | Arun C Murthy | Mahadev konar |
@@ -62,58 +56,52 @@
| JIRA | Summary | Priority | Component | Reporter | Contributor |
|:---- |:---- | :--- |:---- |:---- |:---- |
-| [HADOOP-7680](https://issues.apache.org/jira/browse/HADOOP-7680) | TestHardLink fails on Mac OS X, when gnu stat is in path | Major | . | Milind Bhandarkar | Milind Bhandarkar |
+| [MAPREDUCE-2420](https://issues.apache.org/jira/browse/MAPREDUCE-2420) | JobTracker should be able to renew delegation token over HTTP | Major | . | Boris Shkolnik | Boris Shkolnik |
+| [MAPREDUCE-2452](https://issues.apache.org/jira/browse/MAPREDUCE-2452) | Delegation token cancellation shouldn't hold global JobTracker lock | Major | jobtracker | Devaraj Das | Devaraj Das |
| [HADOOP-7621](https://issues.apache.org/jira/browse/HADOOP-7621) | alfredo config should be in a file not readable by users | Critical | security | Alejandro Abdelnur | Aaron T. Myers |
-| [HADOOP-7115](https://issues.apache.org/jira/browse/HADOOP-7115) | Add a cache for getpwuid\_r and getpwgid\_r calls | Major | . | Arun C Murthy | Alejandro Abdelnur |
-| [HDFS-3402](https://issues.apache.org/jira/browse/HDFS-3402) | Fix hdfs scripts for secure datanodes | Minor | scripts, security | Benoy Antony | Benoy Antony |
-| [HDFS-3368](https://issues.apache.org/jira/browse/HDFS-3368) | Missing blocks due to bad DataNodes coming up and down. | Major | namenode | Konstantin Shvachko | Konstantin Shvachko |
-| [HDFS-2991](https://issues.apache.org/jira/browse/HDFS-2991) | failure to load edits: ClassCastException | Blocker | namenode | Todd Lipcon | Todd Lipcon |
-| [HDFS-2877](https://issues.apache.org/jira/browse/HDFS-2877) | If locking of a storage dir fails, it will remove the other NN's lock file on exit | Major | namenode | Todd Lipcon | Todd Lipcon |
-| [HDFS-2718](https://issues.apache.org/jira/browse/HDFS-2718) | Optimize OP\_ADD in edits loading | Major | namenode | Konstantin Shvachko | Konstantin Shvachko |
| [HDFS-2698](https://issues.apache.org/jira/browse/HDFS-2698) | BackupNode is downloading image from NameNode for every checkpoint | Major | namenode | Konstantin Shvachko | Konstantin Shvachko |
| [HDFS-1910](https://issues.apache.org/jira/browse/HDFS-1910) | when dfs.name.dir and dfs.name.edits.dir are same fsimage will be saved twice every time | Minor | namenode | Gokul | |
-| [HDFS-1584](https://issues.apache.org/jira/browse/HDFS-1584) | Need to check TGT and renew if needed when fetching delegation tokens using HFTP | Major | security | Kan Zhang | Benoy Antony |
-| [MAPREDUCE-5706](https://issues.apache.org/jira/browse/MAPREDUCE-5706) | toBeDeleted parent directories aren't being cleaned up | Major | security | Robert Kanter | Robert Kanter |
-| [MAPREDUCE-4404](https://issues.apache.org/jira/browse/MAPREDUCE-4404) | Adding Test case for TestMRJobClient to verify the user name | Minor | client | Mayank Bansal | Mayank Bansal |
-| [MAPREDUCE-4360](https://issues.apache.org/jira/browse/MAPREDUCE-4360) | Capacity Scheduler Hierarchical leaf queue does not honor the max capacity of container queue | Major | . | Mayank Bansal | Mayank Bansal |
-| [MAPREDUCE-4318](https://issues.apache.org/jira/browse/MAPREDUCE-4318) | TestRecoveryManager should not use raw and deprecated configuration parameters. | Major | test | Konstantin Shvachko | Benoy Antony |
-| [MAPREDUCE-4314](https://issues.apache.org/jira/browse/MAPREDUCE-4314) | Synchronization in JvmManager for 0.22 branch | Major | tasktracker | Konstantin Shvachko | Benoy Antony |
+| [MAPREDUCE-3593](https://issues.apache.org/jira/browse/MAPREDUCE-3593) | MAPREDUCE Impersonation is not working in 22 | Major | job submission | Mayank Bansal | Mayank Bansal |
+| [MAPREDUCE-3725](https://issues.apache.org/jira/browse/MAPREDUCE-3725) | Hadoop 22 hadoop job -list returns user name as NULL | Major | client | Mayank Bansal | Mayank Bansal |
+| [HDFS-2718](https://issues.apache.org/jira/browse/HDFS-2718) | Optimize OP\_ADD in edits loading | Major | namenode | Konstantin Shvachko | Konstantin Shvachko |
+| [HDFS-2877](https://issues.apache.org/jira/browse/HDFS-2877) | If locking of a storage dir fails, it will remove the other NN's lock file on exit | Major | namenode | Todd Lipcon | Todd Lipcon |
+| [HADOOP-7680](https://issues.apache.org/jira/browse/HADOOP-7680) | TestHardLink fails on Mac OS X, when gnu stat is in path | Major | . | Milind Bhandarkar | Milind Bhandarkar |
+| [HDFS-2991](https://issues.apache.org/jira/browse/HDFS-2991) | failure to load edits: ClassCastException | Blocker | namenode | Todd Lipcon | Todd Lipcon |
| [MAPREDUCE-4164](https://issues.apache.org/jira/browse/MAPREDUCE-4164) | Hadoop 22 Exception thrown after task completion causes its reexecution | Major | tasktracker | Mayank Bansal | Mayank Bansal |
| [MAPREDUCE-3863](https://issues.apache.org/jira/browse/MAPREDUCE-3863) | 0.22 branch mvn deploy is not publishing hadoop-streaming JAR | Critical | build | Alejandro Abdelnur | Benoy Antony |
-| [MAPREDUCE-3725](https://issues.apache.org/jira/browse/MAPREDUCE-3725) | Hadoop 22 hadoop job -list returns user name as NULL | Major | client | Mayank Bansal | Mayank Bansal |
-| [MAPREDUCE-3593](https://issues.apache.org/jira/browse/MAPREDUCE-3593) | MAPREDUCE Impersonation is not working in 22 | Major | job submission | Mayank Bansal | Mayank Bansal |
-| [MAPREDUCE-2452](https://issues.apache.org/jira/browse/MAPREDUCE-2452) | Delegation token cancellation shouldn't hold global JobTracker lock | Major | jobtracker | Devaraj Das | Devaraj Das |
-| [MAPREDUCE-2420](https://issues.apache.org/jira/browse/MAPREDUCE-2420) | JobTracker should be able to renew delegation token over HTTP | Major | . | Boris Shkolnik | Boris Shkolnik |
-| [MAPREDUCE-2377](https://issues.apache.org/jira/browse/MAPREDUCE-2377) | task-controller fails to parse configuration if it doesn't end in \n | Major | task-controller | Todd Lipcon | Benoy Antony |
+| [HDFS-3368](https://issues.apache.org/jira/browse/HDFS-3368) | Missing blocks due to bad DataNodes coming up and down. | Major | namenode | Konstantin Shvachko | Konstantin Shvachko |
| [MAPREDUCE-2178](https://issues.apache.org/jira/browse/MAPREDUCE-2178) | Race condition in LinuxTaskController permissions handling | Major | security, task-controller | Todd Lipcon | Benoy Antony |
-
-
-### TESTS:
-
-| JIRA | Summary | Priority | Component | Reporter | Contributor |
-|:---- |:---- | :--- |:---- |:---- |:---- |
+| [MAPREDUCE-4314](https://issues.apache.org/jira/browse/MAPREDUCE-4314) | Synchronization in JvmManager for 0.22 branch | Major | tasktracker | Konstantin Shvachko | Benoy Antony |
+| [MAPREDUCE-2377](https://issues.apache.org/jira/browse/MAPREDUCE-2377) | task-controller fails to parse configuration if it doesn't end in \\n | Major | task-controller | Todd Lipcon | Benoy Antony |
+| [MAPREDUCE-4318](https://issues.apache.org/jira/browse/MAPREDUCE-4318) | TestRecoveryManager should not use raw and deprecated configuration parameters. | Major | test | Konstantin Shvachko | Benoy Antony |
+| [MAPREDUCE-4360](https://issues.apache.org/jira/browse/MAPREDUCE-4360) | Capacity Scheduler Hierarchical leaf queue does not honor the max capacity of container queue | Major | . | Mayank Bansal | Mayank Bansal |
+| [HDFS-1584](https://issues.apache.org/jira/browse/HDFS-1584) | Need to check TGT and renew if needed when fetching delegation tokens using HFTP | Major | security | Kan Zhang | Benoy Antony |
+| [HDFS-3402](https://issues.apache.org/jira/browse/HDFS-3402) | Fix hdfs scripts for secure datanodes | Minor | scripts, security | Benoy Antony | Benoy Antony |
+| [HADOOP-7115](https://issues.apache.org/jira/browse/HADOOP-7115) | Add a cache for getpwuid\_r and getpwgid\_r calls | Major | . | Arun C Murthy | Alejandro Abdelnur |
+| [MAPREDUCE-4404](https://issues.apache.org/jira/browse/MAPREDUCE-4404) | Adding Test case for TestMRJobClient to verify the user name | Minor | client | Mayank Bansal | Mayank Bansal |
+| [MAPREDUCE-5706](https://issues.apache.org/jira/browse/MAPREDUCE-5706) | toBeDeleted parent directories aren't being cleaned up | Major | security | Robert Kanter | Robert Kanter |
### SUB-TASKS:
| JIRA | Summary | Priority | Component | Reporter | Contributor |
|:---- |:---- | :--- |:---- |:---- |:---- |
-| [HADOOP-8383](https://issues.apache.org/jira/browse/HADOOP-8383) | TestKerberosAuthenticator fails | Minor | security | Benoy Antony | Benoy Antony |
-| [HADOOP-8381](https://issues.apache.org/jira/browse/HADOOP-8381) | Substitute \_HOST with hostname for HTTP principals | Minor | security | Benoy Antony | Benoy Antony |
| [HDFS-2886](https://issues.apache.org/jira/browse/HDFS-2886) | CreateEditLogs should generate a realistic edit log. | Major | namenode | Konstantin Shvachko | Konstantin Shvachko |
+| [HADOOP-8381](https://issues.apache.org/jira/browse/HADOOP-8381) | Substitute \_HOST with hostname for HTTP principals | Minor | security | Benoy Antony | Benoy Antony |
+| [HADOOP-8383](https://issues.apache.org/jira/browse/HADOOP-8383) | TestKerberosAuthenticator fails | Minor | security | Benoy Antony | Benoy Antony |
### OTHER:
| JIRA | Summary | Priority | Component | Reporter | Contributor |
|:---- |:---- | :--- |:---- |:---- |:---- |
-| [HADOOP-8357](https://issues.apache.org/jira/browse/HADOOP-8357) | Restore security in Hadoop 0.22 branch | Major | security | Konstantin Shvachko | Benoy Antony |
-| [MAPREDUCE-4249](https://issues.apache.org/jira/browse/MAPREDUCE-4249) | Fix failures in streaming test TestFileArgs | Minor | security | Benoy Antony | Benoy Antony |
-| [MAPREDUCE-4248](https://issues.apache.org/jira/browse/MAPREDUCE-4248) | TestRecoveryManager fails | Minor | security | Benoy Antony | Benoy Antony |
-| [MAPREDUCE-4247](https://issues.apache.org/jira/browse/MAPREDUCE-4247) | TestTaskTrackerLocalization fails | Minor | security | Benoy Antony | Benoy Antony |
-| [MAPREDUCE-4246](https://issues.apache.org/jira/browse/MAPREDUCE-4246) | Failure in deleting user directories in Secure hadoop | Major | security | Benoy Antony | Benoy Antony |
-| [MAPREDUCE-4244](https://issues.apache.org/jira/browse/MAPREDUCE-4244) | Fix an issue related to do with setting of correct groups for tasks | Minor | security | Benoy Antony | Benoy Antony |
-| [MAPREDUCE-4243](https://issues.apache.org/jira/browse/MAPREDUCE-4243) | Modify mapreduce build to include task-controller | Minor | build | Benoy Antony | Benoy Antony |
| [MAPREDUCE-4240](https://issues.apache.org/jira/browse/MAPREDUCE-4240) | Revert MAPREDUCE-2767 | Minor | security | Benoy Antony | Benoy Antony |
+| [MAPREDUCE-4243](https://issues.apache.org/jira/browse/MAPREDUCE-4243) | Modify mapreduce build to include task-controller | Minor | build | Benoy Antony | Benoy Antony |
+| [MAPREDUCE-4244](https://issues.apache.org/jira/browse/MAPREDUCE-4244) | Fix an issue related to do with setting of correct groups for tasks | Minor | security | Benoy Antony | Benoy Antony |
+| [MAPREDUCE-4246](https://issues.apache.org/jira/browse/MAPREDUCE-4246) | Failure in deleting user directories in Secure hadoop | Major | security | Benoy Antony | Benoy Antony |
+| [MAPREDUCE-4247](https://issues.apache.org/jira/browse/MAPREDUCE-4247) | TestTaskTrackerLocalization fails | Minor | security | Benoy Antony | Benoy Antony |
+| [MAPREDUCE-4248](https://issues.apache.org/jira/browse/MAPREDUCE-4248) | TestRecoveryManager fails | Minor | security | Benoy Antony | Benoy Antony |
+| [MAPREDUCE-4249](https://issues.apache.org/jira/browse/MAPREDUCE-4249) | Fix failures in streaming test TestFileArgs | Minor | security | Benoy Antony | Benoy Antony |
+| [HADOOP-8357](https://issues.apache.org/jira/browse/HADOOP-8357) | Restore security in Hadoop 0.22 branch | Major | security | Konstantin Shvachko | Benoy Antony |
http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/0.22.1/RELEASENOTES.0.22.1.md
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.22.1/RELEASENOTES.0.22.1.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.22.1/RELEASENOTES.0.22.1.md
index f08f501..9230479 100644
--- a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.22.1/RELEASENOTES.0.22.1.md
+++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.22.1/RELEASENOTES.0.22.1.md
@@ -23,13 +23,6 @@ These release notes cover new developer and user-facing incompatibilities, impor
---
-* [HADOOP-7119](https://issues.apache.org/jira/browse/HADOOP-7119) | *Major* | **add Kerberos HTTP SPNEGO authentication support to Hadoop JT/NN/DN/TT web-consoles**
-
-Adding support for Kerberos HTTP SPNEGO authentication to the Hadoop web-consoles
-
-
----
-
* [HADOOP-6995](https://issues.apache.org/jira/browse/HADOOP-6995) | *Minor* | **Allow wildcards to be used in ProxyUsers configurations**
When configuring proxy users and hosts, the special wildcard value "\*" may be specified to match any host or any user.
@@ -37,9 +30,9 @@ When configuring proxy users and hosts, the special wildcard value "\*" may be s
---
-* [HADOOP-6453](https://issues.apache.org/jira/browse/HADOOP-6453) | *Minor* | **Hadoop wrapper script shouldn't ignore an existing JAVA\_LIBRARY\_PATH**
+* [HADOOP-7119](https://issues.apache.org/jira/browse/HADOOP-7119) | *Major* | **add Kerberos HTTP SPNEGO authentication support to Hadoop JT/NN/DN/TT web-consoles**
-**WARNING: No release note provided for this incompatible change.**
+Adding support for Kerberos HTTP SPNEGO authentication to the Hadoop web-consoles
---
@@ -61,4 +54,9 @@ c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum chec
Submitting the patch after setting the user name at the client side
+---
+
+* [HADOOP-6453](https://issues.apache.org/jira/browse/HADOOP-6453) | *Minor* | **Hadoop wrapper script shouldn't ignore an existing JAVA\_LIBRARY\_PATH**
+
+**WARNING: No release note provided for this change.**
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org