You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by ae...@apache.org on 2017/09/01 20:01:21 UTC
[33/74] [abbrv] [partial] hadoop git commit: HADOOP-14364. refresh
changelog/release notes with newer Apache Yetus build
http://git-wip-us.apache.org/repos/asf/hadoop/blob/19041008/hadoop-common-project/hadoop-common/src/site/markdown/release/0.21.0/RELEASENOTES.0.21.0.md
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.21.0/RELEASENOTES.0.21.0.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.21.0/RELEASENOTES.0.21.0.md
index 9f341c1..8a8bef3 100644
--- a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.21.0/RELEASENOTES.0.21.0.md
+++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.21.0/RELEASENOTES.0.21.0.md
@@ -23,298 +23,298 @@ These release notes cover new developer and user-facing incompatibilities, impor
---
-* [HADOOP-6813](https://issues.apache.org/jira/browse/HADOOP-6813) | *Blocker* | **Add a new newInstance method in FileSystem that takes a "user" as argument**
+* [HADOOP-4895](https://issues.apache.org/jira/browse/HADOOP-4895) | *Major* | **Remove deprecated methods in DFSClient**
-I've just committed this to 0.21.
+Removed deprecated methods DFSClient.getHints() and DFSClient.isDirectory().
---
-* [HADOOP-6748](https://issues.apache.org/jira/browse/HADOOP-6748) | *Major* | **Remove hadoop.cluster.administrators**
+* [HADOOP-4941](https://issues.apache.org/jira/browse/HADOOP-4941) | *Major* | **Remove getBlockSize(Path f), getLength(Path f) and getReplication(Path src)**
-Removed configuration property "hadoop.cluster.administrators". Added constructor public HttpServer(String name, String bindAddress, int port, boolean findPort, Configuration conf, AccessControlList adminsAcl) in HttpServer, which takes cluster administrators acl as a parameter.
+Removed deprecated FileSystem methods getBlockSize(Path f), getLength(Path f), and getReplication(Path src).
---
-* [HADOOP-6701](https://issues.apache.org/jira/browse/HADOOP-6701) | *Minor* | ** Incorrect exit codes for "dfs -chown", "dfs -chgrp"**
+* [HADOOP-4268](https://issues.apache.org/jira/browse/HADOOP-4268) | *Major* | **Permission checking in fsck**
-Commands chmod, chown and chgrp now returns non zero exit code and an error message on failure instead of returning zero.
+Fsck now checks permissions as directories are traversed. Any user can now use fsck, but information is provided only for directories the user has permission to read.
---
-* [HADOOP-6692](https://issues.apache.org/jira/browse/HADOOP-6692) | *Major* | **Add FileContext#listStatus that returns an iterator**
+* [HADOOP-4648](https://issues.apache.org/jira/browse/HADOOP-4648) | *Major* | **Remove ChecksumDistriubtedFileSystem and InMemoryFileSystem**
-This issue adds Iterator\<FileStatus\> listStatus(Path) to FileContext, moves FileStatus[] listStatus(Path) to FileContext#Util, and adds Iterator\<FileStatus\> listStatusItor(Path) to AbstractFileSystem which provides a default implementation by using FileStatus[] listStatus(Path).
+Removed obsolete, deprecated subclasses of ChecksumFileSystem (InMemoryFileSystem, ChecksumDistributedFileSystem).
---
-* [HADOOP-6686](https://issues.apache.org/jira/browse/HADOOP-6686) | *Major* | **Remove redundant exception class name in unwrapped exceptions thrown at the RPC client**
+* [HADOOP-4940](https://issues.apache.org/jira/browse/HADOOP-4940) | *Major* | **Remove delete(Path f)**
-The exceptions thrown by the RPC client no longer carries a redundant exception class name in exception message.
+Removed deprecated method FileSystem.delete(Path).
---
-* [HADOOP-6577](https://issues.apache.org/jira/browse/HADOOP-6577) | *Major* | **IPC server response buffer reset threshold should be configurable**
+* [HADOOP-3953](https://issues.apache.org/jira/browse/HADOOP-3953) | *Major* | **Sticky bit for directories**
-Add hidden configuration option "ipc.server.max.response.size" to change the default 1 MB, the maximum size when large IPC handler response buffer is reset.
+UNIX-style sticky bit implemented for HDFS directories. When the sticky bit is set on a directory, files in that directory may be deleted or renamed only by a superuser or the file's owner.
---
-* [HADOOP-6569](https://issues.apache.org/jira/browse/HADOOP-6569) | *Major* | **FsShell#cat should avoid calling unecessary getFileStatus before opening a file to read**
+* [HADOOP-5022](https://issues.apache.org/jira/browse/HADOOP-5022) | *Blocker* | **[HOD] logcondense should delete all hod logs for a user, including jobtracker logs**
-**WARNING: No release note provided for this incompatible change.**
+New logcondense option retain-master-logs indicates whether the script should delete master logs as part of its cleanup process. By default this option is false; master logs are deleted. Earlier versions of logcondense did not delete master logs.
---
-* [HADOOP-6568](https://issues.apache.org/jira/browse/HADOOP-6568) | *Major* | **Authorization for default servlets**
+* [HADOOP-4368](https://issues.apache.org/jira/browse/HADOOP-4368) | *Minor* | **Superuser privileges required to do "df"**
-Added web-authorization for the default servlets - /logs, /stacks, /logLevel, /metrics, /conf, so that only cluster administrators can access these servlets. hadoop.cluster.administrators is the new configuration in core-default.xml that can be used to specify the ACL against which an authenticated user should be verified if he/she is an administrator.
+New filesystem shell command -df reports capacity, space used and space free. Any user may execute this command without special privileges.
---
-* [HADOOP-6537](https://issues.apache.org/jira/browse/HADOOP-6537) | *Major* | **Proposal for exceptions thrown by FileContext and Abstract File System**
+* [HADOOP-5094](https://issues.apache.org/jira/browse/HADOOP-5094) | *Minor* | **Show dead nodes information in dfsadmin -report**
-Detailed exceptions declared in FileContext and AbstractFileSystem
+Changed df dfsadmin -report to list live and dead nodes, and attempt to resolve the hostname of datanode ip addresses.
---
-* [HADOOP-6531](https://issues.apache.org/jira/browse/HADOOP-6531) | *Minor* | **add FileUtil.fullyDeleteContents(dir) api to delete contents of a directory**
+* [HADOOP-3741](https://issues.apache.org/jira/browse/HADOOP-3741) | *Major* | **SecondaryNameNode has http server on dfs.secondary.http.address but without any contents**
-Added an api FileUtil.fullyDeleteContents(String dir) to delete contents of the directory.
+Backup namenode's web UI default page now has some useful content.
---
-* [HADOOP-6515](https://issues.apache.org/jira/browse/HADOOP-6515) | *Major* | **Make maximum number of http threads configurable**
+* [HADOOP-5176](https://issues.apache.org/jira/browse/HADOOP-5176) | *Trivial* | **TestDFSIO reports itself as TestFDSIO**
-HADOOP-6515. Make maximum number of http threads configurable (Scott Chen via zshao)
+**WARNING: No release note provided for this change.**
---
-* [HADOOP-6489](https://issues.apache.org/jira/browse/HADOOP-6489) | *Major* | **Findbug report: LI\_LAZY\_INIT\_STATIC, OBL\_UNSATISFIED\_OBLIGATION**
+* [HADOOP-5018](https://issues.apache.org/jira/browse/HADOOP-5018) | *Major* | **Chukwa should support pipelined writers**
-Fix 3 findsbugs warnings.
+Chukwa supports pipelined writers for improved extensibility.
---
-* [HADOOP-6441](https://issues.apache.org/jira/browse/HADOOP-6441) | *Major* | **Prevent remote CSS attacks in Hostname and UTF-7.**
+* [HADOOP-4942](https://issues.apache.org/jira/browse/HADOOP-4942) | *Major* | **Remove getName() and getNamed(String name, Configuration conf)**
-Quotes the characters coming out of getRequestUrl and getServerName in HttpServer.java as per the specification in HADOOP-6151.
+Removed deprecated methods getName() and getNamed(String, Configuration) from FileSystem and descendant classes.
---
-* [HADOOP-6433](https://issues.apache.org/jira/browse/HADOOP-6433) | *Major* | **Add AsyncDiskService that is used in both hdfs and mapreduce**
+* [HADOOP-4779](https://issues.apache.org/jira/browse/HADOOP-4779) | *Major* | **Remove deprecated FileSystem methods**
-HADOOP-6433. Add AsyncDiskService for asynchronous disk services.
+Removed deprecated FileSystem methods .
---
-* [HADOOP-6386](https://issues.apache.org/jira/browse/HADOOP-6386) | *Blocker* | **NameNode's HttpServer can't instantiate InetSocketAddress: IllegalArgumentException is thrown**
+* [HADOOP-1722](https://issues.apache.org/jira/browse/HADOOP-1722) | *Major* | **Make streaming to handle non-utf8 byte array**
-Improved initialization sequence so that Port Out of Range error when starting web server will less likely interrupt testing.
+Streaming allows binary (or other non-UTF8) streams.
---
-* [HADOOP-6367](https://issues.apache.org/jira/browse/HADOOP-6367) | *Major* | **Move Access Token implementation from Common to HDFS**
+* [HADOOP-4933](https://issues.apache.org/jira/browse/HADOOP-4933) | *Blocker* | **ConcurrentModificationException in JobHistory.java**
-**WARNING: No release note provided for this incompatible change.**
+Fixed a synchronization bug in job history content parsing that could result in garbled history data or a ConcurrentModificationException.
---
-* [HADOOP-6344](https://issues.apache.org/jira/browse/HADOOP-6344) | *Major* | **rm and rmr fail to correctly move the user's files to the trash prior to deleting when they are over quota.**
+* [HADOOP-4885](https://issues.apache.org/jira/browse/HADOOP-4885) | *Major* | **Try to restore failed replicas of Name Node storage (at checkpoint time)**
-Trash feature notifies user of over-quota condition rather than silently deleting files/directories; deletion can be compelled with "rm -skiptrash".
+Patch introduces new configuration switch dfs.name.dir.restore (boolean) enabling this functionality. Documentation needs to be updated.
+
+UPDATE: Config key is now "dfs.namenode.name.dir.restore" for 1.x and 2.x+ versions of HDFS
---
-* [HADOOP-6343](https://issues.apache.org/jira/browse/HADOOP-6343) | *Major* | **Stack trace of any runtime exceptions should be recorded in the server logs.**
+* [HADOOP-5222](https://issues.apache.org/jira/browse/HADOOP-5222) | *Minor* | **Add offset in client trace**
-Record runtime exceptions in server log to facilitate fault analysis.
+Include IO offset to client trace logging output.
---
-* [HADOOP-6313](https://issues.apache.org/jira/browse/HADOOP-6313) | *Major* | **Expose flush APIs to application users**
+* [HADOOP-5052](https://issues.apache.org/jira/browse/HADOOP-5052) | *Major* | **Add an example for computing exact digits of Pi**
-FSOutputDataStream implement Syncable interface to provide hflush and hsync APIs to the application users.
+New example BaileyBorweinPlouffe computes digits of pi. (World record!)
---
-* [HADOOP-6299](https://issues.apache.org/jira/browse/HADOOP-6299) | *Major* | **Use JAAS LoginContext for our login**
+* [HADOOP-4927](https://issues.apache.org/jira/browse/HADOOP-4927) | *Major* | **Part files on the output filesystem are created irrespective of whether the corresponding task has anything to write there**
-**WARNING: No release note provided for this incompatible change.**
+All output part files are created regardless of whether the corresponding task has output.
---
-* [HADOOP-6281](https://issues.apache.org/jira/browse/HADOOP-6281) | *Major* | **HtmlQuoting throws NullPointerException**
+* [HADOOP-5219](https://issues.apache.org/jira/browse/HADOOP-5219) | *Major* | **SequenceFile is using mapred property**
-Fixed null pointer error when quoting HTML in the case JSP has no parameters.
+New configuration parameter io.seqfile.local.dir for use by SequenceFile replaces mapred.local.dir.
---
-* [HADOOP-6235](https://issues.apache.org/jira/browse/HADOOP-6235) | *Major* | **Adding a new method for getting server default values from a FileSystem**
+* [HADOOP-5042](https://issues.apache.org/jira/browse/HADOOP-5042) | *Major* | ** Add expiration handling to the chukwa log4j appender**
-New FileSystem method reports default parameters that would be used by server. See also HDFS-578.
+Chukwwa Log4J appender options allow a retention policy to limit number of files.
---
-* [HADOOP-6234](https://issues.apache.org/jira/browse/HADOOP-6234) | *Major* | **Permission configuration files should use octal and symbolic**
+* [HADOOP-5144](https://issues.apache.org/jira/browse/HADOOP-5144) | *Major* | **manual way of turning on restore of failed storage replicas for namenode**
-New configuration option dfs.umaskmode sets umask with octal or symbolic value.
+New DFSAdmin command -restoreFailedStorage true\|false\|check sets policy for restoring failed fsimage/editslog volumes.
---
-* [HADOOP-6230](https://issues.apache.org/jira/browse/HADOOP-6230) | *Major* | **Move process tree, and memory calculator classes out of Common into Map/Reduce.**
+* [HADOOP-5258](https://issues.apache.org/jira/browse/HADOOP-5258) | *Major* | **Provide dfsadmin functionality to report on namenode's view of network topology**
-Moved process tree, and memory calculator classes out of Common project into the Map/Reduce project.
+New dfsAdmin command -printTopology shows topology as understood by the namenode.
---
-* [HADOOP-6226](https://issues.apache.org/jira/browse/HADOOP-6226) | *Major* | **Create a LimitedByteArrayOutputStream that does not expand its buffer on write**
+* [HADOOP-4756](https://issues.apache.org/jira/browse/HADOOP-4756) | *Major* | **Create a command line tool to access JMX exported properties from a NameNode server**
-New LimitedByteArrayOutputStream does not expand buffer on writes.
+New HDFS tool JMXGet facilitates command line access to statistics via JMX.
---
-* [HADOOP-6223](https://issues.apache.org/jira/browse/HADOOP-6223) | *Major* | **New improved FileSystem interface for those implementing new files systems.**
+* [HADOOP-4539](https://issues.apache.org/jira/browse/HADOOP-4539) | *Major* | **Streaming Edits to a Backup Node.**
-Add new file system interface AbstractFileSystem with implementation of some file systems that delegate to old FileSystem.
+Introduced backup node which maintains the up-to-date state of the namespace by receiving edits from the namenode, and checkpoint node, which creates checkpoints of the name space. These facilities replace the secondary namenode.
---
-* [HADOOP-6203](https://issues.apache.org/jira/browse/HADOOP-6203) | *Major* | **Improve error message when moving to trash fails due to quota issue**
+* [HADOOP-4842](https://issues.apache.org/jira/browse/HADOOP-4842) | *Major* | **Streaming combiner should allow command, not just JavaClass**
-Improved error message suggests using -skpTrash option when hdfs -rm fails to move to trash because of quota.
+Streaming option -combiner allows any streaming command (not just Java class) to be a combiner.
---
-* [HADOOP-6201](https://issues.apache.org/jira/browse/HADOOP-6201) | *Major* | **FileSystem::ListStatus should throw FileNotFoundException**
+* [HADOOP-4655](https://issues.apache.org/jira/browse/HADOOP-4655) | *Major* | **FileSystem.CACHE should be ref-counted**
-FileSystem listStatus method throws FileNotFoundException for all implementations. Application code should catch or propagate FileNotFoundException.
+Every invocation of FileSystem.newInstance() returns a newly allocated FileSystem object. This may be an incompatible change for applications that relied on FileSystem object identity.
---
-* [HADOOP-6184](https://issues.apache.org/jira/browse/HADOOP-6184) | *Major* | **Provide a configuration dump in json format.**
+* [HADOOP-5191](https://issues.apache.org/jira/browse/HADOOP-5191) | *Minor* | **After creation and startup of the hadoop namenode on AIX or Solaris, you will only be allowed to connect to the namenode via hostname but not IP.**
-New Configuration.dumpConfiguration(Configuration, Writer) writes configuration parameters in the JSON format.
+Accessing HDFS with any ip, hostname, or proxy should work as long as it points to the interface NameNode is listening on.
---
-* [HADOOP-6170](https://issues.apache.org/jira/browse/HADOOP-6170) | *Major* | **add Avro-based RPC serialization**
+* [HADOOP-5363](https://issues.apache.org/jira/browse/HADOOP-5363) | *Major* | **Proxying for multiple HDFS clusters of different versions**
-RPC can use Avro serialization.
+New HDFS proxy server (Tomcat based) allows clients controlled access to clusters with different versions. See Hadoop-5366 for information on using curl and wget.
---
-* [HADOOP-6161](https://issues.apache.org/jira/browse/HADOOP-6161) | *Minor* | **Add get/setEnum to Configuration**
+* [HADOOP-5464](https://issues.apache.org/jira/browse/HADOOP-5464) | *Major* | **DFSClient does not treat write timeout of 0 properly**
-Added following APIs to Configuration:
-- public \<T extends Enum\<T\>\> T getEnum(String name, T defaultValue)
-- public \<T extends Enum\<T\>\> void setEnum(String name, T value)
+Zero values for dfs.socket.timeout and dfs.datanode.socket.write.timeout are now respected. Previously zero values for these parameters resulted in a 5 second timeout.
---
-* [HADOOP-6151](https://issues.apache.org/jira/browse/HADOOP-6151) | *Critical* | **The servlets should quote html characters**
+* [HADOOP-2827](https://issues.apache.org/jira/browse/HADOOP-2827) | *Major* | **Remove deprecated NetUtils.getServerAddress**
-The input parameters for all of the servlets will have the 5 html meta characters quoted. The characters are '&', '\<', '\>', '"' and the apostrophe. The goal is to ensure that our web ui servlets can't be used for cross site scripting (XSS) attacks. In particular, it blocks the frequent (especially for errors) case where the servlet echos back the parameters to the user.
+Removed deprecated NetUtils.getServerAddress.
---
-* [HADOOP-6120](https://issues.apache.org/jira/browse/HADOOP-6120) | *Major* | **Add support for Avro types in hadoop**
+* [HADOOP-5528](https://issues.apache.org/jira/browse/HADOOP-5528) | *Major* | **Binary partitioner**
-New Avro serialization in .../io/serializer/avro.
+New BinaryPartitioner that partitions BinaryComparable keys by hashing a configurable part of the bytes array corresponding to the key.
---
-* [HADOOP-5976](https://issues.apache.org/jira/browse/HADOOP-5976) | *Major* | **create script to provide classpath for external tools**
+* [HADOOP-5518](https://issues.apache.org/jira/browse/HADOOP-5518) | *Major* | **MRUnit unit test library**
-New Hadoop script command classpath prints the path to the Hadoop jar and libraries.
+New contribution MRUnit helps authors of map-reduce programs write unit tests with JUnit.
---
-* [HADOOP-5913](https://issues.apache.org/jira/browse/HADOOP-5913) | *Major* | **Allow administrators to be able to start and stop queues**
+* [HADOOP-5257](https://issues.apache.org/jira/browse/HADOOP-5257) | *Minor* | **Export namenode/datanode functionality through a pluggable RPC layer**
-New mradmin command -refreshQueues reads new configuration of ACLs and queue states from mapred-queues.xml. If the new queue state is not "running," jobs in progress will continue, but no other jobs from that queue will be started.
+New plugin facility for namenode and datanode instantiates classes named in new configuration properties dfs.datanode.plugins and dfs.namenode.plugins.
---
-* [HADOOP-5887](https://issues.apache.org/jira/browse/HADOOP-5887) | *Major* | **Sqoop should create tables in Hive metastore after importing to HDFS**
+* [HADOOP-5469](https://issues.apache.org/jira/browse/HADOOP-5469) | *Major* | **Exposing Hadoop metrics via HTTP**
-New Sqoop argument --hive-import facilitates loading data into Hive.
+New server web page .../metrics allows convenient access to metrics data via JSON and text.
---
-* [HADOOP-5879](https://issues.apache.org/jira/browse/HADOOP-5879) | *Major* | **GzipCodec should read compression level etc from configuration**
+* [HADOOP-5485](https://issues.apache.org/jira/browse/HADOOP-5485) | *Major* | **Authorisation machanism required for acceesing jobtracker url :- jobtracker.com:port/scheduler**
-Provide an ability to configure the compression level and strategy for codecs. Compressors need to be 'reinited' with new characteristics such as compression level etc. and hence an incompatible addition to the api.
+New Fair Scheduler configuration parameter webinterface.private.actions controls whether changes to pools and priorities are permitted from the web interface. Changes are not permitted by default.
---
-* [HADOOP-5861](https://issues.apache.org/jira/browse/HADOOP-5861) | *Major* | **s3n files are not getting split by default**
+* [HADOOP-5396](https://issues.apache.org/jira/browse/HADOOP-5396) | *Major* | **Queue ACLs should be refreshed without requiring a restart of the job tracker**
-Files stored on the native S3 filesystem (s3n:// URIs) now report a block size determined by the fs.s3n.block.size property (default 64MB).
+Job Tracker queue ACLs can be changed without restarting Job Tracker.
---
-* [HADOOP-5815](https://issues.apache.org/jira/browse/HADOOP-5815) | *Major* | **Sqoop: A database import tool for Hadoop**
+* [HADOOP-5467](https://issues.apache.org/jira/browse/HADOOP-5467) | *Major* | **Create an offline fsimage image viewer**
-New contribution Sqoop is a JDBC-based database import tool for Hadoop.
+New Offline Image Viewer (oiv) tool reads an fsimage file and writes the data in a variety of user-friendly formats, including XML.
---
-* [HADOOP-5784](https://issues.apache.org/jira/browse/HADOOP-5784) | *Major* | **The length of the heartbeat cycle should be configurable.**
+* [HADOOP-5752](https://issues.apache.org/jira/browse/HADOOP-5752) | *Major* | **Provide examples of using offline image viewer (oiv) to analyze hadoop file systems**
-Introduced a configuration parameter, mapred.heartbeats.in.second, as an expert option, that defines how many heartbeats a jobtracker can process in a second. Administrators can set this to an appropriate value based on cluster size and expected processing time on the jobtracker to achieve a balance between jobtracker scalability and latency of jobs.
+Additional examples and documentation for HDFS Offline Image Viewer Tool show how to generate Pig-friendly data and to do analysis with Pig.
---
-* [HADOOP-5771](https://issues.apache.org/jira/browse/HADOOP-5771) | *Major* | **Create unit test for LinuxTaskController**
+* [HADOOP-5592](https://issues.apache.org/jira/browse/HADOOP-5592) | *Minor* | **Hadoop Streaming - GzipCodec**
-Added unit tests for verifying LinuxTaskController functionality.
+Updates streaming documentation to correct the name used for the GZipCodec.
---
-* [HADOOP-5752](https://issues.apache.org/jira/browse/HADOOP-5752) | *Major* | **Provide examples of using offline image viewer (oiv) to analyze hadoop file systems**
+* [HADOOP-5738](https://issues.apache.org/jira/browse/HADOOP-5738) | *Major* | **Split waiting tasks field in JobTracker metrics to individual tasks**
-Additional examples and documentation for HDFS Offline Image Viewer Tool show how to generate Pig-friendly data and to do analysis with Pig.
+**WARNING: No release note provided for this change.**
---
@@ -326,23 +326,23 @@ New Fair Scheduler configuration parameter sets a default limit on number of run
---
-* [HADOOP-5738](https://issues.apache.org/jira/browse/HADOOP-5738) | *Major* | **Split waiting tasks field in JobTracker metrics to individual tasks**
+* [HADOOP-5679](https://issues.apache.org/jira/browse/HADOOP-5679) | *Major* | **Resolve findbugs warnings in core/streaming/pipes/examples**
-**WARNING: No release note provided for this incompatible change.**
+**WARNING: No release note provided for this change.**
---
-* [HADOOP-5737](https://issues.apache.org/jira/browse/HADOOP-5737) | *Major* | **UGI checks in testcases are broken**
+* [HADOOP-5643](https://issues.apache.org/jira/browse/HADOOP-5643) | *Major* | **Ability to blacklist tasktracker**
-Fixed JobTracker to use it's own credentials instead of the job's credentials for accessing mapred.system.dir. Also added APIs in the JobTracker to get the FileSystem objects as per the JobTracker's configuration.
+New mradmin command -refreshNodes updates the job tracker's node lists.
---
-* [HADOOP-5679](https://issues.apache.org/jira/browse/HADOOP-5679) | *Major* | **Resolve findbugs warnings in core/streaming/pipes/examples**
+* [HADOOP-5771](https://issues.apache.org/jira/browse/HADOOP-5771) | *Major* | **Create unit test for LinuxTaskController**
-**WARNING: No release note provided for this incompatible change.**
+Added unit tests for verifying LinuxTaskController functionality.
---
@@ -354,23 +354,16 @@ Distcp will no longer start jobs that move no data.
---
-* [HADOOP-5643](https://issues.apache.org/jira/browse/HADOOP-5643) | *Major* | **Ability to blacklist tasktracker**
-
-New mradmin command -refreshNodes updates the job tracker's node lists.
-
-
----
-
-* [HADOOP-5620](https://issues.apache.org/jira/browse/HADOOP-5620) | *Major* | **discp can preserve modification times of files**
+* [HADOOP-5737](https://issues.apache.org/jira/browse/HADOOP-5737) | *Major* | **UGI checks in testcases are broken**
-New DistCp option -pt preserves last modification and last access times of copied files.
+Fixed JobTracker to use it's own credentials instead of the job's credentials for accessing mapred.system.dir. Also added APIs in the JobTracker to get the FileSystem objects as per the JobTracker's configuration.
---
-* [HADOOP-5592](https://issues.apache.org/jira/browse/HADOOP-5592) | *Minor* | **Hadoop Streaming - GzipCodec**
+* [HADOOP-4359](https://issues.apache.org/jira/browse/HADOOP-4359) | *Major* | **Access Token: Support for data access authorization checking on DataNodes**
-Updates streaming documentation to correct the name used for the GZipCodec.
+Introduced access tokens as capabilities for accessing datanodes. This change to internal protocols does not affect client applications.
---
@@ -382,996 +375,1035 @@ Fixed error parsing job history counters after change of counter format.
---
-* [HADOOP-5528](https://issues.apache.org/jira/browse/HADOOP-5528) | *Major* | **Binary partitioner**
+* [HADOOP-4829](https://issues.apache.org/jira/browse/HADOOP-4829) | *Minor* | **Allow FileSystem shutdown hook to be disabled**
-New BinaryPartitioner that partitions BinaryComparable keys by hashing a configurable part of the bytes array corresponding to the key.
+New configuration parameter fs.automatic.close can be set false to disable the JVM shutdown hook that automatically closes FileSystems.
---
-* [HADOOP-5518](https://issues.apache.org/jira/browse/HADOOP-5518) | *Major* | **MRUnit unit test library**
+* [HADOOP-5438](https://issues.apache.org/jira/browse/HADOOP-5438) | *Major* | **Merge FileSystem.create and FileSystem.append**
-New contribution MRUnit helps authors of map-reduce programs write unit tests with JUnit.
+**WARNING: No release note provided for this change.**
---
-* [HADOOP-5485](https://issues.apache.org/jira/browse/HADOOP-5485) | *Major* | **Authorisation machanism required for acceesing jobtracker url :- jobtracker.com:port/scheduler**
+* [HADOOP-5815](https://issues.apache.org/jira/browse/HADOOP-5815) | *Major* | **Sqoop: A database import tool for Hadoop**
-New Fair Scheduler configuration parameter webinterface.private.actions controls whether changes to pools and priorities are permitted from the web interface. Changes are not permitted by default.
+New contribution Sqoop is a JDBC-based database import tool for Hadoop.
---
-* [HADOOP-5469](https://issues.apache.org/jira/browse/HADOOP-5469) | *Major* | **Exposing Hadoop metrics via HTTP**
+* [HADOOP-4861](https://issues.apache.org/jira/browse/HADOOP-4861) | *Trivial* | **Add disk usage with human-readable size (-duh)**
-New server web page .../metrics allows convenient access to metrics data via JSON and text.
+Output of hadoop fs -dus changed to be consistent with hadoop fs -du and with Linux du. Users who previously parsed this output should update their scripts. New feature hadoop fs -du -h may be used for human readable output.
---
-* [HADOOP-5467](https://issues.apache.org/jira/browse/HADOOP-5467) | *Major* | **Create an offline fsimage image viewer**
+* [HADOOP-5175](https://issues.apache.org/jira/browse/HADOOP-5175) | *Major* | **Option to prohibit jars unpacking**
-New Offline Image Viewer (oiv) tool reads an fsimage file and writes the data in a variety of user-friendly formats, including XML.
+Jars passed to the -libjars option of hadoop jars are no longer unpacked inside mapred.local.dir.
---
-* [HADOOP-5464](https://issues.apache.org/jira/browse/HADOOP-5464) | *Major* | **DFSClient does not treat write timeout of 0 properly**
+* [HADOOP-5620](https://issues.apache.org/jira/browse/HADOOP-5620) | *Major* | **discp can preserve modification times of files**
-Zero values for dfs.socket.timeout and dfs.datanode.socket.write.timeout are now respected. Previously zero values for these parameters resulted in a 5 second timeout.
+New DistCp option -pt preserves last modification and last access times of copied files.
---
-* [HADOOP-5457](https://issues.apache.org/jira/browse/HADOOP-5457) | *Major* | **Failing contrib tests should not stop the build**
+* [HADOOP-5784](https://issues.apache.org/jira/browse/HADOOP-5784) | *Major* | **The length of the heartbeat cycle should be configurable.**
-Fixed the build to make sure that all the unit tests in contrib are run, regardless of the success/failure status of the previous projects' tests.
+Introduced a configuration parameter, mapred.heartbeats.in.second, as an expert option, that defines how many heartbeats a jobtracker can process in a second. Administrators can set this to an appropriate value based on cluster size and expected processing time on the jobtracker to achieve a balance between jobtracker scalability and latency of jobs.
---
-* [HADOOP-5438](https://issues.apache.org/jira/browse/HADOOP-5438) | *Major* | **Merge FileSystem.create and FileSystem.append**
+* [HADOOP-5861](https://issues.apache.org/jira/browse/HADOOP-5861) | *Major* | **s3n files are not getting split by default**
-**WARNING: No release note provided for this incompatible change.**
+Files stored on the native S3 filesystem (s3n:// URIs) now report a block size determined by the fs.s3n.block.size property (default 64MB).
---
-* [HADOOP-5396](https://issues.apache.org/jira/browse/HADOOP-5396) | *Major* | **Queue ACLs should be refreshed without requiring a restart of the job tracker**
+* [HADOOP-4768](https://issues.apache.org/jira/browse/HADOOP-4768) | *Major* | **Dynamic Priority Scheduler that allows queue shares to be controlled dynamically by a currency**
-Job Tracker queue ACLs can be changed without restarting Job Tracker.
+New contribution Dynamic Scheduler implements dynamic priorities with a currency model. Usage instructions are in the Jira item.
---
-* [HADOOP-5363](https://issues.apache.org/jira/browse/HADOOP-5363) | *Major* | **Proxying for multiple HDFS clusters of different versions**
+* [HADOOP-5457](https://issues.apache.org/jira/browse/HADOOP-5457) | *Major* | **Failing contrib tests should not stop the build**
-New HDFS proxy server (Tomcat based) allows clients controlled access to clusters with different versions. See Hadoop-5366 for information on using curl and wget.
+Fixed the build to make sure that all the unit tests in contrib are run, regardless of the success/failure status of the previous projects' tests.
---
-* [HADOOP-5258](https://issues.apache.org/jira/browse/HADOOP-5258) | *Major* | **Provide dfsadmin functionality to report on namenode's view of network topology**
+* [HADOOP-4041](https://issues.apache.org/jira/browse/HADOOP-4041) | *Major* | **IsolationRunner does not work as documented**
-New dfsAdmin command -printTopology shows topology as understood by the namenode.
+Fixed a bug in IsolationRunner to make it work for map tasks.
---
-* [HADOOP-5257](https://issues.apache.org/jira/browse/HADOOP-5257) | *Minor* | **Export namenode/datanode functionality through a pluggable RPC layer**
+* [HADOOP-5913](https://issues.apache.org/jira/browse/HADOOP-5913) | *Major* | **Allow administrators to be able to start and stop queues**
-New plugin facility for namenode and datanode instantiates classes named in new configuration properties dfs.datanode.plugins and dfs.namenode.plugins.
+New mradmin command -refreshQueues reads new configuration of ACLs and queue states from mapred-queues.xml. If the new queue state is not "running," jobs in progress will continue, but no other jobs from that queue will be started.
---
-* [HADOOP-5222](https://issues.apache.org/jira/browse/HADOOP-5222) | *Minor* | **Add offset in client trace**
+* [HADOOP-5887](https://issues.apache.org/jira/browse/HADOOP-5887) | *Major* | **Sqoop should create tables in Hive metastore after importing to HDFS**
-Include IO offset to client trace logging output.
+New Sqoop argument --hive-import facilitates loading data into Hive.
---
-* [HADOOP-5219](https://issues.apache.org/jira/browse/HADOOP-5219) | *Major* | **SequenceFile is using mapred property**
+* [MAPREDUCE-516](https://issues.apache.org/jira/browse/MAPREDUCE-516) | *Major* | **Fix the 'cluster drain' problem in the Capacity Scheduler wrt High RAM Jobs**
-New configuration parameter io.seqfile.local.dir for use by SequenceFile replaces mapred.local.dir.
+**WARNING: No release note provided for this change.**
---
-* [HADOOP-5191](https://issues.apache.org/jira/browse/HADOOP-5191) | *Minor* | **After creation and startup of the hadoop namenode on AIX or Solaris, you will only be allowed to connect to the namenode via hostname but not IP.**
+* [MAPREDUCE-358](https://issues.apache.org/jira/browse/MAPREDUCE-358) | *Major* | **Change org.apache.hadoop.examples. AggregateWordCount and org.apache.hadoop.examples.AggregateWordHistogram to use new mapreduce api.**
-Accessing HDFS with any ip, hostname, or proxy should work as long as it points to the interface NameNode is listening on.
+Modifies AggregateWordCount and AggregateWordHistogram examples to use the new Map/Reduce API
---
-* [HADOOP-5176](https://issues.apache.org/jira/browse/HADOOP-5176) | *Trivial* | **TestDFSIO reports itself as TestFDSIO**
+* [MAPREDUCE-463](https://issues.apache.org/jira/browse/MAPREDUCE-463) | *Major* | **The job setup and cleanup tasks should be optional**
-**WARNING: No release note provided for this incompatible change.**
+Added Configuration property "mapred.committer.job.setup.cleanup.needed" to specify whether job-setup and job-cleanup is needed for the job output committer. The default value is true.
+Added Job.setJobSetupCleanupNeeded and JobContext.getJobSetupCleanupNeeded api for setting/getting the configuration.
+If the configuration is set to false, no setup or cleanup will be done.
---
-* [HADOOP-5175](https://issues.apache.org/jira/browse/HADOOP-5175) | *Major* | **Option to prohibit jars unpacking**
+* [MAPREDUCE-502](https://issues.apache.org/jira/browse/MAPREDUCE-502) | *Major* | **Allow jobtracker to be configured with zero completed jobs in memory**
-Jars passed to the -libjars option of hadoop jars are no longer unpacked inside mapred.local.dir.
+If the number of jobs per user exceeded mapred.jobtracker.completeuserjobs.maximum then the job was flushed out of the jobtracker's memory after the job finishes min-time (hardcoded to 1 min). This caused jobclient's fail with NPE. In this patch the min-time to retain a job is made configurable (mapred.jobtracker.retirejob.interval.min).
---
-* [HADOOP-5144](https://issues.apache.org/jira/browse/HADOOP-5144) | *Major* | **manual way of turning on restore of failed storage replicas for namenode**
+* [MAPREDUCE-551](https://issues.apache.org/jira/browse/MAPREDUCE-551) | *Major* | **Add preemption to the fair scheduler**
-New DFSAdmin command -restoreFailedStorage true\|false\|check sets policy for restoring failed fsimage/editslog volumes.
+Added support for preemption in the fair scheduler. The new configuration options for enabling this are described in the fair scheduler documentation.
---
-* [HADOOP-5094](https://issues.apache.org/jira/browse/HADOOP-5094) | *Minor* | **Show dead nodes information in dfsadmin -report**
-
-Changed df dfsadmin -report to list live and dead nodes, and attempt to resolve the hostname of datanode ip addresses.
+* [MAPREDUCE-416](https://issues.apache.org/jira/browse/MAPREDUCE-416) | *Major* | **Move the completed jobs' history files to a DONE subdirectory inside the configured history directory**
+Once the job is done, the history file and associated conf file is moved to history.folder/done folder. This is done to avoid garbling the running jobs' folder and the framework no longer gets affected with the files in the done folder. This helps in 2 was
+1) ls on running folder (recovery) is faster with less files
+2) changes in running folder results into FileNotFoundException.
----
-* [HADOOP-5073](https://issues.apache.org/jira/browse/HADOOP-5073) | *Major* | **Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability**
+So with existing code, the best way to keep the running folder clean is to note the id's of running job and then move files that are not in this list to the done folder. Note that on an avg there will be 2 files in the history folder namely
+1) job history file
+2) conf file.
-Annotation mechanism enables interface classification.
+With restart, there might be more than 2 files, mostly the extra conf files. In such a case keep the oldest conf file (based on timestamp) and delete the rest. Note that this its better to do this when the jobtracker is down.
---
-* [HADOOP-5052](https://issues.apache.org/jira/browse/HADOOP-5052) | *Major* | **Add an example for computing exact digits of Pi**
+* [MAPREDUCE-646](https://issues.apache.org/jira/browse/MAPREDUCE-646) | *Major* | **distcp should place the file distcp\_src\_files in distributed cache**
-New example BaileyBorweinPlouffe computes digits of pi. (World record!)
+Patch increases the replication factor of \_distcp\_src\_files to sqrt(min(maxMapsOnCluster, totalMapsInThisJob)) sothat many maps won't access the same replica of the file \_distcp\_src\_files at the same time.
---
-* [HADOOP-5042](https://issues.apache.org/jira/browse/HADOOP-5042) | *Major* | ** Add expiration handling to the chukwa log4j appender**
+* [MAPREDUCE-211](https://issues.apache.org/jira/browse/MAPREDUCE-211) | *Major* | **Provide a node health check script and run it periodically to check the node health status**
-Chukwwa Log4J appender options allow a retention policy to limit number of files.
+Provides ability to run a health check script on the tasktracker nodes and blacklist nodes if they are unhealthy.
---
-* [HADOOP-5022](https://issues.apache.org/jira/browse/HADOOP-5022) | *Blocker* | **[HOD] logcondense should delete all hod logs for a user, including jobtracker logs**
+* [MAPREDUCE-642](https://issues.apache.org/jira/browse/MAPREDUCE-642) | *Major* | **distcp could have an option to preserve the full source path**
-New logcondense option retain-master-logs indicates whether the script should delete master logs as part of its cleanup process. By default this option is false; master logs are deleted. Earlier versions of logcondense did not delete master logs.
+DistCp now has a "-basedir" option that allows you to set the sufix of the source path that will be copied to the destination.
---
-* [HADOOP-5018](https://issues.apache.org/jira/browse/HADOOP-5018) | *Major* | **Chukwa should support pipelined writers**
+* [MAPREDUCE-686](https://issues.apache.org/jira/browse/MAPREDUCE-686) | *Major* | **Move TestSpeculativeExecution.Fake\* into a separate class so that it can be used by other tests also**
-Chukwa supports pipelined writers for improved extensibility.
+Consolidate the Mock Objects used for testing in a separate class(FakeObjectUtiltities) to ease re-usability
---
-* [HADOOP-4952](https://issues.apache.org/jira/browse/HADOOP-4952) | *Major* | **Improved files system interface for the application writer.**
+* [MAPREDUCE-625](https://issues.apache.org/jira/browse/MAPREDUCE-625) | *Minor* | **Modify TestTaskLimits to improve execution time**
-New FileContext API introduced to replace FileSystem API. FileContext will be the version-compatible API for future releases. FileSystem API will be deprecated in the next release.
+Modifies TestTaskLimits to do unit testing instead of running jobs using MR clusters
---
-* [HADOOP-4942](https://issues.apache.org/jira/browse/HADOOP-4942) | *Major* | **Remove getName() and getNamed(String name, Configuration conf)**
+* [MAPREDUCE-532](https://issues.apache.org/jira/browse/MAPREDUCE-532) | *Major* | **Allow admins of the Capacity Scheduler to set a hard-limit on the capacity of a queue**
-Removed deprecated methods getName() and getNamed(String, Configuration) from FileSystem and descendant classes.
+Provided ability in the capacity scheduler to limit the number of slots that can be concurrently used per queue at any given time.
---
-* [HADOOP-4941](https://issues.apache.org/jira/browse/HADOOP-4941) | *Major* | **Remove getBlockSize(Path f), getLength(Path f) and getReplication(Path src)**
+* [MAPREDUCE-701](https://issues.apache.org/jira/browse/MAPREDUCE-701) | *Minor* | **Make TestRackAwareTaskPlacement a unit test**
-Removed deprecated FileSystem methods getBlockSize(Path f), getLength(Path f), and getReplication(Path src).
+Modifies TestRackAwareTaskPlacement to not use MiniMR/DFS Cluster for testing, thereby making it a unit test
---
-* [HADOOP-4940](https://issues.apache.org/jira/browse/HADOOP-4940) | *Major* | **Remove delete(Path f)**
+* [MAPREDUCE-683](https://issues.apache.org/jira/browse/MAPREDUCE-683) | *Major* | **TestJobTrackerRestart fails with Map task completion events ordering mismatch**
-Removed deprecated method FileSystem.delete(Path).
+TestJobTrackerRestart failed because of stale filemanager cache (which was created once per jvm). This patch makes sure that the filemanager is inited upon every JobHistory.init() and hence upon every restart. Note that this wont happen in production as upon a restart the new jobtracker will start in a new jvm and hence a new cache will be created.
---
-* [HADOOP-4933](https://issues.apache.org/jira/browse/HADOOP-4933) | *Blocker* | **ConcurrentModificationException in JobHistory.java**
+* [MAPREDUCE-676](https://issues.apache.org/jira/browse/MAPREDUCE-676) | *Major* | **Existing diagnostic rules fail for MAP ONLY jobs**
-Fixed a synchronization bug in job history content parsing that could result in garbled history data or a ConcurrentModificationException.
+hadoop vaidya counter names LOCAL\_BYTES\_READ and LOCAL\_BYTES\_WRITTEN are changed to respectively FILE\_BYTES\_READ, FILE\_BYTES\_WRITTEN as per current hadoop counter names.
---
-* [HADOOP-4927](https://issues.apache.org/jira/browse/HADOOP-4927) | *Major* | **Part files on the output filesystem are created irrespective of whether the corresponding task has anything to write there**
+* [HADOOP-5976](https://issues.apache.org/jira/browse/HADOOP-5976) | *Major* | **create script to provide classpath for external tools**
-All output part files are created regardless of whether the corresponding task has output.
+New Hadoop script command classpath prints the path to the Hadoop jar and libraries.
---
-* [HADOOP-4895](https://issues.apache.org/jira/browse/HADOOP-4895) | *Major* | **Remove deprecated methods in DFSClient**
+* [MAPREDUCE-371](https://issues.apache.org/jira/browse/MAPREDUCE-371) | *Major* | **Change org.apache.hadoop.mapred.lib.KeyFieldBasedComparator and org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner to use new api**
-Removed deprecated methods DFSClient.getHints() and DFSClient.isDirectory().
+Ports KeyFieldBasedComparator and KeyFieldBasedPartitioner to the new Map/Reduce API
---
-* [HADOOP-4885](https://issues.apache.org/jira/browse/HADOOP-4885) | *Major* | **Try to restore failed replicas of Name Node storage (at checkpoint time)**
-
-Patch introduces new configuration switch dfs.name.dir.restore (boolean) enabling this functionality. Documentation needs to be updated.
+* [MAPREDUCE-732](https://issues.apache.org/jira/browse/MAPREDUCE-732) | *Minor* | **node health check script should not log "UNHEALTHY" status for every heartbeat in INFO mode**
-UPDATE: Config key is now "dfs.namenode.name.dir.restore" for 1.x and 2.x+ versions of HDFS
+Changed log level of addition of blacklisted reason in the JobTracker log to debug instead of INFO
---
-* [HADOOP-4861](https://issues.apache.org/jira/browse/HADOOP-4861) | *Trivial* | **Add disk usage with human-readable size (-duh)**
+* [MAPREDUCE-655](https://issues.apache.org/jira/browse/MAPREDUCE-655) | *Major* | **Change KeyValueLineRecordReader and KeyValueTextInputFormat to use new api.**
-Output of hadoop fs -dus changed to be consistent with hadoop fs -du and with Linux du. Users who previously parsed this output should update their scripts. New feature hadoop fs -du -h may be used for human readable output.
+Ports KeyValueLineRecordReader and KeyValueTextInputFormat the new Map/Reduce API
---
-* [HADOOP-4842](https://issues.apache.org/jira/browse/HADOOP-4842) | *Major* | **Streaming combiner should allow command, not just JavaClass**
+* [MAPREDUCE-153](https://issues.apache.org/jira/browse/MAPREDUCE-153) | *Major* | **TestJobInProgressListener sometimes timesout**
-Streaming option -combiner allows any streaming command (not just Java class) to be a combiner.
+Only one MR cluster is brought up and hence there is no scope of jobid clashing.
---
-* [HADOOP-4829](https://issues.apache.org/jira/browse/HADOOP-4829) | *Minor* | **Allow FileSystem shutdown hook to be disabled**
+* [MAPREDUCE-632](https://issues.apache.org/jira/browse/MAPREDUCE-632) | *Major* | **Merge TestCustomOutputCommitter with TestCommandLineJobSubmission**
-New configuration parameter fs.automatic.close can be set false to disable the JVM shutdown hook that automatically closes FileSystems.
+Modifies TestCommandLineJobSubmission to add a test for testing custom output committer and removes TestCustomOutputCommitter
---
-* [HADOOP-4779](https://issues.apache.org/jira/browse/HADOOP-4779) | *Major* | **Remove deprecated FileSystem methods**
+* [MAPREDUCE-467](https://issues.apache.org/jira/browse/MAPREDUCE-467) | *Major* | **Collect information about number of tasks succeeded / total per time unit for a tasktracker.**
-Removed deprecated FileSystem methods .
+Provide ability to collect statistics about tasks completed and succeeded for each tracker in time windows. The statistics is available on the jobtrackers' nodes UI page.
---
-* [HADOOP-4768](https://issues.apache.org/jira/browse/HADOOP-4768) | *Major* | **Dynamic Priority Scheduler that allows queue shares to be controlled dynamically by a currency**
+* [MAPREDUCE-677](https://issues.apache.org/jira/browse/MAPREDUCE-677) | *Major* | **TestNodeRefresh timesout**
-New contribution Dynamic Scheduler implements dynamic priorities with a currency model. Usage instructions are in the Jira item.
+TestNodeRefresh sometimes timed out. This happened because the test started a MR cluster with 2 trackers and ran a half-waiting-mapper job. Tasks that have id \> total-maps/2 wait for a signal. Because of 2 trackers, the tasks got scheduled out of order (locality) and hence the job got stuck. The fix is to start only one tracker and then add a new tracker later.
---
-* [HADOOP-4756](https://issues.apache.org/jira/browse/HADOOP-4756) | *Major* | **Create a command line tool to access JMX exported properties from a NameNode server**
+* [MAPREDUCE-627](https://issues.apache.org/jira/browse/MAPREDUCE-627) | *Minor* | **Modify TestTrackerBlacklistAcrossJobs to improve execution time**
-New HDFS tool JMXGet facilitates command line access to statistics via JMX.
+Modifies TestTrackerBlacklistAcrossJobs to use mock objects for testing instead of running a full-fledged job using MiniMR clusters.
---
-* [HADOOP-4655](https://issues.apache.org/jira/browse/HADOOP-4655) | *Major* | **FileSystem.CACHE should be ref-counted**
+* [MAPREDUCE-630](https://issues.apache.org/jira/browse/MAPREDUCE-630) | *Minor* | **TestKillCompletedJob can be modified to improve execution times**
-Every invocation of FileSystem.newInstance() returns a newly allocated FileSystem object. This may be an incompatible change for applications that relied on FileSystem object identity.
+Modifies TestKillCompletedJob to rid of its dependence on MiniMR clusters and makes it a unit test
---
-* [HADOOP-4648](https://issues.apache.org/jira/browse/HADOOP-4648) | *Major* | **Remove ChecksumDistriubtedFileSystem and InMemoryFileSystem**
+* [MAPREDUCE-626](https://issues.apache.org/jira/browse/MAPREDUCE-626) | *Minor* | **Modify TestLostTracker to improve execution time**
-Removed obsolete, deprecated subclasses of ChecksumFileSystem (InMemoryFileSystem, ChecksumDistributedFileSystem).
+Modifies TestLostTracker to use Mock objects instead of running full-fledged jobs using the MiniMR clusters.
---
-* [HADOOP-4539](https://issues.apache.org/jira/browse/HADOOP-4539) | *Major* | **Streaming Edits to a Backup Node.**
+* [MAPREDUCE-353](https://issues.apache.org/jira/browse/MAPREDUCE-353) | *Major* | **Allow shuffle read and connection timeouts to be configurable**
-Introduced backup node which maintains the up-to-date state of the namespace by receiving edits from the namenode, and checkpoint node, which creates checkpoints of the name space. These facilities replace the secondary namenode.
+Expert level config properties mapred.shuffle.connect.timeout and mapred.shuffle.read.timeout that are to be used at cluster level are added by this patch.
---
-* [HADOOP-4368](https://issues.apache.org/jira/browse/HADOOP-4368) | *Minor* | **Superuser privileges required to do "df"**
+* [MAPREDUCE-739](https://issues.apache.org/jira/browse/MAPREDUCE-739) | *Major* | **Allow relative paths to be created inside archives.**
-New filesystem shell command -df reports capacity, space used and space free. Any user may execute this command without special privileges.
+Allow creating archives with relative paths with a -p option on the command line.
---
-* [HADOOP-4359](https://issues.apache.org/jira/browse/HADOOP-4359) | *Major* | **Access Token: Support for data access authorization checking on DataNodes**
+* [MAPREDUCE-740](https://issues.apache.org/jira/browse/MAPREDUCE-740) | *Major* | **Provide summary information per job once a job is finished.**
-Introduced access tokens as capabilities for accessing datanodes. This change to internal protocols does not affect client applications.
+Log a job-summary at the end of a job, while allowing it to be configured to use a custom appender if desired.
---
-* [HADOOP-4268](https://issues.apache.org/jira/browse/HADOOP-4268) | *Major* | **Permission checking in fsck**
+* [MAPREDUCE-245](https://issues.apache.org/jira/browse/MAPREDUCE-245) | *Major* | **Job and JobControl classes should return interfaces rather than implementations**
-Fsck now checks permissions as directories are traversed. Any user can now use fsck, but information is provided only for directories the user has permission to read.
+**WARNING: No release note provided for this change.**
---
-* [HADOOP-4041](https://issues.apache.org/jira/browse/HADOOP-4041) | *Major* | **IsolationRunner does not work as documented**
+* [HADOOP-6161](https://issues.apache.org/jira/browse/HADOOP-6161) | *Minor* | **Add get/setEnum to Configuration**
-Fixed a bug in IsolationRunner to make it work for map tasks.
+Added following APIs to Configuration:
+- public \<T extends Enum\<T\>\> T getEnum(String name, T defaultValue)
+- public \<T extends Enum\<T\>\> void setEnum(String name, T value)
---
-* [HADOOP-4012](https://issues.apache.org/jira/browse/HADOOP-4012) | *Major* | **Providing splitting support for bzip2 compressed files**
+* [MAPREDUCE-772](https://issues.apache.org/jira/browse/MAPREDUCE-772) | *Major* | **Chaging LineRecordReader algo so that it does not need to skip backwards in the stream**
-BZip2 files can now be split.
+**WARNING: No release note provided for this change.**
---
-* [HADOOP-3953](https://issues.apache.org/jira/browse/HADOOP-3953) | *Major* | **Sticky bit for directories**
+* [MAPREDUCE-717](https://issues.apache.org/jira/browse/MAPREDUCE-717) | *Major* | **Fix some corner case issues in speculative execution (post hadoop-2141)**
-UNIX-style sticky bit implemented for HDFS directories. When the sticky bit is set on a directory, files in that directory may be deleted or renamed only by a superuser or the file's owner.
+Fixes some edge cases while using speculative execution
---
-* [HADOOP-3741](https://issues.apache.org/jira/browse/HADOOP-3741) | *Major* | **SecondaryNameNode has http server on dfs.secondary.http.address but without any contents**
+* [MAPREDUCE-785](https://issues.apache.org/jira/browse/MAPREDUCE-785) | *Major* | **Refactor TestReduceFetchFromPartialMem into a separate test**
-Backup namenode's web UI default page now has some useful content.
+Moves TestReduceFetchFromPartialMem out of TestReduceFetch into a separate test to enable it to be included in the commit-tests target.
---
-* [HADOOP-2827](https://issues.apache.org/jira/browse/HADOOP-2827) | *Major* | **Remove deprecated NetUtils.getServerAddress**
+* [MAPREDUCE-682](https://issues.apache.org/jira/browse/MAPREDUCE-682) | *Major* | **Reserved tasktrackers should be removed when a node is globally blacklisted**
-Removed deprecated NetUtils.getServerAddress.
+Jobtracker was modified to cleanup reservations created on tasktracker nodes to support high RAM jobs, when the nodes are blacklisted.
---
-* [HADOOP-1722](https://issues.apache.org/jira/browse/HADOOP-1722) | *Major* | **Make streaming to handle non-utf8 byte array**
+* [HADOOP-6120](https://issues.apache.org/jira/browse/HADOOP-6120) | *Major* | **Add support for Avro types in hadoop**
-Streaming allows binary (or other non-UTF8) streams.
+New Avro serialization in .../io/serializer/avro.
---
-* [HDFS-1024](https://issues.apache.org/jira/browse/HDFS-1024) | *Blocker* | **SecondaryNamenode fails to checkpoint because namenode fails with CancelledKeyException**
+* [MAPREDUCE-784](https://issues.apache.org/jira/browse/MAPREDUCE-784) | *Major* | **Modify TestUserDefinedCounters to use LocalJobRunner instead of MiniMR**
-**WARNING: No release note provided for this incompatible change.**
+Modifies TestUserDefinedCounters to use LocalJobRunner instead of using MiniMR cluster
---
-* [HDFS-1016](https://issues.apache.org/jira/browse/HDFS-1016) | *Major* | **HDFS side change for HADOOP-6569**
+* [MAPREDUCE-369](https://issues.apache.org/jira/browse/MAPREDUCE-369) | *Major* | **Change org.apache.hadoop.mapred.lib.MultipleInputs to use new api.**
-When cat a directory or a non-existent file from the command line, the error message gets printed becomes
-cat: io.java.FileNotFoundException: File does not exist: \<absolute path name\>
+Patch that ports MultipleInputs, DelegatingInputFormat, DelegatingMapper and TaggedInputSplit to the new Map/Reduce API
---
-* [HDFS-1012](https://issues.apache.org/jira/browse/HDFS-1012) | *Major* | **documentLocation attribute in LdapEntry for HDFSProxy isn't specific to a cluster**
+* [MAPREDUCE-373](https://issues.apache.org/jira/browse/MAPREDUCE-373) | *Major* | **Change org.apache.hadoop.mapred.lib. FieldSelectionMapReduce to use new api.**
-Support for fully qualified HDFS path in addition to simple unqualified path.
-The qualified path indicates that the path is accessible on the specific HDFS. Non qualified path is qualified in all clusters.
+Ports FieldSelectionMapReduce to the new Map/Reduce API
---
-* [HDFS-998](https://issues.apache.org/jira/browse/HDFS-998) | *Major* | **The servlets should quote server generated strings sent in the response**
+* [MAPREDUCE-793](https://issues.apache.org/jira/browse/MAPREDUCE-793) | *Major* | **Create a new test that consolidates a few tests to be included in the commit-test list**
-The servlets should quote server generated strings sent in the response.
+Creates a new test to test several miscellaneous functionality at one shot instead of running a job for each, to be used as a fast test for the ant commit-tests target.
---
-* [HDFS-985](https://issues.apache.org/jira/browse/HDFS-985) | *Major* | **HDFS should issue multiple RPCs for listing a large directory**
+* [MAPREDUCE-809](https://issues.apache.org/jira/browse/MAPREDUCE-809) | *Major* | **Job summary logs show status of completed jobs as RUNNING**
-**WARNING: No release note provided for this incompatible change.**
+Fix job-summary logs to correctly record final status of FAILED and KILLED jobs.
---
-* [HDFS-984](https://issues.apache.org/jira/browse/HDFS-984) | *Major* | **Delegation Tokens should be persisted in Namenode**
+* [MAPREDUCE-797](https://issues.apache.org/jira/browse/MAPREDUCE-797) | *Major* | **MRUnit MapReduceDriver should support combiners**
-Layout version is set to -24 reflecting changes in edits log and fsimage format related to persisting delegation tokens.
+Add Combiner support to MapReduceDriver in MRUnit
---
-* [HDFS-946](https://issues.apache.org/jira/browse/HDFS-946) | *Major* | **NameNode should not return full path name when lisitng a diretory or getting the status of a file**
+* [HDFS-514](https://issues.apache.org/jira/browse/HDFS-514) | *Major* | **DFSClient.namenode is a public field. Should be private.**
-**WARNING: No release note provided for this incompatible change.**
+**WARNING: No release note provided for this change.**
---
-* [HDFS-913](https://issues.apache.org/jira/browse/HDFS-913) | *Major* | **TestRename won't run automatically from 'run-test-hdfs-faul-inject' target**
+* [MAPREDUCE-760](https://issues.apache.org/jira/browse/MAPREDUCE-760) | *Major* | **TestNodeRefresh might not work as expected**
-HDFS-913. Rename fault injection test TestRename.java to TestFiRename.java to include it in tests run by ant target run-test-hdfs-fault-inject.
+TestNodeRefresh waits for the newly added tracker to join before starting the testing.
---
-* [HDFS-897](https://issues.apache.org/jira/browse/HDFS-897) | *Major* | **ReplicasMap remove has a bug in generation stamp comparison**
+* [MAPREDUCE-766](https://issues.apache.org/jira/browse/MAPREDUCE-766) | *Major* | **Enhance -list-blacklisted-trackers to display host name, blacklisted reason and blacklist report.**
-Fixed a bug in ReplicasMap.remove method, which compares the generation stamp of the replica removed to itself instead of the the block passed to the method to identify the replica to be removed.
+Enhanced -list-blacklisted-trackers to include the reason for blacklisting a node. Modified JobSubmissionProtocol's version as the ClusterStatus is changed to have a new class. The format of the -list-blacklisted-trackers command line interface has also changed to show the reason.
---
-* [HDFS-892](https://issues.apache.org/jira/browse/HDFS-892) | *Major* | **optionally use Avro for namenode RPC**
+* [MAPREDUCE-656](https://issues.apache.org/jira/browse/MAPREDUCE-656) | *Major* | **Change org.apache.hadoop.mapred.SequenceFile\* classes to use new api**
-**WARNING: No release note provided for this incompatible change.**
+Ports the SequenceFile\* classes to the new Map/Reduce API
---
-* [HDFS-873](https://issues.apache.org/jira/browse/HDFS-873) | *Major* | **DataNode directories as URIs**
+* [MAPREDUCE-670](https://issues.apache.org/jira/browse/MAPREDUCE-670) | *Major* | ** Create target for 10 minute patch test build for mapreduce**
-**WARNING: No release note provided for this incompatible change.**
+Added a new target 'test-commit' to the build.xml file which runs tests specified in the file src/test/commit-tests. The tests specified in src/test/commit-tests should provide maximum coverage and all the tests should run within 10mins.
---
-* [HDFS-850](https://issues.apache.org/jira/browse/HDFS-850) | *Minor* | **Display more memory details on the web ui**
+* [MAPREDUCE-408](https://issues.apache.org/jira/browse/MAPREDUCE-408) | *Major* | **TestKillSubProcesses fails with assertion failure sometimes**
-Changes the format of the message with Heap usage on the NameNode web page.
+Fixed a bug in the testcase TestKillSubProcesses.
---
-* [HDFS-814](https://issues.apache.org/jira/browse/HDFS-814) | *Major* | **Add an api to get the visible length of a DFSDataInputStream.**
+* [MAPREDUCE-375](https://issues.apache.org/jira/browse/MAPREDUCE-375) | *Major* | ** Change org.apache.hadoop.mapred.lib.NLineInputFormat and org.apache.hadoop.mapred.MapFileOutputFormat to use new api.**
-Add an api to get the visible length of a DFSDataInputStream.
+Ports NLineInputFormat and MapFileOutputFormat to the new Map/Reduce API
---
-* [HDFS-793](https://issues.apache.org/jira/browse/HDFS-793) | *Blocker* | **DataNode should first receive the whole packet ack message before it constructs and sends its own ack message for the packet**
+* [MAPREDUCE-814](https://issues.apache.org/jira/browse/MAPREDUCE-814) | *Major* | **Move completed Job history files to HDFS**
-**WARNING: No release note provided for this incompatible change.**
+Provides an ability to move completed job history files to a HDFS location via configuring "mapred.job.tracker.history.completed.location". If the directory location does not already exist, it would be created by jobtracker.
---
-* [HDFS-785](https://issues.apache.org/jira/browse/HDFS-785) | *Minor* | **Missing license header in java source files.**
+* [MAPREDUCE-845](https://issues.apache.org/jira/browse/MAPREDUCE-845) | *Minor* | **build.xml hard codes findbugs heap size, in some configurations 512M is insufficient to successfully build**
-Add the Apache license header to several files that are missing it.
+Changes the heapsize for findbugs to a parameter which can be changed on the build command line.
---
-* [HDFS-781](https://issues.apache.org/jira/browse/HDFS-781) | *Blocker* | **Metrics PendingDeletionBlocks is not decremented**
+* [MAPREDUCE-817](https://issues.apache.org/jira/browse/MAPREDUCE-817) | *Major* | **Add a cache for retired jobs with minimal job info and provide a way to access history file url**
-Correct PendingDeletionBlocks metric to properly decrement counts.
+Provides a way to configure the cache of JobStatus objects for the retired jobs.
+Adds an API in RunningJob to access history file url.
+Adds a LRU based cache for job history files loaded in memory when accessed via JobTracker web UI.
+Adds Retired Jobs table on the Jobtracker UI. The job move from Running to Completed/Failed table. Then job move to Retired table when it is purged from memory. The Retired table shows last 100 retired jobs. The Completed/Failed jobs table are only shown if there are non-zero jobs in the table.
---
-* [HDFS-764](https://issues.apache.org/jira/browse/HDFS-764) | *Major* | **Moving Access Token implementation from Common to HDFS**
+* [MAPREDUCE-848](https://issues.apache.org/jira/browse/MAPREDUCE-848) | *Major* | **TestCapacityScheduler is failing**
-**WARNING: No release note provided for this incompatible change.**
+MAPREDUCE-805 changed the way the job was initialized. Capacity schedulers testcases were not modified as part of MAPREDUCE-805. This patch fixes this bug.
---
-* [HDFS-761](https://issues.apache.org/jira/browse/HDFS-761) | *Major* | **Failure to process rename operation from edits log due to quota verification**
+* [MAPREDUCE-842](https://issues.apache.org/jira/browse/MAPREDUCE-842) | *Major* | **Per-job local data on the TaskTracker node should have right access-control**
-Corrected an error when checking quota policy that resulted in a failure to read the edits log, stopping the primary/secondary name node.
+Modified TaskTracker and related classes so that per-job local data on the TaskTracker node has right access-control. Important changes:
+ - All files/directories of the job on the TaskTracker are now user-owned by the job-owner and group-owner by a special TaskTracker's group.
+ - The permissions of the file/directories are set to the most restrictive permissions possible.
+ - Files/dirs shareable by all tasks of the job on this TT are set proper access control as soon as possible, i.e immediately after job-localization and those that are private to a single task are set access control after the corresponding task's localization.
+ - Also fixes MAPREDUCE-131 which is related to a bug because of which tasks hang when the taskcontroller.cfg has multiple entries for mapred.local.dir
+ - A new configuration entry hadoop.log.dir corresponding to the hadoop.log.dir in TT's configuration is now needed in task-controller.cfg so as to support restricted access control for userlogs of the tasks on the TaskTracker.
---
-* [HDFS-758](https://issues.apache.org/jira/browse/HDFS-758) | *Major* | **Improve reporting of progress of decommissioning**
+* [MAPREDUCE-706](https://issues.apache.org/jira/browse/MAPREDUCE-706) | *Major* | **Support for FIFO pools in the fair scheduler**
-New name node web UI page displays details of decommissioning progress. (dfsnodelist.jsp?whatNodes=DECOMMISSIONING)
+Support for FIFO pools added to the Fair Scheduler.
---
-* [HDFS-737](https://issues.apache.org/jira/browse/HDFS-737) | *Major* | **Improvement in metasave output**
+* [MAPREDUCE-852](https://issues.apache.org/jira/browse/MAPREDUCE-852) | *Major* | **ExampleDriver is incorrectly set as a Main-Class in tools in build.xml**
-Add full path name of the file to the under replicated block information and summary of total number of files, blocks, live and dead datanodes to metasave output.
+Changed the target name from "tools-jar" to "tools" in build.xml.
---
-* [HDFS-702](https://issues.apache.org/jira/browse/HDFS-702) | *Major* | **Add Hdfs Impl for the new file system interface**
+* [HDFS-457](https://issues.apache.org/jira/browse/HDFS-457) | *Major* | **better handling of volume failure in Data Node storage**
-Add HDFS implementation of AbstractFileSystem.
+Datanode can continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
---
-* [HDFS-677](https://issues.apache.org/jira/browse/HDFS-677) | *Blocker* | **Rename failure due to quota results in deletion of src directory**
+* [MAPREDUCE-773](https://issues.apache.org/jira/browse/MAPREDUCE-773) | *Major* | **LineRecordReader can report non-zero progress while it is processing a compressed stream**
-Rename properly considers the case where both source and destination are over quota; operation will fail with error indication.
+Modifies LineRecordReader to report an approximate progress, instead of just returning 0, when using compressed streams.
---
-* [HDFS-660](https://issues.apache.org/jira/browse/HDFS-660) | *Major* | **Remove deprecated methods from InterDatanodeProtocol.**
+* [MAPREDUCE-711](https://issues.apache.org/jira/browse/MAPREDUCE-711) | *Major* | **Move Distributed Cache from Common to Map/Reduce**
-**WARNING: No release note provided for this incompatible change.**
+- Removed distributed cache classes and package from the Common project.
+- Added the same to the mapreduce project.
+- This will mean that users using Distributed Cache will now necessarily need the mapreduce jar in Hadoop 0.21.
+- Modified the package name to o.a.h.mapreduce.filecache from o.a.h.filecache and deprecated the old package name.
---
-* [HDFS-631](https://issues.apache.org/jira/browse/HDFS-631) | *Major* | **Changes in HDFS to rename the config keys as detailed in HDFS-531.**
+* [MAPREDUCE-870](https://issues.apache.org/jira/browse/MAPREDUCE-870) | *Major* | **Clean up the job Retire code**
-File system configuration keys renamed as a step toward API standardization and backward compatibility.
+Removed the Job Retire thread and the associated configuration parameters. Job is purged from memory as soon as the history file is copied to HDFS. Only JobStatus object is retained in the retired jobs cache.
---
-* [HDFS-630](https://issues.apache.org/jira/browse/HDFS-630) | *Major* | **In DFSOutputStream.nextBlockOutputStream(), the client can exclude specific datanodes when locating the next block.**
+* [MAPREDUCE-800](https://issues.apache.org/jira/browse/MAPREDUCE-800) | *Major* | **MRUnit should support the new API**
-**WARNING: No release note provided for this incompatible change.**
+Support new API in unit tests developed with MRUnit.
---
-* [HDFS-625](https://issues.apache.org/jira/browse/HDFS-625) | *Major* | **ListPathsServlet throws NullPointerException**
+* [HADOOP-6201](https://issues.apache.org/jira/browse/HADOOP-6201) | *Major* | **FileSystem::ListStatus should throw FileNotFoundException**
-Corrected error where listing path no longer in name space could stop ListPathsServlet until system restarted.
+FileSystem listStatus method throws FileNotFoundException for all implementations. Application code should catch or propagate FileNotFoundException.
---
-* [HDFS-618](https://issues.apache.org/jira/browse/HDFS-618) | *Major* | **Support for non-recursive mkdir in HDFS**
+* [HDFS-538](https://issues.apache.org/jira/browse/HDFS-538) | *Major* | **DistributedFileSystem::listStatus incorrectly returns null for empty result sets**
-New DFSClient.mkdir(...) allows option of not creating missing parent(s).
+FileSystem.listStatus() previously returned null for empty or nonexistent directories; will now return empty array for empty directories and throw FileNotFoundException for non-existent directory. Client code should be updated for new semantics.
---
-* [HDFS-617](https://issues.apache.org/jira/browse/HDFS-617) | *Major* | **Support for non-recursive create() in HDFS**
+* [MAPREDUCE-895](https://issues.apache.org/jira/browse/MAPREDUCE-895) | *Major* | **FileSystem::ListStatus will now throw FileNotFoundException, MapRed needs updated**
-New DFSClient.create(...) allows option of not creating missing parent(s).
+The semantics for dealing with non-existent paths passed to FileSystem::listStatus() were updated and solidified in HADOOP-6201 and HDFS-538. Existing code within MapReduce that relied on the previous behavior of some FileSystem implementations of returning null has been updated to catch or propagate a FileNotFoundException, per the method's contract.
---
-* [HDFS-602](https://issues.apache.org/jira/browse/HDFS-602) | *Major* | **Atempt to make a directory under an existing file on DistributedFileSystem should throw an FileAlreadyExistsException instead of FileNotFoundException**
+* [MAPREDUCE-336](https://issues.apache.org/jira/browse/MAPREDUCE-336) | *Major* | **The logging level of the tasks should be configurable by the job**
-**WARNING: No release note provided for this incompatible change.**
+Allow logging level of map/reduce tasks to be configurable.
+Configuration changes:
+ add mapred.map.child.log.level
+ add mapred.reduce.child.log.level
---
-* [HDFS-596](https://issues.apache.org/jira/browse/HDFS-596) | *Blocker* | **Memory leak in libhdfs: hdfsFreeFileInfo() in libhdfs does not free memory for mOwner and mGroup**
+* [MAPREDUCE-479](https://issues.apache.org/jira/browse/MAPREDUCE-479) | *Minor* | **Add reduce ID to shuffle clienttrace**
-Memory leak in function hdfsFreeFileInfo in libhdfs. This bug affects fuse-dfs severely.
+Adds Reduce Attempt ID to ClientTrace log messages, and adds Reduce Attempt ID to HTTP query string sent to mapOutputServlet. Extracts partition number from attempt ID.
---
-* [HDFS-595](https://issues.apache.org/jira/browse/HDFS-595) | *Major* | **FsPermission tests need to be updated for new octal configuration parameter from HADOOP-6234**
+* [MAPREDUCE-355](https://issues.apache.org/jira/browse/MAPREDUCE-355) | *Major* | **Change org.apache.hadoop.mapred.join to use new api**
-Unit tests updated to match syntax of new configuration parameters.
+Ports the mapred.join library to the new Map/Reduce API
---
-* [HDFS-578](https://issues.apache.org/jira/browse/HDFS-578) | *Major* | **Support for using server default values for blockSize and replication when creating a file**
+* [HADOOP-6184](https://issues.apache.org/jira/browse/HADOOP-6184) | *Major* | **Provide a configuration dump in json format.**
-New FileSystem.getServerDefaults() reports the server's default file creation parameters.
+New Configuration.dumpConfiguration(Configuration, Writer) writes configuration parameters in the JSON format.
---
-* [HDFS-567](https://issues.apache.org/jira/browse/HDFS-567) | *Major* | **Two contrib tools to facilitate searching for block history information**
+* [MAPREDUCE-798](https://issues.apache.org/jira/browse/MAPREDUCE-798) | *Major* | **MRUnit should be able to test a succession of MapReduce passes**
-New contribution Block Forensics aids investigation of missing blocks.
+Add PipelineMapReduceDriver to MRUnit to support testing a pipeline of MapReduce passes
---
-* [HDFS-538](https://issues.apache.org/jira/browse/HDFS-538) | *Major* | **DistributedFileSystem::listStatus incorrectly returns null for empty result sets**
+* [MAPREDUCE-476](https://issues.apache.org/jira/browse/MAPREDUCE-476) | *Minor* | **extend DistributedCache to work locally (LocalJobRunner)**
-FileSystem.listStatus() previously returned null for empty or nonexistent directories; will now return empty array for empty directories and throw FileNotFoundException for non-existent directory. Client code should be updated for new semantics.
+Extended DistributedCache to work with LocalJobRunner.
---
-* [HDFS-514](https://issues.apache.org/jira/browse/HDFS-514) | *Major* | **DFSClient.namenode is a public field. Should be private.**
+* [MAPREDUCE-768](https://issues.apache.org/jira/browse/MAPREDUCE-768) | *Major* | **Configuration information should generate dump in a standard format.**
-**WARNING: No release note provided for this incompatible change.**
+Provides an ability to dump jobtracker configuration in JSON format to standard output and exits.
+To dump, use hadoop jobtracker -dumpConfiguration
+The format of the dump is {"properties":[{"key":\<key\>,"value":\<value\>,"isFinal":\<true/false\>,"resource" : \<resource\>}] }
---
-* [HDFS-512](https://issues.apache.org/jira/browse/HDFS-512) | *Major* | **Set block id as the key to Block**
+* [MAPREDUCE-788](https://issues.apache.org/jira/browse/MAPREDUCE-788) | *Major* | **Modify gridmix2 to use new api.**
-**WARNING: No release note provided for this incompatible change.**
+Modifies Gridmix2 to use the new Map/Reduce API
---
-* [HDFS-503](https://issues.apache.org/jira/browse/HDFS-503) | *Major* | **Implement erasure coding as a layer on HDFS**
+* [MAPREDUCE-824](https://issues.apache.org/jira/browse/MAPREDUCE-824) | *Major* | **Support a hierarchy of queues in the capacity scheduler**
-This patch implements an optional layer over HDFS that implements offline erasure-coding. It can be used to reduce the total storage requirements of DFS.
+Support hierarchical queues in the CapacityScheduler to allow for more predictable sharing of cluster resources.
---
-* [HDFS-492](https://issues.apache.org/jira/browse/HDFS-492) | *Major* | **Expose corrupt replica/block information**
+* [MAPREDUCE-871](https://issues.apache.org/jira/browse/MAPREDUCE-871) | *Major* | **Job/Task local files have incorrect group ownership set by LinuxTaskController binary**
-New server web pages provide block information: corrupt\_replicas\_xml and block\_info\_xml.
+Fixed LinuxTaskController binary so that permissions of local files on TT are set correctly: user owned by the job-owner and group-owned by the group owner of the binary and \_not\_ the primary group of the TaskTracker.
---
-* [HDFS-457](https://issues.apache.org/jira/browse/HDFS-457) | *Major* | **better handling of volume failure in Data Node storage**
+* [HDFS-492](https://issues.apache.org/jira/browse/HDFS-492) | *Major* | **Expose corrupt replica/block information**
-Datanode can continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.
+New server web pages provide block information: corrupt\_replicas\_xml and block\_info\_xml.
---
-* [HDFS-385](https://issues.apache.org/jira/browse/HDFS-385) | *Major* | **Design a pluggable interface to place replicas of blocks in HDFS**
+* [MAPREDUCE-873](https://issues.apache.org/jira/browse/MAPREDUCE-873) | *Major* | **Simplify Job Recovery**
-New experimental API BlockPlacementPolicy allows investigating alternate rules for locating block replicas.
+Simplifies job recovery. On jobtracker restart, incomplete jobs are resubmitted and all tasks reexecute.
+This JIRA removes a public constructor in JobInProgress.
---
-* [HDFS-288](https://issues.apache.org/jira/browse/HDFS-288) | *Major* | **Redundant computation in hashCode() implemenation**
+* [HADOOP-6226](https://issues.apache.org/jira/browse/HADOOP-6226) | *Major* | **Create a LimitedByteArrayOutputStream that does not expand its buffer on write**
-**WARNING: No release note provided for this incompatible change.**
+New LimitedByteArrayOutputStream does not expand buffer on writes.
---
-* [HDFS-245](https://issues.apache.org/jira/browse/HDFS-245) | *Major* | **Create symbolic links in HDFS**
+* [MAPREDUCE-318](https://issues.apache.org/jira/browse/MAPREDUCE-318) | *Major* | **Refactor reduce shuffle code**
-HDFS-245. Adds a symlink implementation to HDFS. This complements the new symlink feature added in HADOOP-6421
+Refactors shuffle code out of ReduceTask into separate classes in a new package(org.apache.hadoop.mapreduce.task.reduce)
+Incorporates MAPREDUCE-240, batches up several map output files from a TT to a reducer in a single transfer
+Introduces new Shuffle counters to keep track of shuffle errors
---
-* [HDFS-235](https://issues.apache.org/jira/browse/HDFS-235) | *Major* | **Add support for byte-ranges to hftp**
+* [MAPREDUCE-370](https://issues.apache.org/jira/browse/MAPREDUCE-370) | *Major* | **Change org.apache.hadoop.mapred.lib.MultipleOutputs to use new api.**
-HFTP can now serve a specific byte range from a file
+Ports MultipleOutputs to the new Map/Reduce API
---
-* [MAPREDUCE-1747](https://issues.apache.org/jira/browse/MAPREDUCE-1747) | *Blocker* | **Remove documentation for the 'unstable' job-acls feature**
+* [MAPREDUCE-943](https://issues.apache.org/jira/browse/MAPREDUCE-943) | *Major* | **TestNodeRefresh timesout occasionally**
-Removed the documentation for the 'unstable' job-acls feature from branch 0.21.
+TestNodeRefresh timed out as the code to do with node refresh got removed. This patch removes the testcase.
---
-* [MAPREDUCE-1727](https://issues.apache.org/jira/browse/MAPREDUCE-1727) | *Major* | **TestJobACLs fails after HADOOP-6686**
+* [HDFS-288](https://issues.apache.org/jira/browse/HDFS-288) | *Major* | **Redundant computation in hashCode() implemenation**
-Fixed a testcase problem in TestJobACLs.
+**WARNING: No release note provided for this change.**
---
-* [MAPREDUCE-1697](https://issues.apache.org/jira/browse/MAPREDUCE-1697) | *Major* | **Document the behavior of -file option in streaming and deprecate it in favour of generic -files option.**
+* [HADOOP-6230](https://issues.apache.org/jira/browse/HADOOP-6230) | *Major* | **Move process tree, and memory calculator classes out of Common into Map/Reduce.**
-Documented the behavior of -file option in streaming and deprecated it in favor of generic -files option.
+Moved process tree, and memory calculator classes out of Common project into the Map/Reduce project.
---
-* [MAPREDUCE-1692](https://issues.apache.org/jira/browse/MAPREDUCE-1692) | *Minor* | **Remove TestStreamedMerge from the streaming tests**
+* [HADOOP-6203](https://issues.apache.org/jira/browse/HADOOP-6203) | *Major* | **Improve error message when moving to trash fails due to quota issue**
-Removed streaming testcase which tested non-existent functionality in Streaming.
+Improved error message suggests using -skpTrash option when hdfs -rm fails to move to trash because of quota.
---
-* [MAPREDUCE-1657](https://issues.apache.org/jira/browse/MAPREDUCE-1657) | *Major* | **After task logs directory is deleted, tasklog servlet displays wrong error message about job ACLs**
+* [MAPREDUCE-144](https://issues.apache.org/jira/browse/MAPREDUCE-144) | *Major* | **TaskMemoryManager should log process-tree's status while killing tasks.**
-Fixed a bug in tasklog servlet which displayed wrong error message about job ACLs - an access control error instead of the expected log files gone error - after task logs directory is deleted.
+Modified TaskMemoryManager so that it logs a map/reduce task's process-tree's status just before it is killed when it grows out of its configured memory limits. The log dump is in the format " \|- PID PPID PGRPID SESSID CMD\_NAME VMEM\_USAGE(BYTES) FULL\_CMD\_LINE".
+
+This is useful for debugging the cause for a map/reduce task and it's corresponding process-tree to be killed by the TaskMemoryManager.
---
-* [MAPREDUCE-1635](https://issues.apache.org/jira/browse/MAPREDUCE-1635) | *Major* | **ResourceEstimator does not work after MAPREDUCE-842**
+* [HADOOP-6235](https://issues.apache.org/jira/browse/HADOOP-6235) | *Major* | **Adding a new method for getting server default values from a FileSystem**
-Fixed a bug related to resource estimation for disk-based scheduling by modifying TaskTracker to return correct map output size for the completed maps and -1 for other tasks or failures.
+New FileSystem method reports default parameters that would be used by server. See also HDFS-578.
---
-* [MAPREDUCE-1612](https://issues.apache.org/jira/browse/MAPREDUCE-1612) | *Major* | **job conf file is not accessible from job history web page**
+* [HDFS-578](https://issues.apache.org/jira/browse/HDFS-578) | *Major* | **Support for using server default values for blockSize and replication when creating a file**
-Fixed a bug related to access of job\_conf.xml from the history web page of a job.
+New FileSystem.getServerDefaults() reports the server's default file creation parameters.
---
-* [MAPREDUCE-1611](https://issues.apache.org/jira/browse/MAPREDUCE-1611) | *Blocker* | **Refresh nodes and refresh queues doesnt work with service authorization enabled**
+* [HDFS-595](https://issues.apache.org/jira/browse/HDFS-595) | *Major* | **FsPermission tests need to be updated for new octal configuration parameter from HADOOP-6234**
-Fixed a bug that caused all the AdminOperationsProtocol operations to fail when service-level authorization is enabled. The problem is solved by registering AdminOperationsProtocol also with MapReducePolicyProvider.
+Unit tests updated to match syntax of new configuration parameters.
---
-* [MAPREDUCE-1610](https://issues.apache.org/jira/browse/MAPREDUCE-1610) | *Major* | **Forrest documentation should be updated to reflect the changes in MAPREDUCE-856**
+* [HADOOP-6234](https://issues.apache.org/jira/browse/HADOOP-6234) | *Major* | **Permission configuration files should use octal and symbolic**
-Updated forrest documentation to reflect the changes to make localized files from DistributedCache have right access-control on TaskTrackers(MAPREDUCE-856).
+New configuration option dfs.umaskmode sets umask with octal or symbolic value.
---
-* [MAPREDUCE-1609](https://issues.apache.org/jira/browse/MAPREDUCE-1609) | *Major* | **TaskTracker.localizeJob should not set permissions on job log directory recursively**
+* [HDFS-235](https://issues.apache.org/jira/browse/HDFS-235) | *Major* | **Add support for byte-ranges to hftp**
-Fixed TaskTracker so that it does not set permissions on job-log directory recursively. This fix both improves the performance of job localization as well as avoids a bug related to launching of task-cleanup attempts after TaskTracker's restart.
+HFTP can now serve a specific byte range from a file
---
-* [MAPREDUCE-1607](https://issues.apache.org/jira/browse/MAPREDUCE-1607) | *Major* | **Task controller may not set permissions for a task cleanup attempt's log directory**
+* [HADOOP-5073](https://issues.apache.org/jira/browse/HADOOP-5073) | *Major* | **Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability**
-Fixed initialization of a task-cleanup attempt's log directory by setting correct permissions via task-controller. Added new log4j properties hadoop.tasklog.iscleanup and log4j.appender.TLA.isCleanup to conf/log4j.properties. Changed the userlogs for a task-cleanup attempt to go into its own directory instead of the original attempt directory. This is an incompatible change as old userlogs of cleanup attempt-dirs before this release will no longer be visible.
+Annotation mechanism enables interface classification.
---
-* [MAPREDUCE-1606](https://issues.apache.org/jira/browse/MAPREDUCE-1606) | *Major* | **TestJobACLs may timeout as there are no slots for launching JOB\_CLEANUP task**
+* [MAPREDUCE-963](https://issues.apache.org/jira/browse/MAPREDUCE-963) | *Major* | **mapred's FileAlreadyExistsException should be deprecated in favor of hadoop-common's one.**
-Fixed TestJobACLs test timeout failure because of no slots for launching JOB\_CLEANUP task.
+Deprecate o.a.h.mapred.FileAlreadyExistsException and replace it with o.a.h.fs.FileAlreadyExistsException.
---
-* [MAPREDUCE-1568](https://issues.apache.org/jira/browse/MAPREDUCE-1568) | *Major* | **TrackerDistributedCacheManager should clean up cache in a background thread**
+* [HADOOP-4012](https://issues.apache.org/jira/browse/HADOOP-4012) | *Major* | **Providing splitting support for bzip2 compressed files**
-MAPREDUCE-1568. TrackerDistributedCacheManager should clean up cache in a background thread. (Scott Chen via zshao)
+BZip2 files can now be split.
---
-* [MAPREDUCE-1493](https://issues.apache.org/jira/browse/MAPREDUCE-1493) | *Major* | **Authorization for job-history pages**
+* [MAPREDUCE-830](https://issues.apache.org/jira/browse/MAPREDUCE-830) | *Major* | **Providing BZip2 splitting support for Text data**
-Added web-authorization for job-history pages. This is an incompatible change - it changes the JobHistory format by adding job-acls to job-history files and JobHistory currently does not have the support to read older versions of history files.
+Splitting support for BZip2 Text data
---
-* [MAPREDUCE-1482](https://issues.apache.org/jira/browse/MAPREDUCE-1482) | *Major* | **Better handling of task diagnostic information stored in the TaskInProgress**
+* [HDFS-602](https://issues.apache.org/jira/browse/HDFS-602) | *Major* | **Atempt to make a directory under an existing file on DistributedFileSystem should throw an FileAlreadyExistsException instead of FileNotFoundException**
-Limit the size of diagnostics-string and state-string shipped as part of task status. This will help keep the JobTracker's memory usage under control. Diagnostic string and state string are capped to 1024 chars.
+**WARNING: No release note provided for this change.**
---
-* [MAPREDUCE-1476](https://issues.apache.org/jira/browse/MAPREDUCE-1476) | *Major* | **committer.needsTaskCommit should not be called for a task cleanup attempt**
+* [HADOOP-5879](https://issues.apache.org/jira/browse/HADOOP-5879) | *Major* | **GzipCodec should read compression level etc from configuration**
-Fixed Map/Reduce framework to not call commit task for special tasks like job setup/cleanup and task cleanup.
+Provide an ability to configure the compression level and strategy for codecs. Compressors need to be 'reinited' with new characteristics such as compression level etc. and hence an incompatible addition to the api.
---
-* [MAPREDUCE-1466](https://issues.apache.org/jira/browse/MAPREDUCE-1466) | *Minor* | **FileInputFormat should save #input-files in JobConf**
+* [MAPREDUCE-856](https://issues.apache.org/jira/browse/MAPREDUCE-856) | *Major* | **Localized files from DistributedCache should have right access-control**
-Added a private configuration variable mapreduce.input.num.files, to store num
<TRUNCATED>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org