You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by zh...@apache.org on 2015/02/23 20:36:49 UTC
[01/52] [abbrv] hadoop git commit: HADOOP-11593. Convert site
documentation from apt to markdown (stragglers) (Masatake Iwasaki via aw)
Repository: hadoop
Updated Branches:
refs/heads/HDFS-7285 49204fcf1 -> e6d064b3b (forced update)
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm b/hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm
new file mode 100644
index 0000000..0b64586
--- /dev/null
+++ b/hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm
@@ -0,0 +1,559 @@
+%<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+#set ( $H3 = '###' )
+#set ( $H4 = '####' )
+#set ( $H5 = '#####' )
+
+Hadoop Streaming
+================
+
+* [Hadoop Streaming](#Hadoop_Streaming)
+ * [Hadoop Streaming](#Hadoop_Streaming)
+ * [How Streaming Works](#How_Streaming_Works)
+ * [Streaming Command Options](#Streaming_Command_Options)
+ * [Specifying a Java Class as the Mapper/Reducer](#Specifying_a_Java_Class_as_the_MapperReducer)
+ * [Packaging Files With Job Submissions](#Packaging_Files_With_Job_Submissions)
+ * [Specifying Other Plugins for Jobs](#Specifying_Other_Plugins_for_Jobs)
+ * [Setting Environment Variables](#Setting_Environment_Variables)
+ * [Generic Command Options](#Generic_Command_Options)
+ * [Specifying Configuration Variables with the -D Option](#Specifying_Configuration_Variables_with_the_-D_Option)
+ * [Specifying Directories](#Specifying_Directories)
+ * [Specifying Map-Only Jobs](#Specifying_Map-Only_Jobs)
+ * [Specifying the Number of Reducers](#Specifying_the_Number_of_Reducers)
+ * [Customizing How Lines are Split into Key/Value Pairs](#Customizing_How_Lines_are_Split_into_KeyValue_Pairs)
+ * [Working with Large Files and Archives](#Working_with_Large_Files_and_Archives)
+ * [Making Files Available to Tasks](#Making_Files_Available_to_Tasks)
+ * [Making Archives Available to Tasks](#Making_Archives_Available_to_Tasks)
+ * [More Usage Examples](#More_Usage_Examples)
+ * [Hadoop Partitioner Class](#Hadoop_Partitioner_Class)
+ * [Hadoop Comparator Class](#Hadoop_Comparator_Class)
+ * [Hadoop Aggregate Package](#Hadoop_Aggregate_Package)
+ * [Hadoop Field Selection Class](#Hadoop_Field_Selection_Class)
+ * [Frequently Asked Questions](#Frequently_Asked_Questions)
+ * [How do I use Hadoop Streaming to run an arbitrary set of (semi) independent tasks?](#How_do_I_use_Hadoop_Streaming_to_run_an_arbitrary_set_of_semi_independent_tasks)
+ * [How do I process files, one per map?](#How_do_I_process_files_one_per_map)
+ * [How many reducers should I use?](#How_many_reducers_should_I_use)
+ * [If I set up an alias in my shell script, will that work after -mapper?](#If_I_set_up_an_alias_in_my_shell_script_will_that_work_after_-mapper)
+ * [Can I use UNIX pipes?](#Can_I_use_UNIX_pipes)
+ * [What do I do if I get the "No space left on device" error?](#What_do_I_do_if_I_get_the_No_space_left_on_device_error)
+ * [How do I specify multiple input directories?](#How_do_I_specify_multiple_input_directories)
+ * [How do I generate output files with gzip format?](#How_do_I_generate_output_files_with_gzip_format)
+ * [How do I provide my own input/output format with streaming?](#How_do_I_provide_my_own_inputoutput_format_with_streaming)
+ * [How do I parse XML documents using streaming?](#How_do_I_parse_XML_documents_using_streaming)
+ * [How do I update counters in streaming applications?](#How_do_I_update_counters_in_streaming_applications)
+ * [How do I update status in streaming applications?](#How_do_I_update_status_in_streaming_applications)
+ * [How do I get the Job variables in a streaming job's mapper/reducer?](#How_do_I_get_the_Job_variables_in_a_streaming_jobs_mapperreducer)
+
+Hadoop Streaming
+----------------
+
+Hadoop streaming is a utility that comes with the Hadoop distribution. The utility allows you to create and run Map/Reduce jobs with any executable or script as the mapper and/or the reducer. For example:
+
+ hadoop jar hadoop-streaming-${project.version}.jar \
+ -input myInputDirs \
+ -output myOutputDir \
+ -mapper /bin/cat \
+ -reducer /usr/bin/wc
+
+How Streaming Works
+-------------------
+
+In the above example, both the mapper and the reducer are executables that read the input from stdin (line by line) and emit the output to stdout. The utility will create a Map/Reduce job, submit the job to an appropriate cluster, and monitor the progress of the job until it completes.
+
+When an executable is specified for mappers, each mapper task will launch the executable as a separate process when the mapper is initialized. As the mapper task runs, it converts its inputs into lines and feed the lines to the stdin of the process. In the meantime, the mapper collects the line oriented outputs from the stdout of the process and converts each line into a key/value pair, which is collected as the output of the mapper. By default, the *prefix of a line up to the first tab character* is the `key` and the rest of the line (excluding the tab character) will be the `value`. If there is no tab character in the line, then entire line is considered as key and the value is null. However, this can be customized by setting `-inputformat` command option, as discussed later.
+
+When an executable is specified for reducers, each reducer task will launch the executable as a separate process then the reducer is initialized. As the reducer task runs, it converts its input key/values pairs into lines and feeds the lines to the stdin of the process. In the meantime, the reducer collects the line oriented outputs from the stdout of the process, converts each line into a key/value pair, which is collected as the output of the reducer. By default, the prefix of a line up to the first tab character is the key and the rest of the line (excluding the tab character) is the value. However, this can be customized by setting `-outputformat` command option, as discussed later.
+
+This is the basis for the communication protocol between the Map/Reduce framework and the streaming mapper/reducer.
+
+User can specify `stream.non.zero.exit.is.failure` as `true` or `false` to make a streaming task that exits with a non-zero status to be `Failure` or `Success` respectively. By default, streaming tasks exiting with non-zero status are considered to be failed tasks.
+
+Streaming Command Options
+-------------------------
+
+Streaming supports streaming command options as well as [generic command options](#Generic_Command_Options). The general command line syntax is shown below.
+
+**Note:** Be sure to place the generic options before the streaming options, otherwise the command will fail. For an example, see [Making Archives Available to Tasks](#Making_Archives_Available_to_Tasks).
+
+ hadoop command [genericOptions] [streamingOptions]
+
+The Hadoop streaming command options are listed here:
+
+| Parameter | Optional/Required | Description |
+|:---- |:---- |:---- |
+| -input directoryname or filename | Required | Input location for mapper |
+| -output directoryname | Required | Output location for reducer |
+| -mapper executable or JavaClassName | Required | Mapper executable |
+| -reducer executable or JavaClassName | Required | Reducer executable |
+| -file filename | Optional | Make the mapper, reducer, or combiner executable available locally on the compute nodes |
+| -inputformat JavaClassName | Optional | Class you supply should return key/value pairs of Text class. If not specified, TextInputFormat is used as the default |
+| -outputformat JavaClassName | Optional | Class you supply should take key/value pairs of Text class. If not specified, TextOutputformat is used as the default |
+| -partitioner JavaClassName | Optional | Class that determines which reduce a key is sent to |
+| -combiner streamingCommand or JavaClassName | Optional | Combiner executable for map output |
+| -cmdenv name=value | Optional | Pass environment variable to streaming commands |
+| -inputreader | Optional | For backwards-compatibility: specifies a record reader class (instead of an input format class) |
+| -verbose | Optional | Verbose output |
+| -lazyOutput | Optional | Create output lazily. For example, if the output format is based on FileOutputFormat, the output file is created only on the first call to Context.write |
+| -numReduceTasks | Optional | Specify the number of reducers |
+| -mapdebug | Optional | Script to call when map task fails |
+| -reducedebug | Optional | Script to call when reduce task fails |
+
+$H3 Specifying a Java Class as the Mapper/Reducer
+
+You can supply a Java class as the mapper and/or the reducer.
+
+ hadoop jar hadoop-streaming-${project.version}.jar \
+ -input myInputDirs \
+ -output myOutputDir \
+ -inputformat org.apache.hadoop.mapred.KeyValueTextInputFormat \
+ -mapper org.apache.hadoop.mapred.lib.IdentityMapper \
+ -reducer /usr/bin/wc
+
+You can specify `stream.non.zero.exit.is.failure` as `true` or `false` to make a streaming task that exits with a non-zero status to be `Failure` or `Success` respectively. By default, streaming tasks exiting with non-zero status are considered to be failed tasks.
+
+$H3 Packaging Files With Job Submissions
+
+You can specify any executable as the mapper and/or the reducer. The executables do not need to pre-exist on the machines in the cluster; however, if they don't, you will need to use "-file" option to tell the framework to pack your executable files as a part of job submission. For example:
+
+ hadoop jar hadoop-streaming-${project.version}.jar \
+ -input myInputDirs \
+ -output myOutputDir \
+ -mapper myPythonScript.py \
+ -reducer /usr/bin/wc \
+ -file myPythonScript.py
+
+The above example specifies a user defined Python executable as the mapper. The option "-file myPythonScript.py" causes the python executable shipped to the cluster machines as a part of job submission.
+
+In addition to executable files, you can also package other auxiliary files (such as dictionaries, configuration files, etc) that may be used by the mapper and/or the reducer. For example:
+
+ hadoop jar hadoop-streaming-${project.version}.jar \
+ -input myInputDirs \
+ -output myOutputDir \
+ -mapper myPythonScript.py \
+ -reducer /usr/bin/wc \
+ -file myPythonScript.py \
+ -file myDictionary.txt
+
+$H3 Specifying Other Plugins for Jobs
+
+Just as with a normal Map/Reduce job, you can specify other plugins for a streaming job:
+
+ -inputformat JavaClassName
+ -outputformat JavaClassName
+ -partitioner JavaClassName
+ -combiner streamingCommand or JavaClassName
+
+The class you supply for the input format should return key/value pairs of Text class. If you do not specify an input format class, the TextInputFormat is used as the default. Since the TextInputFormat returns keys of LongWritable class, which are actually not part of the input data, the keys will be discarded; only the values will be piped to the streaming mapper.
+
+The class you supply for the output format is expected to take key/value pairs of Text class. If you do not specify an output format class, the TextOutputFormat is used as the default.
+
+$H3 Setting Environment Variables
+
+To set an environment variable in a streaming command use:
+
+ -cmdenv EXAMPLE_DIR=/home/example/dictionaries/
+
+Generic Command Options
+-----------------------
+
+Streaming supports [streaming command options](#Streaming_Command_Options) as well as generic command options. The general command line syntax is shown below.
+
+**Note:** Be sure to place the generic options before the streaming options, otherwise the command will fail. For an example, see [Making Archives Available to Tasks](#Making_Archives_Available_to_Tasks).
+
+ hadoop command [genericOptions] [streamingOptions]
+
+The Hadoop generic command options you can use with streaming are listed here:
+
+| Parameter | Optional/Required | Description |
+|:---- |:---- |:---- |
+| -conf configuration\_file | Optional | Specify an application configuration file |
+| -D property=value | Optional | Use value for given property |
+| -fs host:port or local | Optional | Specify a namenode |
+| -files | Optional | Specify comma-separated files to be copied to the Map/Reduce cluster |
+| -libjars | Optional | Specify comma-separated jar files to include in the classpath |
+| -archives | Optional | Specify comma-separated archives to be unarchived on the compute machines |
+
+$H3 Specifying Configuration Variables with the -D Option
+
+You can specify additional configuration variables by using "-D \<property\>=\<value\>".
+
+$H4 Specifying Directories
+
+To change the local temp directory use:
+
+ -D dfs.data.dir=/tmp
+
+To specify additional local temp directories use:
+
+ -D mapred.local.dir=/tmp/local
+ -D mapred.system.dir=/tmp/system
+ -D mapred.temp.dir=/tmp/temp
+
+**Note:** For more details on job configuration parameters see: [mapred-default.xml](./mapred-default.xml)
+
+$H4 Specifying Map-Only Jobs
+
+Often, you may want to process input data using a map function only. To do this, simply set `mapreduce.job.reduces` to zero. The Map/Reduce framework will not create any reducer tasks. Rather, the outputs of the mapper tasks will be the final output of the job.
+
+ -D mapreduce.job.reduces=0
+
+To be backward compatible, Hadoop Streaming also supports the "-reducer NONE" option, which is equivalent to "-D mapreduce.job.reduces=0".
+
+$H4 Specifying the Number of Reducers
+
+To specify the number of reducers, for example two, use:
+
+ hadoop jar hadoop-streaming-${project.version}.jar \
+ -D mapreduce.job.reduces=2 \
+ -input myInputDirs \
+ -output myOutputDir \
+ -mapper /bin/cat \
+ -reducer /usr/bin/wc
+
+$H4 Customizing How Lines are Split into Key/Value Pairs
+
+As noted earlier, when the Map/Reduce framework reads a line from the stdout of the mapper, it splits the line into a key/value pair. By default, the prefix of the line up to the first tab character is the key and the rest of the line (excluding the tab character) is the value.
+
+However, you can customize this default. You can specify a field separator other than the tab character (the default), and you can specify the nth (n \>= 1) character rather than the first character in a line (the default) as the separator between the key and value. For example:
+
+ hadoop jar hadoop-streaming-${project.version}.jar \
+ -D stream.map.output.field.separator=. \
+ -D stream.num.map.output.key.fields=4 \
+ -input myInputDirs \
+ -output myOutputDir \
+ -mapper /bin/cat \
+ -reducer /bin/cat
+
+In the above example, "-D stream.map.output.field.separator=." specifies "." as the field separator for the map outputs, and the prefix up to the fourth "." in a line will be the key and the rest of the line (excluding the fourth ".") will be the value. If a line has less than four "."s, then the whole line will be the key and the value will be an empty Text object (like the one created by new Text("")).
+
+Similarly, you can use "-D stream.reduce.output.field.separator=SEP" and "-D stream.num.reduce.output.fields=NUM" to specify the nth field separator in a line of the reduce outputs as the separator between the key and the value.
+
+Similarly, you can specify "stream.map.input.field.separator" and "stream.reduce.input.field.separator" as the input separator for Map/Reduce inputs. By default the separator is the tab character.
+
+$H3 Working with Large Files and Archives
+
+The -files and -archives options allow you to make files and archives available to the tasks. The argument is a URI to the file or archive that you have already uploaded to HDFS. These files and archives are cached across jobs. You can retrieve the host and fs\_port values from the fs.default.name config variable.
+
+**Note:** The -files and -archives options are generic options. Be sure to place the generic options before the command options, otherwise the command will fail.
+
+$H4 Making Files Available to Tasks
+
+The -files option creates a symlink in the current working directory of the tasks that points to the local copy of the file.
+
+In this example, Hadoop automatically creates a symlink named testfile.txt in the current working directory of the tasks. This symlink points to the local copy of testfile.txt.
+
+ -files hdfs://host:fs_port/user/testfile.txt
+
+User can specify a different symlink name for -files using \#.
+
+ -files hdfs://host:fs_port/user/testfile.txt#testfile
+
+Multiple entries can be specified like this:
+
+ -files hdfs://host:fs_port/user/testfile1.txt,hdfs://host:fs_port/user/testfile2.txt
+
+$H4 Making Archives Available to Tasks
+
+The -archives option allows you to copy jars locally to the current working directory of tasks and automatically unjar the files.
+
+In this example, Hadoop automatically creates a symlink named testfile.jar in the current working directory of tasks. This symlink points to the directory that stores the unjarred contents of the uploaded jar file.
+
+ -archives hdfs://host:fs_port/user/testfile.jar
+
+User can specify a different symlink name for -archives using \#.
+
+ -archives hdfs://host:fs_port/user/testfile.tgz#tgzdir
+
+In this example, the input.txt file has two lines specifying the names of the two files: cachedir.jar/cache.txt and cachedir.jar/cache2.txt. "cachedir.jar" is a symlink to the archived directory, which has the files "cache.txt" and "cache2.txt".
+
+ hadoop jar hadoop-streaming-${project.version}.jar \
+ -archives 'hdfs://hadoop-nn1.example.com/user/me/samples/cachefile/cachedir.jar' \
+ -D mapreduce.job.maps=1 \
+ -D mapreduce.job.reduces=1 \
+ -D mapreduce.job.name="Experiment" \
+ -input "/user/me/samples/cachefile/input.txt" \
+ -output "/user/me/samples/cachefile/out" \
+ -mapper "xargs cat" \
+ -reducer "cat"
+
+ $ ls test_jar/
+ cache.txt cache2.txt
+
+ $ jar cvf cachedir.jar -C test_jar/ .
+ added manifest
+ adding: cache.txt(in = 30) (out= 29)(deflated 3%)
+ adding: cache2.txt(in = 37) (out= 35)(deflated 5%)
+
+ $ hdfs dfs -put cachedir.jar samples/cachefile
+
+ $ hdfs dfs -cat /user/me/samples/cachefile/input.txt
+ cachedir.jar/cache.txt
+ cachedir.jar/cache2.txt
+
+ $ cat test_jar/cache.txt
+ This is just the cache string
+
+ $ cat test_jar/cache2.txt
+ This is just the second cache string
+
+ $ hdfs dfs -ls /user/me/samples/cachefile/out
+ Found 2 items
+ -rw-r--r-* 1 me supergroup 0 2013-11-14 17:00 /user/me/samples/cachefile/out/_SUCCESS
+ -rw-r--r-* 1 me supergroup 69 2013-11-14 17:00 /user/me/samples/cachefile/out/part-00000
+
+ $ hdfs dfs -cat /user/me/samples/cachefile/out/part-00000
+ This is just the cache string
+ This is just the second cache string
+
+More Usage Examples
+-------------------
+
+$H3 Hadoop Partitioner Class
+
+Hadoop has a library class, [KeyFieldBasedPartitioner](../../api/org/apache/hadoop/mapred/lib/KeyFieldBasedPartitioner.html), that is useful for many applications. This class allows the Map/Reduce framework to partition the map outputs based on certain key fields, not the whole keys. For example:
+
+ hadoop jar hadoop-streaming-${project.version}.jar \
+ -D stream.map.output.field.separator=. \
+ -D stream.num.map.output.key.fields=4 \
+ -D map.output.key.field.separator=. \
+ -D mapreduce.partition.keypartitioner.options=-k1,2 \
+ -D mapreduce.job.reduces=12 \
+ -input myInputDirs \
+ -output myOutputDir \
+ -mapper /bin/cat \
+ -reducer /bin/cat \
+ -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner
+
+Here, *-D stream.map.output.field.separator=.* and *-D stream.num.map.output.key.fields=4* are as explained in previous example. The two variables are used by streaming to identify the key/value pair of mapper.
+
+The map output keys of the above Map/Reduce job normally have four fields separated by ".". However, the Map/Reduce framework will partition the map outputs by the first two fields of the keys using the *-D mapred.text.key.partitioner.options=-k1,2* option. Here, *-D map.output.key.field.separator=.* specifies the separator for the partition. This guarantees that all the key/value pairs with the same first two fields in the keys will be partitioned into the same reducer.
+
+*This is effectively equivalent to specifying the first two fields as the primary key and the next two fields as the secondary. The primary key is used for partitioning, and the combination of the primary and secondary keys is used for sorting.* A simple illustration is shown here:
+
+Output of map (the keys)
+
+ 11.12.1.2
+ 11.14.2.3
+ 11.11.4.1
+ 11.12.1.1
+ 11.14.2.2
+
+Partition into 3 reducers (the first 2 fields are used as keys for partition)
+
+ 11.11.4.1
+ -----------
+ 11.12.1.2
+ 11.12.1.1
+ -----------
+ 11.14.2.3
+ 11.14.2.2
+
+Sorting within each partition for the reducer(all 4 fields used for sorting)
+
+ 11.11.4.1
+ -----------
+ 11.12.1.1
+ 11.12.1.2
+ -----------
+ 11.14.2.2
+ 11.14.2.3
+
+$H3 Hadoop Comparator Class
+
+Hadoop has a library class, [KeyFieldBasedComparator](../../api/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedComparator.html), that is useful for many applications. This class provides a subset of features provided by the Unix/GNU Sort. For example:
+
+ hadoop jar hadoop-streaming-${project.version}.jar \
+ -D mapreduce.job.output.key.comparator.class=org.apache.hadoop.mapreduce.lib.partition.KeyFieldBasedComparator \
+ -D stream.map.output.field.separator=. \
+ -D stream.num.map.output.key.fields=4 \
+ -D mapreduce.map.output.key.field.separator=. \
+ -D mapreduce.partition.keycomparator.options=-k2,2nr \
+ -D mapreduce.job.reduces=1 \
+ -input myInputDirs \
+ -output myOutputDir \
+ -mapper /bin/cat \
+ -reducer /bin/cat
+
+The map output keys of the above Map/Reduce job normally have four fields separated by ".". However, the Map/Reduce framework will sort the outputs by the second field of the keys using the *-D mapreduce.partition.keycomparator.options=-k2,2nr* option. Here, *-n* specifies that the sorting is numerical sorting and *-r* specifies that the result should be reversed. A simple illustration is shown below:
+
+Output of map (the keys)
+
+ 11.12.1.2
+ 11.14.2.3
+ 11.11.4.1
+ 11.12.1.1
+ 11.14.2.2
+
+Sorting output for the reducer (where second field used for sorting)
+
+ 11.14.2.3
+ 11.14.2.2
+ 11.12.1.2
+ 11.12.1.1
+ 11.11.4.1
+
+$H3 Hadoop Aggregate Package
+
+Hadoop has a library package called [Aggregate](../../org/apache/hadoop/mapred/lib/aggregate/package-summary.html). Aggregate provides a special reducer class and a special combiner class, and a list of simple aggregators that perform aggregations such as "sum", "max", "min" and so on over a sequence of values. Aggregate allows you to define a mapper plugin class that is expected to generate "aggregatable items" for each input key/value pair of the mappers. The combiner/reducer will aggregate those aggregatable items by invoking the appropriate aggregators.
+
+To use Aggregate, simply specify "-reducer aggregate":
+
+ hadoop jar hadoop-streaming-${project.version}.jar \
+ -input myInputDirs \
+ -output myOutputDir \
+ -mapper myAggregatorForKeyCount.py \
+ -reducer aggregate \
+ -file myAggregatorForKeyCount.py \
+
+The python program myAggregatorForKeyCount.py looks like:
+
+ #!/usr/bin/python
+
+ import sys;
+
+ def generateLongCountToken(id):
+ return "LongValueSum:" + id + "\t" + "1"
+
+ def main(argv):
+ line = sys.stdin.readline();
+ try:
+ while line:
+ line = line[:-1];
+ fields = line.split("\t");
+ print generateLongCountToken(fields[0]);
+ line = sys.stdin.readline();
+ except "end of file":
+ return None
+ if __name__ == "__main__":
+ main(sys.argv)
+
+$H3 Hadoop Field Selection Class
+
+Hadoop has a library class, [FieldSelectionMapReduce](../../api/org/apache/hadoop/mapred/lib/FieldSelectionMapReduce.html), that effectively allows you to process text data like the unix "cut" utility. The map function defined in the class treats each input key/value pair as a list of fields. You can specify the field separator (the default is the tab character). You can select an arbitrary list of fields as the map output key, and an arbitrary list of fields as the map output value. Similarly, the reduce function defined in the class treats each input key/value pair as a list of fields. You can select an arbitrary list of fields as the reduce output key, and an arbitrary list of fields as the reduce output value. For example:
+
+ hadoop jar hadoop-streaming-${project.version}.jar \
+ -D mapreduce.map.output.key.field.separator=. \
+ -D mapreduce.partition.keypartitioner.options=-k1,2 \
+ -D mapreduce.fieldsel.data.field.separator=. \
+ -D mapreduce.fieldsel.map.output.key.value.fields.spec=6,5,1-3:0- \
+ -D mapreduce.fieldsel.reduce.output.key.value.fields.spec=0-2:5- \
+ -D mapreduce.map.output.key.class=org.apache.hadoop.io.Text \
+ -D mapreduce.job.reduces=12 \
+ -input myInputDirs \
+ -output myOutputDir \
+ -mapper org.apache.hadoop.mapred.lib.FieldSelectionMapReduce \
+ -reducer org.apache.hadoop.mapred.lib.FieldSelectionMapReduce \
+ -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner
+
+The option "-D mapreduce.fieldsel.map.output.key.value.fields.spec=6,5,1-3:0-" specifies key/value selection for the map outputs. Key selection spec and value selection spec are separated by ":". In this case, the map output key will consist of fields 6, 5, 1, 2, and 3. The map output value will consist of all fields (0- means field 0 and all the subsequent fields).
+
+The option "-D mapreduce.fieldsel.reduce.output.key.value.fields.spec=0-2:5-" specifies key/value selection for the reduce outputs. In this case, the reduce output key will consist of fields 0, 1, 2 (corresponding to the original fields 6, 5, 1). The reduce output value will consist of all fields starting from field 5 (corresponding to all the original fields).
+
+Frequently Asked Questions
+--------------------------
+
+$H3 How do I use Hadoop Streaming to run an arbitrary set of (semi) independent tasks?
+
+Often you do not need the full power of Map Reduce, but only need to run multiple instances of the same program - either on different parts of the data, or on the same data, but with different parameters. You can use Hadoop Streaming to do this.
+
+$H3 How do I process files, one per map?
+
+As an example, consider the problem of zipping (compressing) a set of files across the hadoop cluster. You can achieve this by using Hadoop Streaming and custom mapper script:
+
+* Generate a file containing the full HDFS path of the input files. Each map
+ task would get one file name as input.
+
+* Create a mapper script which, given a filename, will get the file to local
+ disk, gzip the file and put it back in the desired output directory.
+
+$H3 How many reducers should I use?
+
+See MapReduce Tutorial for details: [Reducer](./MapReduceTutorial.html#Reducer)
+
+$H3 If I set up an alias in my shell script, will that work after -mapper?
+
+For example, say I do: alias c1='cut -f1'. Will -mapper "c1" work?
+
+Using an alias will not work, but variable substitution is allowed as shown in this example:
+
+ $ hdfs dfs -cat /user/me/samples/student_marks
+ alice 50
+ bruce 70
+ charlie 80
+ dan 75
+
+ $ c2='cut -f2'; hadoop jar hadoop-streaming-${project.version}.jar \
+ -D mapreduce.job.name='Experiment' \
+ -input /user/me/samples/student_marks \
+ -output /user/me/samples/student_out \
+ -mapper "$c2" -reducer 'cat'
+
+ $ hdfs dfs -cat /user/me/samples/student_out/part-00000
+ 50
+ 70
+ 75
+ 80
+
+$H3 Can I use UNIX pipes?
+
+For example, will -mapper "cut -f1 | sed s/foo/bar/g" work?
+
+Currently this does not work and gives an "java.io.IOException: Broken pipe" error. This is probably a bug that needs to be investigated.
+
+$H3 What do I do if I get the "No space left on device" error?
+
+For example, when I run a streaming job by distributing large executables (for example, 3.6G) through the -file option, I get a "No space left on device" error.
+
+The jar packaging happens in a directory pointed to by the configuration variable stream.tmpdir. The default value of stream.tmpdir is /tmp. Set the value to a directory with more space:
+
+ -D stream.tmpdir=/export/bigspace/...
+
+$H3 How do I specify multiple input directories?
+
+You can specify multiple input directories with multiple '-input' options:
+
+ hadoop jar hadoop-streaming-${project.version}.jar \
+ -input '/user/foo/dir1' -input '/user/foo/dir2' \
+ (rest of the command)
+
+$H3 How do I generate output files with gzip format?
+
+Instead of plain text files, you can generate gzip files as your generated output. Pass '-D mapreduce.output.fileoutputformat.compress=true -D mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec' as option to your streaming job.
+
+$H3 How do I provide my own input/output format with streaming?
+
+You can specify your own custom class by packing them and putting the custom jar to `$HADOOP_CLASSPATH`.
+
+$H3 How do I parse XML documents using streaming?
+
+You can use the record reader StreamXmlRecordReader to process XML documents.
+
+ hadoop jar hadoop-streaming-${project.version}.jar \
+ -inputreader "StreamXmlRecord,begin=BEGIN_STRING,end=END_STRING" \
+ (rest of the command)
+
+Anything found between BEGIN\_STRING and END\_STRING would be treated as one record for map tasks.
+
+$H3 How do I update counters in streaming applications?
+
+A streaming process can use the stderr to emit counter information. `reporter:counter:<group>,<counter>,<amount>` should be sent to stderr to update the counter.
+
+$H3 How do I update status in streaming applications?
+
+A streaming process can use the stderr to emit status information. To set a status, `reporter:status:<message>` should be sent to stderr.
+
+$H3 How do I get the Job variables in a streaming job's mapper/reducer?
+
+See [Configured Parameters](./MapReduceTutorial.html#Configured_Parameters). During the execution of a streaming job, the names of the "mapred" parameters are transformed. The dots ( . ) become underscores ( \_ ). For example, mapreduce.job.id becomes mapreduce\_job\_id and mapreduce.job.jar becomes mapreduce\_job\_jar. In your code, use the parameter names with the underscores.
[31/52] [abbrv] hadoop git commit: HADOOP-11604. Prevent
ConcurrentModificationException while closing domain sockets during shutdown
of DomainSocketWatcher thread. Contributed by Chris Nauroth.
Posted by zh...@apache.org.
HADOOP-11604. Prevent ConcurrentModificationException while closing domain sockets during shutdown of DomainSocketWatcher thread. Contributed by Chris Nauroth.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/3c5ff075
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/3c5ff075
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/3c5ff075
Branch: refs/heads/HDFS-7285
Commit: 3c5ff0759c4f4e10c97c6d9036add00edb8be2b5
Parents: aa1c437
Author: cnauroth <cn...@apache.org>
Authored: Fri Feb 20 13:07:16 2015 -0800
Committer: cnauroth <cn...@apache.org>
Committed: Fri Feb 20 13:07:16 2015 -0800
----------------------------------------------------------------------
hadoop-common-project/hadoop-common/CHANGES.txt | 3 +
.../hadoop/net/unix/DomainSocketWatcher.java | 45 ++++++++++++--
.../net/unix/TestDomainSocketWatcher.java | 65 ++++++++++++++++++--
3 files changed, 105 insertions(+), 8 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3c5ff075/hadoop-common-project/hadoop-common/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt b/hadoop-common-project/hadoop-common/CHANGES.txt
index 763377c..b09868a 100644
--- a/hadoop-common-project/hadoop-common/CHANGES.txt
+++ b/hadoop-common-project/hadoop-common/CHANGES.txt
@@ -978,6 +978,9 @@ Release 2.7.0 - UNRELEASED
HADOOP-9087. Queue size metric for metric sinks isn't actually maintained
(Akira AJISAKA via jlowe)
+ HADOOP-11604. Prevent ConcurrentModificationException while closing domain
+ sockets during shutdown of DomainSocketWatcher thread. (cnauroth)
+
Release 2.6.1 - UNRELEASED
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3c5ff075/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/unix/DomainSocketWatcher.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/unix/DomainSocketWatcher.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/unix/DomainSocketWatcher.java
index 0172f6b..8c617dc 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/unix/DomainSocketWatcher.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/unix/DomainSocketWatcher.java
@@ -246,6 +246,13 @@ public final class DomainSocketWatcher implements Closeable {
this.interruptCheckPeriodMs = interruptCheckPeriodMs;
notificationSockets = DomainSocket.socketpair();
watcherThread.setDaemon(true);
+ watcherThread.setUncaughtExceptionHandler(
+ new Thread.UncaughtExceptionHandler() {
+ @Override
+ public void uncaughtException(Thread thread, Throwable t) {
+ LOG.error(thread + " terminating on unexpected exception", t);
+ }
+ });
watcherThread.start();
}
@@ -372,7 +379,17 @@ public final class DomainSocketWatcher implements Closeable {
}
}
- private void sendCallback(String caller, TreeMap<Integer, Entry> entries,
+ /**
+ * Send callback and return whether or not the domain socket was closed as a
+ * result of processing.
+ *
+ * @param caller reason for call
+ * @param entries mapping of file descriptor to entry
+ * @param fdSet set of file descriptors
+ * @param fd file descriptor
+ * @return true if the domain socket was closed as a result of processing
+ */
+ private boolean sendCallback(String caller, TreeMap<Integer, Entry> entries,
FdSet fdSet, int fd) {
if (LOG.isTraceEnabled()) {
LOG.trace(this + ": " + caller + " starting sendCallback for fd " + fd);
@@ -401,13 +418,30 @@ public final class DomainSocketWatcher implements Closeable {
"still in the poll(2) loop.");
}
IOUtils.cleanup(LOG, sock);
- entries.remove(fd);
fdSet.remove(fd);
+ return true;
} else {
if (LOG.isTraceEnabled()) {
LOG.trace(this + ": " + caller + ": sendCallback not " +
"closing fd " + fd);
}
+ return false;
+ }
+ }
+
+ /**
+ * Send callback, and if the domain socket was closed as a result of
+ * processing, then also remove the entry for the file descriptor.
+ *
+ * @param caller reason for call
+ * @param entries mapping of file descriptor to entry
+ * @param fdSet set of file descriptors
+ * @param fd file descriptor
+ */
+ private void sendCallbackAndRemove(String caller,
+ TreeMap<Integer, Entry> entries, FdSet fdSet, int fd) {
+ if (sendCallback(caller, entries, fdSet, fd)) {
+ entries.remove(fd);
}
}
@@ -427,7 +461,8 @@ public final class DomainSocketWatcher implements Closeable {
lock.lock();
try {
for (int fd : fdSet.getAndClearReadableFds()) {
- sendCallback("getAndClearReadableFds", entries, fdSet, fd);
+ sendCallbackAndRemove("getAndClearReadableFds", entries, fdSet,
+ fd);
}
if (!(toAdd.isEmpty() && toRemove.isEmpty())) {
// Handle pending additions (before pending removes).
@@ -448,7 +483,7 @@ public final class DomainSocketWatcher implements Closeable {
while (true) {
Map.Entry<Integer, DomainSocket> entry = toRemove.firstEntry();
if (entry == null) break;
- sendCallback("handlePendingRemovals",
+ sendCallbackAndRemove("handlePendingRemovals",
entries, fdSet, entry.getValue().fd);
}
processedCond.signalAll();
@@ -482,6 +517,8 @@ public final class DomainSocketWatcher implements Closeable {
try {
kick(); // allow the handler for notificationSockets[0] to read a byte
for (Entry entry : entries.values()) {
+ // We do not remove from entries as we iterate, because that can
+ // cause a ConcurrentModificationException.
sendCallback("close", entries, fdSet, entry.getDomainSocket().fd);
}
entries.clear();
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3c5ff075/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/unix/TestDomainSocketWatcher.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/unix/TestDomainSocketWatcher.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/unix/TestDomainSocketWatcher.java
index 7c5b42d..e85e414 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/unix/TestDomainSocketWatcher.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/unix/TestDomainSocketWatcher.java
@@ -17,6 +17,8 @@
*/
package org.apache.hadoop.net.unix;
+import static org.junit.Assert.assertFalse;
+
import java.util.ArrayList;
import java.util.Random;
import java.util.concurrent.CountDownLatch;
@@ -25,6 +27,7 @@ import java.util.concurrent.locks.ReentrantLock;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
+import org.junit.After;
import org.junit.Assume;
import org.junit.Before;
import org.junit.Test;
@@ -34,17 +37,28 @@ import com.google.common.util.concurrent.Uninterruptibles;
public class TestDomainSocketWatcher {
static final Log LOG = LogFactory.getLog(TestDomainSocketWatcher.class);
+ private Throwable trappedException = null;
+
@Before
public void before() {
Assume.assumeTrue(DomainSocket.getLoadingFailureReason() == null);
}
+ @After
+ public void after() {
+ if (trappedException != null) {
+ throw new IllegalStateException(
+ "DomainSocketWatcher thread terminated with unexpected exception.",
+ trappedException);
+ }
+ }
+
/**
* Test that we can create a DomainSocketWatcher and then shut it down.
*/
@Test(timeout=60000)
public void testCreateShutdown() throws Exception {
- DomainSocketWatcher watcher = new DomainSocketWatcher(10000000);
+ DomainSocketWatcher watcher = newDomainSocketWatcher(10000000);
watcher.close();
}
@@ -53,7 +67,7 @@ public class TestDomainSocketWatcher {
*/
@Test(timeout=180000)
public void testDeliverNotifications() throws Exception {
- DomainSocketWatcher watcher = new DomainSocketWatcher(10000000);
+ DomainSocketWatcher watcher = newDomainSocketWatcher(10000000);
DomainSocket pair[] = DomainSocket.socketpair();
final CountDownLatch latch = new CountDownLatch(1);
watcher.add(pair[1], new DomainSocketWatcher.Handler() {
@@ -73,17 +87,35 @@ public class TestDomainSocketWatcher {
*/
@Test(timeout=60000)
public void testInterruption() throws Exception {
- final DomainSocketWatcher watcher = new DomainSocketWatcher(10);
+ final DomainSocketWatcher watcher = newDomainSocketWatcher(10);
watcher.watcherThread.interrupt();
Uninterruptibles.joinUninterruptibly(watcher.watcherThread);
watcher.close();
}
+
+ /**
+ * Test that domain sockets are closed when the watcher is closed.
+ */
+ @Test(timeout=300000)
+ public void testCloseSocketOnWatcherClose() throws Exception {
+ final DomainSocketWatcher watcher = newDomainSocketWatcher(10000000);
+ DomainSocket pair[] = DomainSocket.socketpair();
+ watcher.add(pair[1], new DomainSocketWatcher.Handler() {
+ @Override
+ public boolean handle(DomainSocket sock) {
+ return true;
+ }
+ });
+ watcher.close();
+ Uninterruptibles.joinUninterruptibly(watcher.watcherThread);
+ assertFalse(pair[1].isOpen());
+ }
@Test(timeout=300000)
public void testStress() throws Exception {
final int SOCKET_NUM = 250;
final ReentrantLock lock = new ReentrantLock();
- final DomainSocketWatcher watcher = new DomainSocketWatcher(10000000);
+ final DomainSocketWatcher watcher = newDomainSocketWatcher(10000000);
final ArrayList<DomainSocket[]> pairs = new ArrayList<DomainSocket[]>();
final AtomicInteger handled = new AtomicInteger(0);
@@ -148,4 +180,29 @@ public class TestDomainSocketWatcher {
Uninterruptibles.joinUninterruptibly(removerThread);
watcher.close();
}
+
+ /**
+ * Creates a new DomainSocketWatcher and tracks its thread for termination due
+ * to an unexpected exception. At the end of each test, if there was an
+ * unexpected exception, then that exception is thrown to force a failure of
+ * the test.
+ *
+ * @param interruptCheckPeriodMs interrupt check period passed to
+ * DomainSocketWatcher
+ * @return new DomainSocketWatcher
+ * @throws Exception if there is any failure
+ */
+ private DomainSocketWatcher newDomainSocketWatcher(int interruptCheckPeriodMs)
+ throws Exception {
+ DomainSocketWatcher watcher = new DomainSocketWatcher(
+ interruptCheckPeriodMs);
+ watcher.watcherThread.setUncaughtExceptionHandler(
+ new Thread.UncaughtExceptionHandler() {
+ @Override
+ public void uncaughtException(Thread thread, Throwable t) {
+ trappedException = t;
+ }
+ });
+ return watcher;
+ }
}
[11/52] [abbrv] hadoop git commit: HDFS-7808. Remove obsolete -ns
options in in DFSHAAdmin.java. Contributed by Arshad Mohammad.
Posted by zh...@apache.org.
HDFS-7808. Remove obsolete -ns options in in DFSHAAdmin.java. Contributed by Arshad Mohammad.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9a3e2920
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9a3e2920
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9a3e2920
Branch: refs/heads/HDFS-7285
Commit: 9a3e29208740da94d0cca5bb1c8163bea60d1387
Parents: 2aa9979
Author: Haohui Mai <wh...@apache.org>
Authored: Wed Feb 18 15:14:39 2015 -0800
Committer: Haohui Mai <wh...@apache.org>
Committed: Wed Feb 18 15:15:08 2015 -0800
----------------------------------------------------------------------
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 +++
.../apache/hadoop/hdfs/tools/DFSHAAdmin.java | 20 --------------------
.../hadoop/hdfs/tools/TestDFSHAAdmin.java | 20 --------------------
.../hdfs/tools/TestDFSHAAdminMiniCluster.java | 3 ---
4 files changed, 3 insertions(+), 43 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9a3e2920/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index 70eae1c..3735e90 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -982,6 +982,9 @@ Release 2.7.0 - UNRELEASED
HDFS-6662. WebHDFS cannot open a file if its path contains "%".
(Gerson Carlos via wheat9)
+ HDFS-7808. Remove obsolete -ns options in in DFSHAAdmin.java.
+ (Arshad Mohammad via wheat9)
+
BREAKDOWN OF HDFS-7584 SUBTASKS AND RELATED JIRAS
HDFS-7720. Quota by Storage Type API, tools and ClientNameNode
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9a3e2920/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSHAAdmin.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSHAAdmin.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSHAAdmin.java
index 1ec6d35..6b6fb30 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSHAAdmin.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSHAAdmin.java
@@ -18,7 +18,6 @@
package org.apache.hadoop.hdfs.tools;
import java.io.PrintStream;
-import java.util.Arrays;
import java.util.Collection;
import org.apache.commons.logging.Log;
@@ -98,25 +97,6 @@ public class DFSHAAdmin extends HAAdmin {
printUsage(errOut);
return -1;
}
-
- int i = 0;
- String cmd = argv[i++];
-
- if ("-ns".equals(cmd)) {
- if (i == argv.length) {
- errOut.println("Missing nameservice ID");
- printUsage(errOut);
- return -1;
- }
- nameserviceId = argv[i++];
- if (i >= argv.length) {
- errOut.println("Missing command");
- printUsage(errOut);
- return -1;
- }
- argv = Arrays.copyOfRange(argv, i, argv.length);
- }
-
return super.runCmd(argv);
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9a3e2920/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdmin.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdmin.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdmin.java
index 33da4d4..8ecc71a 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdmin.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdmin.java
@@ -147,17 +147,6 @@ public class TestDFSHAAdmin {
}
@Test
- public void testNameserviceOption() throws Exception {
- assertEquals(-1, runTool("-ns"));
- assertOutputContains("Missing nameservice ID");
- assertEquals(-1, runTool("-ns", "ns1"));
- assertOutputContains("Missing command");
- // "ns1" isn't defined but we check this lazily and help doesn't use the ns
- assertEquals(0, runTool("-ns", "ns1", "-help", "transitionToActive"));
- assertOutputContains("Transitions the service into Active");
- }
-
- @Test
public void testNamenodeResolution() throws Exception {
Mockito.doReturn(STANDBY_READY_RESULT).when(mockProtocol).getServiceStatus();
assertEquals(0, runTool("-getServiceState", "nn1"));
@@ -279,15 +268,6 @@ public class TestDFSHAAdmin {
}
@Test
- public void testFailoverWithFencerAndNameservice() throws Exception {
- Mockito.doReturn(STANDBY_READY_RESULT).when(mockProtocol).getServiceStatus();
- HdfsConfiguration conf = getHAConf();
- conf.set(DFSConfigKeys.DFS_HA_FENCE_METHODS_KEY, getFencerTrueCommand());
- tool.setConf(conf);
- assertEquals(0, runTool("-ns", "ns1", "-failover", "nn1", "nn2"));
- }
-
- @Test
public void testFailoverWithFencerConfiguredAndForce() throws Exception {
Mockito.doReturn(STANDBY_READY_RESULT).when(mockProtocol).getServiceStatus();
HdfsConfiguration conf = getHAConf();
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9a3e2920/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java
index ee1c184..b4d45c3 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java
@@ -156,9 +156,6 @@ public class TestDFSHAAdminMiniCluster {
assertEquals(0, runTool("-transitionToActive", "nn1"));
assertEquals(0, runTool("-failover", "nn1", "nn2"));
- // Test failover with fencer and nameservice
- assertEquals(0, runTool("-ns", "minidfs-ns", "-failover", "nn2", "nn1"));
-
// Fencer has not run yet, since none of the above required fencing
assertEquals("", Files.toString(tmpFile, Charsets.UTF_8));
[16/52] [abbrv] hadoop git commit: HADOOP-11440. Use
"test.build.data" instead of "build.test.dir" for testing in
ClientBaseWithFixes. Contributed by Kengo Seki.
Posted by zh...@apache.org.
HADOOP-11440. Use "test.build.data" instead of "build.test.dir" for testing in ClientBaseWithFixes. Contributed by Kengo Seki.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/18fb421f
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/18fb421f
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/18fb421f
Branch: refs/heads/HDFS-7285
Commit: 18fb421fab73789a9b692f21a99d619b5dc5c9ff
Parents: b8a14ef
Author: Akira Ajisaka <aa...@apache.org>
Authored: Wed Feb 18 17:55:04 2015 -0800
Committer: Akira Ajisaka <aa...@apache.org>
Committed: Wed Feb 18 17:55:04 2015 -0800
----------------------------------------------------------------------
hadoop-common-project/hadoop-common/CHANGES.txt | 3 +++
.../src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java | 2 +-
2 files changed, 4 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/18fb421f/hadoop-common-project/hadoop-common/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt b/hadoop-common-project/hadoop-common/CHANGES.txt
index 92bd48d..7a065d5 100644
--- a/hadoop-common-project/hadoop-common/CHANGES.txt
+++ b/hadoop-common-project/hadoop-common/CHANGES.txt
@@ -621,6 +621,9 @@ Release 2.7.0 - UNRELEASED
HADOOP-11521. Make connection timeout configurable in s3a.
(Thomas Demoor via stevel)
+ HADOOP-11440. Use "test.build.data" instead of "build.test.dir" for testing
+ in ClientBaseWithFixes. (Kengo Seki via aajisaka)
+
OPTIMIZATIONS
HADOOP-11323. WritableComparator#compare keeps reference to byte array.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/18fb421f/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java
index 11d4657..7d0727a 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java
@@ -66,7 +66,7 @@ public abstract class ClientBaseWithFixes extends ZKTestCase {
public static int CONNECTION_TIMEOUT = 30000;
static final File BASETEST =
- new File(System.getProperty("build.test.dir", "build"));
+ new File(System.getProperty("test.build.data", "build"));
protected final String hostPort = initHostPort();
protected int maxCnxns = 0;
[07/52] [abbrv] hadoop git commit: HADOOP-11545.
ArrayIndexOutOfBoundsException is thrown with "hadoop credential list
-provider". Contributed by Brahma Reddy Battula.
Posted by zh...@apache.org.
HADOOP-11545. ArrayIndexOutOfBoundsException is thrown with "hadoop credential list -provider". Contributed by Brahma Reddy Battula.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/17146099
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/17146099
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/17146099
Branch: refs/heads/HDFS-7285
Commit: 17146099197000d85b3aedc84a672111f2c7908a
Parents: 2ecea5a
Author: Akira Ajisaka <aa...@apache.org>
Authored: Wed Feb 18 11:17:10 2015 -0800
Committer: Akira Ajisaka <aa...@apache.org>
Committed: Wed Feb 18 11:17:10 2015 -0800
----------------------------------------------------------------------
hadoop-common-project/hadoop-common/CHANGES.txt | 3 +++
.../hadoop/security/alias/CredentialShell.java | 19 ++++++++++++++++---
.../hadoop/security/alias/TestCredShell.java | 15 +++++++++++++++
3 files changed, 34 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/17146099/hadoop-common-project/hadoop-common/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt b/hadoop-common-project/hadoop-common/CHANGES.txt
index f248555..92bd48d 100644
--- a/hadoop-common-project/hadoop-common/CHANGES.txt
+++ b/hadoop-common-project/hadoop-common/CHANGES.txt
@@ -962,6 +962,9 @@ Release 2.7.0 - UNRELEASED
HADOOP-11599. Client#getTimeout should use IPC_CLIENT_PING_DEFAULT when
IPC_CLIENT_PING_KEY is not configured. (zhihai xu via ozawa)
+ HADOOP-11545. ArrayIndexOutOfBoundsException is thrown with "hadoop
+ credential list -provider". (Brahma Reddy Battula via aajisaka)
+
Release 2.6.1 - UNRELEASED
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/17146099/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialShell.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialShell.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialShell.java
index f397403..e8a721f 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialShell.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialShell.java
@@ -97,6 +97,10 @@ public class CredentialShell extends Configured implements Tool {
for (int i = 0; i < args.length; i++) { // parse command line
if (args[i].equals("create")) {
+ if (i == args.length - 1) {
+ printCredShellUsage();
+ return 1;
+ }
String alias = args[++i];
command = new CreateCommand(alias);
if (alias.equals("-help")) {
@@ -104,6 +108,10 @@ public class CredentialShell extends Configured implements Tool {
return 0;
}
} else if (args[i].equals("delete")) {
+ if (i == args.length - 1) {
+ printCredShellUsage();
+ return 1;
+ }
String alias = args[++i];
command = new DeleteCommand(alias);
if (alias.equals("-help")) {
@@ -113,6 +121,10 @@ public class CredentialShell extends Configured implements Tool {
} else if (args[i].equals("list")) {
command = new ListCommand();
} else if (args[i].equals("-provider")) {
+ if (i == args.length - 1) {
+ printCredShellUsage();
+ return 1;
+ }
userSuppliedProvider = true;
getConf().set(CredentialProviderFactory.CREDENTIAL_PROVIDER_PATH,
args[++i]);
@@ -195,7 +207,7 @@ public class CredentialShell extends Configured implements Tool {
}
private class ListCommand extends Command {
- public static final String USAGE = "list [-provider] [-help]";
+ public static final String USAGE = "list [-provider provider-path]";
public static final String DESC =
"The list subcommand displays the aliases contained within \n" +
"a particular provider - as configured in core-site.xml or " +
@@ -237,7 +249,7 @@ public class CredentialShell extends Configured implements Tool {
private class DeleteCommand extends Command {
public static final String USAGE =
- "delete <alias> [-provider] [-f] [-help]";
+ "delete <alias> [-f] [-provider provider-path]";
public static final String DESC =
"The delete subcommand deletes the credential\n" +
"specified as the <alias> argument from within the provider\n" +
@@ -308,7 +320,8 @@ public class CredentialShell extends Configured implements Tool {
}
private class CreateCommand extends Command {
- public static final String USAGE = "create <alias> [-provider] [-help]";
+ public static final String USAGE =
+ "create <alias> [-provider provider-path]";
public static final String DESC =
"The create subcommand creates a new credential for the name specified\n" +
"as the <alias> argument within the provider indicated through\n" +
http://git-wip-us.apache.org/repos/asf/hadoop/blob/17146099/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredShell.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredShell.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredShell.java
index 7ba4bc1..7551df6 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredShell.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/alias/TestCredShell.java
@@ -212,4 +212,19 @@ public class TestCredShell {
0, shell.init(new String[] {cmd, "-help"}));
}
}
+
+ @Test
+ public void testEmptyArgForCommands() throws Exception {
+ CredentialShell shell = new CredentialShell();
+ String[] command = { "list", "-provider" };
+ assertEquals("Expected empty argument on " + command + " to return 1", 1,
+ shell.init(command));
+
+ for (String cmd : Arrays.asList("create", "delete")) {
+ shell.setConf(new Configuration());
+ assertEquals("Expected empty argument on " + cmd + " to return 1", 1,
+ shell.init(new String[] { cmd }));
+ }
+
+ }
}
[18/52] [abbrv] hadoop git commit: HADOOP-11602. Fix
toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa)
Posted by zh...@apache.org.
HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/946456c6
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/946456c6
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/946456c6
Branch: refs/heads/HDFS-7285
Commit: 946456c6d88780abe0251b098dd771e9e1e93ab3
Parents: 18fb421
Author: Tsuyoshi Ozawa <oz...@apache.org>
Authored: Thu Feb 19 12:46:46 2015 +0900
Committer: Tsuyoshi Ozawa <oz...@apache.org>
Committed: Thu Feb 19 13:06:53 2015 +0900
----------------------------------------------------------------------
.../classification/tools/StabilityOptions.java | 5 +--
.../AltKerberosAuthenticationHandler.java | 6 ++--
.../authentication/util/TestKerberosUtil.java | 14 ++++----
hadoop-common-project/hadoop-common/CHANGES.txt | 2 ++
.../org/apache/hadoop/conf/Configuration.java | 3 +-
.../org/apache/hadoop/crypto/CipherSuite.java | 3 +-
.../hadoop/crypto/key/JavaKeyStoreProvider.java | 3 +-
.../java/org/apache/hadoop/fs/FileSystem.java | 7 ++--
.../apache/hadoop/fs/permission/AclEntry.java | 6 ++--
.../org/apache/hadoop/fs/shell/find/Name.java | 5 +--
.../io/compress/CompressionCodecFactory.java | 6 ++--
.../hadoop/metrics2/impl/MetricsConfig.java | 7 ++--
.../hadoop/metrics2/impl/MetricsSystemImpl.java | 2 +-
.../apache/hadoop/security/SecurityUtil.java | 5 +--
.../hadoop/security/WhitelistBasedResolver.java | 4 ++-
.../security/ssl/FileBasedKeyStoresFactory.java | 4 ++-
.../apache/hadoop/security/ssl/SSLFactory.java | 3 +-
.../security/ssl/SSLHostnameVerifier.java | 10 +++---
.../DelegationTokenAuthenticationHandler.java | 3 +-
.../web/DelegationTokenAuthenticator.java | 5 +--
.../org/apache/hadoop/util/StringUtils.java | 2 +-
.../java/org/apache/hadoop/ipc/TestIPC.java | 3 +-
.../java/org/apache/hadoop/ipc/TestSaslRPC.java | 3 +-
.../hadoop/security/TestSecurityUtil.java | 6 ++--
.../security/TestUserGroupInformation.java | 5 +--
.../hadoop/test/TimedOutTestsListener.java | 4 ++-
.../org/apache/hadoop/util/TestWinUtils.java | 7 ++--
.../java/org/apache/hadoop/nfs/NfsExports.java | 5 +--
.../server/CheckUploadContentTypeFilter.java | 4 ++-
.../hadoop/fs/http/server/FSOperations.java | 4 ++-
.../http/server/HttpFSParametersProvider.java | 4 ++-
.../org/apache/hadoop/lib/server/Server.java | 3 +-
.../service/hadoop/FileSystemAccessService.java | 5 +--
.../org/apache/hadoop/lib/wsrs/EnumParam.java | 3 +-
.../apache/hadoop/lib/wsrs/EnumSetParam.java | 3 +-
.../hadoop/lib/wsrs/ParametersProvider.java | 3 +-
.../org/apache/hadoop/hdfs/StorageType.java | 3 +-
.../org/apache/hadoop/hdfs/XAttrHelper.java | 21 ++++++++----
.../hadoop/hdfs/protocol/HdfsConstants.java | 3 +-
.../BlockStoragePolicySuite.java | 4 ++-
.../hdfs/server/common/HdfsServerConstants.java | 5 +--
.../hdfs/server/datanode/StorageLocation.java | 4 ++-
.../hdfs/server/namenode/FSEditLogOp.java | 3 +-
.../namenode/QuotaByStorageTypeEntry.java | 3 +-
.../hdfs/server/namenode/SecondaryNameNode.java | 2 +-
.../org/apache/hadoop/hdfs/tools/GetConf.java | 17 +++++-----
.../OfflineEditsVisitorFactory.java | 7 ++--
.../offlineImageViewer/FSImageHandler.java | 3 +-
.../org/apache/hadoop/hdfs/web/AuthFilter.java | 3 +-
.../org/apache/hadoop/hdfs/web/ParamFilter.java | 3 +-
.../hadoop/hdfs/web/WebHdfsFileSystem.java | 5 +--
.../hadoop/hdfs/web/resources/EnumParam.java | 3 +-
.../hadoop/hdfs/web/resources/EnumSetParam.java | 4 ++-
.../namenode/snapshot/TestSnapshotManager.java | 5 +--
.../jobhistory/JobHistoryEventHandler.java | 4 ++-
.../mapreduce/v2/app/webapp/AppController.java | 2 +-
.../apache/hadoop/mapreduce/TypeConverter.java | 3 +-
.../apache/hadoop/mapreduce/v2/util/MRApps.java | 5 +--
.../hadoop/mapreduce/TestTypeConverter.java | 4 ++-
.../java/org/apache/hadoop/mapred/Task.java | 3 +-
.../counters/FileSystemCounterGroup.java | 2 +-
.../mapreduce/filecache/DistributedCache.java | 4 +--
.../hadoop/mapreduce/lib/db/DBInputFormat.java | 4 ++-
.../org/apache/hadoop/mapreduce/tools/CLI.java | 8 +++--
.../java/org/apache/hadoop/fs/TestDFSIO.java | 20 ++++++------
.../org/apache/hadoop/fs/TestFileSystem.java | 6 +++-
.../org/apache/hadoop/fs/slive/Constants.java | 6 ++--
.../apache/hadoop/fs/slive/OperationData.java | 4 ++-
.../apache/hadoop/fs/slive/OperationOutput.java | 4 ++-
.../org/apache/hadoop/fs/slive/SliveTest.java | 3 +-
.../java/org/apache/hadoop/io/FileBench.java | 17 ++++++----
.../org/apache/hadoop/mapred/TestMapRed.java | 3 +-
.../apache/hadoop/examples/DBCountPageView.java | 3 +-
.../plugin/versioninfo/VersionInfoMojo.java | 4 ++-
.../fs/azure/AzureNativeFileSystemStore.java | 4 +--
.../apache/hadoop/tools/util/DistCpUtils.java | 11 ++++---
.../java/org/apache/hadoop/tools/DistCpV1.java | 5 ++-
.../gridmix/GridmixJobSubmissionPolicy.java | 3 +-
.../hadoop/tools/rumen/HadoopLogsAnalyzer.java | 34 +++++++++++---------
.../apache/hadoop/tools/rumen/JobBuilder.java | 3 +-
.../apache/hadoop/tools/rumen/LoggedTask.java | 3 +-
.../hadoop/tools/rumen/LoggedTaskAttempt.java | 3 +-
.../apache/hadoop/streaming/Environment.java | 2 +-
.../hadoop/yarn/client/cli/ApplicationCLI.java | 5 +--
.../apache/hadoop/yarn/client/cli/NodeCLI.java | 4 ++-
.../impl/pb/GetApplicationsRequestPBImpl.java | 6 ++--
.../pb/ApplicationSubmissionContextPBImpl.java | 3 +-
.../hadoop/yarn/webapp/hamlet/HamletGen.java | 5 +--
.../webapp/AHSWebServices.java | 4 ++-
.../timeline/webapp/TimelineWebServices.java | 3 +-
.../hadoop/yarn/server/webapp/WebServices.java | 10 +++---
.../server/resourcemanager/ClientRMService.java | 3 +-
.../resource/ResourceWeights.java | 3 +-
.../CapacitySchedulerConfiguration.java | 6 ++--
.../fair/FairSchedulerConfiguration.java | 3 +-
.../scheduler/fair/SchedulingPolicy.java | 3 +-
.../resourcemanager/webapp/NodesPage.java | 3 +-
.../resourcemanager/webapp/RMWebServices.java | 19 +++++++----
98 files changed, 331 insertions(+), 191 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/StabilityOptions.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/StabilityOptions.java b/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/StabilityOptions.java
index dbce31e..657dbce 100644
--- a/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/StabilityOptions.java
+++ b/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/StabilityOptions.java
@@ -21,6 +21,7 @@ import com.sun.javadoc.DocErrorReporter;
import java.util.ArrayList;
import java.util.List;
+import java.util.Locale;
class StabilityOptions {
public static final String STABLE_OPTION = "-stable";
@@ -28,7 +29,7 @@ class StabilityOptions {
public static final String UNSTABLE_OPTION = "-unstable";
public static Integer optionLength(String option) {
- String opt = option.toLowerCase();
+ String opt = option.toLowerCase(Locale.ENGLISH);
if (opt.equals(UNSTABLE_OPTION)) return 1;
if (opt.equals(EVOLVING_OPTION)) return 1;
if (opt.equals(STABLE_OPTION)) return 1;
@@ -38,7 +39,7 @@ class StabilityOptions {
public static void validOptions(String[][] options,
DocErrorReporter reporter) {
for (int i = 0; i < options.length; i++) {
- String opt = options[i][0].toLowerCase();
+ String opt = options[i][0].toLowerCase(Locale.ENGLISH);
if (opt.equals(UNSTABLE_OPTION)) {
RootDocProcessor.stability = UNSTABLE_OPTION;
} else if (opt.equals(EVOLVING_OPTION)) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AltKerberosAuthenticationHandler.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AltKerberosAuthenticationHandler.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AltKerberosAuthenticationHandler.java
index 987330f..dae3b50 100644
--- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AltKerberosAuthenticationHandler.java
+++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AltKerberosAuthenticationHandler.java
@@ -14,6 +14,7 @@
package org.apache.hadoop.security.authentication.server;
import java.io.IOException;
+import java.util.Locale;
import java.util.Properties;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
@@ -68,7 +69,8 @@ public abstract class AltKerberosAuthenticationHandler
NON_BROWSER_USER_AGENTS, NON_BROWSER_USER_AGENTS_DEFAULT)
.split("\\W*,\\W*");
for (int i = 0; i < nonBrowserUserAgents.length; i++) {
- nonBrowserUserAgents[i] = nonBrowserUserAgents[i].toLowerCase();
+ nonBrowserUserAgents[i] =
+ nonBrowserUserAgents[i].toLowerCase(Locale.ENGLISH);
}
}
@@ -120,7 +122,7 @@ public abstract class AltKerberosAuthenticationHandler
if (userAgent == null) {
return false;
}
- userAgent = userAgent.toLowerCase();
+ userAgent = userAgent.toLowerCase(Locale.ENGLISH);
boolean isBrowser = true;
for (String nonBrowserUserAgent : nonBrowserUserAgents) {
if (userAgent.contains(nonBrowserUserAgent)) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosUtil.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosUtil.java b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosUtil.java
index b0e8f04..89e07d1 100644
--- a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosUtil.java
+++ b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosUtil.java
@@ -21,6 +21,7 @@ import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
+import java.util.Locale;
import java.util.regex.Pattern;
import org.apache.directory.server.kerberos.shared.keytab.Keytab;
@@ -58,24 +59,25 @@ public class TestKerberosUtil {
// send null hostname
Assert.assertEquals("When no hostname is sent",
- service + "/" + localHostname.toLowerCase(),
+ service + "/" + localHostname.toLowerCase(Locale.ENGLISH),
KerberosUtil.getServicePrincipal(service, null));
// send empty hostname
Assert.assertEquals("When empty hostname is sent",
- service + "/" + localHostname.toLowerCase(),
+ service + "/" + localHostname.toLowerCase(Locale.ENGLISH),
KerberosUtil.getServicePrincipal(service, ""));
// send 0.0.0.0 hostname
Assert.assertEquals("When 0.0.0.0 hostname is sent",
- service + "/" + localHostname.toLowerCase(),
+ service + "/" + localHostname.toLowerCase(Locale.ENGLISH),
KerberosUtil.getServicePrincipal(service, "0.0.0.0"));
// send uppercase hostname
Assert.assertEquals("When uppercase hostname is sent",
- service + "/" + testHost.toLowerCase(),
+ service + "/" + testHost.toLowerCase(Locale.ENGLISH),
KerberosUtil.getServicePrincipal(service, testHost));
// send lowercase hostname
Assert.assertEquals("When lowercase hostname is sent",
- service + "/" + testHost.toLowerCase(),
- KerberosUtil.getServicePrincipal(service, testHost.toLowerCase()));
+ service + "/" + testHost.toLowerCase(Locale.ENGLISH),
+ KerberosUtil.getServicePrincipal(
+ service, testHost.toLowerCase(Locale.ENGLISH)));
}
@Test
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt b/hadoop-common-project/hadoop-common/CHANGES.txt
index 7a065d5..e6d560a 100644
--- a/hadoop-common-project/hadoop-common/CHANGES.txt
+++ b/hadoop-common-project/hadoop-common/CHANGES.txt
@@ -405,6 +405,8 @@ Trunk (Unreleased)
HADOOP-11585. Fix formatting in Tracing.md (Masatake Iwasaki via aw)
+ HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa)
+
OPTIMIZATIONS
HADOOP-7761. Improve the performance of raw comparisons. (todd)
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index 02654b7..5909e62 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -46,6 +46,7 @@ import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
import java.util.ListIterator;
+import java.util.Locale;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Properties;
@@ -1451,7 +1452,7 @@ public class Configuration implements Iterable<Map.Entry<String,String>>,
return defaultValue;
}
- valueString = valueString.toLowerCase();
+ valueString = valueString.toLowerCase(Locale.ENGLISH);
if ("true".equals(valueString))
return true;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CipherSuite.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CipherSuite.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CipherSuite.java
index c9355d7..c5601eb 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CipherSuite.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CipherSuite.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.crypto;
+import java.util.Locale;
import org.apache.hadoop.classification.InterfaceAudience;
/**
@@ -97,7 +98,7 @@ public enum CipherSuite {
String[] parts = name.split("/");
StringBuilder suffix = new StringBuilder();
for (String part : parts) {
- suffix.append(".").append(part.toLowerCase());
+ suffix.append(".").append(part.toLowerCase(Locale.ENGLISH));
}
return suffix.toString();
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/JavaKeyStoreProvider.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/JavaKeyStoreProvider.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/JavaKeyStoreProvider.java
index bfec1ef..9e09b6e 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/JavaKeyStoreProvider.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/JavaKeyStoreProvider.java
@@ -53,6 +53,7 @@ import java.util.Date;
import java.util.Enumeration;
import java.util.HashMap;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReadWriteLock;
@@ -422,7 +423,7 @@ public class JavaKeyStoreProvider extends KeyProvider {
@Override
public KeyVersion createKey(String name, byte[] material,
Options options) throws IOException {
- Preconditions.checkArgument(name.equals(name.toLowerCase()),
+ Preconditions.checkArgument(name.equals(name.toLowerCase(Locale.ENGLISH)),
"Uppercase key names are unsupported: %s", name);
writeLock.lock();
try {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
index cfa5198..f52ecad 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
@@ -33,6 +33,7 @@ import java.util.IdentityHashMap;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.NoSuchElementException;
import java.util.ServiceLoader;
@@ -2795,8 +2796,10 @@ public abstract class FileSystem extends Configured implements Closeable {
}
Key(URI uri, Configuration conf, long unique) throws IOException {
- scheme = uri.getScheme()==null?"":uri.getScheme().toLowerCase();
- authority = uri.getAuthority()==null?"":uri.getAuthority().toLowerCase();
+ scheme = uri.getScheme() == null ?
+ "" : uri.getScheme().toLowerCase(Locale.ENGLISH);
+ authority = uri.getAuthority() == null ?
+ "" : uri.getAuthority().toLowerCase(Locale.ENGLISH);
this.unique = unique;
this.ugi = UserGroupInformation.getCurrentUser();
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/AclEntry.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/AclEntry.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/AclEntry.java
index b9def64..6397564 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/AclEntry.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/AclEntry.java
@@ -20,6 +20,7 @@ package org.apache.hadoop.fs.permission;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
+import java.util.Locale;
import com.google.common.base.Objects;
@@ -106,7 +107,7 @@ public class AclEntry {
sb.append("default:");
}
if (type != null) {
- sb.append(type.toString().toLowerCase());
+ sb.append(type.toString().toLowerCase(Locale.ENGLISH));
}
sb.append(':');
if (name != null) {
@@ -263,7 +264,8 @@ public class AclEntry {
AclEntryType aclType = null;
try {
- aclType = Enum.valueOf(AclEntryType.class, split[index].toUpperCase());
+ aclType = Enum.valueOf(
+ AclEntryType.class, split[index].toUpperCase(Locale.ENGLISH));
builder.setType(aclType);
index++;
} catch (IllegalArgumentException iae) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/find/Name.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/find/Name.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/find/Name.java
index 88314c6..4c937ef 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/find/Name.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/find/Name.java
@@ -19,6 +19,7 @@ package org.apache.hadoop.fs.shell.find;
import java.io.IOException;
import java.util.Deque;
+import java.util.Locale;
import org.apache.hadoop.fs.GlobPattern;
import org.apache.hadoop.fs.shell.PathData;
@@ -73,7 +74,7 @@ final class Name extends BaseExpression {
public void prepare() throws IOException {
String argPattern = getArgument(1);
if (!caseSensitive) {
- argPattern = argPattern.toLowerCase();
+ argPattern = argPattern.toLowerCase(Locale.ENGLISH);
}
globPattern = new GlobPattern(argPattern);
}
@@ -82,7 +83,7 @@ final class Name extends BaseExpression {
public Result apply(PathData item, int depth) throws IOException {
String name = getPath(item).getName();
if (!caseSensitive) {
- name = name.toLowerCase();
+ name = name.toLowerCase(Locale.ENGLISH);
}
if (globPattern.matches(name)) {
return Result.PASS;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/CompressionCodecFactory.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/CompressionCodecFactory.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/CompressionCodecFactory.java
index eb35759..6d16823 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/CompressionCodecFactory.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/CompressionCodecFactory.java
@@ -65,10 +65,10 @@ public class CompressionCodecFactory {
codecsByClassName.put(codec.getClass().getCanonicalName(), codec);
String codecName = codec.getClass().getSimpleName();
- codecsByName.put(codecName.toLowerCase(), codec);
+ codecsByName.put(codecName.toLowerCase(Locale.ENGLISH), codec);
if (codecName.endsWith("Codec")) {
codecName = codecName.substring(0, codecName.length() - "Codec".length());
- codecsByName.put(codecName.toLowerCase(), codec);
+ codecsByName.put(codecName.toLowerCase(Locale.ENGLISH), codec);
}
}
@@ -240,7 +240,7 @@ public class CompressionCodecFactory {
CompressionCodec codec = getCodecByClassName(codecName);
if (codec == null) {
// trying to get the codec by name in case the name was specified instead a class
- codec = codecsByName.get(codecName.toLowerCase());
+ codec = codecsByName.get(codecName.toLowerCase(Locale.ENGLISH));
}
return codec;
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
index 167205e..f87ef4d 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
@@ -85,12 +85,13 @@ class MetricsConfig extends SubsetConfiguration {
private ClassLoader pluginLoader;
MetricsConfig(Configuration c, String prefix) {
- super(c, prefix.toLowerCase(Locale.US), ".");
+ super(c, prefix.toLowerCase(Locale.ENGLISH), ".");
}
static MetricsConfig create(String prefix) {
- return loadFirst(prefix, "hadoop-metrics2-"+ prefix.toLowerCase(Locale.US)
- +".properties", DEFAULT_FILE_NAME);
+ return loadFirst(prefix, "hadoop-metrics2" + "-"
+ + prefix.toLowerCase(Locale.ENGLISH)
+ +".properties", DEFAULT_FILE_NAME);
}
static MetricsConfig create(String prefix, String... fileNames) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
index 32b00f3..8964934 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
@@ -617,6 +617,6 @@ public class MetricsSystemImpl extends MetricsSystem implements MetricsSource {
String m = System.getProperty(MS_INIT_MODE_KEY);
String m2 = m == null ? System.getenv(MS_INIT_MODE_KEY) : m;
return InitMode.valueOf((m2 == null ? InitMode.NORMAL.name() : m2)
- .toUpperCase(Locale.US));
+ .toUpperCase(Locale.ENGLISH));
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
index 7cbee26..355ea91 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
@@ -182,7 +182,8 @@ public class SecurityUtil {
if (fqdn == null || fqdn.isEmpty() || fqdn.equals("0.0.0.0")) {
fqdn = getLocalHostName();
}
- return components[0] + "/" + fqdn.toLowerCase(Locale.US) + "@" + components[2];
+ return components[0] + "/" + fqdn.toLowerCase(Locale.ENGLISH) + "@"
+ + components[2];
}
static String getLocalHostName() throws UnknownHostException {
@@ -379,7 +380,7 @@ public class SecurityUtil {
}
host = addr.getAddress().getHostAddress();
} else {
- host = addr.getHostName().toLowerCase();
+ host = addr.getHostName().toLowerCase(Locale.ENGLISH);
}
return new Text(host + ":" + addr.getPort());
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/WhitelistBasedResolver.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/WhitelistBasedResolver.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/WhitelistBasedResolver.java
index dc0815e..59d1492 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/WhitelistBasedResolver.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/WhitelistBasedResolver.java
@@ -19,6 +19,7 @@ package org.apache.hadoop.security;
import java.net.InetAddress;
import java.net.UnknownHostException;
+import java.util.Locale;
import java.util.Map;
import java.util.TreeMap;
@@ -138,7 +139,8 @@ public class WhitelistBasedResolver extends SaslPropertiesResolver {
QualityOfProtection.PRIVACY.toString());
for (int i=0; i < qop.length; i++) {
- qop[i] = QualityOfProtection.valueOf(qop[i].toUpperCase()).getSaslQop();
+ qop[i] = QualityOfProtection.valueOf(
+ qop[i].toUpperCase(Locale.ENGLISH)).getSaslQop();
}
saslProps.put(Sasl.QOP, StringUtils.join(",", qop));
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/FileBasedKeyStoresFactory.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/FileBasedKeyStoresFactory.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/FileBasedKeyStoresFactory.java
index 4b81e17..50f96d6 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/FileBasedKeyStoresFactory.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/FileBasedKeyStoresFactory.java
@@ -33,6 +33,7 @@ import java.io.InputStream;
import java.security.GeneralSecurityException;
import java.security.KeyStore;
import java.text.MessageFormat;
+import java.util.Locale;
/**
* {@link KeyStoresFactory} implementation that reads the certificates from
@@ -94,7 +95,8 @@ public class FileBasedKeyStoresFactory implements KeyStoresFactory {
@VisibleForTesting
public static String resolvePropertyName(SSLFactory.Mode mode,
String template) {
- return MessageFormat.format(template, mode.toString().toLowerCase());
+ return MessageFormat.format(
+ template, mode.toString().toLowerCase(Locale.ENGLISH));
}
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/SSLFactory.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/SSLFactory.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/SSLFactory.java
index bbea33b..370f09f 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/SSLFactory.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/SSLFactory.java
@@ -33,6 +33,7 @@ import javax.net.ssl.SSLSocketFactory;
import java.io.IOException;
import java.net.HttpURLConnection;
import java.security.GeneralSecurityException;
+import java.util.Locale;
/**
* Factory that creates SSLEngine and SSLSocketFactory instances using
@@ -138,7 +139,7 @@ public class SSLFactory implements ConnectionConfigurator {
private HostnameVerifier getHostnameVerifier(Configuration conf)
throws GeneralSecurityException, IOException {
return getHostnameVerifier(conf.get(SSL_HOSTNAME_VERIFIER_KEY, "DEFAULT").
- trim().toUpperCase());
+ trim().toUpperCase(Locale.ENGLISH));
}
public static HostnameVerifier getHostnameVerifier(String verifier)
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/SSLHostnameVerifier.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/SSLHostnameVerifier.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/SSLHostnameVerifier.java
index dd5e67b..7a905f1 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/SSLHostnameVerifier.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/SSLHostnameVerifier.java
@@ -41,6 +41,7 @@ import java.util.Collection;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
+import java.util.Locale;
import java.util.Set;
import java.util.StringTokenizer;
import java.util.TreeSet;
@@ -365,7 +366,7 @@ public interface SSLHostnameVerifier extends javax.net.ssl.HostnameVerifier {
buf.append('<');
for (int i = 0; i < hosts.length; i++) {
String h = hosts[i];
- h = h != null ? h.trim().toLowerCase() : "";
+ h = h != null ? h.trim().toLowerCase(Locale.ENGLISH) : "";
hosts[i] = h;
if (i > 0) {
buf.append('/');
@@ -406,7 +407,7 @@ public interface SSLHostnameVerifier extends javax.net.ssl.HostnameVerifier {
out:
for (Iterator<String> it = names.iterator(); it.hasNext();) {
// Don't trim the CN, though!
- final String cn = it.next().toLowerCase();
+ final String cn = it.next().toLowerCase(Locale.ENGLISH);
// Store CN in StringBuffer in case we need to report an error.
buf.append(" <");
buf.append(cn);
@@ -424,7 +425,8 @@ public interface SSLHostnameVerifier extends javax.net.ssl.HostnameVerifier {
acceptableCountryWildcard(cn);
for (int i = 0; i < hosts.length; i++) {
- final String hostName = hosts[i].trim().toLowerCase();
+ final String hostName =
+ hosts[i].trim().toLowerCase(Locale.ENGLISH);
if (doWildcard) {
match = hostName.endsWith(cn.substring(1));
if (match && strictWithSubDomains) {
@@ -479,7 +481,7 @@ public interface SSLHostnameVerifier extends javax.net.ssl.HostnameVerifier {
}
public static boolean isLocalhost(String host) {
- host = host != null ? host.trim().toLowerCase() : "";
+ host = host != null ? host.trim().toLowerCase(Locale.ENGLISH) : "";
if (host.startsWith("::1")) {
int x = host.lastIndexOf('%');
if (x >= 0) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java
index c18b5d3..f1bcd5d 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java
@@ -23,6 +23,7 @@ import java.text.MessageFormat;
import java.util.HashMap;
import java.util.HashSet;
import java.util.LinkedHashMap;
+import java.util.Locale;
import java.util.Map;
import java.util.Properties;
import java.util.Set;
@@ -169,7 +170,7 @@ public abstract class DelegationTokenAuthenticationHandler
boolean requestContinues = true;
String op = ServletUtils.getParameter(request,
KerberosDelegationTokenAuthenticator.OP_PARAM);
- op = (op != null) ? op.toUpperCase() : null;
+ op = (op != null) ? op.toUpperCase(Locale.ENGLISH) : null;
if (DELEGATION_TOKEN_OPS.contains(op) &&
!request.getMethod().equals("OPTIONS")) {
KerberosDelegationTokenAuthenticator.DelegationTokenOperation dtOp =
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.java
index d93f7ac..5d826b7 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.java
@@ -37,6 +37,7 @@ import java.net.InetSocketAddress;
import java.net.URL;
import java.net.URLEncoder;
import java.util.HashMap;
+import java.util.Locale;
import java.util.Map;
/**
@@ -286,8 +287,8 @@ public abstract class DelegationTokenAuthenticator implements Authenticator {
HttpExceptionUtils.validateResponse(conn, HttpURLConnection.HTTP_OK);
if (hasResponse) {
String contentType = conn.getHeaderField(CONTENT_TYPE);
- contentType = (contentType != null) ? contentType.toLowerCase()
- : null;
+ contentType = (contentType != null) ?
+ contentType.toLowerCase(Locale.ENGLISH) : null;
if (contentType != null &&
contentType.contains(APPLICATION_JSON_MIME)) {
try {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
index ff8edc3..c1acc7e 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
@@ -901,7 +901,7 @@ public class StringUtils {
*/
public static String camelize(String s) {
StringBuilder sb = new StringBuilder();
- String[] words = split(s.toLowerCase(Locale.US), ESCAPE_CHAR, '_');
+ String[] words = split(s.toLowerCase(Locale.ENGLISH), ESCAPE_CHAR, '_');
for (String word : words)
sb.append(org.apache.commons.lang.StringUtils.capitalize(word));
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
index eb19f48..e52cb26 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
@@ -43,6 +43,7 @@ import java.net.SocketTimeoutException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.Locale;
import java.util.Random;
import java.util.concurrent.BrokenBarrierException;
import java.util.concurrent.CountDownLatch;
@@ -1296,7 +1297,7 @@ public class TestIPC {
StringBuilder hexString = new StringBuilder();
- for (String line : hexdump.toUpperCase().split("\n")) {
+ for (String line : hexdump.toUpperCase(Locale.ENGLISH).split("\n")) {
hexString.append(line.substring(0, LAST_HEX_COL).replace(" ", ""));
}
return StringUtils.hexStringToByte(hexString.toString());
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestSaslRPC.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestSaslRPC.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestSaslRPC.java
index 903990b..9b25b77 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestSaslRPC.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestSaslRPC.java
@@ -41,6 +41,7 @@ import java.security.Security;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
+import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.regex.Pattern;
@@ -181,7 +182,7 @@ public class TestSaslRPC {
StringBuilder sb = new StringBuilder();
int i = 0;
for (QualityOfProtection qop:qops){
- sb.append(qop.name().toLowerCase());
+ sb.append(qop.name().toLowerCase(Locale.ENGLISH));
if (++i < qops.length){
sb.append(",");
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
index 4616c90..1d68f8a 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
@@ -103,13 +103,13 @@ public class TestSecurityUtil {
String realm = "@REALM";
String principalInConf = service + SecurityUtil.HOSTNAME_PATTERN + realm;
String hostname = "FooHost";
- String principal = service + hostname.toLowerCase() + realm;
+ String principal = service + hostname.toLowerCase(Locale.ENGLISH) + realm;
verify(principalInConf, hostname, principal);
}
@Test
public void testLocalHostNameForNullOrWild() throws Exception {
- String local = SecurityUtil.getLocalHostName().toLowerCase(Locale.US);
+ String local = SecurityUtil.getLocalHostName().toLowerCase(Locale.ENGLISH);
assertEquals("hdfs/" + local + "@REALM",
SecurityUtil.getServerPrincipal("hdfs/_HOST@REALM", (String)null));
assertEquals("hdfs/" + local + "@REALM",
@@ -260,7 +260,7 @@ public class TestSecurityUtil {
//LOG.info("address:"+addr+" host:"+host+" ip:"+ip+" port:"+port);
SecurityUtil.setTokenServiceUseIp(useIp);
- String serviceHost = useIp ? ip : host.toLowerCase();
+ String serviceHost = useIp ? ip : host.toLowerCase(Locale.ENGLISH);
Token<?> token = new Token<TokenIdentifier>();
Text service = new Text(serviceHost+":"+port);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java
index 48b9b99..d77d9b5 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java
@@ -41,6 +41,7 @@ import java.security.PrivilegedExceptionAction;
import java.util.Collection;
import java.util.ConcurrentModificationException;
import java.util.LinkedHashSet;
+import java.util.Locale;
import java.util.Set;
import static org.apache.hadoop.fs.CommonConfigurationKeys.HADOOP_USER_GROUP_METRICS_PERCENTILES_INTERVALS;
@@ -213,7 +214,7 @@ public class TestUserGroupInformation {
userName = userName.substring(sp + 1);
}
// user names are case insensitive on Windows. Make consistent
- userName = userName.toLowerCase();
+ userName = userName.toLowerCase(Locale.ENGLISH);
}
// get the groups
pp = Runtime.getRuntime().exec(Shell.WINDOWS ?
@@ -233,7 +234,7 @@ public class TestUserGroupInformation {
String loginUserName = login.getShortUserName();
if(Shell.WINDOWS) {
// user names are case insensitive on Windows. Make consistent
- loginUserName = loginUserName.toLowerCase();
+ loginUserName = loginUserName.toLowerCase(Locale.ENGLISH);
}
assertEquals(userName, loginUserName);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/TimedOutTestsListener.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/TimedOutTestsListener.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/TimedOutTestsListener.java
index 220ab1d..4ebf29a 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/TimedOutTestsListener.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/TimedOutTestsListener.java
@@ -27,6 +27,7 @@ import java.lang.management.ThreadMXBean;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
+import java.util.Locale;
import java.util.Map;
import org.junit.runner.notification.Failure;
@@ -93,7 +94,8 @@ public class TimedOutTestsListener extends RunListener {
thread.getPriority(),
thread.getId(),
Thread.State.WAITING.equals(thread.getState()) ?
- "in Object.wait()" : thread.getState().name().toLowerCase(),
+ "in Object.wait()" :
+ thread.getState().name().toLowerCase(Locale.ENGLISH),
Thread.State.WAITING.equals(thread.getState()) ?
"WAITING (on object monitor)" : thread.getState()));
for (StackTraceElement stackTraceElement : e.getValue()) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java
index 2d4e442..708fc4c 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java
@@ -27,6 +27,7 @@ import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.FileWriter;
import java.io.IOException;
+import java.util.Locale;
import org.apache.commons.io.FileUtils;
import org.apache.commons.logging.Log;
@@ -382,8 +383,10 @@ public class TestWinUtils {
private void assertOwners(File file, String expectedUser,
String expectedGroup) throws IOException {
String [] args = lsF(file).trim().split("[\\|]");
- assertEquals(expectedUser.toLowerCase(), args[2].toLowerCase());
- assertEquals(expectedGroup.toLowerCase(), args[3].toLowerCase());
+ assertEquals(expectedUser.toLowerCase(Locale.ENGLISH),
+ args[2].toLowerCase(Locale.ENGLISH));
+ assertEquals(expectedGroup.toLowerCase(Locale.ENGLISH),
+ args[3].toLowerCase(Locale.ENGLISH));
}
@Test (timeout = 30000)
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsExports.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsExports.java b/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsExports.java
index b617ae5..cef8f99 100644
--- a/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsExports.java
+++ b/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsExports.java
@@ -20,6 +20,7 @@ package org.apache.hadoop.nfs;
import java.net.InetAddress;
import java.util.ArrayList;
import java.util.List;
+import java.util.Locale;
import java.util.regex.Pattern;
import org.apache.commons.logging.Log;
@@ -359,10 +360,10 @@ public class NfsExports {
AccessPrivilege privilege = AccessPrivilege.READ_ONLY;
switch (parts.length) {
case 1:
- host = parts[0].toLowerCase().trim();
+ host = parts[0].toLowerCase(Locale.ENGLISH).trim();
break;
case 2:
- host = parts[0].toLowerCase().trim();
+ host = parts[0].toLowerCase(Locale.ENGLISH).trim();
String option = parts[1].trim();
if ("rw".equalsIgnoreCase(option)) {
privilege = AccessPrivilege.READ_WRITE;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/CheckUploadContentTypeFilter.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/CheckUploadContentTypeFilter.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/CheckUploadContentTypeFilter.java
index 836b4ce..7074ba2 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/CheckUploadContentTypeFilter.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/CheckUploadContentTypeFilter.java
@@ -32,6 +32,7 @@ import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.util.HashSet;
+import java.util.Locale;
import java.util.Set;
/**
@@ -82,7 +83,8 @@ public class CheckUploadContentTypeFilter implements Filter {
String method = httpReq.getMethod();
if (method.equals("PUT") || method.equals("POST")) {
String op = httpReq.getParameter(HttpFSFileSystem.OP_PARAM);
- if (op != null && UPLOAD_OPERATIONS.contains(op.toUpperCase())) {
+ if (op != null &&
+ UPLOAD_OPERATIONS.contains(op.toUpperCase(Locale.ENGLISH))) {
if ("true".equalsIgnoreCase(httpReq.getParameter(HttpFSParametersProvider.DataParam.NAME))) {
String contentType = httpReq.getContentType();
contentTypeOK =
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
index e7d92f5..4b72a51 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
@@ -43,6 +43,7 @@ import java.io.OutputStream;
import java.util.EnumSet;
import java.util.LinkedHashMap;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.Map.Entry;
@@ -527,7 +528,8 @@ public class FSOperations {
@Override
public JSONObject execute(FileSystem fs) throws IOException {
boolean deleted = fs.delete(path, recursive);
- return toJSON(HttpFSFileSystem.DELETE_JSON.toLowerCase(), deleted);
+ return toJSON(
+ HttpFSFileSystem.DELETE_JSON.toLowerCase(Locale.ENGLISH), deleted);
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSParametersProvider.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSParametersProvider.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSParametersProvider.java
index 9b0be9b..fb06667 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSParametersProvider.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSParametersProvider.java
@@ -33,6 +33,7 @@ import org.apache.hadoop.lib.wsrs.StringParam;
import javax.ws.rs.ext.Provider;
import java.util.HashMap;
+import java.util.Locale;
import java.util.Map;
import java.util.regex.Pattern;
@@ -167,7 +168,8 @@ public class HttpFSParametersProvider extends ParametersProvider {
*/
public OperationParam(String operation) {
super(NAME, HttpFSFileSystem.Operation.class,
- HttpFSFileSystem.Operation.valueOf(operation.toUpperCase()));
+ HttpFSFileSystem.Operation.valueOf(
+ operation.toUpperCase(Locale.ENGLISH)));
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/server/Server.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/server/Server.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/server/Server.java
index 5c1bb4f..e2ef7a1 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/server/Server.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/server/Server.java
@@ -36,6 +36,7 @@ import java.util.ArrayList;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.Properties;
@@ -202,7 +203,7 @@ public class Server {
* @param config server configuration.
*/
public Server(String name, String homeDir, String configDir, String logDir, String tempDir, Configuration config) {
- this.name = Check.notEmpty(name, "name").trim().toLowerCase();
+ this.name = Check.notEmpty(name, "name").trim().toLowerCase(Locale.ENGLISH);
this.homeDir = Check.notEmpty(homeDir, "homeDir");
this.configDir = Check.notEmpty(configDir, "configDir");
this.logDir = Check.notEmpty(logDir, "logDir");
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/service/hadoop/FileSystemAccessService.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/service/hadoop/FileSystemAccessService.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/service/hadoop/FileSystemAccessService.java
index ccb15a3..fd2e822 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/service/hadoop/FileSystemAccessService.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/service/hadoop/FileSystemAccessService.java
@@ -43,6 +43,7 @@ import java.net.URI;
import java.security.PrivilegedExceptionAction;
import java.util.Collection;
import java.util.HashSet;
+import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
@@ -254,7 +255,7 @@ public class FileSystemAccessService extends BaseService implements FileSystemAc
private Set<String> toLowerCase(Collection<String> collection) {
Set<String> set = new HashSet<String>();
for (String value : collection) {
- set.add(value.toLowerCase());
+ set.add(value.toLowerCase(Locale.ENGLISH));
}
return set;
}
@@ -300,7 +301,7 @@ public class FileSystemAccessService extends BaseService implements FileSystemAc
protected void validateNamenode(String namenode) throws FileSystemAccessException {
if (nameNodeWhitelist.size() > 0 && !nameNodeWhitelist.contains("*")) {
- if (!nameNodeWhitelist.contains(namenode.toLowerCase())) {
+ if (!nameNodeWhitelist.contains(namenode.toLowerCase(Locale.ENGLISH))) {
throw new FileSystemAccessException(FileSystemAccessException.ERROR.H05, namenode, "not in whitelist");
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/wsrs/EnumParam.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/wsrs/EnumParam.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/wsrs/EnumParam.java
index 8baef67..08eb60d 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/wsrs/EnumParam.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/wsrs/EnumParam.java
@@ -22,6 +22,7 @@ import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.util.StringUtils;
import java.util.Arrays;
+import java.util.Locale;
@InterfaceAudience.Private
public abstract class EnumParam<E extends Enum<E>> extends Param<E> {
@@ -34,7 +35,7 @@ public abstract class EnumParam<E extends Enum<E>> extends Param<E> {
@Override
protected E parse(String str) throws Exception {
- return Enum.valueOf(klass, str.toUpperCase());
+ return Enum.valueOf(klass, str.toUpperCase(Locale.ENGLISH));
}
@Override
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/wsrs/EnumSetParam.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/wsrs/EnumSetParam.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/wsrs/EnumSetParam.java
index 8d79b71..25158fd 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/wsrs/EnumSetParam.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/wsrs/EnumSetParam.java
@@ -20,6 +20,7 @@ package org.apache.hadoop.lib.wsrs;
import java.util.Arrays;
import java.util.EnumSet;
import java.util.Iterator;
+import java.util.Locale;
import org.apache.hadoop.classification.InterfaceAudience;
@@ -37,7 +38,7 @@ public abstract class EnumSetParam<E extends Enum<E>> extends Param<EnumSet<E>>
final EnumSet<E> set = EnumSet.noneOf(klass);
if (!str.isEmpty()) {
for (String sub : str.split(",")) {
- set.add(Enum.valueOf(klass, sub.trim().toUpperCase()));
+ set.add(Enum.valueOf(klass, sub.trim().toUpperCase(Locale.ENGLISH)));
}
}
return set;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/wsrs/ParametersProvider.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/wsrs/ParametersProvider.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/wsrs/ParametersProvider.java
index 4703a90..9857244 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/wsrs/ParametersProvider.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/wsrs/ParametersProvider.java
@@ -33,6 +33,7 @@ import java.lang.reflect.Type;
import java.text.MessageFormat;
import java.util.HashMap;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
/**
@@ -70,7 +71,7 @@ public class ParametersProvider
}
Enum op;
try {
- op = Enum.valueOf(enumClass, str.toUpperCase());
+ op = Enum.valueOf(enumClass, str.toUpperCase(Locale.ENGLISH));
} catch (IllegalArgumentException ex) {
throw new IllegalArgumentException(
MessageFormat.format("Invalid Operation [{0}]", str));
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StorageType.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StorageType.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StorageType.java
index a26ed91..745e44b 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StorageType.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StorageType.java
@@ -21,6 +21,7 @@ package org.apache.hadoop.hdfs;
import java.util.Arrays;
import java.util.ArrayList;
import java.util.List;
+import java.util.Locale;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
@@ -78,7 +79,7 @@ public enum StorageType {
}
public static StorageType parseStorageType(String s) {
- return StorageType.valueOf(s.toUpperCase());
+ return StorageType.valueOf(s.toUpperCase(Locale.ENGLISH));
}
private static List<StorageType> getNonTransientTypes() {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
index 04364ccf..1b5b8eb 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.hdfs;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import org.apache.hadoop.HadoopIllegalArgumentException;
@@ -57,16 +58,22 @@ public class XAttrHelper {
}
NameSpace ns;
- final String prefix = name.substring(0, prefixIndex).toLowerCase();
- if (prefix.equals(NameSpace.USER.toString().toLowerCase())) {
+ final String prefix = name.substring(0, prefixIndex)
+ .toLowerCase(Locale.ENGLISH);
+ if (prefix.equals(
+ NameSpace.USER.toString().toLowerCase(Locale.ENGLISH))) {
ns = NameSpace.USER;
- } else if (prefix.equals(NameSpace.TRUSTED.toString().toLowerCase())) {
+ } else if (prefix.equals(
+ NameSpace.TRUSTED.toString().toLowerCase(Locale.ENGLISH))) {
ns = NameSpace.TRUSTED;
- } else if (prefix.equals(NameSpace.SYSTEM.toString().toLowerCase())) {
+ } else if (prefix.equals(
+ NameSpace.SYSTEM.toString().toLowerCase(Locale.ENGLISH))) {
ns = NameSpace.SYSTEM;
- } else if (prefix.equals(NameSpace.SECURITY.toString().toLowerCase())) {
+ } else if (prefix.equals(
+ NameSpace.SECURITY.toString().toLowerCase(Locale.ENGLISH))) {
ns = NameSpace.SECURITY;
- } else if (prefix.equals(NameSpace.RAW.toString().toLowerCase())) {
+ } else if (prefix.equals(
+ NameSpace.RAW.toString().toLowerCase(Locale.ENGLISH))) {
ns = NameSpace.RAW;
} else {
throw new HadoopIllegalArgumentException("An XAttr name must be " +
@@ -145,7 +152,7 @@ public class XAttrHelper {
}
String namespace = xAttr.getNameSpace().toString();
- return namespace.toLowerCase() + "." + xAttr.getName();
+ return namespace.toLowerCase(Locale.ENGLISH) + "." + xAttr.getName();
}
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
index 54da8eb..1769794 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.hdfs.protocol;
import java.util.HashMap;
+import java.util.Locale;
import java.util.Map;
import org.apache.hadoop.classification.InterfaceAudience;
@@ -98,7 +99,7 @@ public class HdfsConstants {
/** Covert the given String to a RollingUpgradeAction. */
public static RollingUpgradeAction fromString(String s) {
- return MAP.get(s.toUpperCase());
+ return MAP.get(s.toUpperCase(Locale.ENGLISH));
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
index ce87b06..a0eddad 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
@@ -30,6 +30,7 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.List;
+import java.util.Locale;
/** A collection of block storage policies. */
public class BlockStoragePolicySuite {
@@ -131,7 +132,8 @@ public class BlockStoragePolicySuite {
}
public static String buildXAttrName() {
- return XAttrNS.toString().toLowerCase() + "." + STORAGE_POLICY_XATTR_NAME;
+ return XAttrNS.toString().toLowerCase(Locale.ENGLISH) + "."
+ + STORAGE_POLICY_XATTR_NAME;
}
public static XAttr buildXAttr(byte policyId) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
index 9bba2c9..3674d2c 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
@@ -20,6 +20,7 @@ package org.apache.hadoop.hdfs.server.common;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
+import java.util.Locale;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
@@ -53,7 +54,7 @@ public final class HdfsServerConstants {
public String getOptionString() {
return StartupOption.ROLLINGUPGRADE.getName() + " "
- + name().toLowerCase();
+ + name().toLowerCase(Locale.ENGLISH);
}
public boolean matches(StartupOption option) {
@@ -76,7 +77,7 @@ public final class HdfsServerConstants {
public static String getAllOptionString() {
final StringBuilder b = new StringBuilder("<");
for(RollingUpgradeStartupOption opt : VALUES) {
- b.append(opt.name().toLowerCase()).append("|");
+ b.append(opt.name().toLowerCase(Locale.ENGLISH)).append("|");
}
b.setCharAt(b.length() - 1, '>');
return b.toString();
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
index feb5ac9..2b2da5c 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.hdfs.server.datanode;
+import java.util.Locale;
import java.util.regex.Pattern;
import java.io.File;
@@ -88,7 +89,8 @@ public class StorageLocation {
String classString = matcher.group(1);
location = matcher.group(2);
if (!classString.isEmpty()) {
- storageType = StorageType.valueOf(classString.toUpperCase());
+ storageType = StorageType.valueOf(
+ classString.toUpperCase(Locale.ENGLISH));
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
index dab10d3..3dcce5f 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
@@ -75,6 +75,7 @@ import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.EnumMap;
+import java.util.Locale;
import java.util.List;
import java.util.zip.CheckedInputStream;
import java.util.zip.Checksum;
@@ -4348,7 +4349,7 @@ public abstract class FSEditLogOp {
public RollingUpgradeOp(FSEditLogOpCodes code, String name) {
super(code);
- this.name = name.toUpperCase();
+ this.name = name.toUpperCase(Locale.ENGLISH);
}
static RollingUpgradeOp getStartInstance(OpInstanceCache cache) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaByStorageTypeEntry.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaByStorageTypeEntry.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaByStorageTypeEntry.java
index d115acc..fe185f6 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaByStorageTypeEntry.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaByStorageTypeEntry.java
@@ -19,6 +19,7 @@ package org.apache.hadoop.hdfs.server.namenode;
import com.google.common.base.Objects;
import org.apache.hadoop.hdfs.StorageType;
+import java.util.Locale;
public class QuotaByStorageTypeEntry {
private StorageType type;
@@ -53,7 +54,7 @@ import org.apache.hadoop.hdfs.StorageType;
public String toString() {
StringBuilder sb = new StringBuilder();
assert (type != null);
- sb.append(type.toString().toLowerCase());
+ sb.append(type.toString().toLowerCase(Locale.ENGLISH));
sb.append(':');
sb.append(quota);
return sb.toString();
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
index 83e6426..1157bb8 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
@@ -587,7 +587,7 @@ public class SecondaryNameNode implements Runnable,
return 0;
}
- String cmd = opts.getCommand().toString().toLowerCase();
+ String cmd = opts.getCommand().toString().toLowerCase(Locale.ENGLISH);
int exitCode = 0;
try {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java
index 92a16cd..c380901 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java
@@ -24,6 +24,7 @@ import java.security.PrivilegedExceptionAction;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import org.apache.hadoop.HadoopIllegalArgumentException;
@@ -79,19 +80,19 @@ public class GetConf extends Configured implements Tool {
private static final Map<String, CommandHandler> map;
static {
map = new HashMap<String, CommandHandler>();
- map.put(NAMENODE.getName().toLowerCase(),
+ map.put(NAMENODE.getName().toLowerCase(Locale.ENGLISH),
new NameNodesCommandHandler());
- map.put(SECONDARY.getName().toLowerCase(),
+ map.put(SECONDARY.getName().toLowerCase(Locale.ENGLISH),
new SecondaryNameNodesCommandHandler());
- map.put(BACKUP.getName().toLowerCase(),
+ map.put(BACKUP.getName().toLowerCase(Locale.ENGLISH),
new BackupNodesCommandHandler());
- map.put(INCLUDE_FILE.getName().toLowerCase(),
+ map.put(INCLUDE_FILE.getName().toLowerCase(Locale.ENGLISH),
new CommandHandler(DFSConfigKeys.DFS_HOSTS));
- map.put(EXCLUDE_FILE.getName().toLowerCase(),
+ map.put(EXCLUDE_FILE.getName().toLowerCase(Locale.ENGLISH),
new CommandHandler(DFSConfigKeys.DFS_HOSTS_EXCLUDE));
- map.put(NNRPCADDRESSES.getName().toLowerCase(),
+ map.put(NNRPCADDRESSES.getName().toLowerCase(Locale.ENGLISH),
new NNRpcAddressesCommandHandler());
- map.put(CONFKEY.getName().toLowerCase(),
+ map.put(CONFKEY.getName().toLowerCase(Locale.ENGLISH),
new PrintConfKeyCommandHandler());
}
@@ -116,7 +117,7 @@ public class GetConf extends Configured implements Tool {
}
public static CommandHandler getHandler(String cmd) {
- return map.get(cmd.toLowerCase());
+ return map.get(cmd.toLowerCase(Locale.ENGLISH));
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsVisitorFactory.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsVisitorFactory.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsVisitorFactory.java
index c4b8424..aa542d3 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsVisitorFactory.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsVisitorFactory.java
@@ -20,6 +20,7 @@ package org.apache.hadoop.hdfs.tools.offlineEditsViewer;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
+import java.util.Locale;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
@@ -43,7 +44,7 @@ public class OfflineEditsVisitorFactory {
*/
static public OfflineEditsVisitor getEditsVisitor(String filename,
String processor, boolean printToScreen) throws IOException {
- if(processor.toLowerCase().equals("binary")) {
+ if(processor.toLowerCase(Locale.ENGLISH).equals("binary")) {
return new BinaryEditsVisitor(filename);
}
OfflineEditsVisitor vis;
@@ -59,9 +60,9 @@ public class OfflineEditsVisitorFactory {
outs[1] = System.out;
out = new TeeOutputStream(outs);
}
- if(processor.toLowerCase().equals("xml")) {
+ if(processor.toLowerCase(Locale.ENGLISH).equals("xml")) {
vis = new XmlEditsVisitor(out);
- } else if(processor.toLowerCase().equals("stats")) {
+ } else if(processor.toLowerCase(Locale.ENGLISH).equals("stats")) {
vis = new StatisticsEditsVisitor(out);
} else {
throw new IOException("Unknown proccesor " + processor +
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageHandler.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageHandler.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageHandler.java
index 43fcd69..aa36516 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageHandler.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageHandler.java
@@ -37,6 +37,7 @@ import org.apache.hadoop.hdfs.web.JsonUtil;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import static io.netty.handler.codec.http.HttpHeaders.Names.CONNECTION;
@@ -141,7 +142,7 @@ class FSImageHandler extends SimpleChannelInboundHandler<HttpRequest> {
private static String getOp(QueryStringDecoder decoder) {
Map<String, List<String>> parameters = decoder.parameters();
return parameters.containsKey("op")
- ? parameters.get("op").get(0).toUpperCase() : null;
+ ? parameters.get("op").get(0).toUpperCase(Locale.ENGLISH) : null;
}
private static String getPath(QueryStringDecoder decoder)
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/AuthFilter.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/AuthFilter.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/AuthFilter.java
index b6ff4b6..1a0aaeb 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/AuthFilter.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/AuthFilter.java
@@ -23,6 +23,7 @@ import java.util.Enumeration;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.Properties;
@@ -96,7 +97,7 @@ public class AuthFilter extends AuthenticationFilter {
final Map<String, List<String>> m = new HashMap<String, List<String>>();
for(Map.Entry<String, String[]> entry : original.entrySet()) {
- final String key = entry.getKey().toLowerCase();
+ final String key = entry.getKey().toLowerCase(Locale.ENGLISH);
List<String> strings = m.get(key);
if (strings == null) {
strings = new ArrayList<String>();
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/ParamFilter.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/ParamFilter.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/ParamFilter.java
index 2ae3445..6a18377 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/ParamFilter.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/ParamFilter.java
@@ -19,6 +19,7 @@ package org.apache.hadoop.hdfs.web;
import java.net.URI;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import javax.ws.rs.core.MultivaluedMap;
@@ -75,7 +76,7 @@ public class ParamFilter implements ResourceFilter {
final MultivaluedMap<String, String> parameters) {
UriBuilder b = UriBuilder.fromUri(uri).replaceQuery("");
for(Map.Entry<String, List<String>> e : parameters.entrySet()) {
- final String key = e.getKey().toLowerCase();
+ final String key = e.getKey().toLowerCase(Locale.ENGLISH);
for(String v : e.getValue()) {
b = b.queryParam(key, v);
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
index 938f7c7..b1026c0 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
@@ -32,6 +32,7 @@ import java.security.PrivilegedExceptionAction;
import java.util.ArrayList;
import java.util.EnumSet;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.StringTokenizer;
@@ -1242,7 +1243,7 @@ public class WebHdfsFileSystem extends FileSystem
if (query == null) {
return url;
}
- final String lower = query.toLowerCase();
+ final String lower = query.toLowerCase(Locale.ENGLISH);
if (!lower.startsWith(OFFSET_PARAM_PREFIX)
&& !lower.contains("&" + OFFSET_PARAM_PREFIX)) {
return url;
@@ -1253,7 +1254,7 @@ public class WebHdfsFileSystem extends FileSystem
for(final StringTokenizer st = new StringTokenizer(query, "&");
st.hasMoreTokens();) {
final String token = st.nextToken();
- if (!token.toLowerCase().startsWith(OFFSET_PARAM_PREFIX)) {
+ if (!token.toLowerCase(Locale.ENGLISH).startsWith(OFFSET_PARAM_PREFIX)) {
if (b == null) {
b = new StringBuilder("?").append(token);
} else {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/EnumParam.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/EnumParam.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/EnumParam.java
index 1703e3b..6b4ec2c 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/EnumParam.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/EnumParam.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.hdfs.web.resources;
import java.util.Arrays;
+import java.util.Locale;
abstract class EnumParam<E extends Enum<E>> extends Param<E, EnumParam.Domain<E>> {
EnumParam(final Domain<E> domain, final E value) {
@@ -40,7 +41,7 @@ abstract class EnumParam<E extends Enum<E>> extends Param<E, EnumParam.Domain<E>
@Override
final E parse(final String str) {
- return Enum.valueOf(enumClass, str.toUpperCase());
+ return Enum.valueOf(enumClass, str.toUpperCase(Locale.ENGLISH));
}
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/EnumSetParam.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/EnumSetParam.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/EnumSetParam.java
index 5adb5a6..23ba16c 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/EnumSetParam.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/EnumSetParam.java
@@ -20,6 +20,7 @@ package org.apache.hadoop.hdfs.web.resources;
import java.util.Arrays;
import java.util.EnumSet;
import java.util.Iterator;
+import java.util.Locale;
abstract class EnumSetParam<E extends Enum<E>> extends Param<EnumSet<E>, EnumSetParam.Domain<E>> {
/** Convert an EnumSet to a string of comma separated values. */
@@ -82,7 +83,8 @@ abstract class EnumSetParam<E extends Enum<E>> extends Param<EnumSet<E>, EnumSet
i = j > 0 ? j + 1 : 0;
j = str.indexOf(',', i);
final String sub = j >= 0? str.substring(i, j): str.substring(i);
- set.add(Enum.valueOf(enumClass, sub.trim().toUpperCase()));
+ set.add(
+ Enum.valueOf(enumClass, sub.trim().toUpperCase(Locale.ENGLISH)));
}
}
return set;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotManager.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotManager.java
index ac6acf9..db8f290 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotManager.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotManager.java
@@ -25,6 +25,7 @@ import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.spy;
import java.util.ArrayList;
+import java.util.Locale;
import org.apache.hadoop.hdfs.protocol.SnapshotException;
import org.apache.hadoop.hdfs.server.namenode.FSDirectory;
@@ -70,7 +71,7 @@ public class TestSnapshotManager {
Assert.fail("Expected SnapshotException not thrown");
} catch (SnapshotException se) {
Assert.assertTrue(
- se.getMessage().toLowerCase().contains("rollover"));
+ se.getMessage().toLowerCase(Locale.ENGLISH).contains("rollover"));
}
// Delete a snapshot to free up a slot.
@@ -86,7 +87,7 @@ public class TestSnapshotManager {
Assert.fail("Expected SnapshotException not thrown");
} catch (SnapshotException se) {
Assert.assertTrue(
- se.getMessage().toLowerCase().contains("rollover"));
+ se.getMessage().toLowerCase(Locale.ENGLISH).contains("rollover"));
}
}
}
[04/52] [abbrv] hadoop git commit: HADOOP-11593. Convert site
documentation from apt to markdown (stragglers) (Masatake Iwasaki via aw)
Posted by zh...@apache.org.
HADOOP-11593. Convert site documentation from apt to markdown (stragglers) (Masatake Iwasaki via aw)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/b6fc1f3e
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/b6fc1f3e
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/b6fc1f3e
Branch: refs/heads/HDFS-7285
Commit: b6fc1f3e4355be913b7d4f6ccd48c0c26b66d039
Parents: 7c78204
Author: Allen Wittenauer <aw...@apache.org>
Authored: Tue Feb 17 21:30:24 2015 -1000
Committer: Allen Wittenauer <aw...@apache.org>
Committed: Tue Feb 17 21:30:24 2015 -1000
----------------------------------------------------------------------
.../hadoop-auth/src/site/apt/BuildingIt.apt.vm | 70 --
.../src/site/apt/Configuration.apt.vm | 377 -------
.../hadoop-auth/src/site/apt/Examples.apt.vm | 133 ---
.../hadoop-auth/src/site/apt/index.apt.vm | 59 -
.../hadoop-auth/src/site/markdown/BuildingIt.md | 56 +
.../src/site/markdown/Configuration.md | 341 ++++++
.../hadoop-auth/src/site/markdown/Examples.md | 109 ++
.../hadoop-auth/src/site/markdown/index.md | 43 +
hadoop-common-project/hadoop-common/CHANGES.txt | 3 +
.../hadoop-kms/src/site/apt/index.apt.vm | 1020 ------------------
.../hadoop-kms/src/site/markdown/index.md.vm | 864 +++++++++++++++
hadoop-project/src/site/apt/index.apt.vm | 73 --
hadoop-project/src/site/markdown/index.md.vm | 72 ++
.../hadoop-openstack/src/site/apt/index.apt.vm | 686 ------------
.../hadoop-openstack/src/site/markdown/index.md | 544 ++++++++++
.../src/site/resources/css/site.css | 30 +
.../src/site/apt/SchedulerLoadSimulator.apt.vm | 439 --------
.../src/site/markdown/SchedulerLoadSimulator.md | 357 ++++++
.../src/site/apt/HadoopStreaming.apt.vm | 792 --------------
.../src/site/markdown/HadoopStreaming.md.vm | 559 ++++++++++
20 files changed, 2978 insertions(+), 3649 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-common-project/hadoop-auth/src/site/apt/BuildingIt.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-auth/src/site/apt/BuildingIt.apt.vm b/hadoop-common-project/hadoop-auth/src/site/apt/BuildingIt.apt.vm
deleted file mode 100644
index 2ca2f0a..0000000
--- a/hadoop-common-project/hadoop-auth/src/site/apt/BuildingIt.apt.vm
+++ /dev/null
@@ -1,70 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- Hadoop Auth, Java HTTP SPNEGO ${project.version} - Building It
- ---
- ---
- ${maven.build.timestamp}
-
-Hadoop Auth, Java HTTP SPNEGO ${project.version} - Building It
-
-* Requirements
-
- * Java 6+
-
- * Maven 3+
-
- * Kerberos KDC (for running Kerberos test cases)
-
-* Building
-
- Use Maven goals: clean, test, compile, package, install
-
- Available profiles: docs, testKerberos
-
-* Testing
-
- By default Kerberos testcases are not run.
-
- The requirements to run Kerberos testcases are a running KDC, a keytab
- file with a client principal and a kerberos principal.
-
- To run Kerberos tescases use the <<<testKerberos>>> Maven profile:
-
-+---+
-$ mvn test -PtestKerberos
-+---+
-
- The following Maven <<<-D>>> options can be used to change the default
- values:
-
- * <<<hadoop-auth.test.kerberos.realm>>>: default value <<LOCALHOST>>
-
- * <<<hadoop-auth.test.kerberos.client.principal>>>: default value <<client>>
-
- * <<<hadoop-auth.test.kerberos.server.principal>>>: default value
- <<HTTP/localhost>> (it must start 'HTTP/')
-
- * <<<hadoop-auth.test.kerberos.keytab.file>>>: default value
- <<${HOME}/${USER}.keytab>>
-
-** Generating Documentation
-
- To create the documentation use the <<<docs>>> Maven profile:
-
-+---+
-$ mvn package -Pdocs
-+---+
-
- The generated documentation is available at
- <<<hadoop-auth/target/site/>>>.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-common-project/hadoop-auth/src/site/apt/Configuration.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-auth/src/site/apt/Configuration.apt.vm b/hadoop-common-project/hadoop-auth/src/site/apt/Configuration.apt.vm
deleted file mode 100644
index 88248e5..0000000
--- a/hadoop-common-project/hadoop-auth/src/site/apt/Configuration.apt.vm
+++ /dev/null
@@ -1,377 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- Hadoop Auth, Java HTTP SPNEGO ${project.version} - Server Side
- Configuration
- ---
- ---
- ${maven.build.timestamp}
-
-Hadoop Auth, Java HTTP SPNEGO ${project.version} - Server Side
-Configuration
-
-* Server Side Configuration Setup
-
- The AuthenticationFilter filter is Hadoop Auth's server side component.
-
- This filter must be configured in front of all the web application resources
- that required authenticated requests. For example:
-
- The Hadoop Auth and dependent JAR files must be in the web application
- classpath (commonly the <<<WEB-INF/lib>>> directory).
-
- Hadoop Auth uses SLF4J-API for logging. Auth Maven POM dependencies define
- the SLF4J API dependency but it does not define the dependency on a concrete
- logging implementation, this must be addded explicitly to the web
- application. For example, if the web applicationan uses Log4j, the
- SLF4J-LOG4J12 and LOG4J jar files must be part part of the web application
- classpath as well as the Log4j configuration file.
-
-** Common Configuration parameters
-
- * <<<config.prefix>>>: If specified, all other configuration parameter names
- must start with the prefix. The default value is no prefix.
-
- * <<<[PREFIX.]type>>>: the authentication type keyword (<<<simple>>> or
- <<<kerberos>>>) or a Authentication handler implementation.
-
- * <<<[PREFIX.]signature.secret>>>: When <<<signer.secret.provider>>> is set to
- <<<string>>> or not specified, this is the value for the secret used to sign
- the HTTP cookie.
-
- * <<<[PREFIX.]token.validity>>>: The validity -in seconds- of the generated
- authentication token. The default value is <<<3600>>> seconds. This is also
- used for the rollover interval when <<<signer.secret.provider>>> is set to
- <<<random>>> or <<<zookeeper>>>.
-
- * <<<[PREFIX.]cookie.domain>>>: domain to use for the HTTP cookie that stores
- the authentication token.
-
- * <<<[PREFIX.]cookie.path>>>: path to use for the HTTP cookie that stores the
- authentication token.
-
- * <<<signer.secret.provider>>>: indicates the name of the SignerSecretProvider
- class to use. Possible values are: <<<string>>>, <<<random>>>,
- <<<zookeeper>>>, or a classname. If not specified, the <<<string>>>
- implementation will be used; and failing that, the <<<random>>>
- implementation will be used.
-
-** Kerberos Configuration
-
- <<IMPORTANT>>: A KDC must be configured and running.
-
- To use Kerberos SPNEGO as the authentication mechanism, the authentication
- filter must be configured with the following init parameters:
-
- * <<<[PREFIX.]type>>>: the keyword <<<kerberos>>>.
-
- * <<<[PREFIX.]kerberos.principal>>>: The web-application Kerberos principal
- name. The Kerberos principal name must start with <<<HTTP/...>>>. For
- example: <<<...@LOCALHOST>>>. There is no default value.
-
- * <<<[PREFIX.]kerberos.keytab>>>: The path to the keytab file containing
- the credentials for the kerberos principal. For example:
- <<</Users/tucu/tucu.keytab>>>. There is no default value.
-
- <<Example>>:
-
-+---+
-<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
- ...
-
- <filter>
- <filter-name>kerberosFilter</filter-name>
- <filter-class>org.apache.hadoop.security.auth.server.AuthenticationFilter</filter-class>
- <init-param>
- <param-name>type</param-name>
- <param-value>kerberos</param-value>
- </init-param>
- <init-param>
- <param-name>token.validity</param-name>
- <param-value>30</param-value>
- </init-param>
- <init-param>
- <param-name>cookie.domain</param-name>
- <param-value>.foo.com</param-value>
- </init-param>
- <init-param>
- <param-name>cookie.path</param-name>
- <param-value>/</param-value>
- </init-param>
- <init-param>
- <param-name>kerberos.principal</param-name>
- <param-value>HTTP/localhost@LOCALHOST</param-value>
- </init-param>
- <init-param>
- <param-name>kerberos.keytab</param-name>
- <param-value>/tmp/auth.keytab</param-value>
- </init-param>
- </filter>
-
- <filter-mapping>
- <filter-name>kerberosFilter</filter-name>
- <url-pattern>/kerberos/*</url-pattern>
- </filter-mapping>
-
- ...
-</web-app>
-+---+
-
-** Pseudo/Simple Configuration
-
- To use Pseudo/Simple as the authentication mechanism (trusting the value of
- the query string parameter 'user.name'), the authentication filter must be
- configured with the following init parameters:
-
- * <<<[PREFIX.]type>>>: the keyword <<<simple>>>.
-
- * <<<[PREFIX.]simple.anonymous.allowed>>>: is a boolean parameter that
- indicates if anonymous requests are allowed or not. The default value is
- <<<false>>>.
-
- <<Example>>:
-
-+---+
-<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
- ...
-
- <filter>
- <filter-name>simpleFilter</filter-name>
- <filter-class>org.apache.hadoop.security.auth.server.AuthenticationFilter</filter-class>
- <init-param>
- <param-name>type</param-name>
- <param-value>simple</param-value>
- </init-param>
- <init-param>
- <param-name>token.validity</param-name>
- <param-value>30</param-value>
- </init-param>
- <init-param>
- <param-name>cookie.domain</param-name>
- <param-value>.foo.com</param-value>
- </init-param>
- <init-param>
- <param-name>cookie.path</param-name>
- <param-value>/</param-value>
- </init-param>
- <init-param>
- <param-name>simple.anonymous.allowed</param-name>
- <param-value>false</param-value>
- </init-param>
- </filter>
-
- <filter-mapping>
- <filter-name>simpleFilter</filter-name>
- <url-pattern>/simple/*</url-pattern>
- </filter-mapping>
-
- ...
-</web-app>
-+---+
-
-** AltKerberos Configuration
-
- <<IMPORTANT>>: A KDC must be configured and running.
-
- The AltKerberos authentication mechanism is a partially implemented derivative
- of the Kerberos SPNEGO authentication mechanism which allows a "mixed" form of
- authentication where Kerberos SPNEGO is used by non-browsers while an
- alternate form of authentication (to be implemented by the user) is used for
- browsers. To use AltKerberos as the authentication mechanism (besides
- providing an implementation), the authentication filter must be configured
- with the following init parameters, in addition to the previously mentioned
- Kerberos SPNEGO ones:
-
- * <<<[PREFIX.]type>>>: the full class name of the implementation of
- AltKerberosAuthenticationHandler to use.
-
- * <<<[PREFIX.]alt-kerberos.non-browser.user-agents>>>: a comma-separated
- list of which user-agents should be considered non-browsers.
-
- <<Example>>:
-
-+---+
-<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
- ...
-
- <filter>
- <filter-name>kerberosFilter</filter-name>
- <filter-class>org.apache.hadoop.security.auth.server.AuthenticationFilter</filter-class>
- <init-param>
- <param-name>type</param-name>
- <param-value>org.my.subclass.of.AltKerberosAuthenticationHandler</param-value>
- </init-param>
- <init-param>
- <param-name>alt-kerberos.non-browser.user-agents</param-name>
- <param-value>java,curl,wget,perl</param-value>
- </init-param>
- <init-param>
- <param-name>token.validity</param-name>
- <param-value>30</param-value>
- </init-param>
- <init-param>
- <param-name>cookie.domain</param-name>
- <param-value>.foo.com</param-value>
- </init-param>
- <init-param>
- <param-name>cookie.path</param-name>
- <param-value>/</param-value>
- </init-param>
- <init-param>
- <param-name>kerberos.principal</param-name>
- <param-value>HTTP/localhost@LOCALHOST</param-value>
- </init-param>
- <init-param>
- <param-name>kerberos.keytab</param-name>
- <param-value>/tmp/auth.keytab</param-value>
- </init-param>
- </filter>
-
- <filter-mapping>
- <filter-name>kerberosFilter</filter-name>
- <url-pattern>/kerberos/*</url-pattern>
- </filter-mapping>
-
- ...
-</web-app>
-+---+
-
-** SignerSecretProvider Configuration
-
- The SignerSecretProvider is used to provide more advanced behaviors for the
- secret used for signing the HTTP Cookies.
-
- These are the relevant configuration properties:
-
- * <<<signer.secret.provider>>>: indicates the name of the
- SignerSecretProvider class to use. Possible values are: "string",
- "random", "zookeeper", or a classname. If not specified, the "string"
- implementation will be used; and failing that, the "random" implementation
- will be used.
-
- * <<<[PREFIX.]signature.secret>>>: When <<<signer.secret.provider>>> is set
- to <<<string>>> or not specified, this is the value for the secret used to
- sign the HTTP cookie.
-
- * <<<[PREFIX.]token.validity>>>: The validity -in seconds- of the generated
- authentication token. The default value is <<<3600>>> seconds. This is
- also used for the rollover interval when <<<signer.secret.provider>>> is
- set to <<<random>>> or <<<zookeeper>>>.
-
- The following configuration properties are specific to the <<<zookeeper>>>
- implementation:
-
- * <<<signer.secret.provider.zookeeper.connection.string>>>: Indicates the
- ZooKeeper connection string to connect with.
-
- * <<<signer.secret.provider.zookeeper.path>>>: Indicates the ZooKeeper path
- to use for storing and retrieving the secrets. All servers
- that need to coordinate their secret should point to the same path
-
- * <<<signer.secret.provider.zookeeper.auth.type>>>: Indicates the auth type
- to use. Supported values are <<<none>>> and <<<sasl>>>. The default
- value is <<<none>>>.
-
- * <<<signer.secret.provider.zookeeper.kerberos.keytab>>>: Set this to the
- path with the Kerberos keytab file. This is only required if using
- Kerberos.
-
- * <<<signer.secret.provider.zookeeper.kerberos.principal>>>: Set this to the
- Kerberos principal to use. This only required if using Kerberos.
-
- <<Example>>:
-
-+---+
-<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
- ...
-
- <filter>
- <!-- AuthenticationHandler configs not shown -->
- <init-param>
- <param-name>signer.secret.provider</param-name>
- <param-value>string</param-value>
- </init-param>
- <init-param>
- <param-name>signature.secret</param-name>
- <param-value>my_secret</param-value>
- </init-param>
- </filter>
-
- ...
-</web-app>
-+---+
-
- <<Example>>:
-
-+---+
-<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
- ...
-
- <filter>
- <!-- AuthenticationHandler configs not shown -->
- <init-param>
- <param-name>signer.secret.provider</param-name>
- <param-value>random</param-value>
- </init-param>
- <init-param>
- <param-name>token.validity</param-name>
- <param-value>30</param-value>
- </init-param>
- </filter>
-
- ...
-</web-app>
-+---+
-
- <<Example>>:
-
-+---+
-<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
- ...
-
- <filter>
- <!-- AuthenticationHandler configs not shown -->
- <init-param>
- <param-name>signer.secret.provider</param-name>
- <param-value>zookeeper</param-value>
- </init-param>
- <init-param>
- <param-name>token.validity</param-name>
- <param-value>30</param-value>
- </init-param>
- <init-param>
- <param-name>signer.secret.provider.zookeeper.connection.string</param-name>
- <param-value>zoo1:2181,zoo2:2181,zoo3:2181</param-value>
- </init-param>
- <init-param>
- <param-name>signer.secret.provider.zookeeper.path</param-name>
- <param-value>/myapp/secrets</param-value>
- </init-param>
- <init-param>
- <param-name>signer.secret.provider.zookeeper.use.kerberos.acls</param-name>
- <param-value>true</param-value>
- </init-param>
- <init-param>
- <param-name>signer.secret.provider.zookeeper.kerberos.keytab</param-name>
- <param-value>/tmp/auth.keytab</param-value>
- </init-param>
- <init-param>
- <param-name>signer.secret.provider.zookeeper.kerberos.principal</param-name>
- <param-value>HTTP/localhost@LOCALHOST</param-value>
- </init-param>
- </filter>
-
- ...
-</web-app>
-+---+
-
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-common-project/hadoop-auth/src/site/apt/Examples.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-auth/src/site/apt/Examples.apt.vm b/hadoop-common-project/hadoop-auth/src/site/apt/Examples.apt.vm
deleted file mode 100644
index 1b1afd5..0000000
--- a/hadoop-common-project/hadoop-auth/src/site/apt/Examples.apt.vm
+++ /dev/null
@@ -1,133 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- Hadoop Auth, Java HTTP SPNEGO ${project.version} - Examples
- ---
- ---
- ${maven.build.timestamp}
-
-Hadoop Auth, Java HTTP SPNEGO ${project.version} - Examples
-
-* Accessing a Hadoop Auth protected URL Using a browser
-
- <<IMPORTANT:>> The browser must support HTTP Kerberos SPNEGO. For example,
- Firefox or Internet Explorer.
-
- For Firefox access the low level configuration page by loading the
- <<<about:config>>> page. Then go to the
- <<<network.negotiate-auth.trusted-uris>>> preference and add the hostname or
- the domain of the web server that is HTTP Kerberos SPNEGO protected (if using
- multiple domains and hostname use comma to separate them).
-
-* Accessing a Hadoop Auth protected URL Using <<<curl>>>
-
- <<IMPORTANT:>> The <<<curl>>> version must support GSS, run <<<curl -V>>>.
-
-+---+
-$ curl -V
-curl 7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3
-Protocols: tftp ftp telnet dict ldap http file https ftps
-Features: GSS-Negotiate IPv6 Largefile NTLM SSL libz
-+---+
-
- Login to the KDC using <<kinit>> and then use <<<curl>>> to fetch protected
- URL:
-
-+---+
-$ kinit
-Please enter the password for tucu@LOCALHOST:
-$ curl --negotiate -u foo -b ~/cookiejar.txt -c ~/cookiejar.txt http://localhost:8080/hadoop-auth-examples/kerberos/who
-Enter host password for user 'tucu':
-
-Hello Hadoop Auth Examples!
-+---+
-
- * The <<<--negotiate>>> option enables SPNEGO in <<<curl>>>.
-
- * The <<<-u foo>>> option is required but the user ignored (the principal
- that has been kinit-ed is used).
-
- * The <<<-b>>> and <<<-c>>> are use to store and send HTTP Cookies.
-
-* Using the Java Client
-
- Use the <<<AuthenticatedURL>>> class to obtain an authenticated HTTP
- connection:
-
-+---+
-...
-URL url = new URL("http://localhost:8080/hadoop-auth/kerberos/who");
-AuthenticatedURL.Token token = new AuthenticatedURL.Token();
-...
-HttpURLConnection conn = new AuthenticatedURL(url, token).openConnection();
-...
-conn = new AuthenticatedURL(url, token).openConnection();
-...
-+---+
-
-* Building and Running the Examples
-
- Download Hadoop-Auth's source code, the examples are in the
- <<<src/main/examples>>> directory.
-
-** Server Example:
-
- Edit the <<<hadoop-auth-examples/src/main/webapp/WEB-INF/web.xml>>> and set the
- right configuration init parameters for the <<<AuthenticationFilter>>>
- definition configured for Kerberos (the right Kerberos principal and keytab
- file must be specified). Refer to the {{{./Configuration.html}Configuration
- document}} for details.
-
- Create the web application WAR file by running the <<<mvn package>>> command.
-
- Deploy the WAR file in a servlet container. For example, if using Tomcat,
- copy the WAR file to Tomcat's <<<webapps/>>> directory.
-
- Start the servlet container.
-
-** Accessing the server using <<<curl>>>
-
- Try accessing protected resources using <<<curl>>>. The protected resources
- are:
-
-+---+
-$ kinit
-Please enter the password for tucu@LOCALHOST:
-
-$ curl http://localhost:8080/hadoop-auth-examples/anonymous/who
-
-$ curl http://localhost:8080/hadoop-auth-examples/simple/who?user.name=foo
-
-$ curl --negotiate -u foo -b ~/cookiejar.txt -c ~/cookiejar.txt http://localhost:8080/hadoop-auth-examples/kerberos/who
-+---+
-
-** Accessing the server using the Java client example
-
-+---+
-$ kinit
-Please enter the password for tucu@LOCALHOST:
-
-$ cd examples
-
-$ mvn exec:java -Durl=http://localhost:8080/hadoop-auth-examples/kerberos/who
-
-....
-
-Token value: "u=tucu,p=tucu@LOCALHOST,t=kerberos,e=1295305313146,s=sVZ1mpSnC5TKhZQE3QLN5p2DWBo="
-Status code: 200 OK
-
-You are: user[tucu] principal[tucu@LOCALHOST]
-
-....
-
-+---+
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-common-project/hadoop-auth/src/site/apt/index.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-auth/src/site/apt/index.apt.vm b/hadoop-common-project/hadoop-auth/src/site/apt/index.apt.vm
deleted file mode 100644
index bf85f7f..0000000
--- a/hadoop-common-project/hadoop-auth/src/site/apt/index.apt.vm
+++ /dev/null
@@ -1,59 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- Hadoop Auth, Java HTTP SPNEGO ${project.version}
- ---
- ---
- ${maven.build.timestamp}
-
-Hadoop Auth, Java HTTP SPNEGO ${project.version}
-
- Hadoop Auth is a Java library consisting of a client and a server
- components to enable Kerberos SPNEGO authentication for HTTP.
-
- Hadoop Auth also supports additional authentication mechanisms on the client
- and the server side via 2 simple interfaces.
-
- Additionally, it provides a partially implemented derivative of the Kerberos
- SPNEGO authentication to allow a "mixed" form of authentication where Kerberos
- SPNEGO is used by non-browsers while an alternate form of authentication
- (to be implemented by the user) is used for browsers.
-
-* License
-
- Hadoop Auth is distributed under {{{http://www.apache.org/licenses/}Apache
- License 2.0}}.
-
-* How Does Auth Works?
-
- Hadoop Auth enforces authentication on protected resources, once authentiation
- has been established it sets a signed HTTP Cookie that contains an
- authentication token with the user name, user principal, authentication type
- and expiration time.
-
- Subsequent HTTP client requests presenting the signed HTTP Cookie have access
- to the protected resources until the HTTP Cookie expires.
-
- The secret used to sign the HTTP Cookie has multiple implementations that
- provide different behaviors, including a hardcoded secret string, a rolling
- randomly generated secret, and a rolling randomly generated secret
- synchronized between multiple servers using ZooKeeper.
-
-* User Documentation
-
- * {{{./Examples.html}Examples}}
-
- * {{{./Configuration.html}Configuration}}
-
- * {{{./BuildingIt.html}Building It}}
-
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-common-project/hadoop-auth/src/site/markdown/BuildingIt.md
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-auth/src/site/markdown/BuildingIt.md b/hadoop-common-project/hadoop-auth/src/site/markdown/BuildingIt.md
new file mode 100644
index 0000000..53a49d4
--- /dev/null
+++ b/hadoop-common-project/hadoop-auth/src/site/markdown/BuildingIt.md
@@ -0,0 +1,56 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+Hadoop Auth, Java HTTP SPNEGO - Building It
+===========================================
+
+Requirements
+------------
+
+* Java 6+
+* Maven 3+
+* Kerberos KDC (for running Kerberos test cases)
+
+Building
+--------
+
+Use Maven goals: clean, test, compile, package, install
+
+Available profiles: docs, testKerberos
+
+Testing
+-------
+
+By default Kerberos testcases are not run.
+
+The requirements to run Kerberos testcases are a running KDC, a keytab file with a client principal and a kerberos principal.
+
+To run Kerberos tescases use the `testKerberos` Maven profile:
+
+ $ mvn test -PtestKerberos
+
+The following Maven `-D` options can be used to change the default values:
+
+* `hadoop-auth.test.kerberos.realm`: default value **LOCALHOST**
+* `hadoop-auth.test.kerberos.client.principal`: default value **client**
+* `hadoop-auth.test.kerberos.server.principal`: default value **HTTP/localhost** (it must start 'HTTP/')
+* `hadoop-auth.test.kerberos.keytab.file`: default value **$HOME/$USER.keytab**
+
+### Generating Documentation
+
+To create the documentation use the `docs` Maven profile:
+
+ $ mvn package -Pdocs
+
+The generated documentation is available at `hadoop-auth/target/site/`.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-common-project/hadoop-auth/src/site/markdown/Configuration.md
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-auth/src/site/markdown/Configuration.md b/hadoop-common-project/hadoop-auth/src/site/markdown/Configuration.md
new file mode 100644
index 0000000..9d076bb
--- /dev/null
+++ b/hadoop-common-project/hadoop-auth/src/site/markdown/Configuration.md
@@ -0,0 +1,341 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+Hadoop Auth, Java HTTP SPNEGO - Server Side Configuration
+=========================================================
+
+Server Side Configuration Setup
+-------------------------------
+
+The AuthenticationFilter filter is Hadoop Auth's server side component.
+
+This filter must be configured in front of all the web application resources that required authenticated requests. For example:
+
+The Hadoop Auth and dependent JAR files must be in the web application classpath (commonly the `WEB-INF/lib` directory).
+
+Hadoop Auth uses SLF4J-API for logging. Auth Maven POM dependencies define the SLF4J API dependency but it does not define the dependency on a concrete logging implementation, this must be addded explicitly to the web application. For example, if the web applicationan uses Log4j, the SLF4J-LOG4J12 and LOG4J jar files must be part part of the web application classpath as well as the Log4j configuration file.
+
+### Common Configuration parameters
+
+* `config.prefix`: If specified, all other configuration parameter names
+ must start with the prefix. The default value is no prefix.
+
+* `[PREFIX.]type`: the authentication type keyword (`simple` or \
+ `kerberos`) or a Authentication handler implementation.
+
+* `[PREFIX.]signature.secret`: When `signer.secret.provider` is set to
+ `string` or not specified, this is the value for the secret used to sign
+ the HTTP cookie.
+
+* `[PREFIX.]token.validity`: The validity -in seconds- of the generated
+ authentication token. The default value is `3600` seconds. This is also
+ used for the rollover interval when `signer.secret.provider` is set to
+ `random` or `zookeeper`.
+
+* `[PREFIX.]cookie.domain`: domain to use for the HTTP cookie that stores
+ the authentication token.
+
+* `[PREFIX.]cookie.path`: path to use for the HTTP cookie that stores the
+ authentication token.
+
+* `signer.secret.provider`: indicates the name of the SignerSecretProvider
+ class to use. Possible values are: `string`, `random`,
+ `zookeeper`, or a classname. If not specified, the `string`
+ implementation will be used; and failing that, the `random`
+ implementation will be used.
+
+### Kerberos Configuration
+
+**IMPORTANT**: A KDC must be configured and running.
+
+To use Kerberos SPNEGO as the authentication mechanism, the authentication filter must be configured with the following init parameters:
+
+* `[PREFIX.]type`: the keyword `kerberos`.
+
+* `[PREFIX.]kerberos.principal`: The web-application Kerberos principal
+ name. The Kerberos principal name must start with `HTTP/...`. For
+ example: `HTTP/localhost@LOCALHOST`. There is no default value.
+
+* `[PREFIX.]kerberos.keytab`: The path to the keytab file containing
+ the credentials for the kerberos principal. For example:
+ `/Users/tucu/tucu.keytab`. There is no default value.
+
+**Example**:
+
+ <web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
+ ...
+
+ <filter>
+ <filter-name>kerberosFilter</filter-name>
+ <filter-class>org.apache.hadoop.security.auth.server.AuthenticationFilter</filter-class>
+ <init-param>
+ <param-name>type</param-name>
+ <param-value>kerberos</param-value>
+ </init-param>
+ <init-param>
+ <param-name>token.validity</param-name>
+ <param-value>30</param-value>
+ </init-param>
+ <init-param>
+ <param-name>cookie.domain</param-name>
+ <param-value>.foo.com</param-value>
+ </init-param>
+ <init-param>
+ <param-name>cookie.path</param-name>
+ <param-value>/</param-value>
+ </init-param>
+ <init-param>
+ <param-name>kerberos.principal</param-name>
+ <param-value>HTTP/localhost@LOCALHOST</param-value>
+ </init-param>
+ <init-param>
+ <param-name>kerberos.keytab</param-name>
+ <param-value>/tmp/auth.keytab</param-value>
+ </init-param>
+ </filter>
+
+ <filter-mapping>
+ <filter-name>kerberosFilter</filter-name>
+ <url-pattern>/kerberos/*</url-pattern>
+ </filter-mapping>
+
+ ...
+ </web-app>
+
+### Pseudo/Simple Configuration
+
+To use Pseudo/Simple as the authentication mechanism (trusting the value of the query string parameter 'user.name'), the authentication filter must be configured with the following init parameters:
+
+* `[PREFIX.]type`: the keyword `simple`.
+
+* `[PREFIX.]simple.anonymous.allowed`: is a boolean parameter that
+ indicates if anonymous requests are allowed or not. The default value is
+ `false`.
+
+**Example**:
+
+ <web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
+ ...
+
+ <filter>
+ <filter-name>simpleFilter</filter-name>
+ <filter-class>org.apache.hadoop.security.auth.server.AuthenticationFilter</filter-class>
+ <init-param>
+ <param-name>type</param-name>
+ <param-value>simple</param-value>
+ </init-param>
+ <init-param>
+ <param-name>token.validity</param-name>
+ <param-value>30</param-value>
+ </init-param>
+ <init-param>
+ <param-name>cookie.domain</param-name>
+ <param-value>.foo.com</param-value>
+ </init-param>
+ <init-param>
+ <param-name>cookie.path</param-name>
+ <param-value>/</param-value>
+ </init-param>
+ <init-param>
+ <param-name>simple.anonymous.allowed</param-name>
+ <param-value>false</param-value>
+ </init-param>
+ </filter>
+
+ <filter-mapping>
+ <filter-name>simpleFilter</filter-name>
+ <url-pattern>/simple/*</url-pattern>
+ </filter-mapping>
+
+ ...
+ </web-app>
+
+### AltKerberos Configuration
+
+**IMPORTANT**: A KDC must be configured and running.
+
+The AltKerberos authentication mechanism is a partially implemented derivative of the Kerberos SPNEGO authentication mechanism which allows a "mixed" form of authentication where Kerberos SPNEGO is used by non-browsers while an alternate form of authentication (to be implemented by the user) is used for browsers. To use AltKerberos as the authentication mechanism (besides providing an implementation), the authentication filter must be configured with the following init parameters, in addition to the previously mentioned Kerberos SPNEGO ones:
+
+* `[PREFIX.]type`: the full class name of the implementation of
+ AltKerberosAuthenticationHandler to use.
+
+* `[PREFIX.]alt-kerberos.non-browser.user-agents`: a comma-separated
+ list of which user-agents should be considered non-browsers.
+
+**Example**:
+
+ <web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
+ ...
+
+ <filter>
+ <filter-name>kerberosFilter</filter-name>
+ <filter-class>org.apache.hadoop.security.auth.server.AuthenticationFilter</filter-class>
+ <init-param>
+ <param-name>type</param-name>
+ <param-value>org.my.subclass.of.AltKerberosAuthenticationHandler</param-value>
+ </init-param>
+ <init-param>
+ <param-name>alt-kerberos.non-browser.user-agents</param-name>
+ <param-value>java,curl,wget,perl</param-value>
+ </init-param>
+ <init-param>
+ <param-name>token.validity</param-name>
+ <param-value>30</param-value>
+ </init-param>
+ <init-param>
+ <param-name>cookie.domain</param-name>
+ <param-value>.foo.com</param-value>
+ </init-param>
+ <init-param>
+ <param-name>cookie.path</param-name>
+ <param-value>/</param-value>
+ </init-param>
+ <init-param>
+ <param-name>kerberos.principal</param-name>
+ <param-value>HTTP/localhost@LOCALHOST</param-value>
+ </init-param>
+ <init-param>
+ <param-name>kerberos.keytab</param-name>
+ <param-value>/tmp/auth.keytab</param-value>
+ </init-param>
+ </filter>
+
+ <filter-mapping>
+ <filter-name>kerberosFilter</filter-name>
+ <url-pattern>/kerberos/*</url-pattern>
+ </filter-mapping>
+
+ ...
+ </web-app>
+
+### SignerSecretProvider Configuration
+
+The SignerSecretProvider is used to provide more advanced behaviors for the secret used for signing the HTTP Cookies.
+
+These are the relevant configuration properties:
+
+* `signer.secret.provider`: indicates the name of the
+ SignerSecretProvider class to use. Possible values are: "string",
+ "random", "zookeeper", or a classname. If not specified, the "string"
+ implementation will be used; and failing that, the "random" implementation
+ will be used.
+
+* `[PREFIX.]signature.secret`: When `signer.secret.provider` is set
+ to `string` or not specified, this is the value for the secret used to
+ sign the HTTP cookie.
+
+* `[PREFIX.]token.validity`: The validity -in seconds- of the generated
+ authentication token. The default value is `3600` seconds. This is
+ also used for the rollover interval when `signer.secret.provider` is
+ set to `random` or `zookeeper`.
+
+The following configuration properties are specific to the `zookeeper` implementation:
+
+* `signer.secret.provider.zookeeper.connection.string`: Indicates the
+ ZooKeeper connection string to connect with.
+
+* `signer.secret.provider.zookeeper.path`: Indicates the ZooKeeper path
+ to use for storing and retrieving the secrets. All servers
+ that need to coordinate their secret should point to the same path
+
+* `signer.secret.provider.zookeeper.auth.type`: Indicates the auth type
+ to use. Supported values are `none` and `sasl`. The default
+ value is `none`.
+
+* `signer.secret.provider.zookeeper.kerberos.keytab`: Set this to the
+ path with the Kerberos keytab file. This is only required if using
+ Kerberos.
+
+* `signer.secret.provider.zookeeper.kerberos.principal`: Set this to the
+ Kerberos principal to use. This only required if using Kerberos.
+
+**Example**:
+
+ <web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
+ ...
+
+ <filter>
+ <!-- AuthenticationHandler configs not shown -->
+ <init-param>
+ <param-name>signer.secret.provider</param-name>
+ <param-value>string</param-value>
+ </init-param>
+ <init-param>
+ <param-name>signature.secret</param-name>
+ <param-value>my_secret</param-value>
+ </init-param>
+ </filter>
+
+ ...
+ </web-app>
+
+**Example**:
+
+ <web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
+ ...
+
+ <filter>
+ <!-- AuthenticationHandler configs not shown -->
+ <init-param>
+ <param-name>signer.secret.provider</param-name>
+ <param-value>random</param-value>
+ </init-param>
+ <init-param>
+ <param-name>token.validity</param-name>
+ <param-value>30</param-value>
+ </init-param>
+ </filter>
+
+ ...
+ </web-app>
+
+**Example**:
+
+ <web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
+ ...
+
+ <filter>
+ <!-- AuthenticationHandler configs not shown -->
+ <init-param>
+ <param-name>signer.secret.provider</param-name>
+ <param-value>zookeeper</param-value>
+ </init-param>
+ <init-param>
+ <param-name>token.validity</param-name>
+ <param-value>30</param-value>
+ </init-param>
+ <init-param>
+ <param-name>signer.secret.provider.zookeeper.connection.string</param-name>
+ <param-value>zoo1:2181,zoo2:2181,zoo3:2181</param-value>
+ </init-param>
+ <init-param>
+ <param-name>signer.secret.provider.zookeeper.path</param-name>
+ <param-value>/myapp/secrets</param-value>
+ </init-param>
+ <init-param>
+ <param-name>signer.secret.provider.zookeeper.use.kerberos.acls</param-name>
+ <param-value>true</param-value>
+ </init-param>
+ <init-param>
+ <param-name>signer.secret.provider.zookeeper.kerberos.keytab</param-name>
+ <param-value>/tmp/auth.keytab</param-value>
+ </init-param>
+ <init-param>
+ <param-name>signer.secret.provider.zookeeper.kerberos.principal</param-name>
+ <param-value>HTTP/localhost@LOCALHOST</param-value>
+ </init-param>
+ </filter>
+
+ ...
+ </web-app>
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-common-project/hadoop-auth/src/site/markdown/Examples.md
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-auth/src/site/markdown/Examples.md b/hadoop-common-project/hadoop-auth/src/site/markdown/Examples.md
new file mode 100644
index 0000000..7efb642
--- /dev/null
+++ b/hadoop-common-project/hadoop-auth/src/site/markdown/Examples.md
@@ -0,0 +1,109 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+Hadoop Auth, Java HTTP SPNEGO - Examples
+========================================
+
+Accessing a Hadoop Auth protected URL Using a browser
+-----------------------------------------------------
+
+**IMPORTANT:** The browser must support HTTP Kerberos SPNEGO. For example, Firefox or Internet Explorer.
+
+For Firefox access the low level configuration page by loading the `about:config` page. Then go to the `network.negotiate-auth.trusted-uris` preference and add the hostname or the domain of the web server that is HTTP Kerberos SPNEGO protected (if using multiple domains and hostname use comma to separate them).
+
+Accessing a Hadoop Auth protected URL Using `curl`
+--------------------------------------------------
+
+**IMPORTANT:** The `curl` version must support GSS, run `curl -V`.
+
+ $ curl -V
+ curl 7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3
+ Protocols: tftp ftp telnet dict ldap http file https ftps
+ Features: GSS-Negotiate IPv6 Largefile NTLM SSL libz
+
+Login to the KDC using **kinit** and then use `curl` to fetch protected URL:
+
+ $ kinit
+ Please enter the password for tucu@LOCALHOST:
+ $ curl --negotiate -u foo -b ~/cookiejar.txt -c ~/cookiejar.txt http://localhost:8080/hadoop-auth-examples/kerberos/who
+ Enter host password for user 'tucu':
+
+ Hello Hadoop Auth Examples!
+
+* The `--negotiate` option enables SPNEGO in `curl`.
+
+* The `-u foo` option is required but the user ignored (the principal
+ that has been kinit-ed is used).
+
+* The `-b` and `-c` are use to store and send HTTP Cookies.
+
+Using the Java Client
+---------------------
+
+Use the `AuthenticatedURL` class to obtain an authenticated HTTP connection:
+
+ ...
+ URL url = new URL("http://localhost:8080/hadoop-auth/kerberos/who");
+ AuthenticatedURL.Token token = new AuthenticatedURL.Token();
+ ...
+ HttpURLConnection conn = new AuthenticatedURL(url, token).openConnection();
+ ...
+ conn = new AuthenticatedURL(url, token).openConnection();
+ ...
+
+Building and Running the Examples
+---------------------------------
+
+Download Hadoop-Auth's source code, the examples are in the `src/main/examples` directory.
+
+### Server Example:
+
+Edit the `hadoop-auth-examples/src/main/webapp/WEB-INF/web.xml` and set the right configuration init parameters for the `AuthenticationFilter` definition configured for Kerberos (the right Kerberos principal and keytab file must be specified). Refer to the [Configuration document](./Configuration.html) for details.
+
+Create the web application WAR file by running the `mvn package` command.
+
+Deploy the WAR file in a servlet container. For example, if using Tomcat, copy the WAR file to Tomcat's `webapps/` directory.
+
+Start the servlet container.
+
+### Accessing the server using `curl`
+
+Try accessing protected resources using `curl`. The protected resources are:
+
+ $ kinit
+ Please enter the password for tucu@LOCALHOST:
+
+ $ curl http://localhost:8080/hadoop-auth-examples/anonymous/who
+
+ $ curl http://localhost:8080/hadoop-auth-examples/simple/who?user.name=foo
+
+ $ curl --negotiate -u foo -b ~/cookiejar.txt -c ~/cookiejar.txt http://localhost:8080/hadoop-auth-examples/kerberos/who
+
+### Accessing the server using the Java client example
+
+ $ kinit
+ Please enter the password for tucu@LOCALHOST:
+
+ $ cd examples
+
+ $ mvn exec:java -Durl=http://localhost:8080/hadoop-auth-examples/kerberos/who
+
+ ....
+
+ Token value: "u=tucu,p=tucu@LOCALHOST,t=kerberos,e=1295305313146,s=sVZ1mpSnC5TKhZQE3QLN5p2DWBo="
+ Status code: 200 OK
+
+ You are: user[tucu] principal[tucu@LOCALHOST]
+
+ ....
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-common-project/hadoop-auth/src/site/markdown/index.md
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-auth/src/site/markdown/index.md b/hadoop-common-project/hadoop-auth/src/site/markdown/index.md
new file mode 100644
index 0000000..8573b18
--- /dev/null
+++ b/hadoop-common-project/hadoop-auth/src/site/markdown/index.md
@@ -0,0 +1,43 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+Hadoop Auth, Java HTTP SPNEGO
+=============================
+
+Hadoop Auth is a Java library consisting of a client and a server components to enable Kerberos SPNEGO authentication for HTTP.
+
+Hadoop Auth also supports additional authentication mechanisms on the client and the server side via 2 simple interfaces.
+
+Additionally, it provides a partially implemented derivative of the Kerberos SPNEGO authentication to allow a "mixed" form of authentication where Kerberos SPNEGO is used by non-browsers while an alternate form of authentication (to be implemented by the user) is used for browsers.
+
+License
+-------
+
+Hadoop Auth is distributed under [Apache License 2.0](http://www.apache.org/licenses/).
+
+How Does Auth Works?
+--------------------
+
+Hadoop Auth enforces authentication on protected resources, once authentiation has been established it sets a signed HTTP Cookie that contains an authentication token with the user name, user principal, authentication type and expiration time.
+
+Subsequent HTTP client requests presenting the signed HTTP Cookie have access to the protected resources until the HTTP Cookie expires.
+
+The secret used to sign the HTTP Cookie has multiple implementations that provide different behaviors, including a hardcoded secret string, a rolling randomly generated secret, and a rolling randomly generated secret synchronized between multiple servers using ZooKeeper.
+
+User Documentation
+------------------
+
+* [Examples](./Examples.html)
+* [Configuration](./Configuration.html)
+* [Building It](./BuildingIt.html)
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-common-project/hadoop-common/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt b/hadoop-common-project/hadoop-common/CHANGES.txt
index d8a85f7..b3b2c95 100644
--- a/hadoop-common-project/hadoop-common/CHANGES.txt
+++ b/hadoop-common-project/hadoop-common/CHANGES.txt
@@ -181,6 +181,9 @@ Trunk (Unreleased)
HADOOP-11596. Allow smart-apply-patch.sh to add new files in binary git
patches (raviprak)
+ HADOOP-11593. Convert site documentation from apt to markdown (stragglers)
+ (Masatake Iwasaki via aw)
+
BUG FIXES
HADOOP-11473. test-patch says "-1 overall" even when all checks are +1
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-common-project/hadoop-kms/src/site/apt/index.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-kms/src/site/apt/index.apt.vm b/hadoop-common-project/hadoop-kms/src/site/apt/index.apt.vm
deleted file mode 100644
index a2dcce3..0000000
--- a/hadoop-common-project/hadoop-kms/src/site/apt/index.apt.vm
+++ /dev/null
@@ -1,1020 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License.
-
- ---
- Hadoop KMS - Documentation Sets ${project.version}
- ---
- ---
- ${maven.build.timestamp}
-
-Hadoop Key Management Server (KMS) - Documentation Sets ${project.version}
-
- Hadoop KMS is a cryptographic key management server based on Hadoop's
- <<KeyProvider>> API.
-
- It provides a client and a server components which communicate over
- HTTP using a REST API.
-
- The client is a KeyProvider implementation interacts with the KMS
- using the KMS HTTP REST API.
-
- KMS and its client have built-in security and they support HTTP SPNEGO
- Kerberos authentication and HTTPS secure transport.
-
- KMS is a Java web-application and it runs using a pre-configured Tomcat
- bundled with the Hadoop distribution.
-
-* KMS Client Configuration
-
- The KMS client <<<KeyProvider>>> uses the <<kms>> scheme, and the embedded
- URL must be the URL of the KMS. For example, for a KMS running
- on <<<http://localhost:16000/kms>>>, the KeyProvider URI is
- <<<kms://http@localhost:16000/kms>>>. And, for a KMS running on
- <<<https://localhost:16000/kms>>>, the KeyProvider URI is
- <<<kms://https@localhost:16000/kms>>>
-
-* KMS
-
-** KMS Configuration
-
- Configure the KMS backing KeyProvider properties
- in the <<<etc/hadoop/kms-site.xml>>> configuration file:
-
-+---+
- <property>
- <name>hadoop.kms.key.provider.uri</name>
- <value>jceks://file@/${user.home}/kms.keystore</value>
- </property>
-
- <property>
- <name>hadoop.security.keystore.java-keystore-provider.password-file</name>
- <value>kms.keystore.password</value>
- </property>
-+---+
-
- The password file is looked up in the Hadoop's configuration directory via the
- classpath.
-
- NOTE: You need to restart the KMS for the configuration changes to take
- effect.
-
-** KMS Cache
-
- KMS caches keys for short period of time to avoid excessive hits to the
- underlying key provider.
-
- The Cache is enabled by default (can be dissabled by setting the
- <<<hadoop.kms.cache.enable>>> boolean property to false)
-
- The cache is used with the following 3 methods only, <<<getCurrentKey()>>>
- and <<<getKeyVersion()>>> and <<<getMetadata()>>>.
-
- For the <<<getCurrentKey()>>> method, cached entries are kept for a maximum
- of 30000 millisecond regardless the number of times the key is being access
- (to avoid stale keys to be considered current).
-
- For the <<<getKeyVersion()>>> method, cached entries are kept with a default
- inactivity timeout of 600000 milliseconds (10 mins). This time out is
- configurable via the following property in the <<<etc/hadoop/kms-site.xml>>>
- configuration file:
-
-+---+
- <property>
- <name>hadoop.kms.cache.enable</name>
- <value>true</value>
- </property>
-
- <property>
- <name>hadoop.kms.cache.timeout.ms</name>
- <value>600000</value>
- </property>
-
- <property>
- <name>hadoop.kms.current.key.cache.timeout.ms</name>
- <value>30000</value>
- </property>
-+---+
-
-** KMS Aggregated Audit logs
-
- Audit logs are aggregated for API accesses to the GET_KEY_VERSION,
- GET_CURRENT_KEY, DECRYPT_EEK, GENERATE_EEK operations.
-
- Entries are grouped by the (user,key,operation) combined key for a
- configurable aggregation interval after which the number of accesses to the
- specified end-point by the user for a given key is flushed to the audit log.
-
- The Aggregation interval is configured via the property :
-
-+---+
- <property>
- <name>hadoop.kms.aggregation.delay.ms</name>
- <value>10000</value>
- </property>
-+---+
-
-
-** Start/Stop the KMS
-
- To start/stop KMS use KMS's bin/kms.sh script. For example:
-
-+---+
-hadoop-${project.version} $ sbin/kms.sh start
-+---+
-
- NOTE: Invoking the script without any parameters list all possible
- parameters (start, stop, run, etc.). The <<<kms.sh>>> script is a wrapper
- for Tomcat's <<<catalina.sh>>> script that sets the environment variables
- and Java System properties required to run KMS.
-
-** Embedded Tomcat Configuration
-
- To configure the embedded Tomcat go to the <<<share/hadoop/kms/tomcat/conf>>>.
-
- KMS pre-configures the HTTP and Admin ports in Tomcat's <<<server.xml>>> to
- 16000 and 16001.
-
- Tomcat logs are also preconfigured to go to Hadoop's <<<logs/>>> directory.
-
- The following environment variables (which can be set in KMS's
- <<<etc/hadoop/kms-env.sh>>> script) can be used to alter those values:
-
- * KMS_HTTP_PORT
-
- * KMS_ADMIN_PORT
-
- * KMS_MAX_THREADS
-
- * KMS_LOG
-
- NOTE: You need to restart the KMS for the configuration changes to take
- effect.
-
-** Loading native libraries
-
- The following environment variable (which can be set in KMS's
- <<<etc/hadoop/kms-env.sh>>> script) can be used to specify the location
- of any required native libraries. For eg. Tomact native Apache Portable
- Runtime (APR) libraries:
-
- * JAVA_LIBRARY_PATH
-
-** KMS Security Configuration
-
-*** Enabling Kerberos HTTP SPNEGO Authentication
-
- Configure the Kerberos <<<etc/krb5.conf>>> file with the information of your
- KDC server.
-
- Create a service principal and its keytab for the KMS, it must be an
- <<<HTTP>>> service principal.
-
- Configure KMS <<<etc/hadoop/kms-site.xml>>> with the correct security values,
- for example:
-
-+---+
- <property>
- <name>hadoop.kms.authentication.type</name>
- <value>kerberos</value>
- </property>
-
- <property>
- <name>hadoop.kms.authentication.kerberos.keytab</name>
- <value>${user.home}/kms.keytab</value>
- </property>
-
- <property>
- <name>hadoop.kms.authentication.kerberos.principal</name>
- <value>HTTP/localhost</value>
- </property>
-
- <property>
- <name>hadoop.kms.authentication.kerberos.name.rules</name>
- <value>DEFAULT</value>
- </property>
-+---+
-
- NOTE: You need to restart the KMS for the configuration changes to take
- effect.
-
-*** KMS Proxyuser Configuration
-
- Each proxyuser must be configured in <<<etc/hadoop/kms-site.xml>>> using the
- following properties:
-
-+---+
- <property>
- <name>hadoop.kms.proxyuser.#USER#.users</name>
- <value>*</value>
- </property>
-
- <property>
- <name>hadoop.kms.proxyuser.#USER#.groups</name>
- <value>*</value>
- </property>
-
- <property>
- <name>hadoop.kms.proxyuser.#USER#.hosts</name>
- <value>*</value>
- </property>
-+---+
-
- <<<#USER#>>> is the username of the proxyuser to configure.
-
- The <<<users>>> property indicates the users that can be impersonated.
-
- The <<<groups>>> property indicates the groups users being impersonated must
- belong to.
-
- At least one of the <<<users>>> or <<<groups>>> properties must be defined.
- If both are specified, then the configured proxyuser will be able to
- impersonate and user in the <<<users>>> list and any user belonging to one of
- the groups in the <<<groups>>> list.
-
- The <<<hosts>>> property indicates from which host the proxyuser can make
- impersonation requests.
-
- If <<<users>>>, <<<groups>>> or <<<hosts>>> has a <<<*>>>, it means there are
- no restrictions for the proxyuser regarding users, groups or hosts.
-
-*** KMS over HTTPS (SSL)
-
- To configure KMS to work over HTTPS the following 2 properties must be
- set in the <<<etc/hadoop/kms_env.sh>>> script (shown with default values):
-
- * KMS_SSL_KEYSTORE_FILE=${HOME}/.keystore
-
- * KMS_SSL_KEYSTORE_PASS=password
-
- In the KMS <<<tomcat/conf>>> directory, replace the <<<server.xml>>> file
- with the provided <<<ssl-server.xml>>> file.
-
- You need to create an SSL certificate for the KMS. As the
- <<<kms>>> Unix user, using the Java <<<keytool>>> command to create the
- SSL certificate:
-
-+---+
-$ keytool -genkey -alias tomcat -keyalg RSA
-+---+
-
- You will be asked a series of questions in an interactive prompt. It will
- create the keystore file, which will be named <<.keystore>> and located in the
- <<<kms>>> user home directory.
-
- The password you enter for "keystore password" must match the value of the
- <<<KMS_SSL_KEYSTORE_PASS>>> environment variable set in the
- <<<kms-env.sh>>> script in the configuration directory.
-
- The answer to "What is your first and last name?" (i.e. "CN") must be the
- hostname of the machine where the KMS will be running.
-
- NOTE: You need to restart the KMS for the configuration changes to take
- effect.
-
-*** KMS Access Control
-
- KMS ACLs configuration are defined in the KMS <<<etc/hadoop/kms-acls.xml>>>
- configuration file. This file is hot-reloaded when it changes.
-
- KMS supports both fine grained access control as well as blacklist for kms
- operations via a set ACL configuration properties.
-
- A user accessing KMS is first checked for inclusion in the Access Control
- List for the requested operation and then checked for exclusion in the
- Black list for the operation before access is granted.
-
-
-+---+
- <property>
- <name>hadoop.kms.acl.CREATE</name>
- <value>*</value>
- <description>
- ACL for create-key operations.
- If the user is not in the GET ACL, the key material is not returned
- as part of the response.
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.blacklist.CREATE</name>
- <value>hdfs,foo</value>
- <description>
- Blacklist for create-key operations.
- If the user is in the Blacklist, the key material is not returned
- as part of the response.
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.acl.DELETE</name>
- <value>*</value>
- <description>
- ACL for delete-key operations.
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.blacklist.DELETE</name>
- <value>hdfs,foo</value>
- <description>
- Blacklist for delete-key operations.
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.acl.ROLLOVER</name>
- <value>*</value>
- <description>
- ACL for rollover-key operations.
- If the user is not in the GET ACL, the key material is not returned
- as part of the response.
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.blacklist.ROLLOVER</name>
- <value>hdfs,foo</value>
- <description>
- Blacklist for rollover-key operations.
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.acl.GET</name>
- <value>*</value>
- <description>
- ACL for get-key-version and get-current-key operations.
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.blacklist.GET</name>
- <value>hdfs,foo</value>
- <description>
- ACL for get-key-version and get-current-key operations.
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.acl.GET_KEYS</name>
- <value>*</value>
- <description>
- ACL for get-keys operation.
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.blacklist.GET_KEYS</name>
- <value>hdfs,foo</value>
- <description>
- Blacklist for get-keys operation.
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.acl.GET_METADATA</name>
- <value>*</value>
- <description>
- ACL for get-key-metadata and get-keys-metadata operations.
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.blacklist.GET_METADATA</name>
- <value>hdfs,foo</value>
- <description>
- Blacklist for get-key-metadata and get-keys-metadata operations.
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.acl.SET_KEY_MATERIAL</name>
- <value>*</value>
- <description>
- Complimentary ACL for CREATE and ROLLOVER operation to allow the client
- to provide the key material when creating or rolling a key.
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.blacklist.SET_KEY_MATERIAL</name>
- <value>hdfs,foo</value>
- <description>
- Complimentary Blacklist for CREATE and ROLLOVER operation to allow the client
- to provide the key material when creating or rolling a key.
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.acl.GENERATE_EEK</name>
- <value>*</value>
- <description>
- ACL for generateEncryptedKey
- CryptoExtension operations
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.blacklist.GENERATE_EEK</name>
- <value>hdfs,foo</value>
- <description>
- Blacklist for generateEncryptedKey
- CryptoExtension operations
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.acl.DECRYPT_EEK</name>
- <value>*</value>
- <description>
- ACL for decrypt EncryptedKey
- CryptoExtension operations
- </description>
- </property>
-</configuration>
-
- <property>
- <name>hadoop.kms.blacklist.DECRYPT_EEK</name>
- <value>hdfs,foo</value>
- <description>
- Blacklist for decrypt EncryptedKey
- CryptoExtension operations
- </description>
- </property>
-</configuration>
-
-+---+
-
-*** Key Access Control
-
- KMS supports access control for all non-read operations at the Key level.
- All Key Access operations are classified as :
-
- * MANAGEMENT - createKey, deleteKey, rolloverNewVersion
-
- * GENERATE_EEK - generateEncryptedKey, warmUpEncryptedKeys
-
- * DECRYPT_EEK - decryptEncryptedKey
-
- * READ - getKeyVersion, getKeyVersions, getMetadata, getKeysMetadata,
- getCurrentKey
-
- * ALL - all of the above
-
- These can be defined in the KMS <<<etc/hadoop/kms-acls.xml>>> as follows
-
- For all keys for which a key access has not been explicitly configured, It
- is possible to configure a default key access control for a subset of the
- operation types.
-
- It is also possible to configure a "whitelist" key ACL for a subset of the
- operation types. The whitelist key ACL is a whitelist in addition to the
- explicit or default per-key ACL. That is, if no per-key ACL is explicitly
- set, a user will be granted access if they are present in the default per-key
- ACL or the whitelist key ACL. If a per-key ACL is explicitly set, a user
- will be granted access if they are present in the per-key ACL or the
- whitelist key ACL.
-
- If no ACL is configured for a specific key AND no default ACL is configured
- AND no root key ACL is configured for the requested operation,
- then access will be DENIED.
-
- <<NOTE:>> The default and whitelist key ACL does not support <<<ALL>>>
- operation qualifier.
-
-+---+
- <property>
- <name>key.acl.testKey1.MANAGEMENT</name>
- <value>*</value>
- <description>
- ACL for create-key, deleteKey and rolloverNewVersion operations.
- </description>
- </property>
-
- <property>
- <name>key.acl.testKey2.GENERATE_EEK</name>
- <value>*</value>
- <description>
- ACL for generateEncryptedKey operations.
- </description>
- </property>
-
- <property>
- <name>key.acl.testKey3.DECRYPT_EEK</name>
- <value>admink3</value>
- <description>
- ACL for decryptEncryptedKey operations.
- </description>
- </property>
-
- <property>
- <name>key.acl.testKey4.READ</name>
- <value>*</value>
- <description>
- ACL for getKeyVersion, getKeyVersions, getMetadata, getKeysMetadata,
- getCurrentKey operations
- </description>
- </property>
-
- <property>
- <name>key.acl.testKey5.ALL</name>
- <value>*</value>
- <description>
- ACL for ALL operations.
- </description>
- </property>
-
- <property>
- <name>whitelist.key.acl.MANAGEMENT</name>
- <value>admin1</value>
- <description>
- whitelist ACL for MANAGEMENT operations for all keys.
- </description>
- </property>
-
- <!--
- 'testKey3' key ACL is defined. Since a 'whitelist'
- key is also defined for DECRYPT_EEK, in addition to
- admink3, admin1 can also perform DECRYPT_EEK operations
- on 'testKey3'
- -->
- <property>
- <name>whitelist.key.acl.DECRYPT_EEK</name>
- <value>admin1</value>
- <description>
- whitelist ACL for DECRYPT_EEK operations for all keys.
- </description>
- </property>
-
- <property>
- <name>default.key.acl.MANAGEMENT</name>
- <value>user1,user2</value>
- <description>
- default ACL for MANAGEMENT operations for all keys that are not
- explicitly defined.
- </description>
- </property>
-
- <property>
- <name>default.key.acl.GENERATE_EEK</name>
- <value>user1,user2</value>
- <description>
- default ACL for GENERATE_EEK operations for all keys that are not
- explicitly defined.
- </description>
- </property>
-
- <property>
- <name>default.key.acl.DECRYPT_EEK</name>
- <value>user1,user2</value>
- <description>
- default ACL for DECRYPT_EEK operations for all keys that are not
- explicitly defined.
- </description>
- </property>
-
- <property>
- <name>default.key.acl.READ</name>
- <value>user1,user2</value>
- <description>
- default ACL for READ operations for all keys that are not
- explicitly defined.
- </description>
- </property>
-+---+
-
-** KMS Delegation Token Configuration
-
- KMS delegation token secret manager can be configured with the following
- properties:
-
-+---+
- <property>
- <name>hadoop.kms.authentication.delegation-token.update-interval.sec</name>
- <value>86400</value>
- <description>
- How often the master key is rotated, in seconds. Default value 1 day.
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.authentication.delegation-token.max-lifetime.sec</name>
- <value>604800</value>
- <description>
- Maximum lifetime of a delagation token, in seconds. Default value 7 days.
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.authentication.delegation-token.renew-interval.sec</name>
- <value>86400</value>
- <description>
- Renewal interval of a delagation token, in seconds. Default value 1 day.
- </description>
- </property>
-
- <property>
- <name>hadoop.kms.authentication.delegation-token.removal-scan-interval.sec</name>
- <value>3600</value>
- <description>
- Scan interval to remove expired delegation tokens.
- </description>
- </property>
-+---+
-
-
-** Using Multiple Instances of KMS Behind a Load-Balancer or VIP
-
- KMS supports multiple KMS instances behind a load-balancer or VIP for
- scalability and for HA purposes.
-
- When using multiple KMS instances behind a load-balancer or VIP, requests from
- the same user may be handled by different KMS instances.
-
- KMS instances behind a load-balancer or VIP must be specially configured to
- work properly as a single logical service.
-
-*** HTTP Kerberos Principals Configuration
-
- When KMS instances are behind a load-balancer or VIP, clients will use the
- hostname of the VIP. For Kerberos SPNEGO authentication, the hostname of the
- URL is used to construct the Kerberos service name of the server,
- <<<HTTP/#HOSTNAME#>>>. This means that all KMS instances must have a Kerberos
- service name with the load-balancer or VIP hostname.
-
- In order to be able to access directly a specific KMS instance, the KMS
- instance must also have Keberos service name with its own hostname. This is
- required for monitoring and admin purposes.
-
- Both Kerberos service principal credentials (for the load-balancer/VIP
- hostname and for the actual KMS instance hostname) must be in the keytab file
- configured for authentication. And the principal name specified in the
- configuration must be '*'. For example:
-
-+---+
- <property>
- <name>hadoop.kms.authentication.kerberos.principal</name>
- <value>*</value>
- </property>
-+---+
-
- <<NOTE:>> If using HTTPS, the SSL certificate used by the KMS instance must
- be configured to support multiple hostnames (see Java 7
- <<<keytool>>> SAN extension support for details on how to do this).
-
-*** HTTP Authentication Signature
-
- KMS uses Hadoop Authentication for HTTP authentication. Hadoop Authentication
- issues a signed HTTP Cookie once the client has authenticated successfully.
- This HTTP Cookie has an expiration time, after which it will trigger a new
- authentication sequence. This is done to avoid triggering the authentication
- on every HTTP request of a client.
-
- A KMS instance must verify the HTTP Cookie signatures signed by other KMS
- instances. To do this all KMS instances must share the signing secret.
-
- This secret sharing can be done using a Zookeeper service which is configured
- in KMS with the following properties in the <<<kms-site.xml>>>:
-
-+---+
- <property>
- <name>hadoop.kms.authentication.signer.secret.provider</name>
- <value>zookeeper</value>
- <description>
- Indicates how the secret to sign the authentication cookies will be
- stored. Options are 'random' (default), 'string' and 'zookeeper'.
- If using a setup with multiple KMS instances, 'zookeeper' should be used.
- </description>
- </property>
- <property>
- <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.path</name>
- <value>/hadoop-kms/hadoop-auth-signature-secret</value>
- <description>
- The Zookeeper ZNode path where the KMS instances will store and retrieve
- the secret from.
- </description>
- </property>
- <property>
- <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.connection.string</name>
- <value>#HOSTNAME#:#PORT#,...</value>
- <description>
- The Zookeeper connection string, a list of hostnames and port comma
- separated.
- </description>
- </property>
- <property>
- <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.auth.type</name>
- <value>kerberos</value>
- <description>
- The Zookeeper authentication type, 'none' or 'sasl' (Kerberos).
- </description>
- </property>
- <property>
- <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.keytab</name>
- <value>/etc/hadoop/conf/kms.keytab</value>
- <description>
- The absolute path for the Kerberos keytab with the credentials to
- connect to Zookeeper.
- </description>
- </property>
- <property>
- <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.principal</name>
- <value>kms/#HOSTNAME#</value>
- <description>
- The Kerberos service principal used to connect to Zookeeper.
- </description>
- </property>
-+---+
-
-*** Delegation Tokens
-
- TBD
-
-** KMS HTTP REST API
-
-*** Create a Key
-
- <REQUEST:>
-
-+---+
-POST http://HOST:PORT/kms/v1/keys
-Content-Type: application/json
-
-{
- "name" : "<key-name>",
- "cipher" : "<cipher>",
- "length" : <length>, //int
- "material" : "<material>", //base64
- "description" : "<description>"
-}
-+---+
-
- <RESPONSE:>
-
-+---+
-201 CREATED
-LOCATION: http://HOST:PORT/kms/v1/key/<key-name>
-Content-Type: application/json
-
-{
- "name" : "versionName",
- "material" : "<material>", //base64, not present without GET ACL
-}
-+---+
-
-*** Rollover Key
-
- <REQUEST:>
-
-+---+
-POST http://HOST:PORT/kms/v1/key/<key-name>
-Content-Type: application/json
-
-{
- "material" : "<material>",
-}
-+---+
-
- <RESPONSE:>
-
-+---+
-200 OK
-Content-Type: application/json
-
-{
- "name" : "versionName",
- "material" : "<material>", //base64, not present without GET ACL
-}
-+---+
-
-*** Delete Key
-
- <REQUEST:>
-
-+---+
-DELETE http://HOST:PORT/kms/v1/key/<key-name>
-+---+
-
- <RESPONSE:>
-
-+---+
-200 OK
-+---+
-
-*** Get Key Metadata
-
- <REQUEST:>
-
-+---+
-GET http://HOST:PORT/kms/v1/key/<key-name>/_metadata
-+---+
-
- <RESPONSE:>
-
-+---+
-200 OK
-Content-Type: application/json
-
-{
- "name" : "<key-name>",
- "cipher" : "<cipher>",
- "length" : <length>, //int
- "description" : "<description>",
- "created" : <millis-epoc>, //long
- "versions" : <versions> //int
-}
-+---+
-
-*** Get Current Key
-
- <REQUEST:>
-
-+---+
-GET http://HOST:PORT/kms/v1/key/<key-name>/_currentversion
-+---+
-
- <RESPONSE:>
-
-+---+
-200 OK
-Content-Type: application/json
-
-{
- "name" : "versionName",
- "material" : "<material>", //base64
-}
-+---+
-
-
-*** Generate Encrypted Key for Current KeyVersion
-
- <REQUEST:>
-
-+---+
-GET http://HOST:PORT/kms/v1/key/<key-name>/_eek?eek_op=generate&num_keys=<number-of-keys-to-generate>
-+---+
-
- <RESPONSE:>
-
-+---+
-200 OK
-Content-Type: application/json
-[
- {
- "versionName" : "encryptionVersionName",
- "iv" : "<iv>", //base64
- "encryptedKeyVersion" : {
- "versionName" : "EEK",
- "material" : "<material>", //base64
- }
- },
- {
- "versionName" : "encryptionVersionName",
- "iv" : "<iv>", //base64
- "encryptedKeyVersion" : {
- "versionName" : "EEK",
- "material" : "<material>", //base64
- }
- },
- ...
-]
-+---+
-
-*** Decrypt Encrypted Key
-
- <REQUEST:>
-
-+---+
-POST http://HOST:PORT/kms/v1/keyversion/<version-name>/_eek?ee_op=decrypt
-Content-Type: application/json
-
-{
- "name" : "<key-name>",
- "iv" : "<iv>", //base64
- "material" : "<material>", //base64
-}
-
-+---+
-
- <RESPONSE:>
-
-+---+
-200 OK
-Content-Type: application/json
-
-{
- "name" : "EK",
- "material" : "<material>", //base64
-}
-+---+
-
-
-*** Get Key Version
-
- <REQUEST:>
-
-+---+
-GET http://HOST:PORT/kms/v1/keyversion/<version-name>
-+---+
-
- <RESPONSE:>
-
-+---+
-200 OK
-Content-Type: application/json
-
-{
- "name" : "versionName",
- "material" : "<material>", //base64
-}
-+---+
-
-*** Get Key Versions
-
- <REQUEST:>
-
-+---+
-GET http://HOST:PORT/kms/v1/key/<key-name>/_versions
-+---+
-
- <RESPONSE:>
-
-+---+
-200 OK
-Content-Type: application/json
-
-[
- {
- "name" : "versionName",
- "material" : "<material>", //base64
- },
- {
- "name" : "versionName",
- "material" : "<material>", //base64
- },
- ...
-]
-+---+
-
-*** Get Key Names
-
- <REQUEST:>
-
-+---+
-GET http://HOST:PORT/kms/v1/keys/names
-+---+
-
- <RESPONSE:>
-
-+---+
-200 OK
-Content-Type: application/json
-
-[
- "<key-name>",
- "<key-name>",
- ...
-]
-+---+
-
-*** Get Keys Metadata
-
-+---+
-GET http://HOST:PORT/kms/v1/keys/metadata?key=<key-name>&key=<key-name>,...
-+---+
-
- <RESPONSE:>
-
-+---+
-200 OK
-Content-Type: application/json
-
-[
- {
- "name" : "<key-name>",
- "cipher" : "<cipher>",
- "length" : <length>, //int
- "description" : "<description>",
- "created" : <millis-epoc>, //long
- "versions" : <versions> //int
- },
- {
- "name" : "<key-name>",
- "cipher" : "<cipher>",
- "length" : <length>, //int
- "description" : "<description>",
- "created" : <millis-epoc>, //long
- "versions" : <versions> //int
- },
- ...
-]
-+---+
[47/52] [abbrv] hadoop git commit: HADOOP-11534. Minor improvements
for raw erasure coders ( Contributed by Kai Zheng )
Posted by zh...@apache.org.
HADOOP-11534. Minor improvements for raw erasure coders ( Contributed by Kai Zheng )
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/2dbe4c50
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/2dbe4c50
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/2dbe4c50
Branch: refs/heads/HDFS-7285
Commit: 2dbe4c508a63a85c54b631ad6ad56f05594183c7
Parents: 65dca22
Author: Vinayakumar B <vi...@intel.com>
Authored: Mon Feb 2 14:39:53 2015 +0530
Committer: Zhe Zhang <zh...@apache.org>
Committed: Mon Feb 23 11:20:26 2015 -0800
----------------------------------------------------------------------
.../hadoop-common/CHANGES-HDFS-EC-7285.txt | 5 ++++-
.../org/apache/hadoop/io/erasurecode/ECChunk.java | 15 +++++++++++++--
.../rawcoder/AbstractRawErasureCoder.java | 12 ++++++------
3 files changed, 23 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2dbe4c50/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt b/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt
index 8ce5a89..2124800 100644
--- a/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt
+++ b/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt
@@ -1,4 +1,7 @@
BREAKDOWN OF HADOOP-11264 SUBTASKS AND RELATED JIRAS (Common part of HDFS-7285)
HADOOP-11514. Raw Erasure Coder API for concrete encoding and decoding
- (Kai Zheng via umamahesh)
\ No newline at end of file
+ (Kai Zheng via umamahesh)
+
+ HADOOP-11534. Minor improvements for raw erasure coders
+ ( Kai Zheng via vinayakumarb )
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2dbe4c50/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECChunk.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECChunk.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECChunk.java
index f84eb11..01e8f35 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECChunk.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECChunk.java
@@ -66,15 +66,26 @@ public class ECChunk {
}
/**
- * Convert an array of this chunks to an array of byte array
+ * Convert an array of this chunks to an array of byte array.
+ * Note the chunk buffers are not affected.
* @param chunks
* @return an array of byte array
*/
public static byte[][] toArray(ECChunk[] chunks) {
byte[][] bytesArr = new byte[chunks.length][];
+ ByteBuffer buffer;
for (int i = 0; i < chunks.length; i++) {
- bytesArr[i] = chunks[i].getBuffer().array();
+ buffer = chunks[i].getBuffer();
+ if (buffer.hasArray()) {
+ bytesArr[i] = buffer.array();
+ } else {
+ bytesArr[i] = new byte[buffer.remaining()];
+ // Avoid affecting the original one
+ buffer.mark();
+ buffer.get(bytesArr[i]);
+ buffer.reset();
+ }
}
return bytesArr;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2dbe4c50/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureCoder.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureCoder.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureCoder.java
index 474542b..74d2ab6 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureCoder.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureCoder.java
@@ -24,26 +24,26 @@ package org.apache.hadoop.io.erasurecode.rawcoder;
*/
public abstract class AbstractRawErasureCoder implements RawErasureCoder {
- private int dataSize;
- private int paritySize;
+ private int numDataUnits;
+ private int numParityUnits;
private int chunkSize;
@Override
public void initialize(int numDataUnits, int numParityUnits,
int chunkSize) {
- this.dataSize = numDataUnits;
- this.paritySize = numParityUnits;
+ this.numDataUnits = numDataUnits;
+ this.numParityUnits = numParityUnits;
this.chunkSize = chunkSize;
}
@Override
public int getNumDataUnits() {
- return dataSize;
+ return numDataUnits;
}
@Override
public int getNumParityUnits() {
- return paritySize;
+ return numParityUnits;
}
@Override
[51/52] [abbrv] hadoop git commit: HDFS-7716. Erasure Coding: extend
BlockInfo to handle EC info. Contributed by Jing Zhao.
Posted by zh...@apache.org.
HDFS-7716. Erasure Coding: extend BlockInfo to handle EC info. Contributed by Jing Zhao.
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9d1ec74d
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9d1ec74d
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9d1ec74d
Branch: refs/heads/HDFS-7285
Commit: 9d1ec74d005f20d1e371be3026b21595d6106b60
Parents: 0d3b462
Author: Jing Zhao <ji...@apache.org>
Authored: Tue Feb 10 17:54:10 2015 -0800
Committer: Zhe Zhang <zh...@apache.org>
Committed: Mon Feb 23 11:21:13 2015 -0800
----------------------------------------------------------------------
.../hadoop/hdfs/protocol/HdfsConstants.java | 1 +
.../server/blockmanagement/BlockCollection.java | 13 +-
.../server/blockmanagement/BlockIdManager.java | 7 +-
.../hdfs/server/blockmanagement/BlockInfo.java | 339 +++++++++++++++++
.../blockmanagement/BlockInfoContiguous.java | 363 +++----------------
.../BlockInfoContiguousUnderConstruction.java | 140 +------
.../blockmanagement/BlockInfoStriped.java | 179 +++++++++
.../server/blockmanagement/BlockManager.java | 188 +++++-----
.../hdfs/server/blockmanagement/BlocksMap.java | 46 +--
.../CacheReplicationMonitor.java | 10 +-
.../blockmanagement/DatanodeDescriptor.java | 22 +-
.../blockmanagement/DatanodeStorageInfo.java | 38 +-
.../ReplicaUnderConstruction.java | 119 ++++++
.../hdfs/server/namenode/FSDirectory.java | 4 +-
.../hdfs/server/namenode/FSNamesystem.java | 24 +-
.../hdfs/server/namenode/NamenodeFsck.java | 3 +-
.../snapshot/FSImageFormatPBSnapshot.java | 4 +-
.../org/apache/hadoop/hdfs/DFSTestUtil.java | 4 +-
.../server/blockmanagement/TestBlockInfo.java | 6 +-
.../blockmanagement/TestBlockInfoStriped.java | 219 +++++++++++
.../blockmanagement/TestBlockManager.java | 4 +-
.../blockmanagement/TestReplicationPolicy.java | 2 +-
22 files changed, 1127 insertions(+), 608 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
index e3e3f37..acf4853 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
@@ -184,5 +184,6 @@ public class HdfsConstants {
public static final byte NUM_DATA_BLOCKS = 3;
public static final byte NUM_PARITY_BLOCKS = 2;
+ public static final long BLOCK_GROUP_INDEX_MASK = 15;
public static final byte MAX_BLOCKS_IN_GROUP = 16;
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java
index 1547611..974cac3 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java
@@ -39,12 +39,12 @@ public interface BlockCollection {
public ContentSummary computeContentSummary();
/**
- * @return the number of blocks
+ * @return the number of blocks or block groups
*/
public int numBlocks();
/**
- * Get the blocks.
+ * Get the blocks or block groups.
*/
public BlockInfoContiguous[] getBlocks();
@@ -55,8 +55,8 @@ public interface BlockCollection {
public long getPreferredBlockSize();
/**
- * Get block replication for the collection
- * @return block replication value
+ * Get block replication for the collection.
+ * @return block replication value. Return 0 if the file is erasure coded.
*/
public short getBlockReplication();
@@ -71,7 +71,7 @@ public interface BlockCollection {
public String getName();
/**
- * Set the block at the given index.
+ * Set the block/block-group at the given index.
*/
public void setBlock(int index, BlockInfoContiguous blk);
@@ -79,7 +79,8 @@ public interface BlockCollection {
* Convert the last block of the collection to an under-construction block
* and set the locations.
*/
- public BlockInfoContiguousUnderConstruction setLastBlock(BlockInfoContiguous lastBlock,
+ public BlockInfoContiguousUnderConstruction setLastBlock(
+ BlockInfoContiguous lastBlock,
DatanodeStorageInfo[] targets) throws IOException;
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
index e7f8a05..3ae54ce 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
@@ -217,6 +217,11 @@ public class BlockIdManager {
}
public static long convertToGroupID(long id) {
- return id & (~(HdfsConstants.MAX_BLOCKS_IN_GROUP - 1));
+ return id & (~HdfsConstants.BLOCK_GROUP_INDEX_MASK);
+ }
+
+ public static int getBlockIndex(Block reportedBlock) {
+ return (int) (reportedBlock.getBlockId() &
+ HdfsConstants.BLOCK_GROUP_INDEX_MASK);
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
new file mode 100644
index 0000000..f19ad32
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
@@ -0,0 +1,339 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.hdfs.server.common.HdfsServerConstants;
+import org.apache.hadoop.util.LightWeightGSet;
+
+import java.util.LinkedList;
+
+/**
+ * For a given block (or an erasure coding block group), BlockInfo class
+ * maintains 1) the {@link BlockCollection} it is part of, and 2) datanodes
+ * where the replicas of the block, or blocks belonging to the erasure coding
+ * block group, are stored.
+ */
+public abstract class BlockInfo extends Block
+ implements LightWeightGSet.LinkedElement {
+ private BlockCollection bc;
+
+ /** For implementing {@link LightWeightGSet.LinkedElement} interface */
+ private LightWeightGSet.LinkedElement nextLinkedElement;
+
+ /**
+ * This array contains triplets of references. For each i-th storage, the
+ * block belongs to triplets[3*i] is the reference to the
+ * {@link DatanodeStorageInfo} and triplets[3*i+1] and triplets[3*i+2] are
+ * references to the previous and the next blocks, respectively, in the list
+ * of blocks belonging to this storage.
+ *
+ * Using previous and next in Object triplets is done instead of a
+ * {@link LinkedList} list to efficiently use memory. With LinkedList the cost
+ * per replica is 42 bytes (LinkedList#Entry object per replica) versus 16
+ * bytes using the triplets.
+ */
+ protected Object[] triplets;
+
+ /**
+ * Construct an entry for blocksmap
+ * @param size the block's replication factor, or the total number of blocks
+ * in the block group
+ */
+ public BlockInfo(short size) {
+ this.triplets = new Object[3 * size];
+ this.bc = null;
+ }
+
+ public BlockInfo(Block blk, short size) {
+ super(blk);
+ this.triplets = new Object[3 * size];
+ this.bc = null;
+ }
+
+ public BlockCollection getBlockCollection() {
+ return bc;
+ }
+
+ public void setBlockCollection(BlockCollection bc) {
+ this.bc = bc;
+ }
+
+ public DatanodeDescriptor getDatanode(int index) {
+ DatanodeStorageInfo storage = getStorageInfo(index);
+ return storage == null ? null : storage.getDatanodeDescriptor();
+ }
+
+ DatanodeStorageInfo getStorageInfo(int index) {
+ assert this.triplets != null : "BlockInfo is not initialized";
+ assert index >= 0 && index*3 < triplets.length : "Index is out of bound";
+ return (DatanodeStorageInfo)triplets[index*3];
+ }
+
+ BlockInfo getPrevious(int index) {
+ assert this.triplets != null : "BlockInfo is not initialized";
+ assert index >= 0 && index*3+1 < triplets.length : "Index is out of bound";
+ return (BlockInfo) triplets[index*3+1];
+ }
+
+ BlockInfo getNext(int index) {
+ assert this.triplets != null : "BlockInfo is not initialized";
+ assert index >= 0 && index*3+2 < triplets.length : "Index is out of bound";
+ return (BlockInfo) triplets[index*3+2];
+ }
+
+ void setStorageInfo(int index, DatanodeStorageInfo storage) {
+ assert this.triplets != null : "BlockInfo is not initialized";
+ assert index >= 0 && index*3 < triplets.length : "Index is out of bound";
+ triplets[index*3] = storage;
+ }
+
+ /**
+ * Return the previous block on the block list for the datanode at
+ * position index. Set the previous block on the list to "to".
+ *
+ * @param index - the datanode index
+ * @param to - block to be set to previous on the list of blocks
+ * @return current previous block on the list of blocks
+ */
+ BlockInfo setPrevious(int index, BlockInfo to) {
+ assert this.triplets != null : "BlockInfo is not initialized";
+ assert index >= 0 && index*3+1 < triplets.length : "Index is out of bound";
+ BlockInfo info = (BlockInfo) triplets[index*3+1];
+ triplets[index*3+1] = to;
+ return info;
+ }
+
+ /**
+ * Return the next block on the block list for the datanode at
+ * position index. Set the next block on the list to "to".
+ *
+ * @param index - the datanode index
+ * @param to - block to be set to next on the list of blocks
+ * @return current next block on the list of blocks
+ */
+ BlockInfo setNext(int index, BlockInfo to) {
+ assert this.triplets != null : "BlockInfo is not initialized";
+ assert index >= 0 && index*3+2 < triplets.length : "Index is out of bound";
+ BlockInfo info = (BlockInfo) triplets[index*3+2];
+ triplets[index*3+2] = to;
+ return info;
+ }
+
+ public int getCapacity() {
+ assert this.triplets != null : "BlockInfo is not initialized";
+ assert triplets.length % 3 == 0 : "Malformed BlockInfo";
+ return triplets.length / 3;
+ }
+
+ /**
+ * Count the number of data-nodes the block currently belongs to (i.e., NN
+ * has received block reports from the DN).
+ */
+ public abstract int numNodes();
+
+ /**
+ * Add a {@link DatanodeStorageInfo} location for a block
+ * @param storage The storage to add
+ * @param reportedBlock The block reported from the datanode. This is only
+ * used by erasure coded blocks, this block's id contains
+ * information indicating the index of the block in the
+ * corresponding block group.
+ */
+ abstract boolean addStorage(DatanodeStorageInfo storage, Block reportedBlock);
+
+ /**
+ * Remove {@link DatanodeStorageInfo} location for a block
+ */
+ abstract boolean removeStorage(DatanodeStorageInfo storage);
+
+ /**
+ * Replace the current BlockInfo with the new one in corresponding
+ * DatanodeStorageInfo's linked list
+ */
+ abstract void replaceBlock(BlockInfo newBlock);
+
+ /**
+ * Find specified DatanodeDescriptor.
+ * @return index or -1 if not found.
+ */
+ boolean findDatanode(DatanodeDescriptor dn) {
+ int len = getCapacity();
+ for (int idx = 0; idx < len; idx++) {
+ DatanodeDescriptor cur = getDatanode(idx);
+ if(cur == dn) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Find specified DatanodeStorageInfo.
+ * @return DatanodeStorageInfo or null if not found.
+ */
+ DatanodeStorageInfo findStorageInfo(DatanodeDescriptor dn) {
+ int len = getCapacity();
+ for(int idx = 0; idx < len; idx++) {
+ DatanodeStorageInfo cur = getStorageInfo(idx);
+ if(cur != null && cur.getDatanodeDescriptor() == dn) {
+ return cur;
+ }
+ }
+ return null;
+ }
+
+ /**
+ * Find specified DatanodeStorageInfo.
+ * @return index or -1 if not found.
+ */
+ int findStorageInfo(DatanodeStorageInfo storageInfo) {
+ int len = getCapacity();
+ for(int idx = 0; idx < len; idx++) {
+ DatanodeStorageInfo cur = getStorageInfo(idx);
+ if (cur == storageInfo) {
+ return idx;
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * Insert this block into the head of the list of blocks
+ * related to the specified DatanodeStorageInfo.
+ * If the head is null then form a new list.
+ * @return current block as the new head of the list.
+ */
+ BlockInfo listInsert(BlockInfo head, DatanodeStorageInfo storage) {
+ int dnIndex = this.findStorageInfo(storage);
+ assert dnIndex >= 0 : "Data node is not found: current";
+ assert getPrevious(dnIndex) == null && getNext(dnIndex) == null :
+ "Block is already in the list and cannot be inserted.";
+ this.setPrevious(dnIndex, null);
+ this.setNext(dnIndex, head);
+ if (head != null) {
+ head.setPrevious(head.findStorageInfo(storage), this);
+ }
+ return this;
+ }
+
+ /**
+ * Remove this block from the list of blocks
+ * related to the specified DatanodeStorageInfo.
+ * If this block is the head of the list then return the next block as
+ * the new head.
+ * @return the new head of the list or null if the list becomes
+ * empy after deletion.
+ */
+ BlockInfo listRemove(BlockInfo head, DatanodeStorageInfo storage) {
+ if (head == null) {
+ return null;
+ }
+ int dnIndex = this.findStorageInfo(storage);
+ if (dnIndex < 0) { // this block is not on the data-node list
+ return head;
+ }
+
+ BlockInfo next = this.getNext(dnIndex);
+ BlockInfo prev = this.getPrevious(dnIndex);
+ this.setNext(dnIndex, null);
+ this.setPrevious(dnIndex, null);
+ if (prev != null) {
+ prev.setNext(prev.findStorageInfo(storage), next);
+ }
+ if (next != null) {
+ next.setPrevious(next.findStorageInfo(storage), prev);
+ }
+ if (this == head) { // removing the head
+ head = next;
+ }
+ return head;
+ }
+
+ /**
+ * Remove this block from the list of blocks related to the specified
+ * DatanodeDescriptor. Insert it into the head of the list of blocks.
+ *
+ * @return the new head of the list.
+ */
+ public BlockInfo moveBlockToHead(BlockInfo head, DatanodeStorageInfo storage,
+ int curIndex, int headIndex) {
+ if (head == this) {
+ return this;
+ }
+ BlockInfo next = this.setNext(curIndex, head);
+ BlockInfo prev = this.setPrevious(curIndex, null);
+
+ head.setPrevious(headIndex, this);
+ prev.setNext(prev.findStorageInfo(storage), next);
+ if (next != null) {
+ next.setPrevious(next.findStorageInfo(storage), prev);
+ }
+ return this;
+ }
+
+ /**
+ * BlockInfo represents a block that is not being constructed.
+ * In order to start modifying the block, the BlockInfo should be converted
+ * to {@link BlockInfoContiguousUnderConstruction}.
+ * @return {@link HdfsServerConstants.BlockUCState#COMPLETE}
+ */
+ public HdfsServerConstants.BlockUCState getBlockUCState() {
+ return HdfsServerConstants.BlockUCState.COMPLETE;
+ }
+
+ /**
+ * Is this block complete?
+ *
+ * @return true if the state of the block is
+ * {@link HdfsServerConstants.BlockUCState#COMPLETE}
+ */
+ public boolean isComplete() {
+ return getBlockUCState().equals(HdfsServerConstants.BlockUCState.COMPLETE);
+ }
+
+ @Override
+ public int hashCode() {
+ // Super implementation is sufficient
+ return super.hashCode();
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ // Sufficient to rely on super's implementation
+ return (this == obj) || super.equals(obj);
+ }
+
+ @Override
+ public LightWeightGSet.LinkedElement getNext() {
+ return nextLinkedElement;
+ }
+
+ @Override
+ public void setNext(LightWeightGSet.LinkedElement next) {
+ this.nextLinkedElement = next;
+ }
+
+ static BlockInfo copyOf(BlockInfo b) {
+ if (b instanceof BlockInfoContiguous) {
+ return new BlockInfoContiguous((BlockInfoContiguous) b);
+ } else {
+ return new BlockInfoStriped((BlockInfoStriped) b);
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
index 48069c1..e54cba3 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java
@@ -17,148 +17,33 @@
*/
package org.apache.hadoop.hdfs.server.blockmanagement;
-import java.util.LinkedList;
-
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.server.common.HdfsServerConstants.BlockUCState;
-import org.apache.hadoop.util.LightWeightGSet;
/**
- * BlockInfo class maintains for a given block
- * the {@link BlockCollection} it is part of and datanodes where the replicas of
- * the block are stored.
+ * Subclass of {@link BlockInfo}, used for a block with replication scheme.
*/
@InterfaceAudience.Private
-public class BlockInfoContiguous extends Block
- implements LightWeightGSet.LinkedElement {
+public class BlockInfoContiguous extends BlockInfo {
public static final BlockInfoContiguous[] EMPTY_ARRAY = {};
- private BlockCollection bc;
-
- /** For implementing {@link LightWeightGSet.LinkedElement} interface */
- private LightWeightGSet.LinkedElement nextLinkedElement;
-
- /**
- * This array contains triplets of references. For each i-th storage, the
- * block belongs to triplets[3*i] is the reference to the
- * {@link DatanodeStorageInfo} and triplets[3*i+1] and triplets[3*i+2] are
- * references to the previous and the next blocks, respectively, in the list
- * of blocks belonging to this storage.
- *
- * Using previous and next in Object triplets is done instead of a
- * {@link LinkedList} list to efficiently use memory. With LinkedList the cost
- * per replica is 42 bytes (LinkedList#Entry object per replica) versus 16
- * bytes using the triplets.
- */
- private Object[] triplets;
-
- /**
- * Construct an entry for blocksmap
- * @param replication the block's replication factor
- */
- public BlockInfoContiguous(short replication) {
- this.triplets = new Object[3*replication];
- this.bc = null;
+ public BlockInfoContiguous(short size) {
+ super(size);
}
-
- public BlockInfoContiguous(Block blk, short replication) {
- super(blk);
- this.triplets = new Object[3*replication];
- this.bc = null;
+
+ public BlockInfoContiguous(Block blk, short size) {
+ super(blk, size);
}
/**
* Copy construction.
- * This is used to convert BlockInfoUnderConstruction
- * @param from BlockInfo to copy from.
+ * This is used to convert BlockReplicationInfoUnderConstruction
+ * @param from BlockReplicationInfo to copy from.
*/
protected BlockInfoContiguous(BlockInfoContiguous from) {
- this(from, from.bc.getBlockReplication());
- this.bc = from.bc;
- }
-
- public BlockCollection getBlockCollection() {
- return bc;
- }
-
- public void setBlockCollection(BlockCollection bc) {
- this.bc = bc;
- }
-
- public DatanodeDescriptor getDatanode(int index) {
- DatanodeStorageInfo storage = getStorageInfo(index);
- return storage == null ? null : storage.getDatanodeDescriptor();
- }
-
- DatanodeStorageInfo getStorageInfo(int index) {
- assert this.triplets != null : "BlockInfo is not initialized";
- assert index >= 0 && index*3 < triplets.length : "Index is out of bound";
- return (DatanodeStorageInfo)triplets[index*3];
- }
-
- private BlockInfoContiguous getPrevious(int index) {
- assert this.triplets != null : "BlockInfo is not initialized";
- assert index >= 0 && index*3+1 < triplets.length : "Index is out of bound";
- BlockInfoContiguous info = (BlockInfoContiguous)triplets[index*3+1];
- assert info == null ||
- info.getClass().getName().startsWith(BlockInfoContiguous.class.getName()) :
- "BlockInfo is expected at " + index*3;
- return info;
- }
-
- BlockInfoContiguous getNext(int index) {
- assert this.triplets != null : "BlockInfo is not initialized";
- assert index >= 0 && index*3+2 < triplets.length : "Index is out of bound";
- BlockInfoContiguous info = (BlockInfoContiguous)triplets[index*3+2];
- assert info == null || info.getClass().getName().startsWith(
- BlockInfoContiguous.class.getName()) :
- "BlockInfo is expected at " + index*3;
- return info;
- }
-
- private void setStorageInfo(int index, DatanodeStorageInfo storage) {
- assert this.triplets != null : "BlockInfo is not initialized";
- assert index >= 0 && index*3 < triplets.length : "Index is out of bound";
- triplets[index*3] = storage;
- }
-
- /**
- * Return the previous block on the block list for the datanode at
- * position index. Set the previous block on the list to "to".
- *
- * @param index - the datanode index
- * @param to - block to be set to previous on the list of blocks
- * @return current previous block on the list of blocks
- */
- private BlockInfoContiguous setPrevious(int index, BlockInfoContiguous to) {
- assert this.triplets != null : "BlockInfo is not initialized";
- assert index >= 0 && index*3+1 < triplets.length : "Index is out of bound";
- BlockInfoContiguous info = (BlockInfoContiguous)triplets[index*3+1];
- triplets[index*3+1] = to;
- return info;
- }
-
- /**
- * Return the next block on the block list for the datanode at
- * position index. Set the next block on the list to "to".
- *
- * @param index - the datanode index
- * @param to - block to be set to next on the list of blocks
- * * @return current next block on the list of blocks
- */
- private BlockInfoContiguous setNext(int index, BlockInfoContiguous to) {
- assert this.triplets != null : "BlockInfo is not initialized";
- assert index >= 0 && index*3+2 < triplets.length : "Index is out of bound";
- BlockInfoContiguous info = (BlockInfoContiguous)triplets[index*3+2];
- triplets[index*3+2] = to;
- return info;
- }
-
- public int getCapacity() {
- assert this.triplets != null : "BlockInfo is not initialized";
- assert triplets.length % 3 == 0 : "Malformed BlockInfo";
- return triplets.length / 3;
+ this(from, from.getBlockCollection().getBlockReplication());
+ this.setBlockCollection(from.getBlockCollection());
}
/**
@@ -168,9 +53,10 @@ public class BlockInfoContiguous extends Block
private int ensureCapacity(int num) {
assert this.triplets != null : "BlockInfo is not initialized";
int last = numNodes();
- if(triplets.length >= (last+num)*3)
+ if (triplets.length >= (last+num)*3) {
return last;
- /* Not enough space left. Create a new array. Should normally
+ }
+ /* Not enough space left. Create a new array. Should normally
* happen only when replication is manually increased by the user. */
Object[] old = triplets;
triplets = new Object[(last+num)*3];
@@ -178,23 +64,8 @@ public class BlockInfoContiguous extends Block
return last;
}
- /**
- * Count the number of data-nodes the block belongs to.
- */
- public int numNodes() {
- assert this.triplets != null : "BlockInfo is not initialized";
- assert triplets.length % 3 == 0 : "Malformed BlockInfo";
- for(int idx = getCapacity()-1; idx >= 0; idx--) {
- if(getDatanode(idx) != null)
- return idx+1;
- }
- return 0;
- }
-
- /**
- * Add a {@link DatanodeStorageInfo} location for a block
- */
- boolean addStorage(DatanodeStorageInfo storage) {
+ @Override
+ boolean addStorage(DatanodeStorageInfo storage, Block reportedBlock) {
// find the last null node
int lastNode = ensureCapacity(1);
setStorageInfo(lastNode, storage);
@@ -203,167 +74,53 @@ public class BlockInfoContiguous extends Block
return true;
}
- /**
- * Remove {@link DatanodeStorageInfo} location for a block
- */
+ @Override
boolean removeStorage(DatanodeStorageInfo storage) {
int dnIndex = findStorageInfo(storage);
- if(dnIndex < 0) // the node is not found
+ if (dnIndex < 0) { // the node is not found
return false;
- assert getPrevious(dnIndex) == null && getNext(dnIndex) == null :
- "Block is still in the list and must be removed first.";
+ }
+ assert getPrevious(dnIndex) == null && getNext(dnIndex) == null :
+ "Block is still in the list and must be removed first.";
// find the last not null node
- int lastNode = numNodes()-1;
- // replace current node triplet by the lastNode one
+ int lastNode = numNodes()-1;
+ // replace current node triplet by the lastNode one
setStorageInfo(dnIndex, getStorageInfo(lastNode));
- setNext(dnIndex, getNext(lastNode));
- setPrevious(dnIndex, getPrevious(lastNode));
+ setNext(dnIndex, getNext(lastNode));
+ setPrevious(dnIndex, getPrevious(lastNode));
// set the last triplet to null
setStorageInfo(lastNode, null);
- setNext(lastNode, null);
- setPrevious(lastNode, null);
+ setNext(lastNode, null);
+ setPrevious(lastNode, null);
return true;
}
- /**
- * Find specified DatanodeDescriptor.
- * @return index or -1 if not found.
- */
- boolean findDatanode(DatanodeDescriptor dn) {
- int len = getCapacity();
- for(int idx = 0; idx < len; idx++) {
- DatanodeDescriptor cur = getDatanode(idx);
- if(cur == dn) {
- return true;
- }
- if(cur == null) {
- break;
- }
- }
- return false;
- }
+ @Override
+ public int numNodes() {
+ assert this.triplets != null : "BlockInfo is not initialized";
+ assert triplets.length % 3 == 0 : "Malformed BlockInfo";
- /**
- * Find specified DatanodeStorageInfo.
- * @return DatanodeStorageInfo or null if not found.
- */
- DatanodeStorageInfo findStorageInfo(DatanodeDescriptor dn) {
- int len = getCapacity();
- for(int idx = 0; idx < len; idx++) {
- DatanodeStorageInfo cur = getStorageInfo(idx);
- if(cur == null)
- break;
- if(cur.getDatanodeDescriptor() == dn)
- return cur;
- }
- return null;
- }
-
- /**
- * Find specified DatanodeStorageInfo.
- * @return index or -1 if not found.
- */
- int findStorageInfo(DatanodeStorageInfo storageInfo) {
- int len = getCapacity();
- for(int idx = 0; idx < len; idx++) {
- DatanodeStorageInfo cur = getStorageInfo(idx);
- if (cur == storageInfo) {
- return idx;
- }
- if (cur == null) {
- break;
+ for (int idx = getCapacity()-1; idx >= 0; idx--) {
+ if (getDatanode(idx) != null) {
+ return idx + 1;
}
}
- return -1;
- }
-
- /**
- * Insert this block into the head of the list of blocks
- * related to the specified DatanodeStorageInfo.
- * If the head is null then form a new list.
- * @return current block as the new head of the list.
- */
- BlockInfoContiguous listInsert(BlockInfoContiguous head,
- DatanodeStorageInfo storage) {
- int dnIndex = this.findStorageInfo(storage);
- assert dnIndex >= 0 : "Data node is not found: current";
- assert getPrevious(dnIndex) == null && getNext(dnIndex) == null :
- "Block is already in the list and cannot be inserted.";
- this.setPrevious(dnIndex, null);
- this.setNext(dnIndex, head);
- if(head != null)
- head.setPrevious(head.findStorageInfo(storage), this);
- return this;
- }
-
- /**
- * Remove this block from the list of blocks
- * related to the specified DatanodeStorageInfo.
- * If this block is the head of the list then return the next block as
- * the new head.
- * @return the new head of the list or null if the list becomes
- * empy after deletion.
- */
- BlockInfoContiguous listRemove(BlockInfoContiguous head,
- DatanodeStorageInfo storage) {
- if(head == null)
- return null;
- int dnIndex = this.findStorageInfo(storage);
- if(dnIndex < 0) // this block is not on the data-node list
- return head;
-
- BlockInfoContiguous next = this.getNext(dnIndex);
- BlockInfoContiguous prev = this.getPrevious(dnIndex);
- this.setNext(dnIndex, null);
- this.setPrevious(dnIndex, null);
- if(prev != null)
- prev.setNext(prev.findStorageInfo(storage), next);
- if(next != null)
- next.setPrevious(next.findStorageInfo(storage), prev);
- if(this == head) // removing the head
- head = next;
- return head;
+ return 0;
}
- /**
- * Remove this block from the list of blocks related to the specified
- * DatanodeDescriptor. Insert it into the head of the list of blocks.
- *
- * @return the new head of the list.
- */
- public BlockInfoContiguous moveBlockToHead(BlockInfoContiguous head,
- DatanodeStorageInfo storage, int curIndex, int headIndex) {
- if (head == this) {
- return this;
- }
- BlockInfoContiguous next = this.setNext(curIndex, head);
- BlockInfoContiguous prev = this.setPrevious(curIndex, null);
-
- head.setPrevious(headIndex, this);
- prev.setNext(prev.findStorageInfo(storage), next);
- if (next != null) {
- next.setPrevious(next.findStorageInfo(storage), prev);
+ @Override
+ void replaceBlock(BlockInfo newBlock) {
+ assert newBlock instanceof BlockInfoContiguous;
+ for (int i = this.numNodes() - 1; i >= 0; i--) {
+ final DatanodeStorageInfo storage = this.getStorageInfo(i);
+ final boolean removed = storage.removeBlock(this);
+ assert removed : "currentBlock not found.";
+
+ final DatanodeStorageInfo.AddBlockResult result = storage.addBlock(
+ newBlock, newBlock);
+ assert result == DatanodeStorageInfo.AddBlockResult.ADDED :
+ "newBlock already exists.";
}
- return this;
- }
-
- /**
- * BlockInfo represents a block that is not being constructed.
- * In order to start modifying the block, the BlockInfo should be converted
- * to {@link BlockInfoContiguousUnderConstruction}.
- * @return {@link BlockUCState#COMPLETE}
- */
- public BlockUCState getBlockUCState() {
- return BlockUCState.COMPLETE;
- }
-
- /**
- * Is this block complete?
- *
- * @return true if the state of the block is {@link BlockUCState#COMPLETE}
- */
- public boolean isComplete() {
- return getBlockUCState().equals(BlockUCState.COMPLETE);
}
/**
@@ -375,38 +132,16 @@ public class BlockInfoContiguous extends Block
if(isComplete()) {
BlockInfoContiguousUnderConstruction ucBlock =
new BlockInfoContiguousUnderConstruction(this,
- getBlockCollection().getBlockReplication(), s, targets);
+ getBlockCollection().getBlockReplication(), s, targets);
ucBlock.setBlockCollection(getBlockCollection());
return ucBlock;
}
// the block is already under construction
BlockInfoContiguousUnderConstruction ucBlock =
- (BlockInfoContiguousUnderConstruction)this;
+ (BlockInfoContiguousUnderConstruction) this;
ucBlock.setBlockUCState(s);
ucBlock.setExpectedLocations(targets);
ucBlock.setBlockCollection(getBlockCollection());
return ucBlock;
}
-
- @Override
- public int hashCode() {
- // Super implementation is sufficient
- return super.hashCode();
- }
-
- @Override
- public boolean equals(Object obj) {
- // Sufficient to rely on super's implementation
- return (this == obj) || super.equals(obj);
- }
-
- @Override
- public LightWeightGSet.LinkedElement getNext() {
- return nextLinkedElement;
- }
-
- @Override
- public void setNext(LightWeightGSet.LinkedElement next) {
- this.nextLinkedElement = next;
- }
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java
index 91b76cc..5db9841 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java
@@ -60,101 +60,6 @@ public class BlockInfoContiguousUnderConstruction extends BlockInfoContiguous {
private Block truncateBlock;
/**
- * ReplicaUnderConstruction contains information about replicas while
- * they are under construction.
- * The GS, the length and the state of the replica is as reported by
- * the data-node.
- * It is not guaranteed, but expected, that data-nodes actually have
- * corresponding replicas.
- */
- static class ReplicaUnderConstruction extends Block {
- private final DatanodeStorageInfo expectedLocation;
- private ReplicaState state;
- private boolean chosenAsPrimary;
-
- ReplicaUnderConstruction(Block block,
- DatanodeStorageInfo target,
- ReplicaState state) {
- super(block);
- this.expectedLocation = target;
- this.state = state;
- this.chosenAsPrimary = false;
- }
-
- /**
- * Expected block replica location as assigned when the block was allocated.
- * This defines the pipeline order.
- * It is not guaranteed, but expected, that the data-node actually has
- * the replica.
- */
- private DatanodeStorageInfo getExpectedStorageLocation() {
- return expectedLocation;
- }
-
- /**
- * Get replica state as reported by the data-node.
- */
- ReplicaState getState() {
- return state;
- }
-
- /**
- * Whether the replica was chosen for recovery.
- */
- boolean getChosenAsPrimary() {
- return chosenAsPrimary;
- }
-
- /**
- * Set replica state.
- */
- void setState(ReplicaState s) {
- state = s;
- }
-
- /**
- * Set whether this replica was chosen for recovery.
- */
- void setChosenAsPrimary(boolean chosenAsPrimary) {
- this.chosenAsPrimary = chosenAsPrimary;
- }
-
- /**
- * Is data-node the replica belongs to alive.
- */
- boolean isAlive() {
- return expectedLocation.getDatanodeDescriptor().isAlive;
- }
-
- @Override // Block
- public int hashCode() {
- return super.hashCode();
- }
-
- @Override // Block
- public boolean equals(Object obj) {
- // Sufficient to rely on super's implementation
- return (this == obj) || super.equals(obj);
- }
-
- @Override
- public String toString() {
- final StringBuilder b = new StringBuilder(50);
- appendStringTo(b);
- return b.toString();
- }
-
- @Override
- public void appendStringTo(StringBuilder sb) {
- sb.append("ReplicaUC[")
- .append(expectedLocation)
- .append("|")
- .append(state)
- .append("]");
- }
- }
-
- /**
* Create block and set its state to
* {@link BlockUCState#UNDER_CONSTRUCTION}.
*/
@@ -165,7 +70,8 @@ public class BlockInfoContiguousUnderConstruction extends BlockInfoContiguous {
/**
* Create a block that is currently being constructed.
*/
- public BlockInfoContiguousUnderConstruction(Block blk, short replication, BlockUCState state, DatanodeStorageInfo[] targets) {
+ public BlockInfoContiguousUnderConstruction(Block blk, short replication,
+ BlockUCState state, DatanodeStorageInfo[] targets) {
super(blk, replication);
assert getBlockUCState() != BlockUCState.COMPLETE :
"BlockInfoUnderConstruction cannot be in COMPLETE state";
@@ -191,10 +97,11 @@ public class BlockInfoContiguousUnderConstruction extends BlockInfoContiguous {
/** Set expected locations */
public void setExpectedLocations(DatanodeStorageInfo[] targets) {
int numLocations = targets == null ? 0 : targets.length;
- this.replicas = new ArrayList<ReplicaUnderConstruction>(numLocations);
- for(int i = 0; i < numLocations; i++)
- replicas.add(
- new ReplicaUnderConstruction(this, targets[i], ReplicaState.RBW));
+ this.replicas = new ArrayList<>(numLocations);
+ for(int i = 0; i < numLocations; i++) {
+ replicas.add(new ReplicaUnderConstruction(this, targets[i],
+ ReplicaState.RBW));
+ }
}
/**
@@ -204,8 +111,9 @@ public class BlockInfoContiguousUnderConstruction extends BlockInfoContiguous {
public DatanodeStorageInfo[] getExpectedStorageLocations() {
int numLocations = replicas == null ? 0 : replicas.size();
DatanodeStorageInfo[] storages = new DatanodeStorageInfo[numLocations];
- for(int i = 0; i < numLocations; i++)
+ for (int i = 0; i < numLocations; i++) {
storages[i] = replicas.get(i).getExpectedStorageLocation();
+ }
return storages;
}
@@ -293,17 +201,17 @@ public class BlockInfoContiguousUnderConstruction extends BlockInfoContiguous {
+ " No blocks found, lease removed.");
}
boolean allLiveReplicasTriedAsPrimary = true;
- for (int i = 0; i < replicas.size(); i++) {
+ for (ReplicaUnderConstruction replica : replicas) {
// Check if all replicas have been tried or not.
- if (replicas.get(i).isAlive()) {
- allLiveReplicasTriedAsPrimary =
- (allLiveReplicasTriedAsPrimary && replicas.get(i).getChosenAsPrimary());
+ if (replica.isAlive()) {
+ allLiveReplicasTriedAsPrimary = (allLiveReplicasTriedAsPrimary &&
+ replica.getChosenAsPrimary());
}
}
if (allLiveReplicasTriedAsPrimary) {
// Just set all the replicas to be chosen whether they are alive or not.
- for (int i = 0; i < replicas.size(); i++) {
- replicas.get(i).setChosenAsPrimary(false);
+ for (ReplicaUnderConstruction replica : replicas) {
+ replica.setChosenAsPrimary(false);
}
}
long mostRecentLastUpdate = 0;
@@ -315,7 +223,8 @@ public class BlockInfoContiguousUnderConstruction extends BlockInfoContiguous {
continue;
}
final ReplicaUnderConstruction ruc = replicas.get(i);
- final long lastUpdate = ruc.getExpectedStorageLocation().getDatanodeDescriptor().getLastUpdate();
+ final long lastUpdate = ruc.getExpectedStorageLocation()
+ .getDatanodeDescriptor().getLastUpdate();
if (lastUpdate > mostRecentLastUpdate) {
primaryNodeIndex = i;
primary = ruc;
@@ -323,7 +232,8 @@ public class BlockInfoContiguousUnderConstruction extends BlockInfoContiguous {
}
}
if (primary != null) {
- primary.getExpectedStorageLocation().getDatanodeDescriptor().addBlockToBeRecovered(this);
+ primary.getExpectedStorageLocation().getDatanodeDescriptor()
+ .addBlockToBeRecovered(this);
primary.setChosenAsPrimary(true);
NameNode.blockStateChangeLog.info(
"BLOCK* {} recovery started, primary={}", this, primary);
@@ -356,18 +266,6 @@ public class BlockInfoContiguousUnderConstruction extends BlockInfoContiguous {
replicas.add(new ReplicaUnderConstruction(block, storage, rState));
}
- @Override // BlockInfo
- // BlockInfoUnderConstruction participates in maps the same way as BlockInfo
- public int hashCode() {
- return super.hashCode();
- }
-
- @Override // BlockInfo
- public boolean equals(Object obj) {
- // Sufficient to rely on super's implementation
- return (this == obj) || super.equals(obj);
- }
-
@Override
public String toString() {
final StringBuilder b = new StringBuilder(100);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java
new file mode 100644
index 0000000..5fff41e
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java
@@ -0,0 +1,179 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.hdfs.protocol.Block;
+
+/**
+ * Subclass of {@link BlockInfo}, presenting a block group in erasure coding.
+ *
+ * We still use triplets to store DatanodeStorageInfo for each block in the
+ * block group, as well as the previous/next block in the corresponding
+ * DatanodeStorageInfo. For a (m+k) block group, the first (m+k) triplet units
+ * are sorted and strictly mapped to the corresponding block.
+ *
+ * Normally each block belonging to group is stored in only one DataNode.
+ * However, it is possible that some block is over-replicated. Thus the triplet
+ * array's size can be larger than (m+k). Thus currently we use an extra byte
+ * array to record the block index for each triplet.
+ */
+public class BlockInfoStriped extends BlockInfo {
+ private final short dataBlockNum;
+ private final short parityBlockNum;
+ /**
+ * Always the same size with triplets. Record the block index for each triplet
+ * TODO: actually this is only necessary for over-replicated block. Thus can
+ * be further optimized to save memory usage.
+ */
+ private byte[] indices;
+
+ public BlockInfoStriped(Block blk, short dataBlockNum, short parityBlockNum) {
+ super(blk, (short) (dataBlockNum + parityBlockNum));
+ indices = new byte[dataBlockNum + parityBlockNum];
+ initIndices();
+ this.dataBlockNum = dataBlockNum;
+ this.parityBlockNum = parityBlockNum;
+ }
+
+ BlockInfoStriped(BlockInfoStriped b) {
+ this(b, b.dataBlockNum, b.parityBlockNum);
+ this.setBlockCollection(b.getBlockCollection());
+ }
+
+ private short getTotalBlockNum() {
+ return (short) (dataBlockNum + parityBlockNum);
+ }
+
+ private void initIndices() {
+ for (int i = 0; i < indices.length; i++) {
+ indices[i] = -1;
+ }
+ }
+
+ private int findSlot() {
+ int i = getTotalBlockNum();
+ for (; i < getCapacity(); i++) {
+ if (getStorageInfo(i) == null) {
+ return i;
+ }
+ }
+ // need to expand the triplet size
+ ensureCapacity(i + 1, true);
+ return i;
+ }
+
+ @Override
+ boolean addStorage(DatanodeStorageInfo storage, Block reportedBlock) {
+ int blockIndex = BlockIdManager.getBlockIndex(reportedBlock);
+ int index = blockIndex;
+ DatanodeStorageInfo old = getStorageInfo(index);
+ if (old != null && !old.equals(storage)) { // over replicated
+ // check if the storage has been stored
+ int i = findStorageInfo(storage);
+ if (i == -1) {
+ index = findSlot();
+ } else {
+ return true;
+ }
+ }
+ addStorage(storage, index, blockIndex);
+ return true;
+ }
+
+ private void addStorage(DatanodeStorageInfo storage, int index,
+ int blockIndex) {
+ setStorageInfo(index, storage);
+ setNext(index, null);
+ setPrevious(index, null);
+ indices[index] = (byte) blockIndex;
+ }
+
+ private int findStorageInfoFromEnd(DatanodeStorageInfo storage) {
+ final int len = getCapacity();
+ for(int idx = len - 1; idx >= 0; idx--) {
+ DatanodeStorageInfo cur = getStorageInfo(idx);
+ if (storage.equals(cur)) {
+ return idx;
+ }
+ }
+ return -1;
+ }
+
+ @Override
+ boolean removeStorage(DatanodeStorageInfo storage) {
+ int dnIndex = findStorageInfoFromEnd(storage);
+ if (dnIndex < 0) { // the node is not found
+ return false;
+ }
+ assert getPrevious(dnIndex) == null && getNext(dnIndex) == null :
+ "Block is still in the list and must be removed first.";
+ // set the triplet to null
+ setStorageInfo(dnIndex, null);
+ setNext(dnIndex, null);
+ setPrevious(dnIndex, null);
+ indices[dnIndex] = -1;
+ return true;
+ }
+
+ private void ensureCapacity(int totalSize, boolean keepOld) {
+ if (getCapacity() < totalSize) {
+ Object[] old = triplets;
+ byte[] oldIndices = indices;
+ triplets = new Object[totalSize * 3];
+ indices = new byte[totalSize];
+ initIndices();
+
+ if (keepOld) {
+ System.arraycopy(old, 0, triplets, 0, old.length);
+ System.arraycopy(oldIndices, 0, indices, 0, oldIndices.length);
+ }
+ }
+ }
+
+ @Override
+ void replaceBlock(BlockInfo newBlock) {
+ assert newBlock instanceof BlockInfoStriped;
+ BlockInfoStriped newBlockGroup = (BlockInfoStriped) newBlock;
+ final int size = getCapacity();
+ newBlockGroup.ensureCapacity(size, false);
+ for (int i = 0; i < size; i++) {
+ final DatanodeStorageInfo storage = this.getStorageInfo(i);
+ if (storage != null) {
+ final int blockIndex = indices[i];
+ final boolean removed = storage.removeBlock(this);
+ assert removed : "currentBlock not found.";
+
+ newBlockGroup.addStorage(storage, i, blockIndex);
+ storage.insertToList(newBlockGroup);
+ }
+ }
+ }
+
+ @Override
+ public int numNodes() {
+ assert this.triplets != null : "BlockInfo is not initialized";
+ assert triplets.length % 3 == 0 : "Malformed BlockInfo";
+ int num = 0;
+ for (int idx = getCapacity()-1; idx >= 0; idx--) {
+ if (getStorageInfo(idx) != null) {
+ num++;
+ }
+ }
+ return num;
+ }
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
index f4369b6..674b3d9 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
@@ -595,8 +595,8 @@ public class BlockManager {
* of replicas reported from data-nodes.
*/
private static boolean commitBlock(
- final BlockInfoContiguousUnderConstruction block, final Block commitBlock)
- throws IOException {
+ final BlockInfoContiguousUnderConstruction block,
+ final Block commitBlock) throws IOException {
if (block.getBlockUCState() == BlockUCState.COMMITTED)
return false;
assert block.getNumBytes() <= commitBlock.getNumBytes() :
@@ -627,7 +627,7 @@ public class BlockManager {
return false; // already completed (e.g. by syncBlock)
final boolean b = commitBlock(
- (BlockInfoContiguousUnderConstruction) lastBlock, commitBlock);
+ (BlockInfoContiguousUnderConstruction)lastBlock, commitBlock);
if(countNodes(lastBlock).liveReplicas() >= minReplication)
completeBlock(bc, bc.numBlocks()-1, false);
return b;
@@ -640,15 +640,16 @@ public class BlockManager {
* @throws IOException if the block does not have at least a minimal number
* of replicas reported from data-nodes.
*/
- private BlockInfoContiguous completeBlock(final BlockCollection bc,
+ private BlockInfo completeBlock(final BlockCollection bc,
final int blkIndex, boolean force) throws IOException {
if(blkIndex < 0)
return null;
BlockInfoContiguous curBlock = bc.getBlocks()[blkIndex];
- if(curBlock.isComplete())
+ if (curBlock.isComplete())
return curBlock;
+ // TODO: support BlockInfoStripedUC
BlockInfoContiguousUnderConstruction ucBlock =
- (BlockInfoContiguousUnderConstruction) curBlock;
+ (BlockInfoContiguousUnderConstruction)curBlock;
int numNodes = ucBlock.numNodes();
if (!force && numNodes < minReplication)
throw new IOException("Cannot complete block: " +
@@ -674,13 +675,15 @@ public class BlockManager {
return blocksMap.replaceBlock(completeBlock);
}
- private BlockInfoContiguous completeBlock(final BlockCollection bc,
- final BlockInfoContiguous block, boolean force) throws IOException {
+ // TODO: support BlockInfoStrippedUC
+ private BlockInfo completeBlock(final BlockCollection bc,
+ final BlockInfo block, boolean force) throws IOException {
BlockInfoContiguous[] fileBlocks = bc.getBlocks();
- for(int idx = 0; idx < fileBlocks.length; idx++)
- if(fileBlocks[idx] == block) {
+ for (int idx = 0; idx < fileBlocks.length; idx++) {
+ if (fileBlocks[idx] == block) {
return completeBlock(bc, idx, force);
}
+ }
return block;
}
@@ -689,7 +692,7 @@ public class BlockManager {
* regardless of whether enough replicas are present. This is necessary
* when tailing edit logs as a Standby.
*/
- public BlockInfoContiguous forceCompleteBlock(final BlockCollection bc,
+ public BlockInfo forceCompleteBlock(final BlockCollection bc,
final BlockInfoContiguousUnderConstruction block) throws IOException {
block.commitBlock(block);
return completeBlock(bc, block, true);
@@ -721,8 +724,8 @@ public class BlockManager {
DatanodeStorageInfo[] targets = getStorages(oldBlock);
- BlockInfoContiguousUnderConstruction ucBlock =
- bc.setLastBlock(oldBlock, targets);
+ BlockInfoContiguousUnderConstruction ucBlock = bc.setLastBlock(oldBlock,
+ targets);
blocksMap.replaceBlock(ucBlock);
// Remove block from replication queue.
@@ -1022,7 +1025,7 @@ public class BlockManager {
if(numBlocks == 0) {
return new BlocksWithLocations(new BlockWithLocations[0]);
}
- Iterator<BlockInfoContiguous> iter = node.getBlockIterator();
+ Iterator<BlockInfo> iter = node.getBlockIterator();
int startBlock = DFSUtil.getRandom().nextInt(numBlocks); // starting from a random block
// skip blocks
for(int i=0; i<startBlock; i++) {
@@ -1030,7 +1033,7 @@ public class BlockManager {
}
List<BlockWithLocations> results = new ArrayList<BlockWithLocations>();
long totalSize = 0;
- BlockInfoContiguous curBlock;
+ BlockInfo curBlock;
while(totalSize<size && iter.hasNext()) {
curBlock = iter.next();
if(!curBlock.isComplete()) continue;
@@ -1129,7 +1132,8 @@ public class BlockManager {
public void findAndMarkBlockAsCorrupt(final ExtendedBlock blk,
final DatanodeInfo dn, String storageID, String reason) throws IOException {
assert namesystem.hasWriteLock();
- final BlockInfoContiguous storedBlock = getStoredBlock(blk.getLocalBlock());
+ final Block reportedBlock = blk.getLocalBlock();
+ final BlockInfo storedBlock = getStoredBlock(reportedBlock);
if (storedBlock == null) {
// Check if the replica is in the blockMap, if not
// ignore the request for now. This could happen when BlockScanner
@@ -1146,7 +1150,7 @@ public class BlockManager {
+ ") does not exist");
}
- markBlockAsCorrupt(new BlockToMarkCorrupt(storedBlock,
+ markBlockAsCorrupt(new BlockToMarkCorrupt(reportedBlock, storedBlock,
blk.getGenerationStamp(), reason, Reason.CORRUPTION_REPORTED),
storageID == null ? null : node.getStorageInfo(storageID),
node);
@@ -1172,7 +1176,7 @@ public class BlockManager {
// Add replica to the data-node if it is not already there
if (storageInfo != null) {
- storageInfo.addBlock(b.stored);
+ storageInfo.addBlock(b.stored, b.reportedBlock);
}
// Add this replica to corruptReplicas Map
@@ -1717,41 +1721,55 @@ public class BlockManager {
this.reportedState = reportedState;
}
}
-
+
+ private static class BlockInfoToAdd {
+ final BlockInfo stored;
+ final Block reported;
+
+ BlockInfoToAdd(BlockInfo stored, Block reported) {
+ this.stored = stored;
+ this.reported = reported;
+ }
+ }
+
/**
* BlockToMarkCorrupt is used to build the "toCorrupt" list, which is a
* list of blocks that should be considered corrupt due to a block report.
*/
private static class BlockToMarkCorrupt {
/** The corrupted block in a datanode. */
- final BlockInfoContiguous corrupted;
+ final BlockInfo corrupted;
/** The corresponding block stored in the BlockManager. */
- final BlockInfoContiguous stored;
+ final BlockInfo stored;
+ /** The block reported from a datanode */
+ final Block reportedBlock;
/** The reason to mark corrupt. */
final String reason;
/** The reason code to be stored */
final Reason reasonCode;
- BlockToMarkCorrupt(BlockInfoContiguous corrupted,
- BlockInfoContiguous stored, String reason,
- Reason reasonCode) {
+ BlockToMarkCorrupt(Block reported, BlockInfo corrupted,
+ BlockInfo stored, String reason, Reason reasonCode) {
+ Preconditions.checkNotNull(reported, "reported is null");
Preconditions.checkNotNull(corrupted, "corrupted is null");
Preconditions.checkNotNull(stored, "stored is null");
+ this.reportedBlock = reported;
this.corrupted = corrupted;
this.stored = stored;
this.reason = reason;
this.reasonCode = reasonCode;
}
- BlockToMarkCorrupt(BlockInfoContiguous stored, String reason,
+ BlockToMarkCorrupt(Block reported, BlockInfo stored, String reason,
Reason reasonCode) {
- this(stored, stored, reason, reasonCode);
+ this(reported, stored, stored, reason, reasonCode);
}
- BlockToMarkCorrupt(BlockInfoContiguous stored, long gs, String reason,
- Reason reasonCode) {
- this(new BlockInfoContiguous(stored), stored, reason, reasonCode);
+ BlockToMarkCorrupt(Block reported, BlockInfo stored, long gs,
+ String reason, Reason reasonCode) {
+ this(reported, BlockInfo.copyOf(stored), stored, reason,
+ reasonCode);
//the corrupted block in datanode has a different generation stamp
corrupted.setGenerationStamp(gs);
}
@@ -1876,7 +1894,7 @@ public class BlockManager {
break;
}
- BlockInfoContiguous bi = getStoredBlock(b);
+ BlockInfo bi = getStoredBlock(b);
if (bi == null) {
if (LOG.isDebugEnabled()) {
LOG.debug("BLOCK* rescanPostponedMisreplicatedBlocks: " +
@@ -1916,7 +1934,7 @@ public class BlockManager {
// Modify the (block-->datanode) map, according to the difference
// between the old and new block report.
//
- Collection<BlockInfoContiguous> toAdd = new LinkedList<BlockInfoContiguous>();
+ Collection<BlockInfoToAdd> toAdd = new LinkedList<>();
Collection<Block> toRemove = new TreeSet<Block>();
Collection<Block> toInvalidate = new LinkedList<Block>();
Collection<BlockToMarkCorrupt> toCorrupt = new LinkedList<BlockToMarkCorrupt>();
@@ -1933,8 +1951,9 @@ public class BlockManager {
removeStoredBlock(b, node);
}
int numBlocksLogged = 0;
- for (BlockInfoContiguous b : toAdd) {
- addStoredBlock(b, storageInfo, null, numBlocksLogged < maxNumBlocksToLog);
+ for (BlockInfoToAdd b : toAdd) {
+ addStoredBlock(b.stored, b.reported, storageInfo, null,
+ numBlocksLogged < maxNumBlocksToLog);
numBlocksLogged++;
}
if (numBlocksLogged > maxNumBlocksToLog) {
@@ -1981,7 +2000,7 @@ public class BlockManager {
continue;
}
- BlockInfoContiguous storedBlock = getStoredBlock(iblk);
+ BlockInfo storedBlock = getStoredBlock(iblk);
// If block does not belong to any file, we are done.
if (storedBlock == null) continue;
@@ -2004,7 +2023,7 @@ public class BlockManager {
// If block is under construction, add this replica to its list
if (isBlockUnderConstruction(storedBlock, ucState, reportedState)) {
- ((BlockInfoContiguousUnderConstruction)storedBlock)
+ ((BlockInfoContiguousUnderConstruction) storedBlock)
.addReplicaIfNotPresent(storageInfo, iblk, reportedState);
// OpenFileBlocks only inside snapshots also will be added to safemode
// threshold. So we need to update such blocks to safemode
@@ -2019,14 +2038,14 @@ public class BlockManager {
}
//add replica if appropriate
if (reportedState == ReplicaState.FINALIZED) {
- addStoredBlockImmediate(storedBlock, storageInfo);
+ addStoredBlockImmediate(storedBlock, iblk, storageInfo);
}
}
}
private void reportDiff(DatanodeStorageInfo storageInfo,
BlockListAsLongs newReport,
- Collection<BlockInfoContiguous> toAdd, // add to DatanodeDescriptor
+ Collection<BlockInfoToAdd> toAdd, // add to DatanodeDescriptor
Collection<Block> toRemove, // remove from DatanodeDescriptor
Collection<Block> toInvalidate, // should be removed from DN
Collection<BlockToMarkCorrupt> toCorrupt, // add to corrupt replicas list
@@ -2034,8 +2053,10 @@ public class BlockManager {
// place a delimiter in the list which separates blocks
// that have been reported from those that have not
- BlockInfoContiguous delimiter = new BlockInfoContiguous(new Block(), (short) 1);
- AddBlockResult result = storageInfo.addBlock(delimiter);
+ Block delimiterBlock = new Block();
+ BlockInfoContiguous delimiter = new BlockInfoContiguous(delimiterBlock,
+ (short) 1);
+ AddBlockResult result = storageInfo.addBlock(delimiter, delimiterBlock);
assert result == AddBlockResult.ADDED
: "Delimiting block cannot be present in the node";
int headIndex = 0; //currently the delimiter is in the head of the list
@@ -2049,7 +2070,7 @@ public class BlockManager {
while(itBR.hasNext()) {
Block iblk = itBR.next();
ReplicaState iState = itBR.getCurrentReplicaState();
- BlockInfoContiguous storedBlock = processReportedBlock(storageInfo,
+ BlockInfo storedBlock = processReportedBlock(storageInfo,
iblk, iState, toAdd, toInvalidate, toCorrupt, toUC);
// move block to the head of the list
@@ -2061,8 +2082,7 @@ public class BlockManager {
// collect blocks that have not been reported
// all of them are next to the delimiter
- Iterator<BlockInfoContiguous> it =
- storageInfo.new BlockIterator(delimiter.getNext(0));
+ Iterator<BlockInfo> it = storageInfo.new BlockIterator(delimiter.getNext(0));
while(it.hasNext())
toRemove.add(it.next());
storageInfo.removeBlock(delimiter);
@@ -2099,10 +2119,10 @@ public class BlockManager {
* @return the up-to-date stored block, if it should be kept.
* Otherwise, null.
*/
- private BlockInfoContiguous processReportedBlock(
+ private BlockInfo processReportedBlock(
final DatanodeStorageInfo storageInfo,
final Block block, final ReplicaState reportedState,
- final Collection<BlockInfoContiguous> toAdd,
+ final Collection<BlockInfoToAdd> toAdd,
final Collection<Block> toInvalidate,
final Collection<BlockToMarkCorrupt> toCorrupt,
final Collection<StatefulBlockInfo> toUC) {
@@ -2123,7 +2143,7 @@ public class BlockManager {
}
// find block by blockId
- BlockInfoContiguous storedBlock = getStoredBlock(block);
+ BlockInfo storedBlock = getStoredBlock(block);
if(storedBlock == null) {
// If blocksMap does not contain reported block id,
// the replica should be removed from the data-node.
@@ -2177,7 +2197,7 @@ public class BlockManager {
if (reportedState == ReplicaState.FINALIZED
&& (storedBlock.findStorageInfo(storageInfo) == -1 ||
corruptReplicas.isReplicaCorrupt(storedBlock, dn))) {
- toAdd.add(storedBlock);
+ toAdd.add(new BlockInfoToAdd(storedBlock, block));
}
return storedBlock;
}
@@ -2255,7 +2275,7 @@ public class BlockManager {
*/
private BlockToMarkCorrupt checkReplicaCorrupt(
Block reported, ReplicaState reportedState,
- BlockInfoContiguous storedBlock, BlockUCState ucState,
+ BlockInfo storedBlock, BlockUCState ucState,
DatanodeDescriptor dn) {
switch(reportedState) {
case FINALIZED:
@@ -2264,12 +2284,12 @@ public class BlockManager {
case COMMITTED:
if (storedBlock.getGenerationStamp() != reported.getGenerationStamp()) {
final long reportedGS = reported.getGenerationStamp();
- return new BlockToMarkCorrupt(storedBlock, reportedGS,
+ return new BlockToMarkCorrupt(reported, storedBlock, reportedGS,
"block is " + ucState + " and reported genstamp " + reportedGS
+ " does not match genstamp in block map "
+ storedBlock.getGenerationStamp(), Reason.GENSTAMP_MISMATCH);
} else if (storedBlock.getNumBytes() != reported.getNumBytes()) {
- return new BlockToMarkCorrupt(storedBlock,
+ return new BlockToMarkCorrupt(reported, storedBlock,
"block is " + ucState + " and reported length " +
reported.getNumBytes() + " does not match " +
"length in block map " + storedBlock.getNumBytes(),
@@ -2280,8 +2300,8 @@ public class BlockManager {
case UNDER_CONSTRUCTION:
if (storedBlock.getGenerationStamp() > reported.getGenerationStamp()) {
final long reportedGS = reported.getGenerationStamp();
- return new BlockToMarkCorrupt(storedBlock, reportedGS, "block is "
- + ucState + " and reported state " + reportedState
+ return new BlockToMarkCorrupt(reported, storedBlock, reportedGS,
+ "block is " + ucState + " and reported state " + reportedState
+ ", But reported genstamp " + reportedGS
+ " does not match genstamp in block map "
+ storedBlock.getGenerationStamp(), Reason.GENSTAMP_MISMATCH);
@@ -2296,7 +2316,7 @@ public class BlockManager {
return null; // not corrupt
} else if (storedBlock.getGenerationStamp() != reported.getGenerationStamp()) {
final long reportedGS = reported.getGenerationStamp();
- return new BlockToMarkCorrupt(storedBlock, reportedGS,
+ return new BlockToMarkCorrupt(reported, storedBlock, reportedGS,
"reported " + reportedState + " replica with genstamp " + reportedGS
+ " does not match COMPLETE block's genstamp in block map "
+ storedBlock.getGenerationStamp(), Reason.GENSTAMP_MISMATCH);
@@ -2311,7 +2331,7 @@ public class BlockManager {
"complete with the same genstamp");
return null;
} else {
- return new BlockToMarkCorrupt(storedBlock,
+ return new BlockToMarkCorrupt(reported, storedBlock,
"reported replica has invalid state " + reportedState,
Reason.INVALID_STATE);
}
@@ -2324,11 +2344,12 @@ public class BlockManager {
" on " + dn + " size " + storedBlock.getNumBytes();
// log here at WARN level since this is really a broken HDFS invariant
LOG.warn(msg);
- return new BlockToMarkCorrupt(storedBlock, msg, Reason.INVALID_STATE);
+ return new BlockToMarkCorrupt(reported, storedBlock, msg,
+ Reason.INVALID_STATE);
}
}
- private boolean isBlockUnderConstruction(BlockInfoContiguous storedBlock,
+ private boolean isBlockUnderConstruction(BlockInfo storedBlock,
BlockUCState ucState, ReplicaState reportedState) {
switch(reportedState) {
case FINALIZED:
@@ -2357,7 +2378,7 @@ public class BlockManager {
if (ucBlock.reportedState == ReplicaState.FINALIZED &&
!block.findDatanode(storageInfo.getDatanodeDescriptor())) {
- addStoredBlock(block, storageInfo, null, true);
+ addStoredBlock(block, ucBlock.reportedBlock, storageInfo, null, true);
}
}
@@ -2372,18 +2393,18 @@ public class BlockManager {
*
* @throws IOException
*/
- private void addStoredBlockImmediate(BlockInfoContiguous storedBlock,
+ private void addStoredBlockImmediate(BlockInfo storedBlock, Block reported,
DatanodeStorageInfo storageInfo)
throws IOException {
assert (storedBlock != null && namesystem.hasWriteLock());
if (!namesystem.isInStartupSafeMode()
|| namesystem.isPopulatingReplQueues()) {
- addStoredBlock(storedBlock, storageInfo, null, false);
+ addStoredBlock(storedBlock, reported, storageInfo, null, false);
return;
}
// just add it
- storageInfo.addBlock(storedBlock);
+ storageInfo.addBlock(storedBlock, reported);
// Now check for completion of blocks and safe block count
int numCurrentReplica = countLiveNodes(storedBlock);
@@ -2404,13 +2425,14 @@ public class BlockManager {
* needed replications if this takes care of the problem.
* @return the block that is stored in blockMap.
*/
- private Block addStoredBlock(final BlockInfoContiguous block,
+ private Block addStoredBlock(final BlockInfo block,
+ final Block reportedBlock,
DatanodeStorageInfo storageInfo,
DatanodeDescriptor delNodeHint,
boolean logEveryBlock)
throws IOException {
assert block != null && namesystem.hasWriteLock();
- BlockInfoContiguous storedBlock;
+ BlockInfo storedBlock;
DatanodeDescriptor node = storageInfo.getDatanodeDescriptor();
if (block instanceof BlockInfoContiguousUnderConstruction) {
//refresh our copy in case the block got completed in another thread
@@ -2431,7 +2453,7 @@ public class BlockManager {
assert bc != null : "Block must belong to a file";
// add block to the datanode
- AddBlockResult result = storageInfo.addBlock(storedBlock);
+ AddBlockResult result = storageInfo.addBlock(storedBlock, reportedBlock);
int curReplicaDelta;
if (result == AddBlockResult.ADDED) {
@@ -2506,13 +2528,13 @@ public class BlockManager {
storedBlock + "blockMap has " + numCorruptNodes +
" but corrupt replicas map has " + corruptReplicasCount);
}
- if ((corruptReplicasCount > 0) && (numLiveReplicas >= fileReplication))
- invalidateCorruptReplicas(storedBlock);
+ if ((corruptReplicasCount > 0) && (numLiveReplicas >= fileReplication)) {
+ invalidateCorruptReplicas(storedBlock, reportedBlock);
+ }
return storedBlock;
}
- private void logAddStoredBlock(BlockInfoContiguous storedBlock,
- DatanodeDescriptor node) {
+ private void logAddStoredBlock(BlockInfo storedBlock, DatanodeDescriptor node) {
if (!blockLog.isInfoEnabled()) {
return;
}
@@ -2539,7 +2561,7 @@ public class BlockManager {
*
* @param blk Block whose corrupt replicas need to be invalidated
*/
- private void invalidateCorruptReplicas(BlockInfoContiguous blk) {
+ private void invalidateCorruptReplicas(BlockInfo blk, Block reported) {
Collection<DatanodeDescriptor> nodes = corruptReplicas.getNodes(blk);
boolean removedFromBlocksMap = true;
if (nodes == null)
@@ -2549,7 +2571,7 @@ public class BlockManager {
DatanodeDescriptor[] nodesCopy = nodes.toArray(new DatanodeDescriptor[0]);
for (DatanodeDescriptor node : nodesCopy) {
try {
- if (!invalidateBlock(new BlockToMarkCorrupt(blk, null,
+ if (!invalidateBlock(new BlockToMarkCorrupt(reported, blk, null,
Reason.ANY), node)) {
removedFromBlocksMap = false;
}
@@ -2618,7 +2640,7 @@ public class BlockManager {
long nrInvalid = 0, nrOverReplicated = 0;
long nrUnderReplicated = 0, nrPostponed = 0, nrUnderConstruction = 0;
long startTimeMisReplicatedScan = Time.now();
- Iterator<BlockInfoContiguous> blocksItr = blocksMap.getBlocks().iterator();
+ Iterator<BlockInfo> blocksItr = blocksMap.getBlocks().iterator();
long totalBlocks = blocksMap.size();
replicationQueuesInitProgress = 0;
long totalProcessed = 0;
@@ -2630,7 +2652,7 @@ public class BlockManager {
namesystem.writeLockInterruptibly();
try {
while (processed < numBlocksPerIteration && blocksItr.hasNext()) {
- BlockInfoContiguous block = blocksItr.next();
+ BlockInfo block = blocksItr.next();
MisReplicationResult res = processMisReplicatedBlock(block);
if (LOG.isTraceEnabled()) {
LOG.trace("block " + block + ": " + res);
@@ -2704,7 +2726,7 @@ public class BlockManager {
* appropriate queues if necessary, and returns a result code indicating
* what happened with it.
*/
- private MisReplicationResult processMisReplicatedBlock(BlockInfoContiguous block) {
+ private MisReplicationResult processMisReplicatedBlock(BlockInfo block) {
BlockCollection bc = block.getBlockCollection();
if (bc == null) {
// block does not belong to any file
@@ -3033,14 +3055,14 @@ public class BlockManager {
ReplicaState reportedState, DatanodeDescriptor delHintNode)
throws IOException {
// blockReceived reports a finalized block
- Collection<BlockInfoContiguous> toAdd = new LinkedList<BlockInfoContiguous>();
+ Collection<BlockInfoToAdd> toAdd = new LinkedList<>();
Collection<Block> toInvalidate = new LinkedList<Block>();
Collection<BlockToMarkCorrupt> toCorrupt = new LinkedList<BlockToMarkCorrupt>();
Collection<StatefulBlockInfo> toUC = new LinkedList<StatefulBlockInfo>();
final DatanodeDescriptor node = storageInfo.getDatanodeDescriptor();
- processReportedBlock(storageInfo, block, reportedState,
- toAdd, toInvalidate, toCorrupt, toUC);
+ processReportedBlock(storageInfo, block, reportedState, toAdd, toInvalidate,
+ toCorrupt, toUC);
// the block is only in one of the to-do lists
// if it is in none then data-node already has it
assert toUC.size() + toAdd.size() + toInvalidate.size() + toCorrupt.size() <= 1
@@ -3050,8 +3072,9 @@ public class BlockManager {
addStoredBlockUnderConstruction(b, storageInfo);
}
long numBlocksLogged = 0;
- for (BlockInfoContiguous b : toAdd) {
- addStoredBlock(b, storageInfo, delHintNode, numBlocksLogged < maxNumBlocksToLog);
+ for (BlockInfoToAdd b : toAdd) {
+ addStoredBlock(b.stored, b.reported, storageInfo, delHintNode,
+ numBlocksLogged < maxNumBlocksToLog);
numBlocksLogged++;
}
if (numBlocksLogged > maxNumBlocksToLog) {
@@ -3174,7 +3197,7 @@ public class BlockManager {
* @param b - the block being tested
* @return count of live nodes for this block
*/
- int countLiveNodes(BlockInfoContiguous b) {
+ int countLiveNodes(BlockInfo b) {
if (!namesystem.isInStartupSafeMode()) {
return countNodes(b).liveReplicas();
}
@@ -3329,7 +3352,7 @@ public class BlockManager {
return blocksMap.size();
}
- public DatanodeStorageInfo[] getStorages(BlockInfoContiguous block) {
+ public DatanodeStorageInfo[] getStorages(BlockInfo block) {
final DatanodeStorageInfo[] storages = new DatanodeStorageInfo[block.numNodes()];
int i = 0;
for(DatanodeStorageInfo s : blocksMap.getStorages(block)) {
@@ -3359,8 +3382,8 @@ public class BlockManager {
}
}
- public BlockInfoContiguous getStoredBlock(Block block) {
- BlockInfoContiguous info = null;
+ public BlockInfo getStoredBlock(Block block) {
+ BlockInfo info = null;
if (BlockIdManager.isStripedBlockID(block.getBlockId())) {
info = blocksMap.getStoredBlock(
new Block(BlockIdManager.convertToGroupID(block.getBlockId())));
@@ -3517,7 +3540,8 @@ public class BlockManager {
public BlockInfoContiguous addBlockCollection(BlockInfoContiguous block,
BlockCollection bc) {
- return blocksMap.addBlockCollection(block, bc);
+ // TODO
+ return (BlockInfoContiguous) blocksMap.addBlockCollection(block, bc);
}
public BlockCollection getBlockCollection(Block b) {
@@ -3725,7 +3749,7 @@ public class BlockManager {
/**
* A simple result enum for the result of
- * {@link BlockManager#processMisReplicatedBlock(BlockInfoContiguous)}.
+ * {@link BlockManager#processMisReplicatedBlock}.
*/
enum MisReplicationResult {
/** The block should be invalidated since it belongs to a deleted file. */
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
index 806a4cb..d383de8 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
@@ -20,12 +20,10 @@ package org.apache.hadoop.hdfs.server.blockmanagement;
import java.util.Iterator;
import org.apache.hadoop.hdfs.protocol.Block;
-import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo.AddBlockResult;
import org.apache.hadoop.hdfs.server.protocol.DatanodeStorage;
import org.apache.hadoop.util.GSet;
import org.apache.hadoop.util.LightWeightGSet;
-import com.google.common.base.Preconditions;
import com.google.common.base.Predicate;
import com.google.common.collect.Iterables;
@@ -36,10 +34,10 @@ import com.google.common.collect.Iterables;
*/
class BlocksMap {
private static class StorageIterator implements Iterator<DatanodeStorageInfo> {
- private final BlockInfoContiguous blockInfo;
+ private final BlockInfo blockInfo;
private int nextIdx = 0;
- StorageIterator(BlockInfoContiguous blkInfo) {
+ StorageIterator(BlockInfo blkInfo) {
this.blockInfo = blkInfo;
}
@@ -63,14 +61,14 @@ class BlocksMap {
/** Constant {@link LightWeightGSet} capacity. */
private final int capacity;
- private GSet<Block, BlockInfoContiguous> blocks;
+ private GSet<Block, BlockInfo> blocks;
BlocksMap(int capacity) {
// Use 2% of total memory to size the GSet capacity
this.capacity = capacity;
- this.blocks = new LightWeightGSet<Block, BlockInfoContiguous>(capacity) {
+ this.blocks = new LightWeightGSet<Block, BlockInfo>(capacity) {
@Override
- public Iterator<BlockInfoContiguous> iterator() {
+ public Iterator<BlockInfo> iterator() {
SetIterator iterator = new SetIterator();
/*
* Not tracking any modifications to set. As this set will be used
@@ -97,15 +95,15 @@ class BlocksMap {
}
BlockCollection getBlockCollection(Block b) {
- BlockInfoContiguous info = blocks.get(b);
+ BlockInfo info = blocks.get(b);
return (info != null) ? info.getBlockCollection() : null;
}
/**
* Add block b belonging to the specified block collection to the map.
*/
- BlockInfoContiguous addBlockCollection(BlockInfoContiguous b, BlockCollection bc) {
- BlockInfoContiguous info = blocks.get(b);
+ BlockInfo addBlockCollection(BlockInfo b, BlockCollection bc) {
+ BlockInfo info = blocks.get(b);
if (info != b) {
info = b;
blocks.put(info);
@@ -120,11 +118,12 @@ class BlocksMap {
* and remove all data-node locations associated with the block.
*/
void removeBlock(Block block) {
- BlockInfoContiguous blockInfo = blocks.remove(block);
+ BlockInfo blockInfo = blocks.remove(block);
if (blockInfo == null)
return;
blockInfo.setBlockCollection(null);
+ // TODO: fix this logic for block group
for(int idx = blockInfo.numNodes()-1; idx >= 0; idx--) {
DatanodeDescriptor dn = blockInfo.getDatanode(idx);
dn.removeBlock(blockInfo); // remove from the list and wipe the location
@@ -132,7 +131,7 @@ class BlocksMap {
}
/** Returns the block object it it exists in the map. */
- BlockInfoContiguous getStoredBlock(Block b) {
+ BlockInfo getStoredBlock(Block b) {
return blocks.get(b);
}
@@ -164,7 +163,7 @@ class BlocksMap {
* For a block that has already been retrieved from the BlocksMap
* returns {@link Iterable} of the storages the block belongs to.
*/
- Iterable<DatanodeStorageInfo> getStorages(final BlockInfoContiguous storedBlock) {
+ Iterable<DatanodeStorageInfo> getStorages(final BlockInfo storedBlock) {
return new Iterable<DatanodeStorageInfo>() {
@Override
public Iterator<DatanodeStorageInfo> iterator() {
@@ -175,7 +174,7 @@ class BlocksMap {
/** counts number of containing nodes. Better than using iterator. */
int numNodes(Block b) {
- BlockInfoContiguous info = blocks.get(b);
+ BlockInfo info = blocks.get(b);
return info == null ? 0 : info.numNodes();
}
@@ -185,7 +184,7 @@ class BlocksMap {
* only if it does not belong to any file and data-nodes.
*/
boolean removeNode(Block b, DatanodeDescriptor node) {
- BlockInfoContiguous info = blocks.get(b);
+ BlockInfo info = blocks.get(b);
if (info == null)
return false;
@@ -203,7 +202,7 @@ class BlocksMap {
return blocks.size();
}
- Iterable<BlockInfoContiguous> getBlocks() {
+ Iterable<BlockInfo> getBlocks() {
return blocks;
}
@@ -218,20 +217,11 @@ class BlocksMap {
* @param newBlock - block for replacement
* @return new block
*/
- BlockInfoContiguous replaceBlock(BlockInfoContiguous newBlock) {
- BlockInfoContiguous currentBlock = blocks.get(newBlock);
+ BlockInfo replaceBlock(BlockInfo newBlock) {
+ BlockInfo currentBlock = blocks.get(newBlock);
assert currentBlock != null : "the block if not in blocksMap";
// replace block in data-node lists
- for (int i = currentBlock.numNodes() - 1; i >= 0; i--) {
- final DatanodeDescriptor dn = currentBlock.getDatanode(i);
- final DatanodeStorageInfo storage = currentBlock.findStorageInfo(dn);
- final boolean removed = storage.removeBlock(currentBlock);
- Preconditions.checkState(removed, "currentBlock not found.");
-
- final AddBlockResult result = storage.addBlock(newBlock);
- Preconditions.checkState(result == AddBlockResult.ADDED,
- "newBlock already exists.");
- }
+ currentBlock.replaceBlock(newBlock);
// replace block in the map itself
blocks.put(newBlock);
return newBlock;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java
index bf5ece9..79d7713 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CacheReplicationMonitor.java
@@ -513,8 +513,7 @@ public class CacheReplicationMonitor extends Thread implements Closeable {
iter.remove();
}
}
- BlockInfoContiguous blockInfo = blockManager.
- getStoredBlock(new Block(cblock.getBlockId()));
+ BlockInfoContiguous blockInfo = namesystem.getStoredBlock(new Block(cblock.getBlockId()));
String reason = findReasonForNotCaching(cblock, blockInfo);
int neededCached = 0;
if (reason != null) {
@@ -628,8 +627,7 @@ public class CacheReplicationMonitor extends Thread implements Closeable {
List<DatanodeDescriptor> pendingCached) {
// To figure out which replicas can be cached, we consult the
// blocksMap. We don't want to try to cache a corrupt replica, though.
- BlockInfoContiguous blockInfo = blockManager.
- getStoredBlock(new Block(cachedBlock.getBlockId()));
+ BlockInfoContiguous blockInfo = namesystem.getStoredBlock(new Block(cachedBlock.getBlockId()));
if (blockInfo == null) {
LOG.debug("Block {}: can't add new cached replicas," +
" because there is no record of this block " +
@@ -668,7 +666,7 @@ public class CacheReplicationMonitor extends Thread implements Closeable {
while (it.hasNext()) {
CachedBlock cBlock = it.next();
BlockInfoContiguous info =
- blockManager.getStoredBlock(new Block(cBlock.getBlockId()));
+ namesystem.getStoredBlock(new Block(cBlock.getBlockId()));
if (info != null) {
pendingBytes -= info.getNumBytes();
}
@@ -678,7 +676,7 @@ public class CacheReplicationMonitor extends Thread implements Closeable {
while (it.hasNext()) {
CachedBlock cBlock = it.next();
BlockInfoContiguous info =
- blockManager.getStoredBlock(new Block(cBlock.getBlockId()));
+ namesystem.getStoredBlock(new Block(cBlock.getBlockId()));
if (info != null) {
pendingBytes += info.getNumBytes();
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
index c0a17b1..a54d46b 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
@@ -286,7 +286,7 @@ public class DatanodeDescriptor extends DatanodeInfo {
* Remove block from the list of blocks belonging to the data-node. Remove
* data-node from the block.
*/
- boolean removeBlock(BlockInfoContiguous b) {
+ boolean removeBlock(BlockInfo b) {
final DatanodeStorageInfo s = b.findStorageInfo(this);
// if block exists on this datanode
if (s != null) {
@@ -299,12 +299,9 @@ public class DatanodeDescriptor extends DatanodeInfo {
* Remove block from the list of blocks belonging to the data-node. Remove
* data-node from the block.
*/
- boolean removeBlock(String storageID, BlockInfoContiguous b) {
+ boolean removeBlock(String storageID, BlockInfo b) {
DatanodeStorageInfo s = getStorageInfo(storageID);
- if (s != null) {
- return s.removeBlock(b);
- }
- return false;
+ return s != null && s.removeBlock(b);
}
public void resetBlocks() {
@@ -482,12 +479,12 @@ public class DatanodeDescriptor extends DatanodeInfo {
}
}
- private static class BlockIterator implements Iterator<BlockInfoContiguous> {
+ private static class BlockIterator implements Iterator<BlockInfo> {
private int index = 0;
- private final List<Iterator<BlockInfoContiguous>> iterators;
+ private final List<Iterator<BlockInfo>> iterators;
private BlockIterator(final DatanodeStorageInfo... storages) {
- List<Iterator<BlockInfoContiguous>> iterators = new ArrayList<Iterator<BlockInfoContiguous>>();
+ List<Iterator<BlockInfo>> iterators = new ArrayList<>();
for (DatanodeStorageInfo e : storages) {
iterators.add(e.getBlockIterator());
}
@@ -501,7 +498,7 @@ public class DatanodeDescriptor extends DatanodeInfo {
}
@Override
- public BlockInfoContiguous next() {
+ public BlockInfo next() {
update();
return iterators.get(index).next();
}
@@ -518,10 +515,11 @@ public class DatanodeDescriptor extends DatanodeInfo {
}
}
- Iterator<BlockInfoContiguous> getBlockIterator() {
+ Iterator<BlockInfo> getBlockIterator() {
return new BlockIterator(getStorageInfos());
}
- Iterator<BlockInfoContiguous> getBlockIterator(final String storageID) {
+
+ Iterator<BlockInfo> getBlockIterator(final String storageID) {
return new BlockIterator(getStorageInfo(storageID));
}
[19/52] [abbrv] hadoop git commit: MAPREDUCE-6228. Add truncate
operation to SLive. Constributed by Plamen Jeliazkov.
Posted by zh...@apache.org.
MAPREDUCE-6228. Add truncate operation to SLive. Constributed by Plamen Jeliazkov.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/a19820f2
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/a19820f2
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/a19820f2
Branch: refs/heads/HDFS-7285
Commit: a19820f2fb2000a789a114f8ed55cb7e071723c8
Parents: 946456c
Author: Plamen Jeliazkov <pl...@gmail.com>
Authored: Thu Feb 19 00:02:49 2015 -0800
Committer: Konstantin V Shvachko <sh...@apache.org>
Committed: Thu Feb 19 00:02:49 2015 -0800
----------------------------------------------------------------------
hadoop-mapreduce-project/CHANGES.txt | 6 +-
.../apache/hadoop/fs/slive/ArgumentParser.java | 2 +
.../apache/hadoop/fs/slive/ConfigExtractor.java | 59 ++++++++++
.../apache/hadoop/fs/slive/ConfigMerger.java | 35 ++++++
.../apache/hadoop/fs/slive/ConfigOption.java | 9 ++
.../org/apache/hadoop/fs/slive/Constants.java | 4 +-
.../hadoop/fs/slive/OperationFactory.java | 3 +
.../org/apache/hadoop/fs/slive/TestSlive.java | 27 +++++
.../org/apache/hadoop/fs/slive/TruncateOp.java | 114 +++++++++++++++++++
9 files changed, 255 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a19820f2/hadoop-mapreduce-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/CHANGES.txt b/hadoop-mapreduce-project/CHANGES.txt
index 7f4c3e7..fd40439 100644
--- a/hadoop-mapreduce-project/CHANGES.txt
+++ b/hadoop-mapreduce-project/CHANGES.txt
@@ -248,6 +248,10 @@ Release 2.7.0 - UNRELEASED
NEW FEATURES
+ MAPREDUCE-6227. DFSIO for truncate. (shv via yliu)
+
+ MAPREDUCE-6228. Add truncate operation to SLive. (Plamen Jeliazkov via shv)
+
IMPROVEMENTS
MAPREDUCE-6149. Document override log4j.properties in MR job.
@@ -284,8 +288,6 @@ Release 2.7.0 - UNRELEASED
MAPREDUCE-5800. Use Job#getInstance instead of deprecated constructors
(aajisaka)
- MAPREDUCE-6227. DFSIO for truncate. (shv via yliu)
-
MAPREDUCE-6253. Update use of Iterator to Iterable. (Ray Chiang via devaraj)
MAPREDUCE-5335. Rename Job Tracker terminology in ShuffleSchedulerImpl.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a19820f2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ArgumentParser.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ArgumentParser.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ArgumentParser.java
index 19a55ff..12df4dc 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ArgumentParser.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ArgumentParser.java
@@ -144,6 +144,7 @@ class ArgumentParser {
cliopt.addOption(ConfigOption.DURATION);
cliopt.addOption(ConfigOption.EXIT_ON_ERROR);
cliopt.addOption(ConfigOption.SLEEP_TIME);
+ cliopt.addOption(ConfigOption.TRUNCATE_WAIT);
cliopt.addOption(ConfigOption.FILES);
cliopt.addOption(ConfigOption.DIR_SIZE);
cliopt.addOption(ConfigOption.BASE_DIR);
@@ -167,6 +168,7 @@ class ArgumentParser {
cliopt.addOption(ConfigOption.READ_SIZE);
cliopt.addOption(ConfigOption.WRITE_SIZE);
cliopt.addOption(ConfigOption.APPEND_SIZE);
+ cliopt.addOption(ConfigOption.TRUNCATE_SIZE);
cliopt.addOption(ConfigOption.RANDOM_SEED);
cliopt.addOption(ConfigOption.QUEUE_NAME);
cliopt.addOption(ConfigOption.HELP);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a19820f2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ConfigExtractor.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ConfigExtractor.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ConfigExtractor.java
index a03c812..ef4e436 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ConfigExtractor.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ConfigExtractor.java
@@ -131,6 +131,32 @@ class ConfigExtractor {
}
/**
+ * @return whether the mapper or reducer should wait for truncate recovery
+ */
+ boolean shouldWaitOnTruncate() {
+ return shouldWaitOnTruncate(null);
+ }
+
+ /**
+ * @param primary
+ * primary the initial string to be used for the value of this
+ * configuration option (if not provided then config and then the
+ * default are used)
+ *
+ * @return whether the mapper or reducer should wait for truncate recovery
+ */
+ boolean shouldWaitOnTruncate(String primary) {
+ String val = primary;
+ if (val == null) {
+ val = config.get(ConfigOption.EXIT_ON_ERROR.getCfgOption());
+ }
+ if (val == null) {
+ return ConfigOption.EXIT_ON_ERROR.getDefault();
+ }
+ return Boolean.parseBoolean(val);
+ }
+
+ /**
* @return the number of reducers to use
*/
Integer getReducerAmount() {
@@ -537,6 +563,24 @@ class ConfigExtractor {
* @param primary
* the initial string to be used for the value of this configuration
* option (if not provided then config and then the default are used)
+ * @return the truncate byte size range (or null if none)
+ */
+ Range<Long> getTruncateSize(String primary) {
+ return getMinMaxBytes(ConfigOption.TRUNCATE_SIZE, primary);
+ }
+
+ /**
+ * @return the truncate byte size range (or null if none) using config and
+ * default for lookup
+ */
+ Range<Long> getTruncateSize() {
+ return getTruncateSize(null);
+ }
+
+ /**
+ * @param primary
+ * the initial string to be used for the value of this configuration
+ * option (if not provided then config and then the default are used)
* @return the sleep range (or null if none)
*/
Range<Long> getSleepRange(String primary) {
@@ -600,6 +644,21 @@ class ConfigExtractor {
}
/**
+ * Returns whether the truncate range should use the block size range
+ *
+ * @return true|false
+ */
+ boolean shouldTruncateUseBlockSize() {
+ Range<Long> truncateRange = getTruncateSize();
+ if (truncateRange == null
+ || (truncateRange.getLower() == truncateRange.getUpper()
+ && (truncateRange.getUpper() == Long.MAX_VALUE))) {
+ return true;
+ }
+ return false;
+ }
+
+ /**
* Returns whether the read range should use the entire file
*
* @return true|false
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a19820f2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ConfigMerger.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ConfigMerger.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ConfigMerger.java
index b7be8d8..4bb3500 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ConfigMerger.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ConfigMerger.java
@@ -282,6 +282,18 @@ class ConfigMerger {
"Error extracting & merging exit on error value", e);
}
}
+ // overwrite the truncate wait setting
+ {
+ try {
+ boolean waitOnTruncate = extractor.shouldWaitOnTruncate(opts
+ .getValue(ConfigOption.TRUNCATE_WAIT.getOpt()));
+ base.setBoolean(ConfigOption.TRUNCATE_WAIT.getCfgOption(),
+ waitOnTruncate);
+ } catch (Exception e) {
+ throw new ConfigException(
+ "Error extracting & merging wait on truncate value", e);
+ }
+ }
// verify and set file limit and ensure > 0
{
Integer fileAm = null;
@@ -553,6 +565,29 @@ class ConfigMerger {
.set(ConfigOption.APPEND_SIZE.getCfgOption(), appendSize.toString());
}
}
+ // set the truncate size range
+ {
+ Range<Long> truncateSize = null;
+ try {
+ truncateSize = extractor.getTruncateSize(opts
+ .getValue(ConfigOption.TRUNCATE_SIZE.getOpt()));
+ } catch (Exception e) {
+ throw new ConfigException(
+ "Error extracting & merging truncate size range", e);
+ }
+ if (truncateSize != null) {
+ if (truncateSize.getLower() > truncateSize.getUpper()) {
+ throw new ConfigException(
+ "Truncate size minimum is greater than its maximum");
+ }
+ if (truncateSize.getLower() < 0) {
+ throw new ConfigException(
+ "Truncate size minimum must be greater than or equal to zero");
+ }
+ base
+ .set(ConfigOption.TRUNCATE_SIZE.getCfgOption(), truncateSize.toString());
+ }
+ }
// set the seed
{
Long seed = null;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a19820f2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ConfigOption.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ConfigOption.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ConfigOption.java
index 340473a..bd66336 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ConfigOption.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/ConfigOption.java
@@ -97,6 +97,15 @@ class ConfigOption<T> extends Option {
"Min,max for size to append (min=max=MAX_LONG=blocksize)", SLIVE_PREFIX
+ ".op.append.size", null);
+ static final ConfigOption<Boolean> TRUNCATE_WAIT = new ConfigOption<Boolean>(
+ "truncateWait", true, "Should wait for truncate recovery", SLIVE_PREFIX
+ + ".op.truncate.wait", true);
+
+ static final ConfigOption<Long> TRUNCATE_SIZE = new ConfigOption<Long>(
+ "truncateSize", true,
+ "Min,max for size to truncate (min=max=MAX_LONG=blocksize)", SLIVE_PREFIX
+ + ".op.truncate.size", null);
+
static final ConfigOption<Long> RANDOM_SEED = new ConfigOption<Long>(
"seed", true, "Random number seed", SLIVE_PREFIX + ".seed", null);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a19820f2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/Constants.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/Constants.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/Constants.java
index adf5270..be2648e 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/Constants.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/Constants.java
@@ -45,7 +45,7 @@ class Constants {
* Allowed operation types
*/
enum OperationType {
- READ, APPEND, RENAME, LS, MKDIR, DELETE, CREATE;
+ READ, APPEND, RENAME, LS, MKDIR, DELETE, CREATE, TRUNCATE;
String lowerName() {
return this.name().toLowerCase(Locale.ENGLISH);
}
@@ -53,7 +53,7 @@ class Constants {
// program info
static final String PROG_NAME = SliveTest.class.getSimpleName();
- static final String PROG_VERSION = "0.0.2";
+ static final String PROG_VERSION = "0.1.0";
// useful constants
static final int MEGABYTES = 1048576;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a19820f2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/OperationFactory.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/OperationFactory.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/OperationFactory.java
index 52a4c9f..6af825f 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/OperationFactory.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/OperationFactory.java
@@ -75,6 +75,9 @@ class OperationFactory {
case CREATE:
op = new CreateOp(this.config, rnd);
break;
+ case TRUNCATE:
+ op = new TruncateOp(this.config, rnd);
+ break;
}
typedOperations.put(type, op);
return op;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a19820f2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/TestSlive.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/TestSlive.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/TestSlive.java
index 3db7695..25e3340 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/TestSlive.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/TestSlive.java
@@ -136,6 +136,8 @@ public class TestSlive {
args.add("10");
args.add("-" + ConfigOption.FILES.getOpt());
args.add("10");
+ args.add("-" + ConfigOption.TRUNCATE_SIZE.getOpt());
+ args.add("0,1M");
}
return args.toArray(new String[args.size()]);
}
@@ -237,6 +239,9 @@ public class TestSlive {
Range<Long> wRange = extractor.getWriteSize();
assertEquals(wRange.getLower().intValue(), Constants.MEGABYTES * 1);
assertEquals(wRange.getUpper().intValue(), Constants.MEGABYTES * 2);
+ Range<Long> trRange = extractor.getTruncateSize();
+ assertEquals(trRange.getLower().intValue(), 0);
+ assertEquals(trRange.getUpper().intValue(), Constants.MEGABYTES * 1);
Range<Long> bRange = extractor.getBlockSize();
assertEquals(bRange.getLower().intValue(), Constants.MEGABYTES * 1);
assertEquals(bRange.getUpper().intValue(), Constants.MEGABYTES * 2);
@@ -534,4 +539,26 @@ public class TestSlive {
};
runOperationOk(extractor, aop, false);
}
+
+ @Test
+ public void testTruncateOp() throws Exception {
+ // setup a valid config
+ ConfigExtractor extractor = getTestConfig(false);
+ // ensure file created before append
+ final Path fn = new Path(getTestFile().getCanonicalPath());
+ CreateOp op = new CreateOp(extractor, rnd) {
+ protected Path getCreateFile() {
+ return fn;
+ }
+ };
+ runOperationOk(extractor, op, true);
+ // local file system (ChecksumFileSystem) currently doesn't support truncate -
+ // but we'll leave this test here anyways but can't check the results..
+ TruncateOp top = new TruncateOp(extractor, rnd) {
+ protected Path getTruncateFile() {
+ return fn;
+ }
+ };
+ runOperationOk(extractor, top, false);
+ }
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a19820f2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/TruncateOp.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/TruncateOp.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/TruncateOp.java
new file mode 100644
index 0000000..c845ac1
--- /dev/null
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/TruncateOp.java
@@ -0,0 +1,114 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.slive;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.List;
+import java.util.Random;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.slive.OperationOutput.OutputType;
+
+/**
+ * Operation which selects a random file and truncates a random amount of bytes
+ * (selected from the configuration for truncate size) from that file,
+ * if it exists.
+ *
+ * This operation will capture statistics on success for bytes written, time
+ * taken (milliseconds), and success count and on failure it will capture the
+ * number of failures and the time taken (milliseconds) to fail.
+ */
+class TruncateOp extends Operation {
+
+ private static final Log LOG = LogFactory.getLog(TruncateOp.class);
+
+ TruncateOp(ConfigExtractor cfg, Random rnd) {
+ super(TruncateOp.class.getSimpleName(), cfg, rnd);
+ }
+
+ /**
+ * Gets the file to truncate from
+ *
+ * @return Path
+ */
+ protected Path getTruncateFile() {
+ Path fn = getFinder().getFile();
+ return fn;
+ }
+
+ @Override // Operation
+ List<OperationOutput> run(FileSystem fs) {
+ List<OperationOutput> out = super.run(fs);
+ try {
+ Path fn = getTruncateFile();
+ boolean waitOnTruncate = getConfig().shouldWaitOnTruncate();
+ long currentSize = fs.getFileStatus(fn).getLen();
+ // determine file status for file length requirement
+ // to know if should fill in partial bytes
+ Range<Long> truncateSizeRange = getConfig().getTruncateSize();
+ if (getConfig().shouldTruncateUseBlockSize()) {
+ truncateSizeRange = getConfig().getBlockSize();
+ }
+ long truncateSize = Math.max(0L,
+ currentSize - Range.betweenPositive(getRandom(), truncateSizeRange));
+ long timeTaken = 0;
+ LOG.info("Attempting to truncate file at " + fn + " to size "
+ + Helper.toByteInfo(truncateSize));
+ {
+ // truncate
+ long startTime = Timer.now();
+ boolean completed = fs.truncate(fn, truncateSize);
+ if(!completed && waitOnTruncate)
+ waitForRecovery(fs, fn, truncateSize);
+ timeTaken += Timer.elapsed(startTime);
+ }
+ out.add(new OperationOutput(OutputType.LONG, getType(),
+ ReportWriter.BYTES_WRITTEN, 0));
+ out.add(new OperationOutput(OutputType.LONG, getType(),
+ ReportWriter.OK_TIME_TAKEN, timeTaken));
+ out.add(new OperationOutput(OutputType.LONG, getType(),
+ ReportWriter.SUCCESSES, 1L));
+ LOG.info("Truncate file " + fn + " to " + Helper.toByteInfo(truncateSize)
+ + " in " + timeTaken + " milliseconds");
+ } catch (FileNotFoundException e) {
+ out.add(new OperationOutput(OutputType.LONG, getType(),
+ ReportWriter.NOT_FOUND, 1L));
+ LOG.warn("Error with truncating", e);
+ } catch (IOException e) {
+ out.add(new OperationOutput(OutputType.LONG, getType(),
+ ReportWriter.FAILURES, 1L));
+ LOG.warn("Error with truncating", e);
+ }
+ return out;
+ }
+
+ private void waitForRecovery(FileSystem fs, Path fn, long newLength)
+ throws IOException {
+ LOG.info("Waiting on truncate file recovery for " + fn);
+ for(;;) {
+ FileStatus stat = fs.getFileStatus(fn);
+ if(stat.getLen() == newLength) break;
+ try {Thread.sleep(1000);} catch(InterruptedException ignored) {}
+ }
+ }
+}
[42/52] [abbrv] hadoop git commit: HDFS-7347. Configurable erasure
coding policy for individual files and directories ( Contributed by Zhe Zhang
)
Posted by zh...@apache.org.
HDFS-7347. Configurable erasure coding policy for individual files and directories ( Contributed by Zhe Zhang )
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9815e236
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9815e236
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9815e236
Branch: refs/heads/HDFS-7285
Commit: 9815e236202848b0dd76065b7267019567ee4a78
Parents: fe7a302
Author: Vinayakumar B <vi...@apache.org>
Authored: Thu Nov 6 10:03:26 2014 +0530
Committer: Zhe Zhang <zh...@apache.org>
Committed: Mon Feb 23 11:18:12 2015 -0800
----------------------------------------------------------------------
.../hadoop-hdfs/CHANGES-HDFS-EC-7285.txt | 4 ++
.../hadoop/hdfs/protocol/HdfsConstants.java | 2 +
.../BlockStoragePolicySuite.java | 5 ++
.../hadoop/hdfs/TestBlockStoragePolicy.java | 12 +++-
.../TestBlockInitialEncoding.java | 75 ++++++++++++++++++++
5 files changed, 95 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9815e236/hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
new file mode 100644
index 0000000..2ef8527
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
@@ -0,0 +1,4 @@
+ BREAKDOWN OF HDFS-7285 SUBTASKS AND RELATED JIRAS
+
+ HDFS-7347. Configurable erasure coding policy for individual files and
+ directories ( Zhe Zhang via vinayakumarb )
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9815e236/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
index 1769794..6945074 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
@@ -171,6 +171,7 @@ public class HdfsConstants {
public static final String ONESSD_STORAGE_POLICY_NAME = "ONE_SSD";
public static final String HOT_STORAGE_POLICY_NAME = "HOT";
public static final String WARM_STORAGE_POLICY_NAME = "WARM";
+ public static final String EC_STORAGE_POLICY_NAME = "EC";
public static final String COLD_STORAGE_POLICY_NAME = "COLD";
public static final byte MEMORY_STORAGE_POLICY_ID = 15;
@@ -178,5 +179,6 @@ public class HdfsConstants {
public static final byte ONESSD_STORAGE_POLICY_ID = 10;
public static final byte HOT_STORAGE_POLICY_ID = 7;
public static final byte WARM_STORAGE_POLICY_ID = 5;
+ public static final byte EC_STORAGE_POLICY_ID = 4;
public static final byte COLD_STORAGE_POLICY_ID = 2;
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9815e236/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
index 9d8bdb5..2d841ce 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
@@ -78,6 +78,11 @@ public class BlockStoragePolicySuite {
new StorageType[]{StorageType.DISK, StorageType.ARCHIVE},
new StorageType[]{StorageType.DISK, StorageType.ARCHIVE},
new StorageType[]{StorageType.DISK, StorageType.ARCHIVE});
+ final byte ecId = HdfsConstants.EC_STORAGE_POLICY_ID;
+ policies[ecId] = new BlockStoragePolicy(ecId,
+ HdfsConstants.EC_STORAGE_POLICY_NAME,
+ new StorageType[]{StorageType.DISK}, StorageType.EMPTY_ARRAY,
+ new StorageType[]{StorageType.ARCHIVE});
final byte coldId = HdfsConstants.COLD_STORAGE_POLICY_ID;
policies[coldId] = new BlockStoragePolicy(coldId,
HdfsConstants.COLD_STORAGE_POLICY_NAME,
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9815e236/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
index 8f99a85..606a332 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
@@ -68,6 +68,7 @@ public class TestBlockStoragePolicy {
static final short REPLICATION = 3;
static final byte COLD = HdfsConstants.COLD_STORAGE_POLICY_ID;
+ static final byte EC = HdfsConstants.EC_STORAGE_POLICY_ID;
static final byte WARM = HdfsConstants.WARM_STORAGE_POLICY_ID;
static final byte HOT = HdfsConstants.HOT_STORAGE_POLICY_ID;
static final byte ONESSD = HdfsConstants.ONESSD_STORAGE_POLICY_ID;
@@ -115,6 +116,9 @@ public class TestBlockStoragePolicy {
expectedPolicyStrings.put(COLD,
"BlockStoragePolicy{COLD:" + COLD + ", storageTypes=[ARCHIVE], " +
"creationFallbacks=[], replicationFallbacks=[]}");
+ expectedPolicyStrings.put(EC,
+ "BlockStoragePolicy{EC:" + EC + ", storageTypes=[DISK], " +
+ "creationFallbacks=[], replicationFallbacks=[ARCHIVE]}");
expectedPolicyStrings.put(WARM,
"BlockStoragePolicy{WARM:" + WARM + ", storageTypes=[DISK, ARCHIVE], " +
"creationFallbacks=[DISK, ARCHIVE], " +
@@ -1157,13 +1161,15 @@ public class TestBlockStoragePolicy {
final DistributedFileSystem fs = cluster.getFileSystem();
try {
BlockStoragePolicy[] policies = fs.getStoragePolicies();
- Assert.assertEquals(6, policies.length);
+ Assert.assertEquals(7, policies.length);
Assert.assertEquals(POLICY_SUITE.getPolicy(COLD).toString(),
policies[0].toString());
- Assert.assertEquals(POLICY_SUITE.getPolicy(WARM).toString(),
+ Assert.assertEquals(POLICY_SUITE.getPolicy(EC).toString(),
policies[1].toString());
- Assert.assertEquals(POLICY_SUITE.getPolicy(HOT).toString(),
+ Assert.assertEquals(POLICY_SUITE.getPolicy(WARM).toString(),
policies[2].toString());
+ Assert.assertEquals(POLICY_SUITE.getPolicy(HOT).toString(),
+ policies[3].toString());
} finally {
IOUtils.cleanup(null, fs);
cluster.shutdown();
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9815e236/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInitialEncoding.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInitialEncoding.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInitialEncoding.java
new file mode 100644
index 0000000..a84f67b
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInitialEncoding.java
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.hdfs.*;
+import org.apache.hadoop.hdfs.client.HdfsAdmin;
+import org.apache.hadoop.hdfs.server.namenode.FSNamesystem;
+import org.apache.hadoop.hdfs.server.namenode.INode;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+
+import static org.apache.hadoop.hdfs.protocol.HdfsConstants.EC_STORAGE_POLICY_NAME;
+import static org.apache.hadoop.hdfs.protocol.HdfsConstants.EC_STORAGE_POLICY_ID;
+import static org.junit.Assert.assertEquals;
+
+public class TestBlockInitialEncoding {
+ private final int NUM_OF_DATANODES = 3;
+ private Configuration conf;
+ private MiniDFSCluster cluster;
+ private DistributedFileSystem fs;
+ private static final int BLOCK_SIZE = 1024;
+ private HdfsAdmin dfsAdmin;
+ private FSNamesystem namesystem;
+
+ @Before
+ public void setupCluster() throws IOException {
+ conf = new HdfsConfiguration();
+ conf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, BLOCK_SIZE);
+ cluster = new MiniDFSCluster.Builder(conf).
+ numDataNodes(NUM_OF_DATANODES).build();
+ cluster.waitActive();
+ fs = cluster.getFileSystem();
+ dfsAdmin = new HdfsAdmin(cluster.getURI(), conf);
+ namesystem = cluster.getNamesystem();
+ }
+
+ @After
+ public void shutdownCluster() throws IOException {
+ cluster.shutdown();
+ }
+
+ @Test
+ public void testBlockInitialEncoding()
+ throws IOException, InterruptedException {
+ final Path testDir = new Path("/test");
+ fs.mkdir(testDir, FsPermission.getDirDefault());
+ dfsAdmin.setStoragePolicy(testDir, EC_STORAGE_POLICY_NAME);
+ final Path ECFilePath = new Path("/test/foo.ec");
+ DFSTestUtil.createFile(fs, ECFilePath, 4 * BLOCK_SIZE, (short) 3, 0);
+ INode inode = namesystem.getFSDirectory().getINode(ECFilePath.toString());
+ assertEquals(EC_STORAGE_POLICY_ID, inode.getStoragePolicyID());
+ }
+
+}
\ No newline at end of file
[52/52] [abbrv] hadoop git commit: HADOOP-11542. Raw Reed-Solomon
coder in pure Java. Contributed by Kai Zheng
Posted by zh...@apache.org.
HADOOP-11542. Raw Reed-Solomon coder in pure Java. Contributed by Kai Zheng
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/e6d064b3
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/e6d064b3
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/e6d064b3
Branch: refs/heads/HDFS-7285
Commit: e6d064b3b8c81184a1a8eb99fcf1ef00bd1de2d7
Parents: 9d1ec74
Author: drankye <dr...@gmail.com>
Authored: Thu Feb 12 21:12:44 2015 +0800
Committer: Zhe Zhang <zh...@apache.org>
Committed: Mon Feb 23 11:22:50 2015 -0800
----------------------------------------------------------------------
.../hadoop-common/CHANGES-HDFS-EC-7285.txt | 4 +
.../io/erasurecode/rawcoder/JRSRawDecoder.java | 69 +++
.../io/erasurecode/rawcoder/JRSRawEncoder.java | 78 +++
.../erasurecode/rawcoder/RawErasureCoder.java | 2 +-
.../erasurecode/rawcoder/util/GaloisField.java | 497 +++++++++++++++++++
.../io/erasurecode/rawcoder/util/RSUtil.java | 22 +
.../hadoop/io/erasurecode/TestCoderBase.java | 28 +-
.../erasurecode/rawcoder/TestJRSRawCoder.java | 93 ++++
.../erasurecode/rawcoder/TestRawCoderBase.java | 5 +-
.../erasurecode/rawcoder/TestXorRawCoder.java | 1 -
10 files changed, 786 insertions(+), 13 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e6d064b3/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt b/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt
index 9728f97..7bbacf7 100644
--- a/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt
+++ b/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt
@@ -8,3 +8,7 @@
HADOOP-11541. Raw XOR coder
( Kai Zheng )
+
+ HADOOP-11542. Raw Reed-Solomon coder in pure Java. Contributed by Kai Zheng
+ ( Kai Zheng )
+
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e6d064b3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/JRSRawDecoder.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/JRSRawDecoder.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/JRSRawDecoder.java
new file mode 100644
index 0000000..dbb689e
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/JRSRawDecoder.java
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.io.erasurecode.rawcoder;
+
+import org.apache.hadoop.io.erasurecode.rawcoder.util.RSUtil;
+
+import java.nio.ByteBuffer;
+
+/**
+ * A raw erasure decoder in RS code scheme in pure Java in case native one
+ * isn't available in some environment. Please always use native implementations
+ * when possible.
+ */
+public class JRSRawDecoder extends AbstractRawErasureDecoder {
+ // To describe and calculate the needed Vandermonde matrix
+ private int[] errSignature;
+ private int[] primitivePower;
+
+ @Override
+ public void initialize(int numDataUnits, int numParityUnits, int chunkSize) {
+ super.initialize(numDataUnits, numParityUnits, chunkSize);
+ assert (getNumDataUnits() + getNumParityUnits() < RSUtil.GF.getFieldSize());
+
+ this.errSignature = new int[getNumParityUnits()];
+ this.primitivePower = RSUtil.getPrimitivePower(getNumDataUnits(),
+ getNumParityUnits());
+ }
+
+ @Override
+ protected void doDecode(ByteBuffer[] inputs, int[] erasedIndexes,
+ ByteBuffer[] outputs) {
+ for (int i = 0; i < erasedIndexes.length; i++) {
+ errSignature[i] = primitivePower[erasedIndexes[i]];
+ RSUtil.GF.substitute(inputs, outputs[i], primitivePower[i]);
+ }
+
+ int dataLen = inputs[0].remaining();
+ RSUtil.GF.solveVandermondeSystem(errSignature, outputs,
+ erasedIndexes.length, dataLen);
+ }
+
+ @Override
+ protected void doDecode(byte[][] inputs, int[] erasedIndexes,
+ byte[][] outputs) {
+ for (int i = 0; i < erasedIndexes.length; i++) {
+ errSignature[i] = primitivePower[erasedIndexes[i]];
+ RSUtil.GF.substitute(inputs, outputs[i], primitivePower[i]);
+ }
+
+ int dataLen = inputs[0].length;
+ RSUtil.GF.solveVandermondeSystem(errSignature, outputs,
+ erasedIndexes.length, dataLen);
+ }
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e6d064b3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/JRSRawEncoder.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/JRSRawEncoder.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/JRSRawEncoder.java
new file mode 100644
index 0000000..6ea7551
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/JRSRawEncoder.java
@@ -0,0 +1,78 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.io.erasurecode.rawcoder;
+
+import org.apache.hadoop.io.erasurecode.rawcoder.util.RSUtil;
+
+import java.nio.ByteBuffer;
+
+/**
+ * A raw erasure encoder in RS code scheme in pure Java in case native one
+ * isn't available in some environment. Please always use native implementations
+ * when possible.
+ */
+public class JRSRawEncoder extends AbstractRawErasureEncoder {
+ private int[] generatingPolynomial;
+
+ @Override
+ public void initialize(int numDataUnits, int numParityUnits, int chunkSize) {
+ super.initialize(numDataUnits, numParityUnits, chunkSize);
+ assert (getNumDataUnits() + getNumParityUnits() < RSUtil.GF.getFieldSize());
+
+ int[] primitivePower = RSUtil.getPrimitivePower(getNumDataUnits(),
+ getNumParityUnits());
+ // compute generating polynomial
+ int[] gen = {1};
+ int[] poly = new int[2];
+ for (int i = 0; i < getNumParityUnits(); i++) {
+ poly[0] = primitivePower[i];
+ poly[1] = 1;
+ gen = RSUtil.GF.multiply(gen, poly);
+ }
+ // generating polynomial has all generating roots
+ generatingPolynomial = gen;
+ }
+
+ @Override
+ protected void doEncode(ByteBuffer[] inputs, ByteBuffer[] outputs) {
+ ByteBuffer[] data = new ByteBuffer[getNumDataUnits() + getNumParityUnits()];
+ for (int i = 0; i < getNumParityUnits(); i++) {
+ data[i] = outputs[i];
+ }
+ for (int i = 0; i < getNumDataUnits(); i++) {
+ data[i + getNumParityUnits()] = inputs[i];
+ }
+
+ // Compute the remainder
+ RSUtil.GF.remainder(data, generatingPolynomial);
+ }
+
+ @Override
+ protected void doEncode(byte[][] inputs, byte[][] outputs) {
+ byte[][] data = new byte[getNumDataUnits() + getNumParityUnits()][];
+ for (int i = 0; i < getNumParityUnits(); i++) {
+ data[i] = outputs[i];
+ }
+ for (int i = 0; i < getNumDataUnits(); i++) {
+ data[i + getNumParityUnits()] = inputs[i];
+ }
+
+ // Compute the remainder
+ RSUtil.GF.remainder(data, generatingPolynomial);
+ }
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e6d064b3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureCoder.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureCoder.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureCoder.java
index 91a9abf..6e07cf1 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureCoder.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureCoder.java
@@ -71,7 +71,7 @@ public interface RawErasureCoder {
public boolean preferNativeBuffer();
/**
- * Should be called when release this coder. Good chance to release encoding
+ * Should be called when release this blockcoder. Good chance to release encoding
* or decoding buffers
*/
public void release();
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e6d064b3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/util/GaloisField.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/util/GaloisField.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/util/GaloisField.java
new file mode 100644
index 0000000..77544c6
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/util/GaloisField.java
@@ -0,0 +1,497 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.io.erasurecode.rawcoder.util;
+
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Implementation of Galois field arithmetic with 2^p elements. The input must
+ * be unsigned integers. It's ported from HDFS-RAID, slightly adapted.
+ */
+public class GaloisField {
+
+ // Field size 256 is good for byte based system
+ private static final int DEFAULT_FIELD_SIZE = 256;
+ // primitive polynomial 1 + X^2 + X^3 + X^4 + X^8 (substitute 2)
+ private static final int DEFAULT_PRIMITIVE_POLYNOMIAL = 285;
+ static private final Map<Integer, GaloisField> instances =
+ new HashMap<Integer, GaloisField>();
+ private final int[] logTable;
+ private final int[] powTable;
+ private final int[][] mulTable;
+ private final int[][] divTable;
+ private final int fieldSize;
+ private final int primitivePeriod;
+ private final int primitivePolynomial;
+
+ private GaloisField(int fieldSize, int primitivePolynomial) {
+ assert fieldSize > 0;
+ assert primitivePolynomial > 0;
+
+ this.fieldSize = fieldSize;
+ this.primitivePeriod = fieldSize - 1;
+ this.primitivePolynomial = primitivePolynomial;
+ logTable = new int[fieldSize];
+ powTable = new int[fieldSize];
+ mulTable = new int[fieldSize][fieldSize];
+ divTable = new int[fieldSize][fieldSize];
+ int value = 1;
+ for (int pow = 0; pow < fieldSize - 1; pow++) {
+ powTable[pow] = value;
+ logTable[value] = pow;
+ value = value * 2;
+ if (value >= fieldSize) {
+ value = value ^ primitivePolynomial;
+ }
+ }
+ // building multiplication table
+ for (int i = 0; i < fieldSize; i++) {
+ for (int j = 0; j < fieldSize; j++) {
+ if (i == 0 || j == 0) {
+ mulTable[i][j] = 0;
+ continue;
+ }
+ int z = logTable[i] + logTable[j];
+ z = z >= primitivePeriod ? z - primitivePeriod : z;
+ z = powTable[z];
+ mulTable[i][j] = z;
+ }
+ }
+ // building division table
+ for (int i = 0; i < fieldSize; i++) {
+ for (int j = 1; j < fieldSize; j++) {
+ if (i == 0) {
+ divTable[i][j] = 0;
+ continue;
+ }
+ int z = logTable[i] - logTable[j];
+ z = z < 0 ? z + primitivePeriod : z;
+ z = powTable[z];
+ divTable[i][j] = z;
+ }
+ }
+ }
+
+ /**
+ * Get the object performs Galois field arithmetics
+ *
+ * @param fieldSize size of the field
+ * @param primitivePolynomial a primitive polynomial corresponds to the size
+ */
+ public static GaloisField getInstance(int fieldSize,
+ int primitivePolynomial) {
+ int key = ((fieldSize << 16) & 0xFFFF0000)
+ + (primitivePolynomial & 0x0000FFFF);
+ GaloisField gf;
+ synchronized (instances) {
+ gf = instances.get(key);
+ if (gf == null) {
+ gf = new GaloisField(fieldSize, primitivePolynomial);
+ instances.put(key, gf);
+ }
+ }
+ return gf;
+ }
+
+ /**
+ * Get the object performs Galois field arithmetic with default setting
+ */
+ public static GaloisField getInstance() {
+ return getInstance(DEFAULT_FIELD_SIZE, DEFAULT_PRIMITIVE_POLYNOMIAL);
+ }
+
+ /**
+ * Return number of elements in the field
+ *
+ * @return number of elements in the field
+ */
+ public int getFieldSize() {
+ return fieldSize;
+ }
+
+ /**
+ * Return the primitive polynomial in GF(2)
+ *
+ * @return primitive polynomial as a integer
+ */
+ public int getPrimitivePolynomial() {
+ return primitivePolynomial;
+ }
+
+ /**
+ * Compute the sum of two fields
+ *
+ * @param x input field
+ * @param y input field
+ * @return result of addition
+ */
+ public int add(int x, int y) {
+ assert (x >= 0 && x < getFieldSize() && y >= 0 && y < getFieldSize());
+ return x ^ y;
+ }
+
+ /**
+ * Compute the multiplication of two fields
+ *
+ * @param x input field
+ * @param y input field
+ * @return result of multiplication
+ */
+ public int multiply(int x, int y) {
+ assert (x >= 0 && x < getFieldSize() && y >= 0 && y < getFieldSize());
+ return mulTable[x][y];
+ }
+
+ /**
+ * Compute the division of two fields
+ *
+ * @param x input field
+ * @param y input field
+ * @return x/y
+ */
+ public int divide(int x, int y) {
+ assert (x >= 0 && x < getFieldSize() && y > 0 && y < getFieldSize());
+ return divTable[x][y];
+ }
+
+ /**
+ * Compute power n of a field
+ *
+ * @param x input field
+ * @param n power
+ * @return x^n
+ */
+ public int power(int x, int n) {
+ assert (x >= 0 && x < getFieldSize());
+ if (n == 0) {
+ return 1;
+ }
+ if (x == 0) {
+ return 0;
+ }
+ x = logTable[x] * n;
+ if (x < primitivePeriod) {
+ return powTable[x];
+ }
+ x = x % primitivePeriod;
+ return powTable[x];
+ }
+
+ /**
+ * Given a Vandermonde matrix V[i][j]=x[j]^i and vector y, solve for z such
+ * that Vz=y. The output z will be placed in y.
+ *
+ * @param x the vector which describe the Vandermonde matrix
+ * @param y right-hand side of the Vandermonde system equation. will be
+ * replaced the output in this vector
+ */
+ public void solveVandermondeSystem(int[] x, int[] y) {
+ solveVandermondeSystem(x, y, x.length);
+ }
+
+ /**
+ * Given a Vandermonde matrix V[i][j]=x[j]^i and vector y, solve for z such
+ * that Vz=y. The output z will be placed in y.
+ *
+ * @param x the vector which describe the Vandermonde matrix
+ * @param y right-hand side of the Vandermonde system equation. will be
+ * replaced the output in this vector
+ * @param len consider x and y only from 0...len-1
+ */
+ public void solveVandermondeSystem(int[] x, int[] y, int len) {
+ assert (x.length <= len && y.length <= len);
+ for (int i = 0; i < len - 1; i++) {
+ for (int j = len - 1; j > i; j--) {
+ y[j] = y[j] ^ mulTable[x[i]][y[j - 1]];
+ }
+ }
+ for (int i = len - 1; i >= 0; i--) {
+ for (int j = i + 1; j < len; j++) {
+ y[j] = divTable[y[j]][x[j] ^ x[j - i - 1]];
+ }
+ for (int j = i; j < len - 1; j++) {
+ y[j] = y[j] ^ y[j + 1];
+ }
+ }
+ }
+
+ /**
+ * A "bulk" version to the solving of Vandermonde System
+ */
+ public void solveVandermondeSystem(int[] x, byte[][] y,
+ int len, int dataLen) {
+ for (int i = 0; i < len - 1; i++) {
+ for (int j = len - 1; j > i; j--) {
+ for (int k = 0; k < dataLen; k++) {
+ y[j][k] = (byte) (y[j][k] ^ mulTable[x[i]][y[j - 1][k] &
+ 0x000000FF]);
+ }
+ }
+ }
+ for (int i = len - 1; i >= 0; i--) {
+ for (int j = i + 1; j < len; j++) {
+ for (int k = 0; k < dataLen; k++) {
+ y[j][k] = (byte) (divTable[y[j][k] & 0x000000FF][x[j] ^
+ x[j - i - 1]]);
+ }
+ }
+ for (int j = i; j < len - 1; j++) {
+ for (int k = 0; k < dataLen; k++) {
+ y[j][k] = (byte) (y[j][k] ^ y[j + 1][k]);
+ }
+ }
+ }
+ }
+
+ /**
+ * A "bulk" version of the solveVandermondeSystem, using ByteBuffer.
+ */
+ public void solveVandermondeSystem(int[] x, ByteBuffer[] y,
+ int len, int dataLen) {
+ for (int i = 0; i < len - 1; i++) {
+ for (int j = len - 1; j > i; j--) {
+ for (int k = 0; k < dataLen; k++) {
+ y[j].put(k, (byte) (y[j].get(k) ^ mulTable[x[i]][y[j - 1].get(k) &
+ 0x000000FF]));
+ }
+ }
+ }
+ for (int i = len - 1; i >= 0; i--) {
+ for (int j = i + 1; j < len; j++) {
+ for (int k = 0; k < dataLen; k++) {
+ y[j].put(k, (byte) (divTable[y[j].get(k) & 0x000000FF][x[j] ^
+ x[j - i - 1]]));
+ }
+ }
+ for (int j = i; j < len - 1; j++) {
+ for (int k = 0; k < dataLen; k++) {
+ y[j].put(k, (byte) (y[j].get(k) ^ y[j + 1].get(k)));
+ }
+ }
+ }
+ }
+
+ /**
+ * Compute the multiplication of two polynomials. The index in the array
+ * corresponds to the power of the entry. For example p[0] is the constant
+ * term of the polynomial p.
+ *
+ * @param p input polynomial
+ * @param q input polynomial
+ * @return polynomial represents p*q
+ */
+ public int[] multiply(int[] p, int[] q) {
+ int len = p.length + q.length - 1;
+ int[] result = new int[len];
+ for (int i = 0; i < len; i++) {
+ result[i] = 0;
+ }
+ for (int i = 0; i < p.length; i++) {
+
+ for (int j = 0; j < q.length; j++) {
+ result[i + j] = add(result[i + j], multiply(p[i], q[j]));
+ }
+ }
+ return result;
+ }
+
+ /**
+ * Compute the remainder of a dividend and divisor pair. The index in the
+ * array corresponds to the power of the entry. For example p[0] is the
+ * constant term of the polynomial p.
+ *
+ * @param dividend dividend polynomial, the remainder will be placed
+ * here when return
+ * @param divisor divisor polynomial
+ */
+ public void remainder(int[] dividend, int[] divisor) {
+ for (int i = dividend.length - divisor.length; i >= 0; i--) {
+ int ratio = divTable[dividend[i +
+ divisor.length - 1]][divisor[divisor.length - 1]];
+ for (int j = 0; j < divisor.length; j++) {
+ int k = j + i;
+ dividend[k] = dividend[k] ^ mulTable[ratio][divisor[j]];
+ }
+ }
+ }
+
+ /**
+ * Compute the sum of two polynomials. The index in the array corresponds to
+ * the power of the entry. For example p[0] is the constant term of the
+ * polynomial p.
+ *
+ * @param p input polynomial
+ * @param q input polynomial
+ * @return polynomial represents p+q
+ */
+ public int[] add(int[] p, int[] q) {
+ int len = Math.max(p.length, q.length);
+ int[] result = new int[len];
+ for (int i = 0; i < len; i++) {
+ if (i < p.length && i < q.length) {
+ result[i] = add(p[i], q[i]);
+ } else if (i < p.length) {
+ result[i] = p[i];
+ } else {
+ result[i] = q[i];
+ }
+ }
+ return result;
+ }
+
+ /**
+ * Substitute x into polynomial p(x).
+ *
+ * @param p input polynomial
+ * @param x input field
+ * @return p(x)
+ */
+ public int substitute(int[] p, int x) {
+ int result = 0;
+ int y = 1;
+ for (int i = 0; i < p.length; i++) {
+ result = result ^ mulTable[p[i]][y];
+ y = mulTable[x][y];
+ }
+ return result;
+ }
+
+ /**
+ * A "bulk" version of the substitute.
+ * Tends to be 2X faster than the "int" substitute in a loop.
+ *
+ * @param p input polynomial
+ * @param q store the return result
+ * @param x input field
+ */
+ public void substitute(byte[][] p, byte[] q, int x) {
+ int y = 1;
+ for (int i = 0; i < p.length; i++) {
+ byte[] pi = p[i];
+ for (int j = 0; j < pi.length; j++) {
+ int pij = pi[j] & 0x000000FF;
+ q[j] = (byte) (q[j] ^ mulTable[pij][y]);
+ }
+ y = mulTable[x][y];
+ }
+ }
+
+ /**
+ * A "bulk" version of the substitute, using ByteBuffer.
+ * Tends to be 2X faster than the "int" substitute in a loop.
+ *
+ * @param p input polynomial
+ * @param q store the return result
+ * @param x input field
+ */
+ public void substitute(ByteBuffer[] p, ByteBuffer q, int x) {
+ int y = 1;
+ for (int i = 0; i < p.length; i++) {
+ ByteBuffer pi = p[i];
+ int len = pi.remaining();
+ for (int j = 0; j < len; j++) {
+ int pij = pi.get(j) & 0x000000FF;
+ q.put(j, (byte) (q.get(j) ^ mulTable[pij][y]));
+ }
+ y = mulTable[x][y];
+ }
+ }
+
+ /**
+ * The "bulk" version of the remainder.
+ * Warning: This function will modify the "dividend" inputs.
+ */
+ public void remainder(byte[][] dividend, int[] divisor) {
+ for (int i = dividend.length - divisor.length; i >= 0; i--) {
+ for (int j = 0; j < divisor.length; j++) {
+ for (int k = 0; k < dividend[i].length; k++) {
+ int ratio = divTable[dividend[i + divisor.length - 1][k] &
+ 0x00FF][divisor[divisor.length - 1]];
+ dividend[j + i][k] = (byte) ((dividend[j + i][k] & 0x00FF) ^
+ mulTable[ratio][divisor[j]]);
+ }
+ }
+ }
+ }
+
+ /**
+ * The "bulk" version of the remainder, using ByteBuffer.
+ * Warning: This function will modify the "dividend" inputs.
+ */
+ public void remainder(ByteBuffer[] dividend, int[] divisor) {
+ for (int i = dividend.length - divisor.length; i >= 0; i--) {
+ int width = dividend[i].remaining();
+ for (int j = 0; j < divisor.length; j++) {
+ for (int k = 0; k < width; k++) {
+ int ratio = divTable[dividend[i + divisor.length - 1].get(k) &
+ 0x00FF][divisor[divisor.length - 1]];
+ dividend[j + i].put(k, (byte) ((dividend[j + i].get(k) & 0x00FF) ^
+ mulTable[ratio][divisor[j]]));
+ }
+ }
+ }
+ }
+
+ /**
+ * Perform Gaussian elimination on the given matrix. This matrix has to be a
+ * fat matrix (number of rows > number of columns).
+ */
+ public void gaussianElimination(int[][] matrix) {
+ assert(matrix != null && matrix.length > 0 && matrix[0].length > 0
+ && matrix.length < matrix[0].length);
+ int height = matrix.length;
+ int width = matrix[0].length;
+ for (int i = 0; i < height; i++) {
+ boolean pivotFound = false;
+ // scan the column for a nonzero pivot and swap it to the diagonal
+ for (int j = i; j < height; j++) {
+ if (matrix[i][j] != 0) {
+ int[] tmp = matrix[i];
+ matrix[i] = matrix[j];
+ matrix[j] = tmp;
+ pivotFound = true;
+ break;
+ }
+ }
+ if (!pivotFound) {
+ continue;
+ }
+ int pivot = matrix[i][i];
+ for (int j = i; j < width; j++) {
+ matrix[i][j] = divide(matrix[i][j], pivot);
+ }
+ for (int j = i + 1; j < height; j++) {
+ int lead = matrix[j][i];
+ for (int k = i; k < width; k++) {
+ matrix[j][k] = add(matrix[j][k], multiply(lead, matrix[i][k]));
+ }
+ }
+ }
+ for (int i = height - 1; i >=0; i--) {
+ for (int j = 0; j < i; j++) {
+ int lead = matrix[j][i];
+ for (int k = i; k < width; k++) {
+ matrix[j][k] = add(matrix[j][k], multiply(lead, matrix[i][k]));
+ }
+ }
+ }
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e6d064b3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/util/RSUtil.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/util/RSUtil.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/util/RSUtil.java
new file mode 100644
index 0000000..33ba561
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/util/RSUtil.java
@@ -0,0 +1,22 @@
+package org.apache.hadoop.io.erasurecode.rawcoder.util;
+
+/**
+ * Some utilities for Reed-Solomon coding.
+ */
+public class RSUtil {
+
+ // We always use the byte system (with symbol size 8, field size 256,
+ // primitive polynomial 285, and primitive root 2).
+ public static GaloisField GF = GaloisField.getInstance();
+ public static final int PRIMITIVE_ROOT = 2;
+
+ public static int[] getPrimitivePower(int numDataUnits, int numParityUnits) {
+ int[] primitivePower = new int[numDataUnits + numParityUnits];
+ // compute powers of the primitive root
+ for (int i = 0; i < numDataUnits + numParityUnits; i++) {
+ primitivePower[i] = GF.power(PRIMITIVE_ROOT, i);
+ }
+ return primitivePower;
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e6d064b3/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestCoderBase.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestCoderBase.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestCoderBase.java
index 9482b43..3c4288c 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestCoderBase.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestCoderBase.java
@@ -18,9 +18,11 @@
package org.apache.hadoop.io.erasurecode;
import java.nio.ByteBuffer;
+import java.util.Arrays;
import java.util.Random;
import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertTrue;
/**
* Test base of common utilities for tests not only raw coders but also block
@@ -41,6 +43,14 @@ public abstract class TestCoderBase {
// may go to different coding implementations.
protected boolean usingDirectBuffer = true;
+ protected void prepare(int numDataUnits, int numParityUnits,
+ int[] erasedIndexes) {
+ this.numDataUnits = numDataUnits;
+ this.numParityUnits = numParityUnits;
+ this.erasedDataIndexes = erasedIndexes != null ?
+ erasedIndexes : new int[] {0};
+ }
+
/**
* Compare and verify if erased chunks are equal to recovered chunks
* @param erasedChunks
@@ -50,10 +60,8 @@ public abstract class TestCoderBase {
ECChunk[] recoveredChunks) {
byte[][] erased = ECChunk.toArray(erasedChunks);
byte[][] recovered = ECChunk.toArray(recoveredChunks);
- for (int i = 0; i < erasedChunks.length; ++i) {
- assertArrayEquals("Decoding and comparing failed.", erased[i],
- recovered[i]);
- }
+ boolean result = Arrays.deepEquals(erased, recovered);
+ assertTrue("Decoding and comparing failed.", result);
}
/**
@@ -63,7 +71,7 @@ public abstract class TestCoderBase {
*/
protected int[] getErasedIndexesForDecoding() {
int[] erasedIndexesForDecoding = new int[erasedDataIndexes.length];
- for (int i = 0; i < erasedDataIndexes.length; ++i) {
+ for (int i = 0; i < erasedDataIndexes.length; i++) {
erasedIndexesForDecoding[i] = erasedDataIndexes[i] + numParityUnits;
}
return erasedIndexesForDecoding;
@@ -100,7 +108,7 @@ public abstract class TestCoderBase {
ECChunk[] copiedChunks = new ECChunk[erasedDataIndexes.length];
int j = 0;
- for (int i = 0; i < erasedDataIndexes.length; ++i) {
+ for (int i = 0; i < erasedDataIndexes.length; i++) {
copiedChunks[j ++] = cloneChunkWithData(dataChunks[erasedDataIndexes[i]]);
}
@@ -112,7 +120,7 @@ public abstract class TestCoderBase {
* @param dataChunks
*/
protected void eraseSomeDataBlocks(ECChunk[] dataChunks) {
- for (int i = 0; i < erasedDataIndexes.length; ++i) {
+ for (int i = 0; i < erasedDataIndexes.length; i++) {
eraseDataFromChunk(dataChunks[erasedDataIndexes[i]]);
}
}
@@ -122,7 +130,7 @@ public abstract class TestCoderBase {
* @param chunks
*/
protected void eraseDataFromChunks(ECChunk[] chunks) {
- for (int i = 0; i < chunks.length; ++i) {
+ for (int i = 0; i < chunks.length; i++) {
eraseDataFromChunk(chunks[i]);
}
}
@@ -135,7 +143,7 @@ public abstract class TestCoderBase {
ByteBuffer chunkBuffer = chunk.getBuffer();
// erase the data
chunkBuffer.position(0);
- for (int i = 0; i < chunkSize; ++i) {
+ for (int i = 0; i < chunkSize; i++) {
chunkBuffer.put((byte) 0);
}
chunkBuffer.flip();
@@ -150,7 +158,7 @@ public abstract class TestCoderBase {
*/
protected static ECChunk[] cloneChunksWithData(ECChunk[] chunks) {
ECChunk[] results = new ECChunk[chunks.length];
- for (int i = 0; i < chunks.length; ++i) {
+ for (int i = 0; i < chunks.length; i++) {
results[i] = cloneChunkWithData(chunks[i]);
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e6d064b3/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestJRSRawCoder.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestJRSRawCoder.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestJRSRawCoder.java
new file mode 100644
index 0000000..e54f647
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestJRSRawCoder.java
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.io.erasurecode.rawcoder;
+
+import org.apache.hadoop.io.erasurecode.ECChunk;
+import org.apache.hadoop.io.erasurecode.rawcoder.util.RSUtil;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.nio.ByteBuffer;
+
+/**
+ * Test raw Reed-solomon encoding and decoding.
+ */
+public class TestJRSRawCoder extends TestRawCoderBase {
+
+ private static int symbolSize = 0;
+ private static int symbolMax = 0;
+
+ static {
+ symbolSize = (int) Math.round(Math.log(
+ RSUtil.GF.getFieldSize()) / Math.log(2));
+ symbolMax = (int) Math.pow(2, symbolSize);
+ }
+
+ @Before
+ public void setup() {
+ this.encoderClass = JRSRawEncoder.class;
+ this.decoderClass = JRSRawDecoder.class;
+ }
+
+ @Test
+ public void testCodingNoDirectBuffer_10x4() {
+ prepare(10, 4, null);
+ testCoding(false);
+ }
+
+ @Test
+ public void testCodingDirectBuffer_10x4() {
+ prepare(10, 4, null);
+ testCoding(true);
+ }
+
+ @Test
+ public void testCodingDirectBuffer_10x4_erasure_of_2_4() {
+ prepare(10, 4, new int[] {2, 4});
+ testCoding(true);
+ }
+
+ @Test
+ public void testCodingDirectBuffer_10x4_erasing_all() {
+ prepare(10, 4, new int[] {0, 1, 2, 3});
+ testCoding(true);
+ }
+
+ @Test
+ public void testCodingNoDirectBuffer_3x3() {
+ prepare(3, 3, null);
+ testCoding(false);
+ }
+
+ @Test
+ public void testCodingDirectBuffer_3x3() {
+ prepare(3, 3, null);
+ testCoding(true);
+ }
+
+ @Override
+ protected ECChunk generateDataChunk() {
+ ByteBuffer buffer = allocateOutputBuffer();
+ for (int i = 0; i < chunkSize; i++) {
+ buffer.put((byte) RAND.nextInt(symbolMax));
+ }
+ buffer.flip();
+
+ return new ECChunk(buffer);
+ }
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e6d064b3/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestRawCoderBase.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestRawCoderBase.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestRawCoderBase.java
index 9119211..5f6ccda 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestRawCoderBase.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestRawCoderBase.java
@@ -31,10 +31,13 @@ public abstract class TestRawCoderBase extends TestCoderBase {
* Generating source data, encoding, recovering and then verifying.
* RawErasureCoder mainly uses ECChunk to pass input and output data buffers,
* it supports two kinds of ByteBuffers, one is array backed, the other is
- * direct ByteBuffer. Have usingDirectBuffer to indicate which case to test.
+ * direct ByteBuffer. Use usingDirectBuffer indicate which case to test.
+ *
* @param usingDirectBuffer
*/
protected void testCoding(boolean usingDirectBuffer) {
+ this.usingDirectBuffer = usingDirectBuffer;
+
// Generate data and encode
ECChunk[] dataChunks = prepareDataChunksForEncoding();
ECChunk[] parityChunks = prepareParityChunksForEncoding();
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e6d064b3/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestXorRawCoder.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestXorRawCoder.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestXorRawCoder.java
index 8e59b8a..ff48586 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestXorRawCoder.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestXorRawCoder.java
@@ -26,7 +26,6 @@ import java.util.Random;
* Test XOR encoding and decoding.
*/
public class TestXorRawCoder extends TestRawCoderBase {
- private static Random RAND = new Random();
@Before
public void setup() {
[21/52] [abbrv] hadoop git commit: YARN-3076. Add API/Implementation
to YarnClient to retrieve label-to-node mapping (Varun Saxena via wangda)
Posted by zh...@apache.org.
YARN-3076. Add API/Implementation to YarnClient to retrieve label-to-node mapping (Varun Saxena via wangda)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d49ae725
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d49ae725
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d49ae725
Branch: refs/heads/HDFS-7285
Commit: d49ae725d5fa3eecf879ac42c42a368dd811f854
Parents: f0f2992
Author: Wangda Tan <wa...@apache.org>
Authored: Thu Feb 19 11:00:57 2015 -0800
Committer: Wangda Tan <wa...@apache.org>
Committed: Thu Feb 19 11:00:57 2015 -0800
----------------------------------------------------------------------
.../hadoop/mapred/ResourceMgrDelegate.java | 12 ++
.../hadoop/mapred/TestClientRedirect.java | 8 +
hadoop-yarn-project/CHANGES.txt | 3 +
.../yarn/api/ApplicationClientProtocol.java | 18 ++
.../GetLabelsToNodesRequest.java | 41 +++++
.../GetLabelsToNodesResponse.java | 45 +++++
.../main/proto/applicationclient_protocol.proto | 1 +
.../src/main/proto/yarn_protos.proto | 5 +
.../src/main/proto/yarn_service_protos.proto | 8 +
.../hadoop/yarn/client/api/YarnClient.java | 31 ++++
.../yarn/client/api/impl/YarnClientImpl.java | 15 ++
.../yarn/client/api/impl/TestYarnClient.java | 75 +++++++-
.../ApplicationClientProtocolPBClientImpl.java | 19 ++
.../ApplicationClientProtocolPBServiceImpl.java | 21 +++
.../impl/pb/GetLabelsToNodesRequestPBImpl.java | 121 ++++++++++++
.../impl/pb/GetLabelsToNodesResponsePBImpl.java | 184 +++++++++++++++++++
.../hadoop/yarn/api/TestPBImplRecords.java | 12 ++
.../server/resourcemanager/ClientRMService.java | 15 ++
.../resourcemanager/TestClientRMService.java | 75 ++++++++
19 files changed, 708 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
index 06667ee..82e8bdb 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
@@ -440,6 +440,18 @@ public class ResourceMgrDelegate extends YarnClient {
}
@Override
+ public Map<String, Set<NodeId>> getLabelsToNodes() throws YarnException,
+ IOException {
+ return client.getLabelsToNodes();
+ }
+
+ @Override
+ public Map<String, Set<NodeId>> getLabelsToNodes(Set<String> labels)
+ throws YarnException, IOException {
+ return client.getLabelsToNodes(labels);
+ }
+
+ @Override
public Set<String> getClusterNodeLabels()
throws YarnException, IOException {
return client.getClusterNodeLabels();
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java
index 0af5a71..bb00b19 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java
@@ -90,6 +90,8 @@ import org.apache.hadoop.yarn.api.protocolrecords.GetContainersRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetContainersResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetDelegationTokenRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetDelegationTokenResponse;
+import org.apache.hadoop.yarn.api.protocolrecords.GetLabelsToNodesRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetLabelsToNodesResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetNodesToLabelsRequest;
@@ -436,6 +438,12 @@ public class TestClientRedirect {
GetClusterNodeLabelsRequest request) throws YarnException, IOException {
return null;
}
+
+ @Override
+ public GetLabelsToNodesResponse getLabelsToNodes(
+ GetLabelsToNodesRequest request) throws YarnException, IOException {
+ return null;
+ }
}
class HistoryService extends AMService implements HSClientProtocol {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-yarn-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 6b2b878..aab3496 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -304,6 +304,9 @@ Release 2.7.0 - UNRELEASED
YARN-1514. Utility to benchmark ZKRMStateStore#loadState for RM HA.
(Tsuyoshi OZAWA via jianhe)
+ YARN-3076. Add API/Implementation to YarnClient to retrieve label-to-node
+ mapping. (Varun Saxena via wangda)
+
OPTIMIZATIONS
YARN-2990. FairScheduler's delay-scheduling always waits for node-local and
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java
index add35f1..b5f5cc0 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java
@@ -47,6 +47,8 @@ import org.apache.hadoop.yarn.api.protocolrecords.GetContainersRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetContainersResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetDelegationTokenRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetDelegationTokenResponse;
+import org.apache.hadoop.yarn.api.protocolrecords.GetLabelsToNodesRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetLabelsToNodesResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetNodesToLabelsRequest;
@@ -678,6 +680,22 @@ public interface ApplicationClientProtocol {
/**
* <p>
+ * The interface used by client to get labels to nodes mappings
+ * in existing cluster
+ * </p>
+ *
+ * @param request
+ * @return labels to nodes mappings
+ * @throws YarnException
+ * @throws IOException
+ */
+ @Public
+ @Unstable
+ public GetLabelsToNodesResponse getLabelsToNodes(
+ GetLabelsToNodesRequest request) throws YarnException, IOException;
+
+ /**
+ * <p>
* The interface used by client to get node labels in the cluster
* </p>
*
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetLabelsToNodesRequest.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetLabelsToNodesRequest.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetLabelsToNodesRequest.java
new file mode 100644
index 0000000..bf66945
--- /dev/null
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetLabelsToNodesRequest.java
@@ -0,0 +1,41 @@
+/**
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.hadoop.yarn.api.protocolrecords;
+
+import java.util.Set;
+
+import org.apache.hadoop.yarn.util.Records;
+
+public abstract class GetLabelsToNodesRequest {
+
+ public static GetLabelsToNodesRequest newInstance() {
+ return Records.newRecord(GetLabelsToNodesRequest.class);
+ }
+
+ public static GetLabelsToNodesRequest newInstance(Set<String> nodeLabels) {
+ GetLabelsToNodesRequest request =
+ Records.newRecord(GetLabelsToNodesRequest.class);
+ request.setNodeLabels(nodeLabels);
+ return request;
+ }
+
+ public abstract void setNodeLabels(Set<String> nodeLabels);
+
+ public abstract Set<String> getNodeLabels();
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetLabelsToNodesResponse.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetLabelsToNodesResponse.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetLabelsToNodesResponse.java
new file mode 100644
index 0000000..f105359
--- /dev/null
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetLabelsToNodesResponse.java
@@ -0,0 +1,45 @@
+/**
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.hadoop.yarn.api.protocolrecords;
+
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.hadoop.classification.InterfaceAudience.Public;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+import org.apache.hadoop.yarn.api.records.NodeId;
+import org.apache.hadoop.yarn.util.Records;
+
+public abstract class GetLabelsToNodesResponse {
+ public static GetLabelsToNodesResponse newInstance(
+ Map<String, Set<NodeId>> map) {
+ GetLabelsToNodesResponse response =
+ Records.newRecord(GetLabelsToNodesResponse.class);
+ response.setLabelsToNodes(map);
+ return response;
+ }
+
+ @Public
+ @Evolving
+ public abstract void setLabelsToNodes(Map<String, Set<NodeId>> map);
+
+ @Public
+ @Evolving
+ public abstract Map<String, Set<NodeId>> getLabelsToNodes();
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/applicationclient_protocol.proto
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/applicationclient_protocol.proto b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/applicationclient_protocol.proto
index 8e788cd..e7e3654 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/applicationclient_protocol.proto
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/applicationclient_protocol.proto
@@ -53,5 +53,6 @@ service ApplicationClientProtocolService {
rpc updateReservation (ReservationUpdateRequestProto) returns (ReservationUpdateResponseProto);
rpc deleteReservation (ReservationDeleteRequestProto) returns (ReservationDeleteResponseProto);
rpc getNodeToLabels (GetNodesToLabelsRequestProto) returns (GetNodesToLabelsResponseProto);
+ rpc getLabelsToNodes (GetLabelsToNodesRequestProto) returns (GetLabelsToNodesResponseProto);
rpc getClusterNodeLabels (GetClusterNodeLabelsRequestProto) returns (GetClusterNodeLabelsResponseProto);
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
index c4e756d..4e29d2f 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
@@ -238,6 +238,11 @@ message NodeIdToLabelsProto {
repeated string nodeLabels = 2;
}
+message LabelsToNodeIdsProto {
+ optional string nodeLabels = 1;
+ repeated NodeIdProto nodeId = 2;
+}
+
////////////////////////////////////////////////////////////////////////
////// From AM_RM_Protocol /////////////////////////////////////////////
////////////////////////////////////////////////////////////////////////
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
index 94e73e1..33d1207 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
@@ -201,6 +201,14 @@ message GetNodesToLabelsResponseProto {
repeated NodeIdToLabelsProto nodeToLabels = 1;
}
+message GetLabelsToNodesRequestProto {
+ repeated string nodeLabels = 1;
+}
+
+message GetLabelsToNodesResponseProto {
+ repeated LabelsToNodeIdsProto labelsToNodes = 1;
+}
+
message GetClusterNodeLabelsRequestProto {
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java
index bd4ec6d..d96761a 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java
@@ -598,6 +598,37 @@ public abstract class YarnClient extends AbstractService {
/**
* <p>
+ * The interface used by client to get labels to nodes mapping
+ * in existing cluster
+ * </p>
+ *
+ * @return node to labels mappings
+ * @throws YarnException
+ * @throws IOException
+ */
+ @Public
+ @Unstable
+ public abstract Map<String, Set<NodeId>> getLabelsToNodes()
+ throws YarnException, IOException;
+
+ /**
+ * <p>
+ * The interface used by client to get labels to nodes mapping
+ * for specified labels in existing cluster
+ * </p>
+ *
+ * @param labels labels for which labels to nodes mapping has to be retrieved
+ * @return labels to nodes mappings for specific labels
+ * @throws YarnException
+ * @throws IOException
+ */
+ @Public
+ @Unstable
+ public abstract Map<String, Set<NodeId>> getLabelsToNodes(Set<String> labels)
+ throws YarnException, IOException;
+
+ /**
+ * <p>
* The interface used by client to get node labels in the cluster
* </p>
*
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
index 91fbd00..6acf7d8 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
@@ -61,6 +61,7 @@ import org.apache.hadoop.yarn.api.protocolrecords.GetContainersRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetContainersResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetDelegationTokenRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetDelegationTokenResponse;
+import org.apache.hadoop.yarn.api.protocolrecords.GetLabelsToNodesRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetNodesToLabelsRequest;
@@ -778,6 +779,20 @@ public class YarnClientImpl extends YarnClient {
}
@Override
+ public Map<String, Set<NodeId>> getLabelsToNodes() throws YarnException,
+ IOException {
+ return rmClient.getLabelsToNodes(GetLabelsToNodesRequest.newInstance())
+ .getLabelsToNodes();
+ }
+
+ @Override
+ public Map<String, Set<NodeId>> getLabelsToNodes(Set<String> labels)
+ throws YarnException, IOException {
+ return rmClient.getLabelsToNodes(
+ GetLabelsToNodesRequest.newInstance(labels)).getLabelsToNodes();
+ }
+
+ @Override
public Set<String> getClusterNodeLabels() throws YarnException, IOException {
return rmClient.getClusterNodeLabels(
GetClusterNodeLabelsRequest.newInstance()).getNodeLabels();
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
index 73e06af..7e97134 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
@@ -30,6 +30,7 @@ import java.io.IOException;
import java.nio.ByteBuffer;
import java.security.PrivilegedExceptionAction;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.EnumSet;
@@ -37,6 +38,7 @@ import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
+import java.util.Map;
import java.util.Set;
import org.apache.commons.io.IOUtils;
@@ -63,6 +65,8 @@ import org.apache.hadoop.yarn.api.protocolrecords.GetContainerReportRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetContainerReportResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetContainersRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetContainersResponse;
+import org.apache.hadoop.yarn.api.protocolrecords.GetLabelsToNodesRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetLabelsToNodesResponse;
import org.apache.hadoop.yarn.api.protocolrecords.KillApplicationRequest;
import org.apache.hadoop.yarn.api.protocolrecords.KillApplicationResponse;
import org.apache.hadoop.yarn.api.protocolrecords.ReservationDeleteRequest;
@@ -402,6 +406,32 @@ public class TestYarnClient {
client.stop();
}
+ @Test (timeout = 10000)
+ public void testGetLabelsToNodes() throws YarnException, IOException {
+ Configuration conf = new Configuration();
+ final YarnClient client = new MockYarnClient();
+ client.init(conf);
+ client.start();
+
+ // Get labels to nodes mapping
+ Map<String, Set<NodeId>> expectedLabelsToNodes =
+ ((MockYarnClient)client).getLabelsToNodesMap();
+ Map<String, Set<NodeId>> labelsToNodes = client.getLabelsToNodes();
+ Assert.assertEquals(labelsToNodes, expectedLabelsToNodes);
+ Assert.assertEquals(labelsToNodes.size(), 3);
+
+ // Get labels to nodes for selected labels
+ Set<String> setLabels = new HashSet<String>(Arrays.asList("x", "z"));
+ expectedLabelsToNodes =
+ ((MockYarnClient)client).getLabelsToNodesMap(setLabels);
+ labelsToNodes = client.getLabelsToNodes(setLabels);
+ Assert.assertEquals(labelsToNodes, expectedLabelsToNodes);
+ Assert.assertEquals(labelsToNodes.size(), 2);
+
+ client.stop();
+ client.close();
+ }
+
private static class MockYarnClient extends YarnClientImpl {
private ApplicationReport mockReport;
private List<ApplicationReport> reports;
@@ -422,6 +452,8 @@ public class TestYarnClient {
mock(GetContainersResponse.class);
GetContainerReportResponse mockContainerResponse =
mock(GetContainerReportResponse.class);
+ GetLabelsToNodesResponse mockLabelsToNodesResponse =
+ mock(GetLabelsToNodesResponse.class);
public MockYarnClient() {
super();
@@ -457,6 +489,9 @@ public class TestYarnClient {
when(rmClient.getContainerReport(any(GetContainerReportRequest.class)))
.thenReturn(mockContainerResponse);
+
+ when(rmClient.getLabelsToNodes(any(GetLabelsToNodesRequest.class)))
+ .thenReturn(mockLabelsToNodesResponse);
historyClient = mock(AHSClient.class);
@@ -617,7 +652,45 @@ public class TestYarnClient {
}
return appReports;
}
-
+
+ @Override
+ public Map<String, Set<NodeId>> getLabelsToNodes()
+ throws YarnException, IOException {
+ when(mockLabelsToNodesResponse.getLabelsToNodes()).thenReturn(
+ getLabelsToNodesMap());
+ return super.getLabelsToNodes();
+ }
+
+ @Override
+ public Map<String, Set<NodeId>> getLabelsToNodes(Set<String> labels)
+ throws YarnException, IOException {
+ when(mockLabelsToNodesResponse.getLabelsToNodes()).thenReturn(
+ getLabelsToNodesMap(labels));
+ return super.getLabelsToNodes(labels);
+ }
+
+ public Map<String, Set<NodeId>> getLabelsToNodesMap() {
+ Map<String, Set<NodeId>> map = new HashMap<String, Set<NodeId>>();
+ Set<NodeId> setNodeIds =
+ new HashSet<NodeId>(Arrays.asList(
+ NodeId.newInstance("host1", 0), NodeId.newInstance("host2", 0)));
+ map.put("x", setNodeIds);
+ map.put("y", setNodeIds);
+ map.put("z", setNodeIds);
+ return map;
+ }
+
+ public Map<String, Set<NodeId>> getLabelsToNodesMap(Set<String> labels) {
+ Map<String, Set<NodeId>> map = new HashMap<String, Set<NodeId>>();
+ Set<NodeId> setNodeIds =
+ new HashSet<NodeId>(Arrays.asList(
+ NodeId.newInstance("host1", 0), NodeId.newInstance("host2", 0)));
+ for(String label : labels) {
+ map.put(label, setNodeIds);
+ }
+ return map;
+ }
+
@Override
public List<ApplicationAttemptReport> getApplicationAttempts(
ApplicationId appId) throws YarnException, IOException {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ApplicationClientProtocolPBClientImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ApplicationClientProtocolPBClientImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ApplicationClientProtocolPBClientImpl.java
index 0fe5423..959f399 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ApplicationClientProtocolPBClientImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ApplicationClientProtocolPBClientImpl.java
@@ -53,6 +53,8 @@ import org.apache.hadoop.yarn.api.protocolrecords.GetContainersRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetContainersResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetDelegationTokenRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetDelegationTokenResponse;
+import org.apache.hadoop.yarn.api.protocolrecords.GetLabelsToNodesRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetLabelsToNodesResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetNodesToLabelsRequest;
@@ -97,6 +99,8 @@ import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetContainersRequestPB
import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetContainersResponsePBImpl;
import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl;
import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetDelegationTokenResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetLabelsToNodesRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetLabelsToNodesResponsePBImpl;
import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetNewApplicationRequestPBImpl;
import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetNewApplicationResponsePBImpl;
import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetNodesToLabelsRequestPBImpl;
@@ -475,6 +479,21 @@ public class ApplicationClientProtocolPBClientImpl implements ApplicationClientP
}
@Override
+ public GetLabelsToNodesResponse getLabelsToNodes(
+ GetLabelsToNodesRequest request)
+ throws YarnException, IOException {
+ YarnServiceProtos.GetLabelsToNodesRequestProto requestProto =
+ ((GetLabelsToNodesRequestPBImpl) request).getProto();
+ try {
+ return new GetLabelsToNodesResponsePBImpl(proxy.getLabelsToNodes(
+ null, requestProto));
+ } catch (ServiceException e) {
+ RPCUtil.unwrapAndThrowException(e);
+ return null;
+ }
+ }
+
+ @Override
public GetClusterNodeLabelsResponse getClusterNodeLabels(
GetClusterNodeLabelsRequest request) throws YarnException, IOException {
GetClusterNodeLabelsRequestProto
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ApplicationClientProtocolPBServiceImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ApplicationClientProtocolPBServiceImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ApplicationClientProtocolPBServiceImpl.java
index 385fb78..36bd3af 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ApplicationClientProtocolPBServiceImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ApplicationClientProtocolPBServiceImpl.java
@@ -40,6 +40,7 @@ import org.apache.hadoop.yarn.api.protocolrecords.GetClusterNodesResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetContainerReportResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetContainersResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetDelegationTokenResponse;
+import org.apache.hadoop.yarn.api.protocolrecords.GetLabelsToNodesResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetNodesToLabelsResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetQueueInfoResponse;
@@ -73,6 +74,8 @@ import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetContainersRequestPB
import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetContainersResponsePBImpl;
import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl;
import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetDelegationTokenResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetLabelsToNodesRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetLabelsToNodesResponsePBImpl;
import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetNewApplicationRequestPBImpl;
import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetNewApplicationResponsePBImpl;
import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetNodesToLabelsRequestPBImpl;
@@ -114,6 +117,8 @@ import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetContainerReportRequestP
import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetContainerReportResponseProto;
import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetContainersRequestProto;
import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetContainersResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetLabelsToNodesRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetLabelsToNodesResponseProto;
import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetNewApplicationRequestProto;
import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetNewApplicationResponseProto;
import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetNodesToLabelsRequestProto;
@@ -471,6 +476,22 @@ public class ApplicationClientProtocolPBServiceImpl implements ApplicationClient
}
@Override
+ public GetLabelsToNodesResponseProto getLabelsToNodes(
+ RpcController controller, GetLabelsToNodesRequestProto proto)
+ throws ServiceException {
+ GetLabelsToNodesRequestPBImpl request =
+ new GetLabelsToNodesRequestPBImpl(proto);
+ try {
+ GetLabelsToNodesResponse response = real.getLabelsToNodes(request);
+ return ((GetLabelsToNodesResponsePBImpl) response).getProto();
+ } catch (YarnException e) {
+ throw new ServiceException(e);
+ } catch (IOException e) {
+ throw new ServiceException(e);
+ }
+ }
+
+ @Override
public GetClusterNodeLabelsResponseProto getClusterNodeLabels(
RpcController controller, GetClusterNodeLabelsRequestProto proto)
throws ServiceException {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetLabelsToNodesRequestPBImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetLabelsToNodesRequestPBImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetLabelsToNodesRequestPBImpl.java
new file mode 100644
index 0000000..19ef550
--- /dev/null
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetLabelsToNodesRequestPBImpl.java
@@ -0,0 +1,121 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.api.protocolrecords.impl.pb;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.hadoop.yarn.api.protocolrecords.GetLabelsToNodesRequest;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetLabelsToNodesRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetLabelsToNodesRequestProtoOrBuilder;
+
+import com.google.protobuf.TextFormat;
+
+public class GetLabelsToNodesRequestPBImpl extends GetLabelsToNodesRequest {
+
+ Set<String> nodeLabels = null;
+
+ GetLabelsToNodesRequestProto proto =
+ GetLabelsToNodesRequestProto.getDefaultInstance();
+ GetLabelsToNodesRequestProto.Builder builder = null;
+ boolean viaProto = false;
+
+ public GetLabelsToNodesRequestPBImpl() {
+ builder = GetLabelsToNodesRequestProto.newBuilder();
+ }
+
+ public GetLabelsToNodesRequestPBImpl(GetLabelsToNodesRequestProto proto) {
+ this.proto = proto;
+ viaProto = true;
+ }
+
+ public GetLabelsToNodesRequestProto getProto() {
+ mergeLocalToProto();
+ proto = viaProto ? proto : builder.build();
+ viaProto = true;
+ return proto;
+ }
+
+ private void mergeLocalToProto() {
+ if (viaProto)
+ maybeInitBuilder();
+ mergeLocalToBuilder();
+ proto = builder.build();
+ viaProto = true;
+ }
+
+ private void mergeLocalToBuilder() {
+ if (nodeLabels != null && !nodeLabels.isEmpty()) {
+ builder.clearNodeLabels();
+ builder.addAllNodeLabels(nodeLabels);
+ }
+ }
+
+ private void maybeInitBuilder() {
+ if (viaProto || builder == null) {
+ builder = GetLabelsToNodesRequestProto.newBuilder(proto);
+ }
+ viaProto = false;
+ }
+
+ private void initNodeLabels() {
+ if (this.nodeLabels != null) {
+ return;
+ }
+ GetLabelsToNodesRequestProtoOrBuilder p = viaProto ? proto : builder;
+ List<String> nodeLabelsList = p.getNodeLabelsList();
+ this.nodeLabels = new HashSet<String>();
+ this.nodeLabels.addAll(nodeLabelsList);
+ }
+
+ @Override
+ public Set<String> getNodeLabels() {
+ initNodeLabels();
+ return this.nodeLabels;
+ }
+
+ @Override
+ public void setNodeLabels(Set<String> nodeLabels) {
+ maybeInitBuilder();
+ if (nodeLabels == null)
+ builder.clearNodeLabels();
+ this.nodeLabels = nodeLabels;
+ }
+
+ @Override
+ public int hashCode() {
+ return getProto().hashCode();
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ if (other == null)
+ return false;
+ if (other.getClass().isAssignableFrom(this.getClass())) {
+ return this.getProto().equals(this.getClass().cast(other).getProto());
+ }
+ return false;
+ }
+
+ @Override
+ public String toString() {
+ return TextFormat.shortDebugString(getProto());
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetLabelsToNodesResponsePBImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetLabelsToNodesResponsePBImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetLabelsToNodesResponsePBImpl.java
new file mode 100644
index 0000000..e197997
--- /dev/null
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetLabelsToNodesResponsePBImpl.java
@@ -0,0 +1,184 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.api.protocolrecords.impl.pb;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+
+import org.apache.hadoop.classification.InterfaceAudience.Public;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+import org.apache.hadoop.yarn.api.records.NodeId;
+import org.apache.hadoop.yarn.api.records.impl.pb.NodeIdPBImpl;
+import org.apache.hadoop.yarn.proto.YarnProtos.NodeIdProto;
+import org.apache.hadoop.yarn.api.protocolrecords.GetLabelsToNodesResponse;
+
+import org.apache.hadoop.yarn.proto.YarnProtos.LabelsToNodeIdsProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetLabelsToNodesResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetLabelsToNodesResponseProtoOrBuilder;
+
+public class GetLabelsToNodesResponsePBImpl extends
+ GetLabelsToNodesResponse {
+ GetLabelsToNodesResponseProto proto = GetLabelsToNodesResponseProto
+ .getDefaultInstance();
+ GetLabelsToNodesResponseProto.Builder builder = null;
+ boolean viaProto = false;
+
+ private Map<String, Set<NodeId>> labelsToNodes;
+
+ public GetLabelsToNodesResponsePBImpl() {
+ this.builder = GetLabelsToNodesResponseProto.newBuilder();
+ }
+
+ public GetLabelsToNodesResponsePBImpl(GetLabelsToNodesResponseProto proto) {
+ this.proto = proto;
+ this.viaProto = true;
+ }
+
+ private void initLabelsToNodes() {
+ if (this.labelsToNodes != null) {
+ return;
+ }
+ GetLabelsToNodesResponseProtoOrBuilder p = viaProto ? proto : builder;
+ List<LabelsToNodeIdsProto> list = p.getLabelsToNodesList();
+ this.labelsToNodes = new HashMap<String, Set<NodeId>>();
+
+ for (LabelsToNodeIdsProto c : list) {
+ Set<NodeId> setNodes = new HashSet<NodeId>();
+ for(NodeIdProto n : c.getNodeIdList()) {
+ NodeId node = new NodeIdPBImpl(n);
+ setNodes.add(node);
+ }
+ if(!setNodes.isEmpty()) {
+ this.labelsToNodes.put(c.getNodeLabels(), setNodes);
+ }
+ }
+ }
+
+ private void maybeInitBuilder() {
+ if (viaProto || builder == null) {
+ builder = GetLabelsToNodesResponseProto.newBuilder(proto);
+ }
+ viaProto = false;
+ }
+
+ private void addLabelsToNodesToProto() {
+ maybeInitBuilder();
+ builder.clearLabelsToNodes();
+ if (labelsToNodes == null) {
+ return;
+ }
+ Iterable<LabelsToNodeIdsProto> iterable =
+ new Iterable<LabelsToNodeIdsProto>() {
+ @Override
+ public Iterator<LabelsToNodeIdsProto> iterator() {
+ return new Iterator<LabelsToNodeIdsProto>() {
+
+ Iterator<Entry<String, Set<NodeId>>> iter =
+ labelsToNodes.entrySet().iterator();
+
+ @Override
+ public void remove() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public LabelsToNodeIdsProto next() {
+ Entry<String, Set<NodeId>> now = iter.next();
+ Set<NodeIdProto> nodeProtoSet = new HashSet<NodeIdProto>();
+ for(NodeId n : now.getValue()) {
+ nodeProtoSet.add(convertToProtoFormat(n));
+ }
+ return LabelsToNodeIdsProto.newBuilder()
+ .setNodeLabels(now.getKey()).addAllNodeId(nodeProtoSet)
+ .build();
+ }
+
+ @Override
+ public boolean hasNext() {
+ return iter.hasNext();
+ }
+ };
+ }
+ };
+ builder.addAllLabelsToNodes(iterable);
+ }
+
+ private void mergeLocalToBuilder() {
+ if (this.labelsToNodes != null) {
+ addLabelsToNodesToProto();
+ }
+ }
+
+ private void mergeLocalToProto() {
+ if (viaProto)
+ maybeInitBuilder();
+ mergeLocalToBuilder();
+ proto = builder.build();
+ viaProto = true;
+ }
+
+ public GetLabelsToNodesResponseProto getProto() {
+ mergeLocalToProto();
+ proto = viaProto ? proto : builder.build();
+ viaProto = true;
+ return proto;
+ }
+
+ private NodeIdProto convertToProtoFormat(NodeId t) {
+ return ((NodeIdPBImpl)t).getProto();
+ }
+
+ @Override
+ public int hashCode() {
+ assert false : "hashCode not designed";
+ return 0;
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ if (other == null)
+ return false;
+ if (other.getClass().isAssignableFrom(this.getClass())) {
+ return this.getProto().equals(this.getClass().cast(other).getProto());
+ }
+ return false;
+ }
+
+ @Override
+ @Public
+ @Evolving
+ public void setLabelsToNodes(Map<String, Set<NodeId>> map) {
+ initLabelsToNodes();
+ labelsToNodes.clear();
+ labelsToNodes.putAll(map);
+ }
+
+ @Override
+ @Public
+ @Evolving
+ public Map<String, Set<NodeId>> getLabelsToNodes() {
+ initLabelsToNodes();
+ return this.labelsToNodes;
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java
index b8f6e9c..8b48798 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java
@@ -1003,4 +1003,16 @@ public class TestPBImplRecords {
validatePBImplRecord(GetNodesToLabelsResponsePBImpl.class,
GetNodesToLabelsResponseProto.class);
}
+
+ @Test
+ public void testGetLabelsToNodesRequestPBImpl() throws Exception {
+ validatePBImplRecord(GetLabelsToNodesRequestPBImpl.class,
+ GetLabelsToNodesRequestProto.class);
+ }
+
+ @Test
+ public void testGetLabelsToNodesResponsePBImpl() throws Exception {
+ validatePBImplRecord(GetLabelsToNodesResponsePBImpl.class,
+ GetLabelsToNodesResponseProto.class);
+ }
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
index 38f7b93..8541766 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
@@ -71,6 +71,8 @@ import org.apache.hadoop.yarn.api.protocolrecords.GetContainersRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetContainersResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetDelegationTokenRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetDelegationTokenResponse;
+import org.apache.hadoop.yarn.api.protocolrecords.GetLabelsToNodesRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetLabelsToNodesResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetNodesToLabelsRequest;
@@ -1223,6 +1225,19 @@ public class ClientRMService extends AbstractService implements
}
@Override
+ public GetLabelsToNodesResponse getLabelsToNodes(
+ GetLabelsToNodesRequest request) throws YarnException, IOException {
+ RMNodeLabelsManager labelsMgr = rmContext.getNodeLabelManager();
+ if (request.getNodeLabels() == null || request.getNodeLabels().isEmpty()) {
+ return GetLabelsToNodesResponse.newInstance(
+ labelsMgr.getLabelsToNodes());
+ } else {
+ return GetLabelsToNodesResponse.newInstance(
+ labelsMgr.getLabelsToNodes(request.getNodeLabels()));
+ }
+ }
+
+ @Override
public GetClusterNodeLabelsResponse getClusterNodeLabels(
GetClusterNodeLabelsRequest request) throws YarnException, IOException {
RMNodeLabelsManager labelsMgr = rmContext.getNodeLabelManager();
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d49ae725/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
index a684346..dd2b3f8 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
@@ -70,6 +70,8 @@ import org.apache.hadoop.yarn.api.protocolrecords.GetContainerReportRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetContainerReportResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetContainersRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetContainersResponse;
+import org.apache.hadoop.yarn.api.protocolrecords.GetLabelsToNodesRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetLabelsToNodesResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetNodesToLabelsRequest;
import org.apache.hadoop.yarn.api.protocolrecords.GetNodesToLabelsResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetQueueInfoRequest;
@@ -1437,4 +1439,77 @@ public class TestClientRMService {
rpc.stopProxy(client, conf);
rm.close();
}
+
+ @Test
+ public void testGetLabelsToNodes() throws Exception {
+ MockRM rm = new MockRM() {
+ protected ClientRMService createClientRMService() {
+ return new ClientRMService(this.rmContext, scheduler,
+ this.rmAppManager, this.applicationACLsManager,
+ this.queueACLsManager, this.getRMContext()
+ .getRMDelegationTokenSecretManager());
+ };
+ };
+ rm.start();
+ RMNodeLabelsManager labelsMgr = rm.getRMContext().getNodeLabelManager();
+ labelsMgr.addToCluserNodeLabels(ImmutableSet.of("x", "y", "z"));
+
+ Map<NodeId, Set<String>> map = new HashMap<NodeId, Set<String>>();
+ map.put(NodeId.newInstance("host1", 0), ImmutableSet.of("x"));
+ map.put(NodeId.newInstance("host1", 1), ImmutableSet.of("z"));
+ map.put(NodeId.newInstance("host2", 0), ImmutableSet.of("y"));
+ map.put(NodeId.newInstance("host3", 0), ImmutableSet.of("y"));
+ map.put(NodeId.newInstance("host3", 1), ImmutableSet.of("z"));
+ labelsMgr.replaceLabelsOnNode(map);
+
+ // Create a client.
+ Configuration conf = new Configuration();
+ YarnRPC rpc = YarnRPC.create(conf);
+ InetSocketAddress rmAddress = rm.getClientRMService().getBindAddress();
+ LOG.info("Connecting to ResourceManager at " + rmAddress);
+ ApplicationClientProtocol client =
+ (ApplicationClientProtocol) rpc.getProxy(
+ ApplicationClientProtocol.class, rmAddress, conf);
+
+ // Get node labels collection
+ GetClusterNodeLabelsResponse response =
+ client.getClusterNodeLabels(GetClusterNodeLabelsRequest.newInstance());
+ Assert.assertTrue(response.getNodeLabels().containsAll(
+ Arrays.asList("x", "y", "z")));
+
+ // Get labels to nodes mapping
+ GetLabelsToNodesResponse response1 =
+ client.getLabelsToNodes(GetLabelsToNodesRequest.newInstance());
+ Map<String, Set<NodeId>> labelsToNodes = response1.getLabelsToNodes();
+ Assert.assertTrue(
+ labelsToNodes.keySet().containsAll(Arrays.asList("x", "y", "z")));
+ Assert.assertTrue(
+ labelsToNodes.get("x").containsAll(Arrays.asList(
+ NodeId.newInstance("host1", 0))));
+ Assert.assertTrue(
+ labelsToNodes.get("y").containsAll(Arrays.asList(
+ NodeId.newInstance("host2", 0), NodeId.newInstance("host3", 0))));
+ Assert.assertTrue(
+ labelsToNodes.get("z").containsAll(Arrays.asList(
+ NodeId.newInstance("host1", 1), NodeId.newInstance("host3", 1))));
+
+ // Get labels to nodes mapping for specific labels
+ Set<String> setlabels =
+ new HashSet<String>(Arrays.asList(new String[]{"x", "z"}));
+ GetLabelsToNodesResponse response2 =
+ client.getLabelsToNodes(GetLabelsToNodesRequest.newInstance(setlabels));
+ labelsToNodes = response2.getLabelsToNodes();
+ Assert.assertTrue(
+ labelsToNodes.keySet().containsAll(Arrays.asList("x", "z")));
+ Assert.assertTrue(
+ labelsToNodes.get("x").containsAll(Arrays.asList(
+ NodeId.newInstance("host1", 0))));
+ Assert.assertTrue(
+ labelsToNodes.get("z").containsAll(Arrays.asList(
+ NodeId.newInstance("host1", 1), NodeId.newInstance("host3", 1))));
+ Assert.assertEquals(labelsToNodes.get("y"), null);
+
+ rpc.stopProxy(client, conf);
+ rm.close();
+ }
}
[22/52] [abbrv] hadoop git commit: YARN-933. Fixed
InvalidStateTransitonException at FINAL_SAVING state in RMApp. Contributed by
Rohith Sharmaks
Posted by zh...@apache.org.
YARN-933. Fixed InvalidStateTransitonException at FINAL_SAVING state in RMApp. Contributed by Rohith Sharmaks
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/c0d9b939
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/c0d9b939
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/c0d9b939
Branch: refs/heads/HDFS-7285
Commit: c0d9b93953767608dfe429ddb9bd4c1c3bd3debf
Parents: d49ae72
Author: Jian He <ji...@apache.org>
Authored: Thu Feb 19 15:42:39 2015 -0800
Committer: Jian He <ji...@apache.org>
Committed: Thu Feb 19 15:42:39 2015 -0800
----------------------------------------------------------------------
hadoop-yarn-project/CHANGES.txt | 3 ++
.../rmapp/attempt/RMAppAttemptImpl.java | 2 ++
.../attempt/TestRMAppAttemptTransitions.java | 30 ++++++++++++++++++++
3 files changed, 35 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c0d9b939/hadoop-yarn-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index aab3496..cac6680 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -623,6 +623,9 @@ Release 2.7.0 - UNRELEASED
YARN-1615. Fix typos in description about delay scheduling. (Akira Ajisaka via
ozawa)
+ YARN-933. Fixed InvalidStateTransitonException at FINAL_SAVING state in
+ RMApp. (Rohith Sharmaks via jianhe)
+
Release 2.6.0 - 2014-11-18
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c0d9b939/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
index 1a19eee..1be1727 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
@@ -354,6 +354,8 @@ public class RMAppAttemptImpl implements RMAppAttempt, Recoverable {
EnumSet.of(
RMAppAttemptEventType.UNREGISTERED,
RMAppAttemptEventType.STATUS_UPDATE,
+ RMAppAttemptEventType.LAUNCHED,
+ RMAppAttemptEventType.LAUNCH_FAILED,
// should be fixed to reject container allocate request at Final
// Saving in scheduler
RMAppAttemptEventType.CONTAINER_ALLOCATED,
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c0d9b939/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
index 9f5ae79..c074ad9 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
@@ -919,6 +919,36 @@ public class TestRMAppAttemptTransitions {
testAppAttemptFailedState(amContainer, diagnostics);
}
+ @Test(timeout = 10000)
+ public void testLaunchedAtFinalSaving() {
+ Container amContainer = allocateApplicationAttempt();
+
+ // ALLOCATED->FINAL_SAVING
+ applicationAttempt.handle(new RMAppAttemptEvent(applicationAttempt
+ .getAppAttemptId(), RMAppAttemptEventType.KILL));
+ assertEquals(RMAppAttemptState.FINAL_SAVING,
+ applicationAttempt.getAppAttemptState());
+
+ // verify for both launched and launch_failed transitions in final_saving
+ applicationAttempt.handle(new RMAppAttemptEvent(applicationAttempt
+ .getAppAttemptId(), RMAppAttemptEventType.LAUNCHED));
+ applicationAttempt.handle(new RMAppAttemptLaunchFailedEvent(
+ applicationAttempt.getAppAttemptId(), "Launch Failed"));
+
+ assertEquals(RMAppAttemptState.FINAL_SAVING,
+ applicationAttempt.getAppAttemptState());
+
+ testAppAttemptKilledState(amContainer, EMPTY_DIAGNOSTICS);
+
+ // verify for both launched and launch_failed transitions in killed
+ applicationAttempt.handle(new RMAppAttemptEvent(applicationAttempt
+ .getAppAttemptId(), RMAppAttemptEventType.LAUNCHED));
+ applicationAttempt.handle(new RMAppAttemptLaunchFailedEvent(
+ applicationAttempt.getAppAttemptId(), "Launch Failed"));
+ assertEquals(RMAppAttemptState.KILLED,
+ applicationAttempt.getAppAttemptState());
+ }
+
@Test
public void testAMCrashAtAllocated() {
Container amContainer = allocateApplicationAttempt();
[08/52] [abbrv] hadoop git commit: MAPREDUCE-6261.
NullPointerException if MapOutputBuffer.flush invoked twice. Contributed by
Tsuyoshi OZAWA
Posted by zh...@apache.org.
MAPREDUCE-6261. NullPointerException if MapOutputBuffer.flush invoked twice. Contributed by Tsuyoshi OZAWA
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/4981d082
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/4981d082
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/4981d082
Branch: refs/heads/HDFS-7285
Commit: 4981d082d4f3c82d1c2c900c7488b83bf20301cc
Parents: 1714609
Author: Jason Lowe <jl...@apache.org>
Authored: Wed Feb 18 19:28:02 2015 +0000
Committer: Jason Lowe <jl...@apache.org>
Committed: Wed Feb 18 19:28:02 2015 +0000
----------------------------------------------------------------------
hadoop-mapreduce-project/CHANGES.txt | 3 +++
.../src/main/java/org/apache/hadoop/mapred/MapTask.java | 4 ++++
2 files changed, 7 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/4981d082/hadoop-mapreduce-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/CHANGES.txt b/hadoop-mapreduce-project/CHANGES.txt
index e944d82..7f4c3e7 100644
--- a/hadoop-mapreduce-project/CHANGES.txt
+++ b/hadoop-mapreduce-project/CHANGES.txt
@@ -379,6 +379,9 @@ Release 2.7.0 - UNRELEASED
MAPREDUCE-4286. TestClientProtocolProviderImpls passes on failure
conditions. (Devaraj K via ozawa)
+ MAPREDUCE-6261. NullPointerException if MapOutputBuffer.flush invoked
+ twice (Tsuyoshi OZAWA via jlowe)
+
Release 2.6.1 - UNRELEASED
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/4981d082/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
index 1a4901b..8094317 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
@@ -1458,6 +1458,10 @@ public class MapTask extends Task {
public void flush() throws IOException, ClassNotFoundException,
InterruptedException {
LOG.info("Starting flush of map output");
+ if (kvbuffer == null) {
+ LOG.info("kvbuffer is null. Skipping flush.");
+ return;
+ }
spillLock.lock();
try {
while (spillInProgress) {
[15/52] [abbrv] hadoop git commit: YARN-1615. Fix typos in delay
scheduler's description. Contributed by Akira Ajisaka.
Posted by zh...@apache.org.
YARN-1615. Fix typos in delay scheduler's description. Contributed by Akira Ajisaka.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/b8a14efd
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/b8a14efd
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/b8a14efd
Branch: refs/heads/HDFS-7285
Commit: b8a14efdf535d42bcafa58d380bd2c7f4d36f8cb
Parents: 1c03376
Author: Tsuyoshi Ozawa <oz...@apache.org>
Authored: Thu Feb 19 10:24:07 2015 +0900
Committer: Tsuyoshi Ozawa <oz...@apache.org>
Committed: Thu Feb 19 10:24:07 2015 +0900
----------------------------------------------------------------------
hadoop-yarn-project/CHANGES.txt | 3 +++
.../server/resourcemanager/scheduler/fair/FSAppAttempt.java | 6 +++---
2 files changed, 6 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b8a14efd/hadoop-yarn-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 91ce11f..6b2b878 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -617,6 +617,9 @@ Release 2.7.0 - UNRELEASED
YARN-3132. RMNodeLabelsManager should remove node from node-to-label mapping
when node becomes deactivated. (Wangda Tan via jianhe)
+ YARN-1615. Fix typos in description about delay scheduling. (Akira Ajisaka via
+ ozawa)
+
Release 2.6.0 - 2014-11-18
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b8a14efd/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
index 7d26396..67103d1 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
@@ -80,10 +80,10 @@ public class FSAppAttempt extends SchedulerApplicationAttempt
/**
* Delay scheduling: We often want to prioritize scheduling of node-local
- * containers over rack-local or off-switch containers. To acheive this
- * we first only allow node-local assigments for a given prioirty level,
+ * containers over rack-local or off-switch containers. To achieve this
+ * we first only allow node-local assignments for a given priority level,
* then relax the locality threshold once we've had a long enough period
- * without succesfully scheduling. We measure both the number of "missed"
+ * without successfully scheduling. We measure both the number of "missed"
* scheduling opportunities since the last container was scheduled
* at the current allowed level and the time since the last container
* was scheduled. Currently we use only the former.
[44/52] [abbrv] hadoop git commit: HDFS-7652. Process block reports
for erasure coded blocks. Contributed by Zhe Zhang
Posted by zh...@apache.org.
HDFS-7652. Process block reports for erasure coded blocks. Contributed by Zhe Zhang
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/6caf9c77
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/6caf9c77
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/6caf9c77
Branch: refs/heads/HDFS-7285
Commit: 6caf9c77c60321897c536101a801fcdba8e981ff
Parents: 6d6fffc
Author: Zhe Zhang <zh...@apache.org>
Authored: Mon Feb 9 10:27:14 2015 -0800
Committer: Zhe Zhang <zh...@apache.org>
Committed: Mon Feb 23 11:19:51 2015 -0800
----------------------------------------------------------------------
.../server/blockmanagement/BlockIdManager.java | 8 ++++++++
.../hdfs/server/blockmanagement/BlockManager.java | 18 +++++++++++++-----
2 files changed, 21 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6caf9c77/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
index c8b9d20..e7f8a05 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
@@ -211,4 +211,12 @@ public class BlockIdManager {
.LAST_RESERVED_BLOCK_ID);
generationStampV1Limit = GenerationStamp.GRANDFATHER_GENERATION_STAMP;
}
+
+ public static boolean isStripedBlockID(long id) {
+ return id < 0;
+ }
+
+ public static long convertToGroupID(long id) {
+ return id & (~(HdfsConstants.MAX_BLOCKS_IN_GROUP - 1));
+ }
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6caf9c77/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
index 58a8b94..f4369b6 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
@@ -1876,7 +1876,7 @@ public class BlockManager {
break;
}
- BlockInfoContiguous bi = blocksMap.getStoredBlock(b);
+ BlockInfoContiguous bi = getStoredBlock(b);
if (bi == null) {
if (LOG.isDebugEnabled()) {
LOG.debug("BLOCK* rescanPostponedMisreplicatedBlocks: " +
@@ -1981,7 +1981,7 @@ public class BlockManager {
continue;
}
- BlockInfoContiguous storedBlock = blocksMap.getStoredBlock(iblk);
+ BlockInfoContiguous storedBlock = getStoredBlock(iblk);
// If block does not belong to any file, we are done.
if (storedBlock == null) continue;
@@ -2123,7 +2123,7 @@ public class BlockManager {
}
// find block by blockId
- BlockInfoContiguous storedBlock = blocksMap.getStoredBlock(block);
+ BlockInfoContiguous storedBlock = getStoredBlock(block);
if(storedBlock == null) {
// If blocksMap does not contain reported block id,
// the replica should be removed from the data-node.
@@ -2414,7 +2414,7 @@ public class BlockManager {
DatanodeDescriptor node = storageInfo.getDatanodeDescriptor();
if (block instanceof BlockInfoContiguousUnderConstruction) {
//refresh our copy in case the block got completed in another thread
- storedBlock = blocksMap.getStoredBlock(block);
+ storedBlock = getStoredBlock(block);
} else {
storedBlock = block;
}
@@ -3360,7 +3360,15 @@ public class BlockManager {
}
public BlockInfoContiguous getStoredBlock(Block block) {
- return blocksMap.getStoredBlock(block);
+ BlockInfoContiguous info = null;
+ if (BlockIdManager.isStripedBlockID(block.getBlockId())) {
+ info = blocksMap.getStoredBlock(
+ new Block(BlockIdManager.convertToGroupID(block.getBlockId())));
+ }
+ if (info == null) {
+ info = blocksMap.getStoredBlock(block);
+ }
+ return info;
}
/** updates a block in under replication queue */
[20/52] [abbrv] hadoop git commit: HADOOP-9087. Queue size metric for
metric sinks isn't actually maintained. Contributed by Akira AJISAKA
Posted by zh...@apache.org.
HADOOP-9087. Queue size metric for metric sinks isn't actually maintained. Contributed by Akira AJISAKA
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f0f29926
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f0f29926
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f0f29926
Branch: refs/heads/HDFS-7285
Commit: f0f299268625af275522f55d5bfc43118c31bdd8
Parents: 2fd02af
Author: Jason Lowe <jl...@apache.org>
Authored: Thu Feb 19 17:30:07 2015 +0000
Committer: Jason Lowe <jl...@apache.org>
Committed: Thu Feb 19 17:30:07 2015 +0000
----------------------------------------------------------------------
hadoop-common-project/hadoop-common/CHANGES.txt | 3 ++
.../metrics2/impl/MetricsSinkAdapter.java | 15 +++++-
.../hadoop-common/src/site/markdown/Metrics.md | 2 +-
.../metrics2/impl/TestMetricsSystemImpl.java | 50 ++++++++++++++++++++
4 files changed, 67 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f0f29926/hadoop-common-project/hadoop-common/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt b/hadoop-common-project/hadoop-common/CHANGES.txt
index c01e3d6..8d3f9f5 100644
--- a/hadoop-common-project/hadoop-common/CHANGES.txt
+++ b/hadoop-common-project/hadoop-common/CHANGES.txt
@@ -973,6 +973,9 @@ Release 2.7.0 - UNRELEASED
HADOOP-11595. Add default implementation for AbstractFileSystem#truncate.
(yliu)
+ HADOOP-9087. Queue size metric for metric sinks isn't actually maintained
+ (Akira AJISAKA via jlowe)
+
Release 2.6.1 - UNRELEASED
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f0f29926/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSinkAdapter.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSinkAdapter.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSinkAdapter.java
index 9add494..ed52317 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSinkAdapter.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSinkAdapter.java
@@ -95,7 +95,10 @@ class MetricsSinkAdapter implements SinkQueue.Consumer<MetricsBuffer> {
boolean putMetrics(MetricsBuffer buffer, long logicalTime) {
if (logicalTime % period == 0) {
LOG.debug("enqueue, logicalTime="+ logicalTime);
- if (queue.enqueue(buffer)) return true;
+ if (queue.enqueue(buffer)) {
+ refreshQueueSizeGauge();
+ return true;
+ }
dropped.incr();
return false;
}
@@ -105,7 +108,9 @@ class MetricsSinkAdapter implements SinkQueue.Consumer<MetricsBuffer> {
public boolean putMetricsImmediate(MetricsBuffer buffer) {
WaitableMetricsBuffer waitableBuffer =
new WaitableMetricsBuffer(buffer);
- if (!queue.enqueue(waitableBuffer)) {
+ if (queue.enqueue(waitableBuffer)) {
+ refreshQueueSizeGauge();
+ } else {
LOG.warn(name + " has a full queue and can't consume the given metrics.");
dropped.incr();
return false;
@@ -127,6 +132,7 @@ class MetricsSinkAdapter implements SinkQueue.Consumer<MetricsBuffer> {
while (!stopping) {
try {
queue.consumeAll(this);
+ refreshQueueSizeGauge();
retryDelay = firstRetryDelay;
n = retryCount;
inError = false;
@@ -151,12 +157,17 @@ class MetricsSinkAdapter implements SinkQueue.Consumer<MetricsBuffer> {
"suppressing further error messages", e);
}
queue.clear();
+ refreshQueueSizeGauge();
inError = true; // Don't keep complaining ad infinitum
}
}
}
}
+ private void refreshQueueSizeGauge() {
+ qsize.set(queue.size());
+ }
+
@Override
public void consume(MetricsBuffer buffer) {
long ts = 0;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f0f29926/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md b/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
index dbcf0d8..6953c3b 100644
--- a/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
+++ b/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
@@ -434,7 +434,7 @@ MetricsSystem shows the statistics for metrics snapshots and publishes. Each met
| `Sink_`*instance*`NumOps` | Total number of sink operations for the *instance* |
| `Sink_`*instance*`AvgTime` | Average time in milliseconds of sink operations for the *instance* |
| `Sink_`*instance*`Dropped` | Total number of dropped sink operations for the *instance* |
-| `Sink_`*instance*`Qsize` | Current queue length of sink operations (BUT always set to 0 because nothing to increment this metrics, see [HADOOP-9941](https://issues.apache.org/jira/browse/HADOOP-9941)) |
+| `Sink_`*instance*`Qsize` | Current queue length of sink operations |
default context
===============
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f0f29926/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
index 4c2ebc8..0f7b15f 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
@@ -29,7 +29,9 @@ import org.junit.runner.RunWith;
import org.mockito.ArgumentCaptor;
import org.mockito.Captor;
+import org.mockito.invocation.InvocationOnMock;
import org.mockito.runners.MockitoJUnitRunner;
+import org.mockito.stubbing.Answer;
import static org.junit.Assert.*;
import static org.mockito.Mockito.*;
@@ -434,6 +436,54 @@ public class TestMetricsSystemImpl {
new MetricGaugeInt(MsInfo.NumActiveSinks, 3)));
}
+ @Test
+ public void testQSize() throws Exception {
+ new ConfigBuilder().add("*.period", 8)
+ .add("test.sink.test.class", TestSink.class.getName())
+ .save(TestMetricsConfig.getTestFilename("hadoop-metrics2-test"));
+ MetricsSystemImpl ms = new MetricsSystemImpl("Test");
+ final CountDownLatch proceedSignal = new CountDownLatch(1);
+ final CountDownLatch reachedPutMetricSignal = new CountDownLatch(1);
+ ms.start();
+ try {
+ MetricsSink slowSink = mock(MetricsSink.class);
+ MetricsSink dataSink = mock(MetricsSink.class);
+ ms.registerSink("slowSink",
+ "The sink that will wait on putMetric", slowSink);
+ ms.registerSink("dataSink",
+ "The sink I'll use to get info about slowSink", dataSink);
+ doAnswer(new Answer() {
+ @Override
+ public Object answer(InvocationOnMock invocation) throws Throwable {
+ reachedPutMetricSignal.countDown();
+ proceedSignal.await();
+ return null;
+ }
+ }).when(slowSink).putMetrics(any(MetricsRecord.class));
+
+ // trigger metric collection first time
+ ms.onTimerEvent();
+ assertTrue(reachedPutMetricSignal.await(1, TimeUnit.SECONDS));
+ // Now that the slow sink is still processing the first metric,
+ // its queue length should be 1 for the second collection.
+ ms.onTimerEvent();
+ verify(dataSink, timeout(500).times(2)).putMetrics(r1.capture());
+ List<MetricsRecord> mr = r1.getAllValues();
+ Number qSize = Iterables.find(mr.get(1).metrics(),
+ new Predicate<AbstractMetric>() {
+ @Override
+ public boolean apply(@Nullable AbstractMetric input) {
+ assert input != null;
+ return input.name().equals("Sink_slowSinkQsize");
+ }
+ }).value();
+ assertEquals(1, qSize);
+ } finally {
+ proceedSignal.countDown();
+ ms.stop();
+ }
+ }
+
@Metrics(context="test")
private static class TestSource {
@Metric("C1 desc") MutableCounterLong c1;
[23/52] [abbrv] hadoop git commit: HDFS-7752. Improve description for
"dfs.namenode.num.extra.edits.retained" and
"dfs.namenode.num.checkpoints.retained" properties on hdfs-default.xml.
Contributed by Wellington Chevreuil.
Posted by zh...@apache.org.
HDFS-7752. Improve description for "dfs.namenode.num.extra.edits.retained" and "dfs.namenode.num.checkpoints.retained" properties on hdfs-default.xml. Contributed by Wellington Chevreuil.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/b9a17909
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/b9a17909
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/b9a17909
Branch: refs/heads/HDFS-7285
Commit: b9a17909ba39898120a096cb6ae90104640690db
Parents: c0d9b93
Author: Harsh J <ha...@cloudera.com>
Authored: Fri Feb 20 19:20:41 2015 +0530
Committer: Harsh J <ha...@cloudera.com>
Committed: Fri Feb 20 19:20:41 2015 +0530
----------------------------------------------------------------------
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 5 +++++
.../hadoop-hdfs/src/main/resources/hdfs-default.xml | 15 +++++++++++----
2 files changed, 16 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b9a17909/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index 80a086a..5f3cc02 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -341,6 +341,11 @@ Release 2.7.0 - UNRELEASED
IMPROVEMENTS
+ HDFS-7752. Improve description for
+ "dfs.namenode.num.extra.edits.retained"
+ and "dfs.namenode.num.checkpoints.retained" properties on
+ hdfs-default.xml (Wellington Chevreuil via harsh)
+
HDFS-7055. Add tracing to DFSInputStream (cmccabe)
HDFS-7186. Document the "hadoop trace" command. (Masatake Iwasaki via Colin
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b9a17909/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
index 9299ea3..85d2273 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
@@ -852,9 +852,9 @@
<property>
<name>dfs.namenode.num.checkpoints.retained</name>
<value>2</value>
- <description>The number of image checkpoint files that will be retained by
+ <description>The number of image checkpoint files (fsimage_*) that will be retained by
the NameNode and Secondary NameNode in their storage directories. All edit
- logs necessary to recover an up-to-date namespace from the oldest retained
+ logs (stored on edits_* files) necessary to recover an up-to-date namespace from the oldest retained
checkpoint will also be retained.
</description>
</property>
@@ -863,8 +863,15 @@
<name>dfs.namenode.num.extra.edits.retained</name>
<value>1000000</value>
<description>The number of extra transactions which should be retained
- beyond what is minimally necessary for a NN restart. This can be useful for
- audit purposes or for an HA setup where a remote Standby Node may have
+ beyond what is minimally necessary for a NN restart.
+ It does not translate directly to file's age, or the number of files kept,
+ but to the number of transactions (here "edits" means transactions).
+ One edit file may contain several transactions (edits).
+ During checkpoint, NameNode will identify the total number of edits to retain as extra by
+ checking the latest checkpoint transaction value, subtracted by the value of this property.
+ Then, it scans edits files to identify the older ones that don't include the computed range of
+ retained transactions that are to be kept around, and purges them subsequently.
+ The retainment can be useful for audit purposes or for an HA setup where a remote Standby Node may have
been offline for some time and need to have a longer backlog of retained
edits in order to start again.
Typically each edit is on the order of a few hundred bytes, so the default
[12/52] [abbrv] hadoop git commit: YARN-1514. Utility to benchmark
ZKRMStateStore#loadState for RM HA. Contributed by Tsuyoshi OZAWA
Posted by zh...@apache.org.
YARN-1514. Utility to benchmark ZKRMStateStore#loadState for RM HA. Contributed by Tsuyoshi OZAWA
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/1c033763
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/1c033763
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/1c033763
Branch: refs/heads/HDFS-7285
Commit: 1c03376300a46722d4147f5b8f37242f68dba0a2
Parents: 9a3e292
Author: Jian He <ji...@apache.org>
Authored: Wed Feb 18 16:06:55 2015 -0800
Committer: Jian He <ji...@apache.org>
Committed: Wed Feb 18 16:06:55 2015 -0800
----------------------------------------------------------------------
hadoop-project/pom.xml | 1 -
hadoop-yarn-project/CHANGES.txt | 3 +
.../hadoop-yarn-server-resourcemanager/pom.xml | 8 +-
.../org/apache/hadoop/test/YarnTestDriver.java | 60 ++++
.../recovery/RMStateStoreTestBase.java | 19 +-
.../recovery/TestZKRMStateStorePerf.java | 277 +++++++++++++++++++
6 files changed, 359 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1c033763/hadoop-project/pom.xml
----------------------------------------------------------------------
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 0c7cfc8..2c0f03a 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -828,7 +828,6 @@
<artifactId>zookeeper</artifactId>
<version>${zookeeper.version}</version>
<type>test-jar</type>
- <scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.jboss.netty</groupId>
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1c033763/hadoop-yarn-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 884b506..91ce11f 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -301,6 +301,9 @@ Release 2.7.0 - UNRELEASED
YARN-1299. Improve a log message in AppSchedulingInfo by adding application
id. (Ashutosh Jindal and Devaraj K via ozawa)
+ YARN-1514. Utility to benchmark ZKRMStateStore#loadState for RM HA.
+ (Tsuyoshi OZAWA via jianhe)
+
OPTIMIZATIONS
YARN-2990. FairScheduler's delay-scheduling always waits for node-local and
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1c033763/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
index 9bcc7c8..ff429cc 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
@@ -186,7 +186,6 @@
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<type>test-jar</type>
- <scope>test</scope>
</dependency>
<!-- 'mvn dependency:analyze' fails to detect use of this dependency -->
<dependency>
@@ -245,6 +244,13 @@
<goal>test-jar</goal>
</goals>
<phase>test-compile</phase>
+ <configuration>
+ <archive>
+ <manifest>
+ <mainClass>org.apache.hadoop.test.YarnTestDriver</mainClass>
+ </manifest>
+ </archive>
+ </configuration>
</execution>
</executions>
</plugin>
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1c033763/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/test/YarnTestDriver.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/test/YarnTestDriver.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/test/YarnTestDriver.java
new file mode 100644
index 0000000..8874ed8
--- /dev/null
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/test/YarnTestDriver.java
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.test;
+
+import org.apache.hadoop.util.ProgramDriver;
+import org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf;
+
+/**
+ * Driver for Yarn tests.
+ *
+ */
+public class YarnTestDriver {
+
+ private ProgramDriver pgd;
+
+ public YarnTestDriver() {
+ this(new ProgramDriver());
+ }
+
+ public YarnTestDriver(ProgramDriver pgd) {
+ this.pgd = pgd;
+ try {
+ pgd.addClass(TestZKRMStateStorePerf.class.getSimpleName(),
+ TestZKRMStateStorePerf.class,
+ "ZKRMStateStore i/o benchmark.");
+ } catch(Throwable e) {
+ e.printStackTrace();
+ }
+ }
+
+ public void run(String argv[]) {
+ int exitCode = -1;
+ try {
+ exitCode = pgd.run(argv);
+ } catch(Throwable e) {
+ e.printStackTrace();
+ }
+ System.exit(exitCode);
+ }
+
+ public static void main(String argv[]){
+ new YarnTestDriver().run(argv);
+ }
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1c033763/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java
index b01969b..5b53a02 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.yarn.server.resourcemanager.recovery;
+import org.apache.hadoop.yarn.event.Event;
import static org.junit.Assert.assertArrayEquals;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
@@ -79,8 +80,7 @@ public class RMStateStoreTestBase extends ClientBaseWithFixes{
public static final Log LOG = LogFactory.getLog(RMStateStoreTestBase.class);
- static class TestDispatcher implements
- Dispatcher, EventHandler<RMAppAttemptEvent> {
+ static class TestDispatcher implements Dispatcher, EventHandler<Event> {
ApplicationAttemptId attemptId;
@@ -93,8 +93,11 @@ public class RMStateStoreTestBase extends ClientBaseWithFixes{
}
@Override
- public void handle(RMAppAttemptEvent event) {
- assertEquals(attemptId, event.getApplicationAttemptId());
+ public void handle(Event event) {
+ if (event instanceof RMAppAttemptEvent) {
+ RMAppAttemptEvent rmAppAttemptEvent = (RMAppAttemptEvent) event;
+ assertEquals(attemptId, rmAppAttemptEvent.getApplicationAttemptId());
+ }
notified = true;
synchronized (this) {
notifyAll();
@@ -134,7 +137,8 @@ public class RMStateStoreTestBase extends ClientBaseWithFixes{
dispatcher.notified = false;
}
- RMApp storeApp(RMStateStore store, ApplicationId appId, long submitTime,
+ protected RMApp storeApp(RMStateStore store, ApplicationId appId,
+ long submitTime,
long startTime) throws Exception {
ApplicationSubmissionContext context =
new ApplicationSubmissionContextPBImpl();
@@ -150,7 +154,8 @@ public class RMStateStoreTestBase extends ClientBaseWithFixes{
return mockApp;
}
- ContainerId storeAttempt(RMStateStore store, ApplicationAttemptId attemptId,
+ protected ContainerId storeAttempt(RMStateStore store,
+ ApplicationAttemptId attemptId,
String containerIdStr, Token<AMRMTokenIdentifier> appToken,
SecretKey clientTokenMasterKey, TestDispatcher dispatcher)
throws Exception {
@@ -474,7 +479,7 @@ public class RMStateStoreTestBase extends ClientBaseWithFixes{
}
- private Token<AMRMTokenIdentifier> generateAMRMToken(
+ protected Token<AMRMTokenIdentifier> generateAMRMToken(
ApplicationAttemptId attemptId,
AMRMTokenSecretManager appTokenMgr) {
Token<AMRMTokenIdentifier> appToken =
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1c033763/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStorePerf.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStorePerf.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStorePerf.java
new file mode 100644
index 0000000..654b357
--- /dev/null
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStorePerf.java
@@ -0,0 +1,277 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.recovery;
+
+import com.google.common.base.Optional;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.Set;
+import javax.crypto.SecretKey;
+import org.apache.hadoop.security.token.Token;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.hadoop.util.Tool;
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ContainerId;
+import org.apache.hadoop.yarn.security.AMRMTokenIdentifier;
+import org.apache.hadoop.yarn.server.resourcemanager.RMContext;
+import org.apache.hadoop.yarn.server.resourcemanager.recovery.records.ApplicationStateData;
+import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp;
+import org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager;
+import org.apache.hadoop.yarn.server.resourcemanager.security.ClientToAMTokenSecretManagerInRM;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.junit.Before;
+import org.junit.After;
+import org.junit.Test;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestZKRMStateStorePerf extends RMStateStoreTestBase
+ implements Tool {
+ public static final Log LOG = LogFactory.getLog(TestZKRMStateStore.class);
+
+ final String version = "0.1";
+
+ // Configurable variables for performance test
+ private int ZK_PERF_NUM_APP_DEFAULT = 1000;
+ private int ZK_PERF_NUM_APPATTEMPT_PER_APP = 10;
+
+ private final long clusterTimeStamp = 1352994193343L;
+
+ private static final String USAGE =
+ "Usage: " + TestZKRMStateStorePerf.class.getSimpleName() +
+ " -appSize numberOfApplications" +
+ " -appAttemptSize numberOfApplicationAttempts" +
+ " [-hostPort Host:Port]" +
+ " [-workingZnode rootZnodeForTesting]\n";
+
+ private YarnConfiguration conf = null;
+ private String workingZnode = "/Test";
+ private ZKRMStateStore store;
+ private AMRMTokenSecretManager appTokenMgr;
+ private ClientToAMTokenSecretManagerInRM clientToAMTokenMgr;
+
+ @Before
+ public void setUpZKServer() throws Exception {
+ super.setUp();
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ if (store != null) {
+ store.stop();
+ }
+ if (appTokenMgr != null) {
+ appTokenMgr.stop();
+ }
+ super.tearDown();
+ }
+
+ private void initStore(String hostPort) {
+ Optional<String> optHostPort = Optional.fromNullable(hostPort);
+ RMContext rmContext = mock(RMContext.class);
+
+ conf = new YarnConfiguration();
+ conf.set(YarnConfiguration.RM_ZK_ADDRESS, optHostPort.or(this.hostPort));
+ conf.set(YarnConfiguration.ZK_RM_STATE_STORE_PARENT_PATH, workingZnode);
+
+ store = new ZKRMStateStore();
+ store.init(conf);
+ store.start();
+ when(rmContext.getStateStore()).thenReturn(store);
+ appTokenMgr = new AMRMTokenSecretManager(conf, rmContext);
+ appTokenMgr.start();
+ clientToAMTokenMgr = new ClientToAMTokenSecretManagerInRM();
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public int run(String[] args) {
+ LOG.info("Starting ZKRMStateStorePerf ver." + version);
+
+ int numApp = ZK_PERF_NUM_APP_DEFAULT;
+ int numAppAttemptPerApp = ZK_PERF_NUM_APPATTEMPT_PER_APP;
+ String hostPort = null;
+ boolean launchLocalZK= true;
+
+ if (args.length == 0) {
+ System.err.println("Missing arguments.");
+ return -1;
+ }
+
+ for (int i = 0; i < args.length; i++) { // parse command line
+ if (args[i].equalsIgnoreCase("-appsize")) {
+ numApp = Integer.parseInt(args[++i]);
+ } else if (args[i].equalsIgnoreCase("-appattemptsize")) {
+ numAppAttemptPerApp = Integer.parseInt(args[++i]);
+ } else if (args[i].equalsIgnoreCase("-hostPort")) {
+ hostPort = args[++i];
+ launchLocalZK = false;
+ } else if (args[i].equalsIgnoreCase("-workingZnode")) {
+ workingZnode = args[++i];
+ } else {
+ System.err.println("Illegal argument: " + args[i]);
+ return -1;
+ }
+ }
+
+ if (launchLocalZK) {
+ try {
+ setUp();
+ } catch (Exception e) {
+ System.err.println("failed to setup. : " + e.getMessage());
+ return -1;
+ }
+ }
+
+ initStore(hostPort);
+
+ long submitTime = System.currentTimeMillis();
+ long startTime = System.currentTimeMillis() + 1234;
+
+ ArrayList<ApplicationId> applicationIds = new ArrayList<>();
+ ArrayList<RMApp> rmApps = new ArrayList<>();
+ ArrayList<ApplicationAttemptId> attemptIds = new ArrayList<>();
+ HashMap<ApplicationId, Set<ApplicationAttemptId>> appIdsToAttemptId =
+ new HashMap<>();
+ TestDispatcher dispatcher = new TestDispatcher();
+ store.setRMDispatcher(dispatcher);
+
+ for (int i = 0; i < numApp; i++) {
+ ApplicationId appId = ApplicationId.newInstance(clusterTimeStamp, i);
+ applicationIds.add(appId);
+ ArrayList<ApplicationAttemptId> attemptIdsForThisApp =
+ new ArrayList<>();
+ for (int j = 0; j < numAppAttemptPerApp; j++) {
+ ApplicationAttemptId attemptId =
+ ApplicationAttemptId.newInstance(appId, j);
+ attemptIdsForThisApp.add(attemptId);
+ }
+ appIdsToAttemptId.put(appId, new LinkedHashSet(attemptIdsForThisApp));
+ attemptIds.addAll(attemptIdsForThisApp);
+ }
+
+ for (ApplicationId appId : applicationIds) {
+ RMApp app = null;
+ try {
+ app = storeApp(store, appId, submitTime, startTime);
+ } catch (Exception e) {
+ System.err.println("failed to create Application Znode. : "
+ + e.getMessage());
+ return -1;
+ }
+ waitNotify(dispatcher);
+ rmApps.add(app);
+ }
+
+ for (ApplicationAttemptId attemptId : attemptIds) {
+ Token<AMRMTokenIdentifier> tokenId =
+ generateAMRMToken(attemptId, appTokenMgr);
+ SecretKey clientTokenKey =
+ clientToAMTokenMgr.createMasterKey(attemptId);
+ try {
+ storeAttempt(store, attemptId,
+ ContainerId.newContainerId(attemptId, 0L).toString(),
+ tokenId, clientTokenKey, dispatcher);
+ } catch (Exception e) {
+ System.err.println("failed to create AppAttempt Znode. : "
+ + e.getMessage());
+ return -1;
+ }
+ }
+
+ long storeStart = System.currentTimeMillis();
+ try {
+ store.loadState();
+ } catch (Exception e) {
+ System.err.println("failed to locaState from ZKRMStateStore. : "
+ + e.getMessage());
+ return -1;
+ }
+ long storeEnd = System.currentTimeMillis();
+
+ long loadTime = storeEnd - storeStart;
+
+ String resultMsg = "ZKRMStateStore takes " + loadTime + " msec to loadState.";
+ LOG.info(resultMsg);
+ System.out.println(resultMsg);
+
+ // cleanup
+ try {
+ for (RMApp app : rmApps) {
+ ApplicationStateData appState =
+ ApplicationStateData.newInstance(app.getSubmitTime(),
+ app.getStartTime(), app.getApplicationSubmissionContext(),
+ app.getUser());
+ ApplicationId appId = app.getApplicationId();
+ Map m = mock(Map.class);
+ when(m.keySet()).thenReturn(appIdsToAttemptId.get(appId));
+ appState.attempts = m;
+ store.removeApplicationStateInternal(appState);
+ }
+ } catch (Exception e) {
+ System.err.println("failed to cleanup. : " + e.getMessage());
+ return -1;
+ }
+
+ return 0;
+ }
+
+ @Override
+ public void setConf(Configuration conf) {
+ // currently this function is just ignored
+ }
+
+ @Override
+ public Configuration getConf() {
+ return conf;
+ }
+
+ @Test
+ public void perfZKRMStateStore() throws Exception {
+ String[] args = {
+ "-appSize", String.valueOf(ZK_PERF_NUM_APP_DEFAULT),
+ "-appAttemptSize", String.valueOf(ZK_PERF_NUM_APPATTEMPT_PER_APP)
+ };
+ run(args);
+ }
+
+ static public void main(String[] args) throws Exception {
+ TestZKRMStateStorePerf perf = new TestZKRMStateStorePerf();
+
+ int res = -1;
+ try {
+ res = ToolRunner.run(perf, args);
+ } catch(Exception e) {
+ System.err.print(StringUtils.stringifyException(e));
+ res = -2;
+ }
+ if(res == -1) {
+ System.err.print(USAGE);
+ }
+ System.exit(res);
+ }
+}
[30/52] [abbrv] hadoop git commit: HADOOP-11607. Reduce log spew in
S3AFileSystem. (Lei (Eddy) Xu via stevel)
Posted by zh...@apache.org.
HADOOP-11607. Reduce log spew in S3AFileSystem. (Lei (Eddy) Xu via stevel)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/aa1c437b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/aa1c437b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/aa1c437b
Branch: refs/heads/HDFS-7285
Commit: aa1c437b6a806de612f030a68984c606c623f1d9
Parents: 02e7dec
Author: Steve Loughran <st...@apache.org>
Authored: Fri Feb 20 20:45:05 2015 +0000
Committer: Steve Loughran <st...@apache.org>
Committed: Fri Feb 20 20:51:24 2015 +0000
----------------------------------------------------------------------
hadoop-common-project/hadoop-common/CHANGES.txt | 2 ++
.../main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java | 10 +++++++---
2 files changed, 9 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/aa1c437b/hadoop-common-project/hadoop-common/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt b/hadoop-common-project/hadoop-common/CHANGES.txt
index 8d3f9f5..763377c 100644
--- a/hadoop-common-project/hadoop-common/CHANGES.txt
+++ b/hadoop-common-project/hadoop-common/CHANGES.txt
@@ -626,6 +626,8 @@ Release 2.7.0 - UNRELEASED
HADOOP-11440. Use "test.build.data" instead of "build.test.dir" for testing
in ClientBaseWithFixes. (Kengo Seki via aajisaka)
+ HADOOP-11607. Reduce log spew in S3AFileSystem. (Lei (Eddy) Xu via stevel)
+
OPTIMIZATIONS
HADOOP-11323. WritableComparator#compare keeps reference to byte array.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/aa1c437b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
index 4de5c13..eaa5f2d 100644
--- a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
+++ b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
@@ -354,7 +354,9 @@ public class S3AFileSystem extends FileSystem {
public FSDataInputStream open(Path f, int bufferSize)
throws IOException {
- LOG.info("Opening '" + f + "' for reading");
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Opening '{}' for reading.", f);
+ }
final FileStatus fileStatus = getFileStatus(f);
if (fileStatus.isDirectory()) {
throw new FileNotFoundException("Can't open " + f + " because it is a directory");
@@ -425,7 +427,9 @@ public class S3AFileSystem extends FileSystem {
* @return true if rename is successful
*/
public boolean rename(Path src, Path dst) throws IOException {
- LOG.info("Rename path " + src + " to " + dst);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Rename path {} to {}", src, dst);
+ }
String srcKey = pathToKey(src);
String dstKey = pathToKey(dst);
@@ -441,7 +445,7 @@ public class S3AFileSystem extends FileSystem {
try {
srcStatus = getFileStatus(src);
} catch (FileNotFoundException e) {
- LOG.info("rename: src not found " + src);
+ LOG.error("rename: src not found {}", src);
return false;
}
[45/52] [abbrv] hadoop git commit: Fix Compilation Error in
TestAddBlockgroup.java after the merge
Posted by zh...@apache.org.
Fix Compilation Error in TestAddBlockgroup.java after the merge
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/08a9ac32
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/08a9ac32
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/08a9ac32
Branch: refs/heads/HDFS-7285
Commit: 08a9ac3227286e953afdb01a93a42adce8fdd6fe
Parents: 6caf9c7
Author: Jing Zhao <ji...@apache.org>
Authored: Sun Feb 8 16:01:03 2015 -0800
Committer: Zhe Zhang <zh...@apache.org>
Committed: Mon Feb 23 11:19:59 2015 -0800
----------------------------------------------------------------------
.../apache/hadoop/hdfs/server/namenode/TestAddBlockgroup.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/08a9ac32/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockgroup.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockgroup.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockgroup.java
index 95133ce..06dfade 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockgroup.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockgroup.java
@@ -26,7 +26,7 @@ import org.apache.hadoop.hdfs.DFSTestUtil;
import org.apache.hadoop.hdfs.DistributedFileSystem;
import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.apache.hadoop.hdfs.protocol.HdfsConstants;
-import org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo;
+import org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguous;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
@@ -75,7 +75,7 @@ public class TestAddBlockgroup {
final Path file1 = new Path("/file1");
DFSTestUtil.createFile(fs, file1, BLOCKSIZE * 2, REPLICATION, 0L);
INodeFile file1Node = fsdir.getINode4Write(file1.toString()).asFile();
- BlockInfo[] file1Blocks = file1Node.getBlocks();
+ BlockInfoContiguous[] file1Blocks = file1Node.getBlocks();
assertEquals(2, file1Blocks.length);
assertEquals(GROUP_SIZE, file1Blocks[0].numNodes());
assertEquals(HdfsConstants.MAX_BLOCKS_IN_GROUP,
[24/52] [abbrv] hadoop git commit: HDFS-7788. Post-2.6 namenode may
not start up with an image containing inodes created with an old release.
Contributed by Rushabh Shah.
Posted by zh...@apache.org.
HDFS-7788. Post-2.6 namenode may not start up with an image containing inodes created with an old release. Contributed by Rushabh Shah.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/7ae5255a
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/7ae5255a
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/7ae5255a
Branch: refs/heads/HDFS-7285
Commit: 7ae5255a1613ccfb43646f33eabacf1062c86e93
Parents: b9a1790
Author: Kihwal Lee <ki...@apache.org>
Authored: Fri Feb 20 09:06:07 2015 -0600
Committer: Kihwal Lee <ki...@apache.org>
Committed: Fri Feb 20 09:09:02 2015 -0600
----------------------------------------------------------------------
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 ++
.../hadoop/hdfs/server/namenode/INodeFile.java | 3 ++
.../apache/hadoop/hdfs/util/LongBitFormat.java | 4 ++
.../hdfs/server/namenode/TestFSImage.java | 48 +++++++++++++++++++
.../resources/image-with-zero-block-size.tar.gz | Bin 0 -> 1452 bytes
5 files changed, 58 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7ae5255a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index 5f3cc02..71ce48f 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -992,6 +992,9 @@ Release 2.7.0 - UNRELEASED
HDFS-7808. Remove obsolete -ns options in in DFSHAAdmin.java.
(Arshad Mohammad via wheat9)
+ HDFS-7788. Post-2.6 namenode may not start up with an image containing
+ inodes created with an old release. (Rushabh Shah via kihwal)
+
BREAKDOWN OF HDFS-7584 SUBTASKS AND RELATED JIRAS
HDFS-7720. Quota by Storage Type API, tools and ClientNameNode
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7ae5255a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
index 24e25ec..3743bf0 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
@@ -107,6 +107,9 @@ public class INodeFile extends INodeWithAdditionalFields
static long toLong(long preferredBlockSize, short replication,
byte storagePolicyID) {
long h = 0;
+ if (preferredBlockSize == 0) {
+ preferredBlockSize = PREFERRED_BLOCK_SIZE.BITS.getMin();
+ }
h = PREFERRED_BLOCK_SIZE.BITS.combine(preferredBlockSize, h);
h = REPLICATION.BITS.combine(replication, h);
h = STORAGE_POLICY_ID.BITS.combine(storagePolicyID, h);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7ae5255a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LongBitFormat.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LongBitFormat.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LongBitFormat.java
index 863d9f7..9399d84 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LongBitFormat.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LongBitFormat.java
@@ -64,4 +64,8 @@ public class LongBitFormat implements Serializable {
}
return (record & ~MASK) | (value << OFFSET);
}
+
+ public long getMin() {
+ return MIN;
+ }
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7ae5255a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
index c68ae04..f7dad18 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
@@ -28,10 +28,13 @@ import org.junit.Assert;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSOutputStream;
import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.apache.hadoop.hdfs.client.HdfsDataOutputStream.SyncFlag;
import org.apache.hadoop.hdfs.protocol.HdfsConstants;
@@ -40,10 +43,14 @@ import org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguous;
import org.apache.hadoop.hdfs.server.common.HdfsServerConstants.BlockUCState;
import org.apache.hadoop.hdfs.server.namenode.LeaseManager.Lease;
import org.apache.hadoop.hdfs.util.MD5FileUtils;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.test.PathUtils;
import org.junit.Test;
public class TestFSImage {
+ private static final String HADOOP_2_7_ZER0_BLOCK_SIZE_TGZ =
+ "image-with-zero-block-size.tar.gz";
@Test
public void testPersist() throws IOException {
Configuration conf = new Configuration();
@@ -183,4 +190,45 @@ public class TestFSImage {
}
}
}
+
+ /**
+ * In this test case, I have created an image with a file having
+ * preferredblockSize = 0. We are trying to read this image (since file with
+ * preferredblockSize = 0 was allowed pre 2.1.0-beta version. The namenode
+ * after 2.6 version will not be able to read this particular file.
+ * See HDFS-7788 for more information.
+ * @throws Exception
+ */
+ @Test
+ public void testZeroBlockSize() throws Exception {
+ final Configuration conf = new HdfsConfiguration();
+ String tarFile = System.getProperty("test.cache.data", "build/test/cache")
+ + "/" + HADOOP_2_7_ZER0_BLOCK_SIZE_TGZ;
+ String testDir = PathUtils.getTestDirName(getClass());
+ File dfsDir = new File(testDir, "image-with-zero-block-size");
+ if (dfsDir.exists() && !FileUtil.fullyDelete(dfsDir)) {
+ throw new IOException("Could not delete dfs directory '" + dfsDir + "'");
+ }
+ FileUtil.unTar(new File(tarFile), new File(testDir));
+ File nameDir = new File(dfsDir, "name");
+ GenericTestUtils.assertExists(nameDir);
+ conf.set(DFSConfigKeys.DFS_NAMENODE_NAME_DIR_KEY,
+ nameDir.getAbsolutePath());
+ MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).numDataNodes(1)
+ .format(false)
+ .manageDataDfsDirs(false)
+ .manageNameDfsDirs(false)
+ .waitSafeMode(false)
+ .build();
+ try {
+ FileSystem fs = cluster.getFileSystem();
+ Path testPath = new Path("/tmp/zeroBlockFile");
+ assertTrue("File /tmp/zeroBlockFile doesn't exist ", fs.exists(testPath));
+ assertTrue("Name node didn't come up", cluster.isNameNodeUp(0));
+ } finally {
+ cluster.shutdown();
+ //Clean up
+ FileUtil.fullyDelete(dfsDir);
+ }
+ }
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7ae5255a/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/image-with-zero-block-size.tar.gz
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/image-with-zero-block-size.tar.gz b/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/image-with-zero-block-size.tar.gz
new file mode 100644
index 0000000..8bfc454
Binary files /dev/null and b/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/image-with-zero-block-size.tar.gz differ
[34/52] [abbrv] hadoop git commit: HDFS-7813.
TestDFSHAAdminMiniCluster#testFencer testcase is failing frequently.
Contributed by Rakesh R.
Posted by zh...@apache.org.
HDFS-7813. TestDFSHAAdminMiniCluster#testFencer testcase is failing frequently. Contributed by Rakesh R.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/0d6af574
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/0d6af574
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/0d6af574
Branch: refs/heads/HDFS-7285
Commit: 0d6af574e0056fc627461eb91ed0c365b026b470
Parents: f56c65b
Author: cnauroth <cn...@apache.org>
Authored: Fri Feb 20 17:01:08 2015 -0800
Committer: cnauroth <cn...@apache.org>
Committed: Fri Feb 20 17:01:08 2015 -0800
----------------------------------------------------------------------
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 +++
.../org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java | 1 +
2 files changed, 4 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/0d6af574/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index 5c472a8..c47b89d 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -1001,6 +1001,9 @@ Release 2.7.0 - UNRELEASED
HDFS-7814. Fix usage string of storageType parameter for
"dfsadmin -setSpaceQuota/clrSpaceQuota". (Xiaoyu Yao via cnauroth)
+ HDFS-7813. TestDFSHAAdminMiniCluster#testFencer testcase is failing
+ frequently. (Rakesh R via cnauroth)
+
BREAKDOWN OF HDFS-7584 SUBTASKS AND RELATED JIRAS
HDFS-7720. Quota by Storage Type API, tools and ClientNameNode
http://git-wip-us.apache.org/repos/asf/hadoop/blob/0d6af574/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java
index b4d45c3..9047279 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java
@@ -155,6 +155,7 @@ public class TestDFSHAAdminMiniCluster {
tool.setConf(conf);
assertEquals(0, runTool("-transitionToActive", "nn1"));
assertEquals(0, runTool("-failover", "nn1", "nn2"));
+ assertEquals(0, runTool("-failover", "nn2", "nn1"));
// Fencer has not run yet, since none of the above required fencing
assertEquals("", Files.toString(tmpFile, Charsets.UTF_8));
[37/52] [abbrv] hadoop git commit: HDFS-7806. Refactor: move
StorageType from hadoop-hdfs to hadoop-common. (Contributed by Xiaoyu Yao)
Posted by zh...@apache.org.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
index 6ba778f..7d0edb2 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
@@ -51,6 +51,7 @@ import org.apache.hadoop.fs.FsStatus;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.shell.Command;
import org.apache.hadoop.fs.shell.CommandFormat;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.client.BlockReportOptions;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSUtil;
@@ -71,7 +72,6 @@ import org.apache.hadoop.hdfs.protocol.RollingUpgradeInfo;
import org.apache.hadoop.hdfs.protocol.SnapshotException;
import org.apache.hadoop.hdfs.server.namenode.NameNode;
import org.apache.hadoop.hdfs.server.namenode.TransferFsImage;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.ipc.GenericRefreshProtocol;
import org.apache.hadoop.ipc.ProtobufRpcEngine;
import org.apache.hadoop.ipc.RPC;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
index c7fd2f8..5b391c5 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
@@ -30,15 +30,24 @@ import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.crypto.key.KeyProvider;
-import org.apache.hadoop.fs.*;
+import org.apache.hadoop.fs.BlockLocation;
+import org.apache.hadoop.fs.CacheFlag;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.hadoop.fs.CreateFlag;
+import org.apache.hadoop.fs.FileContext;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FileSystem.Statistics;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FsShell;
import org.apache.hadoop.fs.Options.Rename;
+import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.permission.AclEntry;
import org.apache.hadoop.fs.permission.AclEntryScope;
import org.apache.hadoop.fs.permission.AclEntryType;
import org.apache.hadoop.fs.permission.FsAction;
import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.MiniDFSCluster.NameNodeInfo;
import org.apache.hadoop.hdfs.client.HdfsDataInputStream;
import org.apache.hadoop.hdfs.protocol.*;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
index 33bd4e5..5297ba2 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
@@ -69,6 +69,7 @@ import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FileUtil;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.ha.HAServiceProtocol.RequestSource;
import org.apache.hadoop.ha.HAServiceProtocol.StateChangeRequestInfo;
import org.apache.hadoop.ha.ServiceFailedException;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSClusterWithNodeGroup.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSClusterWithNodeGroup.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSClusterWithNodeGroup.java
index d5225a4..ca2d8d6 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSClusterWithNodeGroup.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSClusterWithNodeGroup.java
@@ -18,15 +18,14 @@
package org.apache.hadoop.hdfs;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_HOST_NAME_KEY;
-import static org.apache.hadoop.hdfs.server.common.Util.fileAsURI;
-import java.io.File;
import java.io.IOException;
import java.util.List;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.server.common.HdfsServerConstants.StartupOption;
import org.apache.hadoop.hdfs.server.datanode.DataNode;
import org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter;
@@ -37,7 +36,6 @@ import org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl;
import org.apache.hadoop.net.NetUtils;
import org.apache.hadoop.net.StaticMapping;
import org.apache.hadoop.security.UserGroupInformation;
-import org.apache.hadoop.security.ssl.SSLFactory;
public class MiniDFSClusterWithNodeGroup extends MiniDFSCluster {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
index 3d417e6..8f99a85 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
@@ -28,6 +28,7 @@ import com.google.common.collect.Lists;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.*;
import org.apache.hadoop.hdfs.protocol.HdfsConstants.SafeModeAction;
import org.apache.hadoop.hdfs.server.blockmanagement.*;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferProtocol.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferProtocol.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferProtocol.java
index 519c511..040a91c 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferProtocol.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferProtocol.java
@@ -39,6 +39,7 @@ import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.DatanodeID;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteBlockGetsBlockLengthHint.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteBlockGetsBlockLengthHint.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteBlockGetsBlockLengthHint.java
index 7f318df..5c1b38f 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteBlockGetsBlockLengthHint.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteBlockGetsBlockLengthHint.java
@@ -21,6 +21,7 @@ import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
import org.apache.hadoop.hdfs.server.datanode.*;
import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsDatasetSpi;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
index 98fd59a..0236288 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java
@@ -31,8 +31,8 @@ import org.apache.hadoop.fs.permission.AclEntryScope;
import org.apache.hadoop.fs.permission.AclEntryType;
import org.apache.hadoop.fs.permission.AclStatus;
import org.apache.hadoop.fs.permission.FsAction;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSTestUtil;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.DatanodeID;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
index 03d2812..9b62467 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
@@ -17,9 +17,9 @@
*/
package org.apache.hadoop.hdfs.server.balancer;
+import static org.apache.hadoop.fs.StorageType.DEFAULT;
+import static org.apache.hadoop.fs.StorageType.RAM_DISK;
import static org.apache.hadoop.hdfs.DFSConfigKeys.*;
-import static org.apache.hadoop.hdfs.StorageType.DEFAULT;
-import static org.apache.hadoop.hdfs.StorageType.RAM_DISK;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
@@ -46,6 +46,7 @@ import org.apache.commons.logging.impl.Log4JLogger;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSClient;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSTestUtil;
@@ -54,7 +55,6 @@ import org.apache.hadoop.hdfs.DistributedFileSystem;
import org.apache.hadoop.hdfs.HdfsConfiguration;
import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.apache.hadoop.hdfs.NameNodeProxies;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.*;
import org.apache.hadoop.hdfs.protocol.HdfsConstants.DatanodeReportType;
import org.apache.hadoop.hdfs.server.balancer.Balancer.Cli;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
index 698f9f4..f9c2bdc 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
@@ -37,10 +37,10 @@ import java.util.Map.Entry;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSTestUtil;
import org.apache.hadoop.hdfs.HdfsConfiguration;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.BlockListAsLongs;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java
index 059eac0..a4a6263 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java
@@ -31,8 +31,8 @@ import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSConfigKeys;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
import org.apache.hadoop.hdfs.protocol.LocatedBlock;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
index 3e6b85f..24fd81d 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
@@ -41,13 +41,13 @@ import org.apache.commons.logging.impl.Log4JLogger;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.ContentSummary;
import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSTestUtil;
import org.apache.hadoop.hdfs.DFSUtil;
import org.apache.hadoop.hdfs.HdfsConfiguration;
import org.apache.hadoop.hdfs.LogVerificationAppender;
import org.apache.hadoop.hdfs.MiniDFSCluster;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.TestBlockStoragePolicy;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.HdfsConstants;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
index a1ea73c..7708ddc 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
@@ -33,10 +33,10 @@ import java.util.Set;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSTestUtil;
import org.apache.hadoop.hdfs.HdfsConfiguration;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.TestBlockStoragePolicy;
import org.apache.hadoop.hdfs.protocol.HdfsConstants;
import org.apache.hadoop.hdfs.server.namenode.NameNode;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/BlockReportTestBase.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/BlockReportTestBase.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/BlockReportTestBase.java
index a22a71e..8d9de7b 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/BlockReportTestBase.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/BlockReportTestBase.java
@@ -35,16 +35,15 @@ import java.util.concurrent.TimeoutException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
-import org.apache.commons.logging.impl.Log4JLogger;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.AppendTestUtil;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSTestUtil;
import org.apache.hadoop.hdfs.DistributedFileSystem;
import org.apache.hadoop.hdfs.MiniDFSCluster;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.BlockListAsLongs;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
index 991aa9e..8ae5415 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
@@ -37,8 +37,8 @@ import javax.management.StandardMBean;
import org.apache.commons.lang.ArrayUtils;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSConfigKeys;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.BlockListAsLongs;
import org.apache.hadoop.hdfs.protocol.BlockLocalPathInfo;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
index 7490723..e566f2a 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
@@ -50,13 +50,13 @@ import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FileUtil;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.ha.HAServiceProtocol.HAServiceState;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSTestUtil;
import org.apache.hadoop.hdfs.DistributedFileSystem;
import org.apache.hadoop.hdfs.HdfsConfiguration;
import org.apache.hadoop.hdfs.MiniDFSCluster;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.DatanodeID;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java
index fbdfebf..1f2c1b7 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java
@@ -36,13 +36,13 @@ import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSClient;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSTestUtil;
import org.apache.hadoop.hdfs.DistributedFileSystem;
import org.apache.hadoop.hdfs.HdfsConfiguration;
import org.apache.hadoop.hdfs.MiniDFSCluster;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
import org.apache.hadoop.hdfs.protocol.HdfsConstants.DatanodeReportType;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java
index 94af015..396945e 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java
@@ -22,7 +22,7 @@ import java.io.*;
import java.util.*;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.junit.Test;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_DATA_DIR_KEY;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
index 82a1684..681768e 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
@@ -40,7 +40,12 @@ import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hdfs.*;
+import org.apache.hadoop.fs.StorageType;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.DFSClient;
+import org.apache.hadoop.hdfs.DFSTestUtil;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.LocatedBlock;
import org.apache.hadoop.hdfs.server.common.GenerationStamp;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java
index fb219d7..3ec76db 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java
@@ -31,11 +31,11 @@ import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FileUtil;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSTestUtil;
import org.apache.hadoop.hdfs.HdfsConfiguration;
import org.apache.hadoop.hdfs.MiniDFSCluster;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.hdfs.protocol.LocatedBlock;
import org.apache.hadoop.hdfs.protocol.LocatedBlocks;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java
index b9adce4..dd24685 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java
@@ -28,8 +28,8 @@ import java.io.InputStream;
import java.io.OutputStream;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.HdfsConfiguration;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.BlockListAsLongs;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestStorageReport.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestStorageReport.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestStorageReport.java
index ec39892..ecb28dc 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestStorageReport.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestStorageReport.java
@@ -23,7 +23,10 @@ import java.io.IOException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hdfs.*;
+import org.apache.hadoop.fs.StorageType;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB;
import org.apache.hadoop.hdfs.server.namenode.NameNode;
import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java
index fc11886..a3c9935 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java
@@ -22,7 +22,7 @@ import java.io.*;
import java.util.*;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.BlockListAsLongs;
import org.apache.hadoop.hdfs.protocol.BlockLocalPathInfo;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java
index 0ea33bb..ea9e4c1 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java
@@ -22,7 +22,7 @@ import java.io.File;
import java.io.IOException;
import java.nio.channels.ClosedChannelException;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsDatasetSpi;
import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeReference;
import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeSpi;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java
index 6272991..89a70c9 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java
@@ -25,11 +25,11 @@ import org.apache.hadoop.fs.CreateFlag;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSClient;
import org.apache.hadoop.hdfs.DFSTestUtil;
import org.apache.hadoop.hdfs.DistributedFileSystem;
import org.apache.hadoop.hdfs.MiniDFSCluster;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.LocatedBlock;
import org.apache.hadoop.hdfs.protocol.LocatedBlocks;
import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils;
@@ -55,9 +55,9 @@ import java.util.UUID;
import static org.apache.hadoop.fs.CreateFlag.CREATE;
import static org.apache.hadoop.fs.CreateFlag.LAZY_PERSIST;
+import static org.apache.hadoop.fs.StorageType.DEFAULT;
+import static org.apache.hadoop.fs.StorageType.RAM_DISK;
import static org.apache.hadoop.hdfs.DFSConfigKeys.*;
-import static org.apache.hadoop.hdfs.StorageType.DEFAULT;
-import static org.apache.hadoop.hdfs.StorageType.RAM_DISK;
import static org.hamcrest.core.Is.is;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertThat;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
index c3b871c..8c28033 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
@@ -20,8 +20,8 @@ package org.apache.hadoop.hdfs.server.datanode.fsdataset.impl;
import com.google.common.collect.Lists;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystemTestHelper;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSConfigKeys;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
import org.apache.hadoop.hdfs.server.common.HdfsServerConstants;
import org.apache.hadoop.hdfs.server.common.Storage;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
index f87c404..46189ba 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
@@ -19,8 +19,8 @@ package org.apache.hadoop.hdfs.server.datanode.fsdataset.impl;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystemTestHelper;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSConfigKeys;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.server.datanode.BlockScanner;
import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeReference;
import org.apache.hadoop.hdfs.server.datanode.fsdataset.RoundRobinVolumeChoosingPolicy;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
index a4df4ab..bd64cbe 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
@@ -37,9 +37,10 @@ import java.util.concurrent.atomic.AtomicBoolean;
import java.util.ArrayList;
import java.util.Collections;
+import static org.apache.hadoop.fs.StorageType.DEFAULT;
+import static org.apache.hadoop.fs.StorageType.RAM_DISK;
import static org.apache.hadoop.hdfs.DFSConfigKeys.*;
-import static org.apache.hadoop.hdfs.StorageType.DEFAULT;
-import static org.apache.hadoop.hdfs.StorageType.RAM_DISK;
+
import static org.hamcrest.core.Is.is;
import static org.hamcrest.core.IsNot.not;
import static org.junit.Assert.assertThat;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestScrLazyPersistFiles.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestScrLazyPersistFiles.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestScrLazyPersistFiles.java
index be6ca2c..d54e5a4 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestScrLazyPersistFiles.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestScrLazyPersistFiles.java
@@ -20,10 +20,10 @@ import org.apache.commons.io.IOUtils;
import org.apache.hadoop.fs.ChecksumException;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.ClientContext;
import org.apache.hadoop.hdfs.DFSTestUtil;
import org.apache.hadoop.hdfs.MiniDFSCluster;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.client.HdfsDataInputStream;
import org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader;
import org.apache.hadoop.net.unix.DomainSocket;
@@ -40,8 +40,8 @@ import org.junit.rules.ExpectedException;
import java.io.File;
import java.io.IOException;
-import static org.apache.hadoop.hdfs.StorageType.DEFAULT;
-import static org.apache.hadoop.hdfs.StorageType.RAM_DISK;
+import static org.apache.hadoop.fs.StorageType.DEFAULT;
+import static org.apache.hadoop.fs.StorageType.RAM_DISK;
import static org.hamcrest.CoreMatchers.equalTo;
import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java
index 5aafc9c..9325cdc 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java
@@ -19,9 +19,9 @@ package org.apache.hadoop.hdfs.server.datanode.fsdataset.impl;
import java.io.IOException;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.HdfsConfiguration;
import org.apache.hadoop.hdfs.MiniDFSCluster;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
import org.apache.hadoop.hdfs.server.datanode.DataNode;
import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
index f35e1c8..1de236e 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java
@@ -25,7 +25,13 @@ import com.google.common.collect.Maps;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hdfs.*;
+import org.apache.hadoop.fs.StorageType;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.hadoop.hdfs.DFSTestUtil;
+import org.apache.hadoop.hdfs.DFSUtil;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.MiniDFSNNTopology;
import org.apache.hadoop.hdfs.protocol.LocatedBlock;
import org.apache.hadoop.hdfs.server.balancer.Dispatcher.DBlock;
import org.apache.hadoop.hdfs.server.balancer.NameNodeConnector;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java
index 43e96e4..7eaf5c7 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java
@@ -33,6 +33,7 @@ import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSOutputStream;
@@ -41,7 +42,6 @@ import org.apache.hadoop.hdfs.DFSUtil;
import org.apache.hadoop.hdfs.DistributedFileSystem;
import org.apache.hadoop.hdfs.HdfsConfiguration;
import org.apache.hadoop.hdfs.MiniDFSCluster;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.DirectoryListing;
import org.apache.hadoop.hdfs.protocol.HdfsConstants;
import org.apache.hadoop.hdfs.protocol.HdfsFileStatus;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockRetry.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockRetry.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockRetry.java
index 6098ebf..cf37a54 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockRetry.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockRetry.java
@@ -35,7 +35,6 @@ import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.CreateFlag;
import org.apache.hadoop.fs.permission.FsPermission;
import org.apache.hadoop.hdfs.MiniDFSCluster;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.LocatedBlock;
import org.apache.hadoop.hdfs.protocol.LocatedBlocks;
import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaByStorageType.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaByStorageType.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaByStorageType.java
index c69b40c..aee756f 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaByStorageType.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaByStorageType.java
@@ -17,7 +17,6 @@
*/
package org.apache.hadoop.hdfs.server.namenode;
- import static org.apache.hadoop.hdfs.StorageType.DEFAULT;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
@@ -26,7 +25,11 @@ package org.apache.hadoop.hdfs.server.namenode;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
- import org.apache.hadoop.hdfs.*;
+ import org.apache.hadoop.fs.StorageType;
+ import org.apache.hadoop.hdfs.DFSConfigKeys;
+ import org.apache.hadoop.hdfs.DFSTestUtil;
+ import org.apache.hadoop.hdfs.DistributedFileSystem;
+ import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.apache.hadoop.hdfs.protocol.HdfsConstants;
import org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotTestHelper;
import org.apache.hadoop.test.GenericTestUtils;
@@ -61,7 +64,7 @@ public class TestQuotaByStorageType {
cluster = new MiniDFSCluster
.Builder(conf)
.numDataNodes(REPLICATION)
- .storageTypes(new StorageType[]{StorageType.SSD, DEFAULT})
+ .storageTypes(new StorageType[]{StorageType.SSD, StorageType.DEFAULT})
.build();
cluster.waitActive();
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
index f625bb4..fa7a307 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
@@ -33,13 +33,13 @@ import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.AppendTestUtil;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSTestUtil;
import org.apache.hadoop.hdfs.DFSUtil;
import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.apache.hadoop.hdfs.MiniDFSNNTopology;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
import org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB;
[03/52] [abbrv] hadoop git commit: HADOOP-11593. Convert site
documentation from apt to markdown (stragglers) (Masatake Iwasaki via aw)
Posted by zh...@apache.org.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm b/hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm
new file mode 100644
index 0000000..44b5bfb
--- /dev/null
+++ b/hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm
@@ -0,0 +1,864 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+#set ( $H3 = '###' )
+#set ( $H4 = '####' )
+#set ( $H5 = '#####' )
+
+Hadoop Key Management Server (KMS) - Documentation Sets
+=======================================================
+
+Hadoop KMS is a cryptographic key management server based on Hadoop's **KeyProvider** API.
+
+It provides a client and a server components which communicate over HTTP using a REST API.
+
+The client is a KeyProvider implementation interacts with the KMS using the KMS HTTP REST API.
+
+KMS and its client have built-in security and they support HTTP SPNEGO Kerberos authentication and HTTPS secure transport.
+
+KMS is a Java web-application and it runs using a pre-configured Tomcat bundled with the Hadoop distribution.
+
+KMS Client Configuration
+------------------------
+
+The KMS client `KeyProvider` uses the **kms** scheme, and the embedded URL must be the URL of the KMS. For example, for a KMS running on `http://localhost:16000/kms`, the KeyProvider URI is `kms://http@localhost:16000/kms`. And, for a KMS running on `https://localhost:16000/kms`, the KeyProvider URI is `kms://https@localhost:16000/kms`
+
+KMS
+---
+
+$H3 KMS Configuration
+
+Configure the KMS backing KeyProvider properties in the `etc/hadoop/kms-site.xml` configuration file:
+
+```xml
+ <property>
+ <name>hadoop.kms.key.provider.uri</name>
+ <value>jceks://file@/${user.home}/kms.keystore</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.keystore.java-keystore-provider.password-file</name>
+ <value>kms.keystore.password</value>
+ </property>
+```
+
+The password file is looked up in the Hadoop's configuration directory via the classpath.
+
+NOTE: You need to restart the KMS for the configuration changes to take effect.
+
+$H3 KMS Cache
+
+KMS caches keys for short period of time to avoid excessive hits to the underlying key provider.
+
+The Cache is enabled by default (can be dissabled by setting the `hadoop.kms.cache.enable` boolean property to false)
+
+The cache is used with the following 3 methods only, `getCurrentKey()` and `getKeyVersion()` and `getMetadata()`.
+
+For the `getCurrentKey()` method, cached entries are kept for a maximum of 30000 millisecond regardless the number of times the key is being access (to avoid stale keys to be considered current).
+
+For the `getKeyVersion()` method, cached entries are kept with a default inactivity timeout of 600000 milliseconds (10 mins). This time out is configurable via the following property in the `etc/hadoop/kms-site.xml` configuration file:
+
+```xml
+ <property>
+ <name>hadoop.kms.cache.enable</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hadoop.kms.cache.timeout.ms</name>
+ <value>600000</value>
+ </property>
+
+ <property>
+ <name>hadoop.kms.current.key.cache.timeout.ms</name>
+ <value>30000</value>
+ </property>
+```
+
+$H3 KMS Aggregated Audit logs
+
+Audit logs are aggregated for API accesses to the GET\_KEY\_VERSION, GET\_CURRENT\_KEY, DECRYPT\_EEK, GENERATE\_EEK operations.
+
+Entries are grouped by the (user,key,operation) combined key for a configurable aggregation interval after which the number of accesses to the specified end-point by the user for a given key is flushed to the audit log.
+
+The Aggregation interval is configured via the property :
+
+ <property>
+ <name>hadoop.kms.aggregation.delay.ms</name>
+ <value>10000</value>
+ </property>
+
+$H3 Start/Stop the KMS
+
+To start/stop KMS use KMS's bin/kms.sh script. For example:
+
+ hadoop-${project.version} $ sbin/kms.sh start
+
+NOTE: Invoking the script without any parameters list all possible parameters (start, stop, run, etc.). The `kms.sh` script is a wrapper for Tomcat's `catalina.sh` script that sets the environment variables and Java System properties required to run KMS.
+
+$H3 Embedded Tomcat Configuration
+
+To configure the embedded Tomcat go to the `share/hadoop/kms/tomcat/conf`.
+
+KMS pre-configures the HTTP and Admin ports in Tomcat's `server.xml` to 16000 and 16001.
+
+Tomcat logs are also preconfigured to go to Hadoop's `logs/` directory.
+
+The following environment variables (which can be set in KMS's `etc/hadoop/kms-env.sh` script) can be used to alter those values:
+
+* KMS_HTTP_PORT
+* KMS_ADMIN_PORT
+* KMS_MAX_THREADS
+* KMS_LOGNOTE: You need to restart the KMS for the configuration changes to take effect.
+
+$H3 Loading native libraries
+
+The following environment variable (which can be set in KMS's `etc/hadoop/kms-env.sh` script) can be used to specify the location of any required native libraries. For eg. Tomact native Apache Portable Runtime (APR) libraries:
+
+* JAVA_LIBRARY_PATH
+
+$H3 KMS Security Configuration
+
+$H4 Enabling Kerberos HTTP SPNEGO Authentication
+
+Configure the Kerberos `etc/krb5.conf` file with the information of your KDC server.
+
+Create a service principal and its keytab for the KMS, it must be an `HTTP` service principal.
+
+Configure KMS `etc/hadoop/kms-site.xml` with the correct security values, for example:
+
+```xml
+ <property>
+ <name>hadoop.kms.authentication.type</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.kerberos.keytab</name>
+ <value>${user.home}/kms.keytab</value>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.kerberos.principal</name>
+ <value>HTTP/localhost</value>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.kerberos.name.rules</name>
+ <value>DEFAULT</value>
+ </property>
+```
+
+NOTE: You need to restart the KMS for the configuration changes to take effect.
+
+$H4 KMS Proxyuser Configuration
+
+Each proxyuser must be configured in `etc/hadoop/kms-site.xml` using the following properties:
+
+```xml
+ <property>
+ <name>hadoop.kms.proxyuser.#USER#.users</name>
+ <value>*</value>
+ </property>
+
+ <property>
+ <name>hadoop.kms.proxyuser.#USER#.groups</name>
+ <value>*</value>
+ </property>
+
+ <property>
+ <name>hadoop.kms.proxyuser.#USER#.hosts</name>
+ <value>*</value>
+ </property>
+```
+
+`#USER#` is the username of the proxyuser to configure.
+
+The `users` property indicates the users that can be impersonated.
+
+The `groups` property indicates the groups users being impersonated must belong to.
+
+At least one of the `users` or `groups` properties must be defined. If both are specified, then the configured proxyuser will be able to impersonate and user in the `users` list and any user belonging to one of the groups in the `groups` list.
+
+The `hosts` property indicates from which host the proxyuser can make impersonation requests.
+
+If `users`, `groups` or `hosts` has a `*`, it means there are no restrictions for the proxyuser regarding users, groups or hosts.
+
+$H4 KMS over HTTPS (SSL)
+
+To configure KMS to work over HTTPS the following 2 properties must be set in the `etc/hadoop/kms_env.sh` script (shown with default values):
+
+* KMS_SSL_KEYSTORE_FILE=$HOME/.keystore
+* KMS_SSL_KEYSTORE_PASS=password
+
+In the KMS `tomcat/conf` directory, replace the `server.xml` file with the provided `ssl-server.xml` file.
+
+You need to create an SSL certificate for the KMS. As the `kms` Unix user, using the Java `keytool` command to create the SSL certificate:
+
+ $ keytool -genkey -alias tomcat -keyalg RSA
+
+You will be asked a series of questions in an interactive prompt. It will create the keystore file, which will be named **.keystore** and located in the `kms` user home directory.
+
+The password you enter for "keystore password" must match the value of the `KMS_SSL_KEYSTORE_PASS` environment variable set in the `kms-env.sh` script in the configuration directory.
+
+The answer to "What is your first and last name?" (i.e. "CN") must be the hostname of the machine where the KMS will be running.
+
+NOTE: You need to restart the KMS for the configuration changes to take effect.
+
+$H4 KMS Access Control
+
+KMS ACLs configuration are defined in the KMS `etc/hadoop/kms-acls.xml` configuration file. This file is hot-reloaded when it changes.
+
+KMS supports both fine grained access control as well as blacklist for kms operations via a set ACL configuration properties.
+
+A user accessing KMS is first checked for inclusion in the Access Control List for the requested operation and then checked for exclusion in the Black list for the operation before access is granted.
+
+```xml
+<configuration>
+ <property>
+ <name>hadoop.kms.acl.CREATE</name>
+ <value>*</value>
+ <description>
+ ACL for create-key operations.
+ If the user is not in the GET ACL, the key material is not returned
+ as part of the response.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.blacklist.CREATE</name>
+ <value>hdfs,foo</value>
+ <description>
+ Blacklist for create-key operations.
+ If the user is in the Blacklist, the key material is not returned
+ as part of the response.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.acl.DELETE</name>
+ <value>*</value>
+ <description>
+ ACL for delete-key operations.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.blacklist.DELETE</name>
+ <value>hdfs,foo</value>
+ <description>
+ Blacklist for delete-key operations.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.acl.ROLLOVER</name>
+ <value>*</value>
+ <description>
+ ACL for rollover-key operations.
+ If the user is not in the GET ACL, the key material is not returned
+ as part of the response.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.blacklist.ROLLOVER</name>
+ <value>hdfs,foo</value>
+ <description>
+ Blacklist for rollover-key operations.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.acl.GET</name>
+ <value>*</value>
+ <description>
+ ACL for get-key-version and get-current-key operations.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.blacklist.GET</name>
+ <value>hdfs,foo</value>
+ <description>
+ ACL for get-key-version and get-current-key operations.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.acl.GET_KEYS</name>
+ <value>*</value>
+ <description>
+ ACL for get-keys operation.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.blacklist.GET_KEYS</name>
+ <value>hdfs,foo</value>
+ <description>
+ Blacklist for get-keys operation.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.acl.GET_METADATA</name>
+ <value>*</value>
+ <description>
+ ACL for get-key-metadata and get-keys-metadata operations.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.blacklist.GET_METADATA</name>
+ <value>hdfs,foo</value>
+ <description>
+ Blacklist for get-key-metadata and get-keys-metadata operations.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.acl.SET_KEY_MATERIAL</name>
+ <value>*</value>
+ <description>
+ Complimentary ACL for CREATE and ROLLOVER operation to allow the client
+ to provide the key material when creating or rolling a key.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.blacklist.SET_KEY_MATERIAL</name>
+ <value>hdfs,foo</value>
+ <description>
+ Complimentary Blacklist for CREATE and ROLLOVER operation to allow the client
+ to provide the key material when creating or rolling a key.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.acl.GENERATE_EEK</name>
+ <value>*</value>
+ <description>
+ ACL for generateEncryptedKey
+ CryptoExtension operations
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.blacklist.GENERATE_EEK</name>
+ <value>hdfs,foo</value>
+ <description>
+ Blacklist for generateEncryptedKey
+ CryptoExtension operations
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.acl.DECRYPT_EEK</name>
+ <value>*</value>
+ <description>
+ ACL for decrypt EncryptedKey
+ CryptoExtension operations
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.blacklist.DECRYPT_EEK</name>
+ <value>hdfs,foo</value>
+ <description>
+ Blacklist for decrypt EncryptedKey
+ CryptoExtension operations
+ </description>
+ </property>
+</configuration>
+```
+
+$H4 Key Access Control
+
+KMS supports access control for all non-read operations at the Key level. All Key Access operations are classified as :
+
+* MANAGEMENT - createKey, deleteKey, rolloverNewVersion
+* GENERATE_EEK - generateEncryptedKey, warmUpEncryptedKeys
+* DECRYPT_EEK - decryptEncryptedKey
+* READ - getKeyVersion, getKeyVersions, getMetadata, getKeysMetadata, getCurrentKey
+* ALL - all of the above
+
+These can be defined in the KMS `etc/hadoop/kms-acls.xml` as follows
+
+For all keys for which a key access has not been explicitly configured, It is possible to configure a default key access control for a subset of the operation types.
+
+It is also possible to configure a "whitelist" key ACL for a subset of the operation types. The whitelist key ACL is a whitelist in addition to the explicit or default per-key ACL. That is, if no per-key ACL is explicitly set, a user will be granted access if they are present in the default per-key ACL or the whitelist key ACL. If a per-key ACL is explicitly set, a user will be granted access if they are present in the per-key ACL or the whitelist key ACL.
+
+If no ACL is configured for a specific key AND no default ACL is configured AND no root key ACL is configured for the requested operation, then access will be DENIED.
+
+**NOTE:** The default and whitelist key ACL does not support `ALL` operation qualifier.
+
+```xml
+ <property>
+ <name>key.acl.testKey1.MANAGEMENT</name>
+ <value>*</value>
+ <description>
+ ACL for create-key, deleteKey and rolloverNewVersion operations.
+ </description>
+ </property>
+
+ <property>
+ <name>key.acl.testKey2.GENERATE_EEK</name>
+ <value>*</value>
+ <description>
+ ACL for generateEncryptedKey operations.
+ </description>
+ </property>
+
+ <property>
+ <name>key.acl.testKey3.DECRYPT_EEK</name>
+ <value>admink3</value>
+ <description>
+ ACL for decryptEncryptedKey operations.
+ </description>
+ </property>
+
+ <property>
+ <name>key.acl.testKey4.READ</name>
+ <value>*</value>
+ <description>
+ ACL for getKeyVersion, getKeyVersions, getMetadata, getKeysMetadata,
+ getCurrentKey operations
+ </description>
+ </property>
+
+ <property>
+ <name>key.acl.testKey5.ALL</name>
+ <value>*</value>
+ <description>
+ ACL for ALL operations.
+ </description>
+ </property>
+
+ <property>
+ <name>whitelist.key.acl.MANAGEMENT</name>
+ <value>admin1</value>
+ <description>
+ whitelist ACL for MANAGEMENT operations for all keys.
+ </description>
+ </property>
+
+ <!--
+ 'testKey3' key ACL is defined. Since a 'whitelist'
+ key is also defined for DECRYPT_EEK, in addition to
+ admink3, admin1 can also perform DECRYPT_EEK operations
+ on 'testKey3'
+-->
+ <property>
+ <name>whitelist.key.acl.DECRYPT_EEK</name>
+ <value>admin1</value>
+ <description>
+ whitelist ACL for DECRYPT_EEK operations for all keys.
+ </description>
+ </property>
+
+ <property>
+ <name>default.key.acl.MANAGEMENT</name>
+ <value>user1,user2</value>
+ <description>
+ default ACL for MANAGEMENT operations for all keys that are not
+ explicitly defined.
+ </description>
+ </property>
+
+ <property>
+ <name>default.key.acl.GENERATE_EEK</name>
+ <value>user1,user2</value>
+ <description>
+ default ACL for GENERATE_EEK operations for all keys that are not
+ explicitly defined.
+ </description>
+ </property>
+
+ <property>
+ <name>default.key.acl.DECRYPT_EEK</name>
+ <value>user1,user2</value>
+ <description>
+ default ACL for DECRYPT_EEK operations for all keys that are not
+ explicitly defined.
+ </description>
+ </property>
+
+ <property>
+ <name>default.key.acl.READ</name>
+ <value>user1,user2</value>
+ <description>
+ default ACL for READ operations for all keys that are not
+ explicitly defined.
+ </description>
+ </property>
+```
+
+$H3 KMS Delegation Token Configuration
+
+KMS delegation token secret manager can be configured with the following properties:
+
+```xml
+ <property>
+ <name>hadoop.kms.authentication.delegation-token.update-interval.sec</name>
+ <value>86400</value>
+ <description>
+ How often the master key is rotated, in seconds. Default value 1 day.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.delegation-token.max-lifetime.sec</name>
+ <value>604800</value>
+ <description>
+ Maximum lifetime of a delagation token, in seconds. Default value 7 days.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.delegation-token.renew-interval.sec</name>
+ <value>86400</value>
+ <description>
+ Renewal interval of a delagation token, in seconds. Default value 1 day.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.delegation-token.removal-scan-interval.sec</name>
+ <value>3600</value>
+ <description>
+ Scan interval to remove expired delegation tokens.
+ </description>
+ </property>
+```
+
+$H3 Using Multiple Instances of KMS Behind a Load-Balancer or VIP
+
+KMS supports multiple KMS instances behind a load-balancer or VIP for scalability and for HA purposes.
+
+When using multiple KMS instances behind a load-balancer or VIP, requests from the same user may be handled by different KMS instances.
+
+KMS instances behind a load-balancer or VIP must be specially configured to work properly as a single logical service.
+
+$H4 HTTP Kerberos Principals Configuration
+
+When KMS instances are behind a load-balancer or VIP, clients will use the hostname of the VIP. For Kerberos SPNEGO authentication, the hostname of the URL is used to construct the Kerberos service name of the server, `HTTP/#HOSTNAME#`. This means that all KMS instances must have a Kerberos service name with the load-balancer or VIP hostname.
+
+In order to be able to access directly a specific KMS instance, the KMS instance must also have Keberos service name with its own hostname. This is required for monitoring and admin purposes.
+
+Both Kerberos service principal credentials (for the load-balancer/VIP hostname and for the actual KMS instance hostname) must be in the keytab file configured for authentication. And the principal name specified in the configuration must be '\*'. For example:
+
+```xml
+ <property>
+ <name>hadoop.kms.authentication.kerberos.principal</name>
+ <value>*</value>
+ </property>
+```
+
+**NOTE:** If using HTTPS, the SSL certificate used by the KMS instance must be configured to support multiple hostnames (see Java 7 `keytool` SAN extension support for details on how to do this).
+
+$H4 HTTP Authentication Signature
+
+KMS uses Hadoop Authentication for HTTP authentication. Hadoop Authentication issues a signed HTTP Cookie once the client has authenticated successfully. This HTTP Cookie has an expiration time, after which it will trigger a new authentication sequence. This is done to avoid triggering the authentication on every HTTP request of a client.
+
+A KMS instance must verify the HTTP Cookie signatures signed by other KMS instances. To do this all KMS instances must share the signing secret.
+
+This secret sharing can be done using a Zookeeper service which is configured in KMS with the following properties in the `kms-site.xml`:
+
+```xml
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider</name>
+ <value>zookeeper</value>
+ <description>
+ Indicates how the secret to sign the authentication cookies will be
+ stored. Options are 'random' (default), 'string' and 'zookeeper'.
+ If using a setup with multiple KMS instances, 'zookeeper' should be used.
+ </description>
+ </property>
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.path</name>
+ <value>/hadoop-kms/hadoop-auth-signature-secret</value>
+ <description>
+ The Zookeeper ZNode path where the KMS instances will store and retrieve
+ the secret from.
+ </description>
+ </property>
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.connection.string</name>
+ <value>#HOSTNAME#:#PORT#,...</value>
+ <description>
+ The Zookeeper connection string, a list of hostnames and port comma
+ separated.
+ </description>
+ </property>
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.auth.type</name>
+ <value>kerberos</value>
+ <description>
+ The Zookeeper authentication type, 'none' or 'sasl' (Kerberos).
+ </description>
+ </property>
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.keytab</name>
+ <value>/etc/hadoop/conf/kms.keytab</value>
+ <description>
+ The absolute path for the Kerberos keytab with the credentials to
+ connect to Zookeeper.
+ </description>
+ </property>
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.principal</name>
+ <value>kms/#HOSTNAME#</value>
+ <description>
+ The Kerberos service principal used to connect to Zookeeper.
+ </description>
+ </property>
+```
+
+$H4 Delegation Tokens
+
+TBD
+
+$H3 KMS HTTP REST API
+
+$H4 Create a Key
+
+*REQUEST:*
+
+ POST http://HOST:PORT/kms/v1/keys
+ Content-Type: application/json
+
+ {
+ "name" : "<key-name>",
+ "cipher" : "<cipher>",
+ "length" : <length>, //int
+ "material" : "<material>", //base64
+ "description" : "<description>"
+ }
+
+*RESPONSE:*
+
+ 201 CREATED
+ LOCATION: http://HOST:PORT/kms/v1/key/<key-name>
+ Content-Type: application/json
+
+ {
+ "name" : "versionName",
+ "material" : "<material>", //base64, not present without GET ACL
+ }
+
+$H4 Rollover Key
+
+*REQUEST:*
+
+ POST http://HOST:PORT/kms/v1/key/<key-name>
+ Content-Type: application/json
+
+ {
+ "material" : "<material>",
+ }
+
+*RESPONSE:*
+
+ 200 OK
+ Content-Type: application/json
+
+ {
+ "name" : "versionName",
+ "material" : "<material>", //base64, not present without GET ACL
+ }
+
+$H4 Delete Key
+
+*REQUEST:*
+
+ DELETE http://HOST:PORT/kms/v1/key/<key-name>
+
+*RESPONSE:*
+
+ 200 OK
+
+$H4 Get Key Metadata
+
+*REQUEST:*
+
+ GET http://HOST:PORT/kms/v1/key/<key-name>/_metadata
+
+*RESPONSE:*
+
+ 200 OK
+ Content-Type: application/json
+
+ {
+ "name" : "<key-name>",
+ "cipher" : "<cipher>",
+ "length" : <length>, //int
+ "description" : "<description>",
+ "created" : <millis-epoc>, //long
+ "versions" : <versions> //int
+ }
+
+$H4 Get Current Key
+
+*REQUEST:*
+
+ GET http://HOST:PORT/kms/v1/key/<key-name>/_currentversion
+
+*RESPONSE:*
+
+ 200 OK
+ Content-Type: application/json
+
+ {
+ "name" : "versionName",
+ "material" : "<material>", //base64
+ }
+
+$H4 Generate Encrypted Key for Current KeyVersion
+
+*REQUEST:*
+
+ GET http://HOST:PORT/kms/v1/key/<key-name>/_eek?eek_op=generate&num_keys=<number-of-keys-to-generate>
+
+*RESPONSE:*
+
+ 200 OK
+ Content-Type: application/json
+ [
+ {
+ "versionName" : "encryptionVersionName",
+ "iv" : "<iv>", //base64
+ "encryptedKeyVersion" : {
+ "versionName" : "EEK",
+ "material" : "<material>", //base64
+ }
+ },
+ {
+ "versionName" : "encryptionVersionName",
+ "iv" : "<iv>", //base64
+ "encryptedKeyVersion" : {
+ "versionName" : "EEK",
+ "material" : "<material>", //base64
+ }
+ },
+ ...
+ ]
+
+$H4 Decrypt Encrypted Key
+
+*REQUEST:*
+
+ POST http://HOST:PORT/kms/v1/keyversion/<version-name>/_eek?ee_op=decrypt
+ Content-Type: application/json
+
+ {
+ "name" : "<key-name>",
+ "iv" : "<iv>", //base64
+ "material" : "<material>", //base64
+ }
+
+*RESPONSE:*
+
+ 200 OK
+ Content-Type: application/json
+
+ {
+ "name" : "EK",
+ "material" : "<material>", //base64
+ }
+
+$H4 Get Key Version
+
+*REQUEST:*
+
+ GET http://HOST:PORT/kms/v1/keyversion/<version-name>
+
+*RESPONSE:*
+
+ 200 OK
+ Content-Type: application/json
+
+ {
+ "name" : "versionName",
+ "material" : "<material>", //base64
+ }
+
+$H4 Get Key Versions
+
+*REQUEST:*
+
+ GET http://HOST:PORT/kms/v1/key/<key-name>/_versions
+
+*RESPONSE:*
+
+ 200 OK
+ Content-Type: application/json
+
+ [
+ {
+ "name" : "versionName",
+ "material" : "<material>", //base64
+ },
+ {
+ "name" : "versionName",
+ "material" : "<material>", //base64
+ },
+ ...
+ ]
+
+$H4 Get Key Names
+
+*REQUEST:*
+
+ GET http://HOST:PORT/kms/v1/keys/names
+
+*RESPONSE:*
+
+ 200 OK
+ Content-Type: application/json
+
+ [
+ "<key-name>",
+ "<key-name>",
+ ...
+ ]
+
+$H4 Get Keys Metadata
+
+ GET http://HOST:PORT/kms/v1/keys/metadata?key=<key-name>&key=<key-name>,...
+
+*RESPONSE:*
+
+ 200 OK
+ Content-Type: application/json
+
+ [
+ {
+ "name" : "<key-name>",
+ "cipher" : "<cipher>",
+ "length" : <length>, //int
+ "description" : "<description>",
+ "created" : <millis-epoc>, //long
+ "versions" : <versions> //int
+ },
+ {
+ "name" : "<key-name>",
+ "cipher" : "<cipher>",
+ "length" : <length>, //int
+ "description" : "<description>",
+ "created" : <millis-epoc>, //long
+ "versions" : <versions> //int
+ },
+ ...
+ ]
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-project/src/site/apt/index.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-project/src/site/apt/index.apt.vm b/hadoop-project/src/site/apt/index.apt.vm
deleted file mode 100644
index 4167f4d..0000000
--- a/hadoop-project/src/site/apt/index.apt.vm
+++ /dev/null
@@ -1,73 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- Apache Hadoop ${project.version}
- ---
- ---
- ${maven.build.timestamp}
-
-Apache Hadoop ${project.version}
-
- Apache Hadoop ${project.version} consists of significant
- improvements over the previous stable release (hadoop-1.x).
-
- Here is a short overview of the improvments to both HDFS and MapReduce.
-
- * {HDFS Federation}
-
- In order to scale the name service horizontally, federation uses multiple
- independent Namenodes/Namespaces. The Namenodes are federated, that is, the
- Namenodes are independent and don't require coordination with each other.
- The datanodes are used as common storage for blocks by all the Namenodes.
- Each datanode registers with all the Namenodes in the cluster. Datanodes
- send periodic heartbeats and block reports and handles commands from the
- Namenodes.
-
- More details are available in the
- {{{./hadoop-project-dist/hadoop-hdfs/Federation.html}HDFS Federation}}
- document.
-
- * {MapReduce NextGen aka YARN aka MRv2}
-
- The new architecture introduced in hadoop-0.23, divides the two major
- functions of the JobTracker: resource management and job life-cycle management
- into separate components.
-
- The new ResourceManager manages the global assignment of compute resources to
- applications and the per-application ApplicationMaster manages the
- application‚ scheduling and coordination.
-
- An application is either a single job in the sense of classic MapReduce jobs
- or a DAG of such jobs.
-
- The ResourceManager and per-machine NodeManager daemon, which manages the
- user processes on that machine, form the computation fabric.
-
- The per-application ApplicationMaster is, in effect, a framework specific
- library and is tasked with negotiating resources from the ResourceManager and
- working with the NodeManager(s) to execute and monitor the tasks.
-
- More details are available in the
- {{{./hadoop-yarn/hadoop-yarn-site/YARN.html}YARN}}
- document.
-
-Getting Started
-
- The Hadoop documentation includes the information you need to get started using
- Hadoop. Begin with the
- {{{./hadoop-project-dist/hadoop-common/SingleCluster.html}Single Node Setup}} which
- shows you how to set up a single-node Hadoop installation. Then move on to the
- {{{./hadoop-project-dist/hadoop-common/ClusterSetup.html}Cluster Setup}} to learn how
- to set up a multi-node Hadoop installation.
-
-
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-project/src/site/markdown/index.md.vm
----------------------------------------------------------------------
diff --git a/hadoop-project/src/site/markdown/index.md.vm b/hadoop-project/src/site/markdown/index.md.vm
new file mode 100644
index 0000000..c3a93ad
--- /dev/null
+++ b/hadoop-project/src/site/markdown/index.md.vm
@@ -0,0 +1,72 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+Apache Hadoop ${project.version}
+================================
+
+Apache Hadoop ${project.version} consists of significant
+improvements over the previous stable release (hadoop-1.x).
+
+Here is a short overview of the improvments to both HDFS and MapReduce.
+
+* HDFS Federation
+
+ In order to scale the name service horizontally, federation uses
+ multiple independent Namenodes/Namespaces. The Namenodes are
+ federated, that is, the Namenodes are independent and don't require
+ coordination with each other. The datanodes are used as common storage
+ for blocks by all the Namenodes. Each datanode registers with all the
+ Namenodes in the cluster. Datanodes send periodic heartbeats and block
+ reports and handles commands from the Namenodes.
+
+ More details are available in the
+ [HDFS Federation](./hadoop-project-dist/hadoop-hdfs/Federation.html)
+ document.
+
+* MapReduce NextGen aka YARN aka MRv2
+
+ The new architecture introduced in hadoop-0.23, divides the two major
+ functions of the JobTracker: resource management and job life-cycle
+ management into separate components.
+
+ The new ResourceManager manages the global assignment of compute
+ resources to applications and the per-application
+ ApplicationMaster manages the application‚ scheduling and
+ coordination.
+
+ An application is either a single job in the sense of classic
+ MapReduce jobs or a DAG of such jobs.
+
+ The ResourceManager and per-machine NodeManager daemon, which
+ manages the user processes on that machine, form the computation
+ fabric.
+
+ The per-application ApplicationMaster is, in effect, a framework
+ specific library and is tasked with negotiating resources from the
+ ResourceManager and working with the NodeManager(s) to execute and
+ monitor the tasks.
+
+ More details are available in the
+ [YARN](./hadoop-yarn/hadoop-yarn-site/YARN.html) document.
+
+Getting Started
+===============
+
+The Hadoop documentation includes the information you need to get started using
+Hadoop. Begin with the
+[Single Node Setup](./hadoop-project-dist/hadoop-common/SingleCluster.html)
+which shows you how to set up a single-node Hadoop installation.
+Then move on to the
+[Cluster Setup](./hadoop-project-dist/hadoop-common/ClusterSetup.html)
+to learn how to set up a multi-node Hadoop installation.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-tools/hadoop-openstack/src/site/apt/index.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-openstack/src/site/apt/index.apt.vm b/hadoop-tools/hadoop-openstack/src/site/apt/index.apt.vm
deleted file mode 100644
index 29fbd33..0000000
--- a/hadoop-tools/hadoop-openstack/src/site/apt/index.apt.vm
+++ /dev/null
@@ -1,686 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- Hadoop OpenStack Support: Swift Object Store
- ---
- ---
- ${maven.build.timestamp}
-
-%{toc|section=1|fromDepth=0}
-
-Hadoop OpenStack Support: Swift Object Store
-
-* {Introduction}
-
- {{{http://www.openstack.org/}OpenStack}} is an open source cloud infrastructure
- which can be accessed
- from multiple public IaaS providers, and deployed privately. It offers
- infrastructure services such as VM hosting (Nova), authentication (Keystone)
- and storage of binary objects (Swift).
-
- This module enables Apache Hadoop applications -including MapReduce jobs,
- read and write data to and from instances of the
- {{{http://www.openstack.org/software/openstack-storage/}OpenStack Swift object store}}.
-
-* Features
-
- * Read and write of data stored in a Swift object store
-
- * Support of a pseudo-hierachical file system (directories, subdirectories and
- files)
-
- * Standard filesystem operations: <<<create>>>, <<<delete>>>, <<<mkdir>>>,
- <<<ls>>>, <<<mv>>>, <<<stat>>>.
-
- * Can act as a source of data in a MapReduce job, or a sink.
-
- * Support for multiple OpenStack services, and multiple containers from a
- single service.
-
- * Supports in-cluster and remote access to Swift data.
-
- * Supports OpenStack Keystone authentication with password or token.
-
- * Released under the Apache Software License
-
- * Tested against the Hadoop 3.x and 1.x branches, against multiple public
- OpenStack clusters: Rackspace US, Rackspace UK, HP Cloud.
-
- * Tested against private OpenStack clusters, including scalability tests of
- large file uploads.
-
-* Using the Hadoop Swift Filesystem Client
-
-** Concepts: services and containers
-
- OpenStack swift is an <Object Store>; also known as a <blobstore>. It stores
- arbitrary binary objects by name in a <container>.
-
- The Hadoop Swift filesystem library adds another concept, the <service>, which
- defines which Swift blobstore hosts a container -and how to connect to it.
-
-** Containers and Objects
-
- * Containers are created by users with accounts on the Swift filestore, and hold
- <objects>.
-
- * Objects can be zero bytes long, or they can contain data.
-
- * Objects in the container can be up to 5GB; there is a special support for
- larger files than this, which merges multiple objects in to one.
-
- * Each object is referenced by it's <name>; there is no notion of directories.
-
- * You can use any characters in an object name that can be 'URL-encoded'; the
- maximum length of a name is 1034 characters -after URL encoding.
-
- * Names can have <<</>>> characters in them, which are used to create the illusion of
- a directory structure. For example <<<dir/dir2/name>>>. Even though this looks
- like a directory, <it is still just a name>. There is no requirement to have
- any entries in the container called <<<dir>>> or <<<dir/dir2>>>
-
- * That said. if the container has zero-byte objects that look like directory
- names above other objects, they can pretend to be directories. Continuing the
- example, a 0-byte object called <<<dir>>> would tell clients that it is a
- directory while <<<dir/dir2>>> or <<<dir/dir2/name>>> were present. This creates an
- illusion of containers holding a filesystem.
-
- Client applications talk to Swift over HTTP or HTTPS, reading, writing and
- deleting objects using standard HTTP operations (GET, PUT and DELETE,
- respectively). There is also a COPY operation, that creates a new object in the
- container, with a new name, containing the old data. There is no rename
- operation itself, objects need to be copied -then the original entry deleted.
-
-** Eventual Consistency
-
- The Swift Filesystem is *eventually consistent*: an operation on an object may
- not be immediately visible to that client, or other clients. This is a
- consequence of the goal of the filesystem: to span a set of machines, across
- multiple datacenters, in such a way that the data can still be available when
- many of them fail. (In contrast, the Hadoop HDFS filesystem is *immediately
- consistent*, but it does not span datacenters.)
-
- Eventual consistency can cause surprises for client applications that expect
- immediate consistency: after an object is deleted or overwritten, the object
- may still be visible -or the old data still retrievable. The Swift Filesystem
- client for Apache Hadoop attempts to handle this, in conjunction with the
- MapReduce engine, but there may be still be occasions when eventual consistency
- causes surprises.
-
-** Non-atomic "directory" operations.
-
- Hadoop expects some
- operations to be atomic, especially <<<rename()>>>, which is something
- the MapReduce layer relies on to commit the output of a job, renaming data
- from a temp directory to the final path. Because a rename
- is implemented as a copy of every blob under the directory's path, followed
- by a delete of the originals, the intermediate state of the operation
- will be visible to other clients. If two Reducer tasks to rename their temp
- directory to the final path, both operations may succeed, with the result that
- output directory contains mixed data. This can happen if MapReduce jobs
- are being run with <speculation> enabled and Swift used as the direct output
- of the MR job (it can also happen against Amazon S3).
-
- Other consequences of the non-atomic operations are:
-
- 1. If a program is looking for the presence of the directory before acting
- on the data -it may start prematurely. This can be avoided by using
- other mechanisms to co-ordinate the programs, such as the presence of a file
- that is written <after> any bulk directory operations.
-
- 2. A <<<rename()>>> or <<<delete()>>> operation may include files added under
- the source directory tree during the operation, may unintentionally delete
- it, or delete the 0-byte swift entries that mimic directories and act
- as parents for the files. Try to avoid doing this.
-
- The best ways to avoid all these problems is not using Swift as
- the filesystem between MapReduce jobs or other Hadoop workflows. It
- can act as a source of data, and a final destination, but it doesn't meet
- all of Hadoop's expectations of what a filesystem is -it's a <blobstore>.
-
-* Working with Swift Object Stores in Hadoop
-
- Once installed, the Swift FileSystem client can be used by any Hadoop application
- to read from or write to data stored in a Swift container.
-
- Data stored in Swift can be used as the direct input to a MapReduce job
- -simply use the <<<swift:>>> URL (see below) to declare the source of the data.
-
- This Swift Filesystem client is designed to work with multiple
- Swift object stores, both public and private. This allows the client to work
- with different clusters, reading and writing data to and from either of them.
-
- It can also work with the same object stores using multiple login details.
-
- These features are achieved by one basic concept: using a service name in
- the URI referring to a swift filesystem, and looking up all the connection and
- login details for that specific service. Different service names can be defined
- in the Hadoop XML configuration file, so defining different clusters, or
- providing different login details for the same object store(s).
-
-
-** Swift Filesystem URIs
-
- Hadoop uses URIs to refer to files within a filesystem. Some common examples
- are:
-
-+--
- local://etc/hosts
- hdfs://cluster1/users/example/data/set1
- hdfs://cluster2.example.org:8020/users/example/data/set1
-+--
-
- The Swift Filesystem Client adds a new URL type <<<swift>>>. In a Swift Filesystem
- URL, the hostname part of a URL identifies the container and the service to
- work with; the path the name of the object. Here are some examples
-
-+--
- swift://container.rackspace/my-object.csv
- swift://data.hpcloud/data/set1
- swift://dmitry.privatecloud/out/results
-+--
-
- In the last two examples, the paths look like directories: it is not, they are
- simply the objects named <<<data/set1>>> and <<<out/results>>> respectively.
-
-** Installing
-
- The <<<hadoop-openstack>>> JAR must be on the classpath of the Hadoop program trying to
- talk to the Swift service. If installed in the classpath of the Hadoop
- MapReduce service, then all programs started by the MR engine will pick up the
- JAR automatically. This is the easiest way to give all Hadoop jobs access to
- Swift.
-
- Alternatively, the JAR can be included as one of the JAR files that an
- application uses. This lets the Hadoop jobs work with a Swift object store even
- if the Hadoop cluster is not pre-configured for this.
-
- The library also depends upon the Apache HttpComponents library, which
- must also be on the classpath.
-
-** Configuring
-
- To talk to a swift service, the user must must provide:
-
- [[1]] The URL defining the container and the service.
-
- [[1]] In the cluster/job configuration, the login details of that service.
-
- Multiple service definitions can co-exist in the same configuration file: just
- use different names for them.
-
-*** Example: Rackspace US, in-cluster access using API key
-
- This service definition is for use in a Hadoop cluster deployed within Rackspace's
- US infrastructure.
-
-+--
- <property>
- <name>fs.swift.service.rackspace.auth.url</name>
- <value>https://auth.api.rackspacecloud.com/v2.0/tokens</value>
- <description>Rackspace US (multiregion)</description>
- </property>
-
- <property>
- <name>fs.swift.service.rackspace.username</name>
- <value>user4</value>
- </property>
-
- <property>
- <name>fs.swift.service.rackspace.region</name>
- <value>DFW</value>
- </property>
-
- <property>
- <name>fs.swift.service.rackspace.apikey</name>
- <value>fe806aa86dfffe2f6ed8</value>
- </property>
-+--
-
- Here the API key visible in the account settings API keys page is used to log
- in. No property for public/private access -the default is to use the private
- endpoint for Swift operations.
-
- This configuration also selects one of the regions, DFW, for its data.
-
- A reference to this service would use the <<<rackspace>>> service name:
-
----
- swift://hadoop-container.rackspace/
----
-
-*** Example: Rackspace UK: remote access with password authentication
-
- This connects to Rackspace's UK ("LON") datacenter.
-
-+--
- <property>
- <name>fs.swift.service.rackspaceuk.auth.url</name>
- <value>https://lon.identity.api.rackspacecloud.com/v2.0/tokens</value>
- <description>Rackspace UK</description>
- </property>
-
- <property>
- <name>fs.swift.service.rackspaceuk.username</name>
- <value>user4</value>
- </property>
-
- <property>
- <name>fs.swift.service.rackspaceuk.password</name>
- <value>insert-password-here/value>
- </property>
-
- <property>
- <name>fs.swift.service.rackspace.public</name>
- <value>true</value>
- </property>
-+--
-
- This is a public access point connection, using a password over an API key.
-
- A reference to this service would use the <<<rackspaceuk>>> service name:
-
-+--
- swift://hadoop-container.rackspaceuk/
-+--
-
- Because the public endpoint is used, if this service definition is used within
- the London datacenter, all accesses will be billed at the public
- upload/download rates, <irrespective of where the Hadoop cluster is>.
-
-*** Example: HP cloud service definition
-
- Here is an example that connects to the HP Cloud object store.
-
-+--
- <property>
- <name>fs.swift.service.hpcloud.auth.url</name>
- <value>https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0/tokens
- </value>
- <description>HP Cloud</description>
- </property>
-
- <property>
- <name>fs.swift.service.hpcloud.tenant</name>
- <value>FE806AA86</value>
- </property>
-
- <property>
- <name>fs.swift.service.hpcloud.username</name>
- <value>FE806AA86DFFFE2F6ED8</value>
- </property>
-
- <property>
- <name>fs.swift.service.hpcloud.password</name>
- <value>secret-password-goes-here</value>
- </property>
-
- <property>
- <name>fs.swift.service.hpcloud.public</name>
- <value>true</value>
- </property>
-+--
-
- A reference to this service would use the <<<hpcloud>>> service name:
-
-+--
- swift://hadoop-container.hpcloud/
-+--
-
-** General Swift Filesystem configuration options
-
- Some configuration options apply to the Swift client, independent of
- the specific Swift filesystem chosen.
-
-*** Blocksize fs.swift.blocksize
-
- Swift does not break up files into blocks, except in the special case of files
- over 5GB in length. Accordingly, there isn't a notion of a "block size"
- to define where the data is kept.
-
- Hadoop's MapReduce layer depends on files declaring their block size,
- so that it knows how to partition work. Too small a blocksize means that
- many mappers work on small pieces of data; too large a block size means
- that only a few mappers get started.
-
- The block size value reported by Swift, therefore, controls the basic workload
- partioning of the MapReduce engine -and can be an important parameter to
- tune for performance of the cluster.
-
- The property has a unit of kilobytes; the default value is <<<32*1024>>>: 32 MB
-
-+--
- <property>
- <name>fs.swift.blocksize</name>
- <value>32768</value>
- </property>
-+--
-
- This blocksize has no influence on how files are stored in Swift; it only controls
- what the reported size of blocks are - a value used in Hadoop MapReduce to
- divide work.
-
- Note that the MapReduce engine's split logic can be tuned independently by setting
- the <<<mapred.min.split.size>>> and <<<mapred.max.split.size>>> properties,
- which can be done in specific job configurations.
-
-+--
- <property>
- <name>mapred.min.split.size</name>
- <value>524288</value>
- </property>
-
- <property>
- <name>mapred.max.split.size</name>
- <value>1048576</value>
- </property>
-+--
-
- In an Apache Pig script, these properties would be set as:
-
----
- mapred.min.split.size 524288
- mapred.max.split.size 1048576
----
-
-*** Partition size fs.swift.partsize
-
- The Swift filesystem client breaks very large files into partitioned files,
- uploading each as it progresses, and writing any remaning data and an XML
- manifest when a partitioned file is closed.
-
- The partition size defaults to 4608 MB; 4.5GB, the maximum filesize that
- Swift can support.
-
- It is possible to set a smaller partition size, in the <<<fs.swift.partsize>>>
- option. This takes a value in KB.
-
-+--
- <property>
- <name>fs.swift.partsize</name>
- <value>1024</value>
- <description>upload every MB</description>
- </property>
-+--
-
- When should this value be changed from its default?
-
- While there is no need to ever change it for basic operation of
- the Swift filesystem client, it can be tuned
-
- * If a Swift filesystem is location aware, then breaking a file up into
- smaller partitions scatters the data round the cluster. For best performance,
- the property <<<fs.swift.blocksize>>> should be set to a smaller value than the
- partition size of files.
-
- * When writing to an unpartitioned file, the entire write is done in the
- <<<close()>>> operation. When a file is partitioned, the outstanding data to
- be written whenever the outstanding amount of data is greater than the
- partition size. This means that data will be written more incrementally
-
-*** Request size fs.swift.requestsize
-
- The Swift filesystem client reads files in HTTP GET operations, asking for
- a block of data at a time.
-
- The default value is 64KB. A larger value may be more efficient over faster
- networks, as it reduces the overhead of setting up the HTTP operation.
-
- However, if the file is read with many random accesses, requests for
- data will be made from different parts of the file -discarding some of the
- previously requested data. The benefits of larger request sizes may be wasted.
-
- The property <<<fs.swift.requestsize>>> sets the request size in KB.
-
-+--
- <property>
- <name>fs.swift.requestsize</name>
- <value>128</value>
- </property>
-+--
-
-*** Connection timeout fs.swift.connect.timeout
-
- This sets the timeout in milliseconds to connect to a Swift service.
-
-+--
- <property>
- <name>fs.swift.connect.timeout</name>
- <value>15000</value>
- </property>
-+--
-
- A shorter timeout means that connection failures are raised faster -but
- may trigger more false alarms. A longer timeout is more resilient to network
- problems -and may be needed when talking to remote filesystems.
-
-*** Connection timeout fs.swift.socket.timeout
-
- This sets the timeout in milliseconds to wait for data from a connected socket.
-
-+--
- <property>
- <name>fs.swift.socket.timeout</name>
- <value>60000</value>
- </property>
-+--
-
- A shorter timeout means that connection failures are raised faster -but
- may trigger more false alarms. A longer timeout is more resilient to network
- problems -and may be needed when talking to remote filesystems.
-
-*** Connection Retry Count fs.swift.connect.retry.count
-
- This sets the number of times to try to connect to a service whenever
- an HTTP request is made.
-
-+--
- <property>
- <name>fs.swift.connect.retry.count</name>
- <value>3</value>
- </property>
-+--
-
- The more retries, the more resilient it is to transient outages -and the
- less rapid it is at detecting and reporting server connectivity problems.
-
-*** Connection Throttle Delay fs.swift.connect.throttle.delay
-
- This property adds a delay between bulk file copy and delete operations,
- to prevent requests being throttled or blocked by the remote service
-
-+--
- <property>
- <name>fs.swift.connect.throttle.delay</name>
- <value>0</value>
- </property>
-+--
-
- It is measured in milliseconds; "0" means do not add any delay.
-
- Throttling is enabled on the public endpoints of some Swift services.
- If <<<rename()>>> or <<<delete()>>> operations fail with
- <<<SwiftThrottledRequestException>>>
- exceptions, try setting this property.
-
-*** HTTP Proxy
-
- If the client can only access the Swift filesystem via a web proxy
- server, the client configuration must specify the proxy via
- the <<<fs.swift.connect.proxy.host>>> and <<<fs.swift.connect.proxy.port>>>
- properties.
-
-+--
- <property>
- <name>fs.swift.proxy.host</name>
- <value>web-proxy</value>
- </property>
-
- <property>
- <name>fs.swift.proxy.port</name>
- <value>8088</value>
- </property>
-+--
-
- If the host is declared, the proxy port must be set to a valid integer value.
-
-
-** Troubleshooting
-
-*** ClassNotFoundException
-
- The <<<hadoop-openstack>>> JAR -or any dependencies- may not be on your classpath.
-
- If it is a remote MapReduce job that is failing, make sure that the JAR is
- installed on the servers in the cluster -or that the job submission process
- uploads the JAR file to the distributed cache.
-
-*** Failure to Authenticate
-
- A <<<SwiftAuthenticationFailedException>>> is thrown when the client
- cannot authenticate with the OpenStack keystone server. This could be
- because the URL in the service definition is wrong, or because
- the supplied credentials are invalid.
-
- [[1]] Check the authentication URL through <<<curl>>> or your browser
-
- [[1]] Use a Swift client such as CyberDuck to validate your credentials
-
- [[1]] If you have included a tenant ID, try leaving it out. Similarly,
- try adding it if you had not included it.
-
- [[1]] Try switching from API key authentication to password-based authentication,
- by setting the password.
-
- [[1]] Change your credentials. As with Amazon AWS clients, some credentials
- don't seem to like going over the network.
-
-*** Timeout connecting to the Swift Service
-
- This happens if the client application is running outside an OpenStack cluster,
- where it does not have access to the private hostname/IP address for filesystem
- operations. Set the <<<public>>> flag to true -but remember to set it to false
- for use in-cluster.
-
-** Warnings
-
- [[1]] Do not share your login details with anyone, which means do not log the
- details, or check the XML configuration files into any revision control system
- to which you do not have exclusive access.
-
- [[1]] Similarly, do not use your real account details in any documentation *or any
- bug reports submitted online*
-
- [[1]] Prefer the apikey authentication over passwords as it is easier
- to revoke a key -and some service providers allow you to set
- an automatic expiry date on a key when issued.
-
- [[1]] Do not use the public service endpoint from within a public OpenStack
- cluster, as it will run up large bills.
-
- [[1]] Remember: it's not a real filesystem or hierarchical directory structure.
- Some operations (directory rename and delete) take time and are not atomic or
- isolated from other operations taking place.
-
- [[1]] Append is not supported.
-
- [[1]] Unix-style permissions are not supported. All accounts with write access to
- a repository have unlimited access; the same goes for those with read access.
-
- [[1]] In the public clouds, do not make the containers public unless you are happy
- with anyone reading your data, and are prepared to pay the costs of their
- downloads.
-
-** Limits
-
- * Maximum length of an object path: 1024 characters
-
- * Maximum size of a binary object: no absolute limit. Files > 5GB are
- partitioned into separate files in the native filesystem, and merged during
- retrieval. <Warning:> the partitioned/large file support is the
- most complex part of the Hadoop/Swift FS integration, and, along with
- authentication, the most troublesome to support.
-
-** Testing the hadoop-openstack module
-
- The <<<hadoop-openstack>>> can be remotely tested against any public
- or private cloud infrastructure which supports the OpenStack Keystone
- authentication mechanism. It can also be tested against private
- OpenStack clusters. OpenStack Development teams are strongly encouraged to test
- the Hadoop swift filesystem client against any version of Swift that they
- are developing or deploying, to stress their cluster and to identify
- bugs early.
-
- The module comes with a large suite of JUnit tests -tests that are
- only executed if the source tree includes credentials to test against a
- specific cluster.
-
- After checking out the Hadoop source tree, create the file:
-
-+--
- hadoop-tools/hadoop-openstack/src/test/resources/auth-keys.xml
-+--
-
- Into this file, insert the credentials needed to bond to the test filesystem,
- as decribed above.
-
- Next set the property <<<test.fs.swift.name>>> to the URL of a
- swift container to test against. The tests expect exclusive access
- to this container -do not keep any other data on it, or expect it
- to be preserved.
-
-+--
- <property>
- <name>test.fs.swift.name</name>
- <value>swift://test.myswift/</value>
- </property>
-+--
-
- In the base hadoop directory, run:
-
-+--
- mvn clean install -DskipTests
-+--
-
- This builds a set of Hadoop JARs consistent with the <<<hadoop-openstack>>>
- module that is about to be tested.
-
- In the <<<hadoop-tools/hadoop-openstack>>> directory run
-
-+--
- mvn test -Dtest=TestSwiftRestClient
-+--
-
- This runs some simple tests which include authenticating
- against the remote swift service. If these tests fail, so will all
- the rest. If it does fail: check your authentication.
-
- Once this test succeeds, you can run the full test suite
-
-+--
- mvn test
-+--
-
- Be advised that these tests can take an hour or more, especially against a
- remote Swift service -or one that throttles bulk operations.
-
- Once the <<<auth-keys.xml>>> file is in place, the <<<mvn test>>> runs from
- the Hadoop source base directory will automatically run these OpenStack tests
- While this ensures that no regressions have occurred, it can also add significant
- time to test runs, and may run up bills, depending on who is providing\
- the Swift storage service. We recommend having a separate source tree
- set up purely for the Swift tests, and running it manually or by the CI tooling
- at a lower frequency than normal test runs.
-
- Finally: Apache Hadoop is an open source project. Contributions of code
- -including more tests- are very welcome.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-tools/hadoop-openstack/src/site/markdown/index.md
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-openstack/src/site/markdown/index.md b/hadoop-tools/hadoop-openstack/src/site/markdown/index.md
new file mode 100644
index 0000000..0eeb274
--- /dev/null
+++ b/hadoop-tools/hadoop-openstack/src/site/markdown/index.md
@@ -0,0 +1,544 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+* [Hadoop OpenStack Support: Swift Object Store](#Hadoop_OpenStack_Support:_Swift_Object_Store)
+ * [Introduction](#Introduction)
+ * [Features](#Features)
+ * [Using the Hadoop Swift Filesystem Client](#Using_the_Hadoop_Swift_Filesystem_Client)
+ * [Concepts: services and containers](#Concepts:_services_and_containers)
+ * [Containers and Objects](#Containers_and_Objects)
+ * [Eventual Consistency](#Eventual_Consistency)
+ * [Non-atomic "directory" operations.](#Non-atomic_directory_operations.)
+ * [Working with Swift Object Stores in Hadoop](#Working_with_Swift_Object_Stores_in_Hadoop)
+ * [Swift Filesystem URIs](#Swift_Filesystem_URIs)
+ * [Installing](#Installing)
+ * [Configuring](#Configuring)
+ * [Example: Rackspace US, in-cluster access using API key](#Example:_Rackspace_US_in-cluster_access_using_API_key)
+ * [Example: Rackspace UK: remote access with password authentication](#Example:_Rackspace_UK:_remote_access_with_password_authentication)
+ * [Example: HP cloud service definition](#Example:_HP_cloud_service_definition)
+ * [General Swift Filesystem configuration options](#General_Swift_Filesystem_configuration_options)
+ * [Blocksize fs.swift.blocksize](#Blocksize_fs.swift.blocksize)
+ * [Partition size fs.swift.partsize](#Partition_size_fs.swift.partsize)
+ * [Request size fs.swift.requestsize](#Request_size_fs.swift.requestsize)
+ * [Connection timeout fs.swift.connect.timeout](#Connection_timeout_fs.swift.connect.timeout)
+ * [Connection timeout fs.swift.socket.timeout](#Connection_timeout_fs.swift.socket.timeout)
+ * [Connection Retry Count fs.swift.connect.retry.count](#Connection_Retry_Count_fs.swift.connect.retry.count)
+ * [Connection Throttle Delay fs.swift.connect.throttle.delay](#Connection_Throttle_Delay_fs.swift.connect.throttle.delay)
+ * [HTTP Proxy](#HTTP_Proxy)
+ * [Troubleshooting](#Troubleshooting)
+ * [ClassNotFoundException](#ClassNotFoundException)
+ * [Failure to Authenticate](#Failure_to_Authenticate)
+ * [Timeout connecting to the Swift Service](#Timeout_connecting_to_the_Swift_Service)
+ * [Warnings](#Warnings)
+ * [Limits](#Limits)
+ * [Testing the hadoop-openstack module](#Testing_the_hadoop-openstack_module)
+
+Hadoop OpenStack Support: Swift Object Store
+============================================
+
+Introduction
+------------
+
+[OpenStack](http://www.openstack.org/) is an open source cloud infrastructure which can be accessed from multiple public IaaS providers, and deployed privately. It offers infrastructure services such as VM hosting (Nova), authentication (Keystone) and storage of binary objects (Swift).
+
+This module enables Apache Hadoop applications -including MapReduce jobs, read and write data to and from instances of the [OpenStack Swift object store](http://www.openstack.org/software/openstack-storage/).
+
+Features
+--------
+
+* Read and write of data stored in a Swift object store
+
+* Support of a pseudo-hierachical file system (directories, subdirectories and
+ files)
+
+* Standard filesystem operations: `create`, `delete`, `mkdir`,
+ `ls`, `mv`, `stat`.
+
+* Can act as a source of data in a MapReduce job, or a sink.
+
+* Support for multiple OpenStack services, and multiple containers from a
+ single service.
+
+* Supports in-cluster and remote access to Swift data.
+
+* Supports OpenStack Keystone authentication with password or token.
+
+* Released under the Apache Software License
+
+* Tested against the Hadoop 3.x and 1.x branches, against multiple public
+ OpenStack clusters: Rackspace US, Rackspace UK, HP Cloud.
+
+* Tested against private OpenStack clusters, including scalability tests of
+ large file uploads.
+
+Using the Hadoop Swift Filesystem Client
+----------------------------------------
+
+### Concepts: services and containers
+
+OpenStack swift is an *Object Store*; also known as a *blobstore*. It stores arbitrary binary objects by name in a *container*.
+
+The Hadoop Swift filesystem library adds another concept, the *service*, which defines which Swift blobstore hosts a container -and how to connect to it.
+
+### Containers and Objects
+
+* Containers are created by users with accounts on the Swift filestore, and hold
+ *objects*.
+
+* Objects can be zero bytes long, or they can contain data.
+
+* Objects in the container can be up to 5GB; there is a special support for
+ larger files than this, which merges multiple objects in to one.
+
+* Each object is referenced by it's *name*; there is no notion of directories.
+
+* You can use any characters in an object name that can be 'URL-encoded'; the
+ maximum length of a name is 1034 characters -after URL encoding.
+
+* Names can have `/` characters in them, which are used to create the illusion of
+ a directory structure. For example `dir/dir2/name`. Even though this looks
+ like a directory, *it is still just a name*. There is no requirement to have
+ any entries in the container called `dir` or `dir/dir2`
+
+* That said. if the container has zero-byte objects that look like directory
+ names above other objects, they can pretend to be directories. Continuing the
+ example, a 0-byte object called `dir` would tell clients that it is a
+ directory while `dir/dir2` or `dir/dir2/name` were present. This creates an
+ illusion of containers holding a filesystem.
+
+Client applications talk to Swift over HTTP or HTTPS, reading, writing and deleting objects using standard HTTP operations (GET, PUT and DELETE, respectively). There is also a COPY operation, that creates a new object in the container, with a new name, containing the old data. There is no rename operation itself, objects need to be copied -then the original entry deleted.
+
+### Eventual Consistency
+
+The Swift Filesystem is \*eventually consistent\*: an operation on an object may not be immediately visible to that client, or other clients. This is a consequence of the goal of the filesystem: to span a set of machines, across multiple datacenters, in such a way that the data can still be available when many of them fail. (In contrast, the Hadoop HDFS filesystem is \*immediately consistent\*, but it does not span datacenters.)
+
+Eventual consistency can cause surprises for client applications that expect immediate consistency: after an object is deleted or overwritten, the object may still be visible -or the old data still retrievable. The Swift Filesystem client for Apache Hadoop attempts to handle this, in conjunction with the MapReduce engine, but there may be still be occasions when eventual consistency causes surprises.
+
+### Non-atomic "directory" operations.
+
+Hadoop expects some operations to be atomic, especially `rename()`, which is something the MapReduce layer relies on to commit the output of a job, renaming data from a temp directory to the final path. Because a rename is implemented as a copy of every blob under the directory's path, followed by a delete of the originals, the intermediate state of the operation will be visible to other clients. If two Reducer tasks to rename their temp directory to the final path, both operations may succeed, with the result that output directory contains mixed data. This can happen if MapReduce jobs are being run with *speculation* enabled and Swift used as the direct output of the MR job (it can also happen against Amazon S3).
+
+Other consequences of the non-atomic operations are:
+
+1. If a program is looking for the presence of the directory before acting
+ on the data -it may start prematurely. This can be avoided by using
+ other mechanisms to co-ordinate the programs, such as the presence of a file
+ that is written *after* any bulk directory operations.
+
+2. A `rename()` or `delete()` operation may include files added under
+ the source directory tree during the operation, may unintentionally delete
+ it, or delete the 0-byte swift entries that mimic directories and act
+ as parents for the files. Try to avoid doing this.
+
+The best ways to avoid all these problems is not using Swift as the filesystem between MapReduce jobs or other Hadoop workflows. It can act as a source of data, and a final destination, but it doesn't meet all of Hadoop's expectations of what a filesystem is -it's a *blobstore*.
+
+Working with Swift Object Stores in Hadoop
+------------------------------------------
+
+Once installed, the Swift FileSystem client can be used by any Hadoop application to read from or write to data stored in a Swift container.
+
+Data stored in Swift can be used as the direct input to a MapReduce job -simply use the `swift:` URL (see below) to declare the source of the data.
+
+This Swift Filesystem client is designed to work with multiple Swift object stores, both public and private. This allows the client to work with different clusters, reading and writing data to and from either of them.
+
+It can also work with the same object stores using multiple login details.
+
+These features are achieved by one basic concept: using a service name in the URI referring to a swift filesystem, and looking up all the connection and login details for that specific service. Different service names can be defined in the Hadoop XML configuration file, so defining different clusters, or providing different login details for the same object store(s).
+
+### Swift Filesystem URIs
+
+Hadoop uses URIs to refer to files within a filesystem. Some common examples are:
+
+ local://etc/hosts
+ hdfs://cluster1/users/example/data/set1
+ hdfs://cluster2.example.org:8020/users/example/data/set1
+
+The Swift Filesystem Client adds a new URL type `swift`. In a Swift Filesystem URL, the hostname part of a URL identifies the container and the service to work with; the path the name of the object. Here are some examples
+
+ swift://container.rackspace/my-object.csv
+ swift://data.hpcloud/data/set1
+ swift://dmitry.privatecloud/out/results
+
+In the last two examples, the paths look like directories: it is not, they are simply the objects named `data/set1` and `out/results` respectively.
+
+### Installing
+
+The `hadoop-openstack` JAR must be on the classpath of the Hadoop program trying to talk to the Swift service. If installed in the classpath of the Hadoop MapReduce service, then all programs started by the MR engine will pick up the JAR automatically. This is the easiest way to give all Hadoop jobs access to Swift.
+
+Alternatively, the JAR can be included as one of the JAR files that an application uses. This lets the Hadoop jobs work with a Swift object store even if the Hadoop cluster is not pre-configured for this.
+
+The library also depends upon the Apache HttpComponents library, which must also be on the classpath.
+
+### Configuring
+
+To talk to a swift service, the user must must provide:
+
+1. The URL defining the container and the service.
+
+2. In the cluster/job configuration, the login details of that service.
+
+Multiple service definitions can co-exist in the same configuration file: just use different names for them.
+
+#### Example: Rackspace US, in-cluster access using API key
+
+This service definition is for use in a Hadoop cluster deployed within Rackspace's US infrastructure.
+
+ <property>
+ <name>fs.swift.service.rackspace.auth.url</name>
+ <value>https://auth.api.rackspacecloud.com/v2.0/tokens</value>
+ <description>Rackspace US (multiregion)</description>
+ </property>
+
+ <property>
+ <name>fs.swift.service.rackspace.username</name>
+ <value>user4</value>
+ </property>
+
+ <property>
+ <name>fs.swift.service.rackspace.region</name>
+ <value>DFW</value>
+ </property>
+
+ <property>
+ <name>fs.swift.service.rackspace.apikey</name>
+ <value>fe806aa86dfffe2f6ed8</value>
+ </property>
+
+Here the API key visible in the account settings API keys page is used to log in. No property for public/private access -the default is to use the private endpoint for Swift operations.
+
+This configuration also selects one of the regions, DFW, for its data.
+
+A reference to this service would use the `rackspace` service name:
+
+ swift://hadoop-container.rackspace/
+
+#### Example: Rackspace UK: remote access with password authentication
+
+This connects to Rackspace's UK ("LON") datacenter.
+
+ <property>
+ <name>fs.swift.service.rackspaceuk.auth.url</name>
+ <value>https://lon.identity.api.rackspacecloud.com/v2.0/tokens</value>
+ <description>Rackspace UK</description>
+ </property>
+
+ <property>
+ <name>fs.swift.service.rackspaceuk.username</name>
+ <value>user4</value>
+ </property>
+
+ <property>
+ <name>fs.swift.service.rackspaceuk.password</name>
+ <value>insert-password-here/value>
+ </property>
+
+ <property>
+ <name>fs.swift.service.rackspace.public</name>
+ <value>true</value>
+ </property>
+
+This is a public access point connection, using a password over an API key.
+
+A reference to this service would use the `rackspaceuk` service name:
+
+ swift://hadoop-container.rackspaceuk/
+
+Because the public endpoint is used, if this service definition is used within the London datacenter, all accesses will be billed at the public upload/download rates, *irrespective of where the Hadoop cluster is*.
+
+#### Example: HP cloud service definition
+
+Here is an example that connects to the HP Cloud object store.
+
+ <property>
+ <name>fs.swift.service.hpcloud.auth.url</name>
+ <value>https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0/tokens
+ </value>
+ <description>HP Cloud</description>
+ </property>
+
+ <property>
+ <name>fs.swift.service.hpcloud.tenant</name>
+ <value>FE806AA86</value>
+ </property>
+
+ <property>
+ <name>fs.swift.service.hpcloud.username</name>
+ <value>FE806AA86DFFFE2F6ED8</value>
+ </property>
+
+ <property>
+ <name>fs.swift.service.hpcloud.password</name>
+ <value>secret-password-goes-here</value>
+ </property>
+
+ <property>
+ <name>fs.swift.service.hpcloud.public</name>
+ <value>true</value>
+ </property>
+
+A reference to this service would use the `hpcloud` service name:
+
+ swift://hadoop-container.hpcloud/
+
+### General Swift Filesystem configuration options
+
+Some configuration options apply to the Swift client, independent of the specific Swift filesystem chosen.
+
+#### Blocksize fs.swift.blocksize
+
+Swift does not break up files into blocks, except in the special case of files over 5GB in length. Accordingly, there isn't a notion of a "block size" to define where the data is kept.
+
+Hadoop's MapReduce layer depends on files declaring their block size, so that it knows how to partition work. Too small a blocksize means that many mappers work on small pieces of data; too large a block size means that only a few mappers get started.
+
+The block size value reported by Swift, therefore, controls the basic workload partioning of the MapReduce engine -and can be an important parameter to tune for performance of the cluster.
+
+The property has a unit of kilobytes; the default value is `32*1024`: 32 MB
+
+ <property>
+ <name>fs.swift.blocksize</name>
+ <value>32768</value>
+ </property>
+
+This blocksize has no influence on how files are stored in Swift; it only controls what the reported size of blocks are - a value used in Hadoop MapReduce to divide work.
+
+Note that the MapReduce engine's split logic can be tuned independently by setting the `mapred.min.split.size` and `mapred.max.split.size` properties, which can be done in specific job configurations.
+
+ <property>
+ <name>mapred.min.split.size</name>
+ <value>524288</value>
+ </property>
+
+ <property>
+ <name>mapred.max.split.size</name>
+ <value>1048576</value>
+ </property>
+
+In an Apache Pig script, these properties would be set as:
+
+ mapred.min.split.size 524288
+ mapred.max.split.size 1048576
+
+#### Partition size fs.swift.partsize
+
+The Swift filesystem client breaks very large files into partitioned files, uploading each as it progresses, and writing any remaning data and an XML manifest when a partitioned file is closed.
+
+The partition size defaults to 4608 MB; 4.5GB, the maximum filesize that Swift can support.
+
+It is possible to set a smaller partition size, in the `fs.swift.partsize` option. This takes a value in KB.
+
+ <property>
+ <name>fs.swift.partsize</name>
+ <value>1024</value>
+ <description>upload every MB</description>
+ </property>
+
+When should this value be changed from its default?
+
+While there is no need to ever change it for basic operation of the Swift filesystem client, it can be tuned
+
+* If a Swift filesystem is location aware, then breaking a file up into
+ smaller partitions scatters the data round the cluster. For best performance,
+ the property `fs.swift.blocksize` should be set to a smaller value than the
+ partition size of files.
+
+* When writing to an unpartitioned file, the entire write is done in the
+ `close()` operation. When a file is partitioned, the outstanding data to
+ be written whenever the outstanding amount of data is greater than the
+ partition size. This means that data will be written more incrementally
+
+#### Request size fs.swift.requestsize
+
+The Swift filesystem client reads files in HTTP GET operations, asking for a block of data at a time.
+
+The default value is 64KB. A larger value may be more efficient over faster networks, as it reduces the overhead of setting up the HTTP operation.
+
+However, if the file is read with many random accesses, requests for data will be made from different parts of the file -discarding some of the previously requested data. The benefits of larger request sizes may be wasted.
+
+The property `fs.swift.requestsize` sets the request size in KB.
+
+ <property>
+ <name>fs.swift.requestsize</name>
+ <value>128</value>
+ </property>
+
+#### Connection timeout fs.swift.connect.timeout
+
+This sets the timeout in milliseconds to connect to a Swift service.
+
+ <property>
+ <name>fs.swift.connect.timeout</name>
+ <value>15000</value>
+ </property>
+
+A shorter timeout means that connection failures are raised faster -but may trigger more false alarms. A longer timeout is more resilient to network problems -and may be needed when talking to remote filesystems.
+
+#### Connection timeout fs.swift.socket.timeout
+
+This sets the timeout in milliseconds to wait for data from a connected socket.
+
+ <property>
+ <name>fs.swift.socket.timeout</name>
+ <value>60000</value>
+ </property>
+
+A shorter timeout means that connection failures are raised faster -but may trigger more false alarms. A longer timeout is more resilient to network problems -and may be needed when talking to remote filesystems.
+
+#### Connection Retry Count fs.swift.connect.retry.count
+
+This sets the number of times to try to connect to a service whenever an HTTP request is made.
+
+ <property>
+ <name>fs.swift.connect.retry.count</name>
+ <value>3</value>
+ </property>
+
+The more retries, the more resilient it is to transient outages -and the less rapid it is at detecting and reporting server connectivity problems.
+
+#### Connection Throttle Delay fs.swift.connect.throttle.delay
+
+This property adds a delay between bulk file copy and delete operations, to prevent requests being throttled or blocked by the remote service
+
+ <property>
+ <name>fs.swift.connect.throttle.delay</name>
+ <value>0</value>
+ </property>
+
+It is measured in milliseconds; "0" means do not add any delay.
+
+Throttling is enabled on the public endpoints of some Swift services. If `rename()` or `delete()` operations fail with `SwiftThrottledRequestException` exceptions, try setting this property.
+
+#### HTTP Proxy
+
+If the client can only access the Swift filesystem via a web proxy server, the client configuration must specify the proxy via the `fs.swift.connect.proxy.host` and `fs.swift.connect.proxy.port` properties.
+
+ <property>
+ <name>fs.swift.proxy.host</name>
+ <value>web-proxy</value>
+ </property>
+
+ <property>
+ <name>fs.swift.proxy.port</name>
+ <value>8088</value>
+ </property>
+
+If the host is declared, the proxy port must be set to a valid integer value.
+
+### Troubleshooting
+
+#### ClassNotFoundException
+
+The `hadoop-openstack` JAR -or any dependencies- may not be on your classpath.
+
+If it is a remote MapReduce job that is failing, make sure that the JAR is installed on the servers in the cluster -or that the job submission process uploads the JAR file to the distributed cache.
+
+#### Failure to Authenticate
+
+A `SwiftAuthenticationFailedException` is thrown when the client cannot authenticate with the OpenStack keystone server. This could be because the URL in the service definition is wrong, or because the supplied credentials are invalid.
+
+1. Check the authentication URL through `curl` or your browser
+
+2. Use a Swift client such as CyberDuck to validate your credentials
+
+3. If you have included a tenant ID, try leaving it out. Similarly,
+ try adding it if you had not included it.
+
+4. Try switching from API key authentication to password-based authentication,
+ by setting the password.
+
+5. Change your credentials. As with Amazon AWS clients, some credentials
+ don't seem to like going over the network.
+
+#### Timeout connecting to the Swift Service
+
+This happens if the client application is running outside an OpenStack cluster, where it does not have access to the private hostname/IP address for filesystem operations. Set the `public` flag to true -but remember to set it to false for use in-cluster.
+
+### Warnings
+
+1. Do not share your login details with anyone, which means do not log the
+ details, or check the XML configuration files into any revision control system
+ to which you do not have exclusive access.
+
+2. Similarly, do not use your real account details in any
+ documentation \*or any bug reports submitted online\*
+
+3. Prefer the apikey authentication over passwords as it is easier
+ to revoke a key -and some service providers allow you to set
+ an automatic expiry date on a key when issued.
+
+4. Do not use the public service endpoint from within a public OpenStack
+ cluster, as it will run up large bills.
+
+5. Remember: it's not a real filesystem or hierarchical directory structure.
+ Some operations (directory rename and delete) take time and are not atomic or
+ isolated from other operations taking place.
+
+6. Append is not supported.
+
+7. Unix-style permissions are not supported. All accounts with write access to
+ a repository have unlimited access; the same goes for those with read access.
+
+8. In the public clouds, do not make the containers public unless you are happy
+ with anyone reading your data, and are prepared to pay the costs of their
+ downloads.
+
+### Limits
+
+* Maximum length of an object path: 1024 characters
+
+* Maximum size of a binary object: no absolute limit. Files \> 5GB are
+ partitioned into separate files in the native filesystem, and merged during
+ retrieval. *Warning:* the partitioned/large file support is the
+ most complex part of the Hadoop/Swift FS integration, and, along with
+ authentication, the most troublesome to support.
+
+### Testing the hadoop-openstack module
+
+The `hadoop-openstack` can be remotely tested against any public or private cloud infrastructure which supports the OpenStack Keystone authentication mechanism. It can also be tested against private OpenStack clusters. OpenStack Development teams are strongly encouraged to test the Hadoop swift filesystem client against any version of Swift that they are developing or deploying, to stress their cluster and to identify bugs early.
+
+The module comes with a large suite of JUnit tests -tests that are only executed if the source tree includes credentials to test against a specific cluster.
+
+After checking out the Hadoop source tree, create the file:
+
+ hadoop-tools/hadoop-openstack/src/test/resources/auth-keys.xml
+
+Into this file, insert the credentials needed to bond to the test filesystem, as decribed above.
+
+Next set the property `test.fs.swift.name` to the URL of a swift container to test against. The tests expect exclusive access to this container -do not keep any other data on it, or expect it to be preserved.
+
+ <property>
+ <name>test.fs.swift.name</name>
+ <value>swift://test.myswift/</value>
+ </property>
+
+In the base hadoop directory, run:
+
+ mvn clean install -DskipTests
+
+This builds a set of Hadoop JARs consistent with the `hadoop-openstack` module that is about to be tested.
+
+In the `hadoop-tools/hadoop-openstack` directory run
+
+ mvn test -Dtest=TestSwiftRestClient
+
+This runs some simple tests which include authenticating against the remote swift service. If these tests fail, so will all the rest. If it does fail: check your authentication.
+
+Once this test succeeds, you can run the full test suite
+
+ mvn test
+
+Be advised that these tests can take an hour or more, especially against a remote Swift service -or one that throttles bulk operations.
+
+Once the `auth-keys.xml` file is in place, the `mvn test` runs from the Hadoop source base directory will automatically run these OpenStack tests While this ensures that no regressions have occurred, it can also add significant time to test runs, and may run up bills, depending on who is providingthe Swift storage service. We recommend having a separate source tree set up purely for the Swift tests, and running it manually or by the CI tooling at a lower frequency than normal test runs.
+
+Finally: Apache Hadoop is an open source project. Contributions of code -including more tests- are very welcome.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-tools/hadoop-openstack/src/site/resources/css/site.css
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-openstack/src/site/resources/css/site.css b/hadoop-tools/hadoop-openstack/src/site/resources/css/site.css
new file mode 100644
index 0000000..f830baa
--- /dev/null
+++ b/hadoop-tools/hadoop-openstack/src/site/resources/css/site.css
@@ -0,0 +1,30 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+#banner {
+ height: 93px;
+ background: none;
+}
+
+#bannerLeft img {
+ margin-left: 30px;
+ margin-top: 10px;
+}
+
+#bannerRight img {
+ margin: 17px;
+}
+
[17/52] [abbrv] hadoop git commit: HADOOP-11602. Fix
toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa)
Posted by zh...@apache.org.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
index aad63d3..ca204a6 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
@@ -23,6 +23,7 @@ import java.util.Collections;
import java.util.EnumSet;
import java.util.HashMap;
import java.util.Iterator;
+import java.util.Locale;
import java.util.Map;
import java.util.Timer;
import java.util.TimerTask;
@@ -711,7 +712,8 @@ public class JobHistoryEventHandler extends AbstractService
private void processEventForTimelineServer(HistoryEvent event, JobId jobId,
long timestamp) {
TimelineEvent tEvent = new TimelineEvent();
- tEvent.setEventType(event.getEventType().name().toUpperCase());
+ tEvent.setEventType(
+ event.getEventType().name().toUpperCase(Locale.ENGLISH));
tEvent.setTimestamp(timestamp);
TimelineEntity tEntity = new TimelineEntity();
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AppController.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AppController.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AppController.java
index 53f21db..440ff49 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AppController.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AppController.java
@@ -227,7 +227,7 @@ public class AppController extends Controller implements AMParams {
try {
String tt = $(TASK_TYPE);
tt = tt.isEmpty() ? "All" : StringUtils.capitalize(MRApps.taskType(tt).
- toString().toLowerCase(Locale.US));
+ toString().toLowerCase(Locale.ENGLISH));
setTitle(join(tt, " Tasks for ", $(JOB_ID)));
} catch (Exception e) {
LOG.error("Failed to render tasks page with task type : "
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
index 553ba70..e0c4773 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
@@ -21,6 +21,7 @@ package org.apache.hadoop.mapreduce;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
+import java.util.Locale;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapred.JobPriority;
@@ -314,7 +315,7 @@ public class TypeConverter {
QueueState state) {
org.apache.hadoop.mapreduce.QueueState qState =
org.apache.hadoop.mapreduce.QueueState.getState(
- state.toString().toLowerCase());
+ state.toString().toLowerCase(Locale.ENGLISH));
return qState;
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
index 08b44f8..37cfb7a 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
@@ -30,6 +30,7 @@ import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import com.google.common.annotations.VisibleForTesting;
@@ -303,7 +304,7 @@ public class MRApps extends Apps {
remoteFS.getWorkingDirectory()));
String name = (null == u.getFragment())
? p.getName() : u.getFragment();
- if (!name.toLowerCase().endsWith(".jar")) {
+ if (!name.toLowerCase(Locale.ENGLISH).endsWith(".jar")) {
linkLookup.put(p, name);
}
}
@@ -317,7 +318,7 @@ public class MRApps extends Apps {
if (name == null) {
name = p.getName();
}
- if(!name.toLowerCase().endsWith(".jar")) {
+ if(!name.toLowerCase(Locale.ENGLISH).endsWith(".jar")) {
MRApps.addToEnvironment(
environment,
classpathEnvVar,
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
index cc42b9c..d360811 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
@@ -22,6 +22,7 @@ import static org.mockito.Mockito.when;
import java.util.ArrayList;
import java.util.List;
+import java.util.Locale;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.JobStatus.State;
@@ -153,7 +154,8 @@ public class TestTypeConverter {
org.apache.hadoop.mapreduce.QueueInfo returned =
TypeConverter.fromYarn(queueInfo, new Configuration());
Assert.assertEquals("queueInfo translation didn't work.",
- returned.getState().toString(), queueInfo.getQueueState().toString().toLowerCase());
+ returned.getState().toString(),
+ queueInfo.getQueueState().toString().toLowerCase(Locale.ENGLISH));
}
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
index 5274438..73dab4f 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
@@ -28,6 +28,7 @@ import java.util.ArrayList;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.NoSuchElementException;
import java.util.concurrent.atomic.AtomicBoolean;
@@ -116,7 +117,7 @@ abstract public class Task implements Writable, Configurable {
* BYTES_READ counter and second one is of the BYTES_WRITTEN counter.
*/
protected static String[] getFileSystemCounterNames(String uriScheme) {
- String scheme = uriScheme.toUpperCase();
+ String scheme = uriScheme.toUpperCase(Locale.ENGLISH);
return new String[]{scheme+"_BYTES_READ", scheme+"_BYTES_WRITTEN"};
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
index a53b76a..e8d5d1d 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
@@ -227,7 +227,7 @@ public abstract class FileSystemCounterGroup<C extends Counter>
}
private String checkScheme(String scheme) {
- String fixed = scheme.toUpperCase(Locale.US);
+ String fixed = scheme.toUpperCase(Locale.ENGLISH);
String interned = schemes.putIfAbsent(fixed, fixed);
if (schemes.size() > MAX_NUM_SCHEMES) {
// mistakes or abuses
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/DistributedCache.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/DistributedCache.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/DistributedCache.java
index eaa5af8..ef9b0a4 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/DistributedCache.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/DistributedCache.java
@@ -473,7 +473,7 @@ public class DistributedCache {
if (fragment == null) {
return false;
}
- String lowerCaseFragment = fragment.toLowerCase();
+ String lowerCaseFragment = fragment.toLowerCase(Locale.ENGLISH);
if (fragments.contains(lowerCaseFragment)) {
return false;
}
@@ -488,7 +488,7 @@ public class DistributedCache {
if (fragment == null) {
return false;
}
- String lowerCaseFragment = fragment.toLowerCase();
+ String lowerCaseFragment = fragment.toLowerCase(Locale.ENGLISH);
if (fragments.contains(lowerCaseFragment)) {
return false;
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/db/DBInputFormat.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/db/DBInputFormat.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/db/DBInputFormat.java
index 00fbeda..7faa736 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/db/DBInputFormat.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/db/DBInputFormat.java
@@ -29,6 +29,7 @@ import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.List;
+import java.util.Locale;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
@@ -162,7 +163,8 @@ public class DBInputFormat<T extends DBWritable>
this.connection = createConnection();
DatabaseMetaData dbMeta = connection.getMetaData();
- this.dbProductName = dbMeta.getDatabaseProductName().toUpperCase();
+ this.dbProductName =
+ dbMeta.getDatabaseProductName().toUpperCase(Locale.ENGLISH);
}
catch (Exception ex) {
throw new RuntimeException(ex);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
index 37ba5b7..6af815b 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
@@ -22,6 +22,7 @@ import java.io.OutputStreamWriter;
import java.io.PrintWriter;
import java.util.ArrayList;
import java.util.List;
+import java.util.Locale;
import java.util.Set;
import java.util.HashSet;
import java.util.Arrays;
@@ -222,12 +223,12 @@ public class CLI extends Configured implements Tool {
taskType = argv[2];
taskState = argv[3];
displayTasks = true;
- if (!taskTypes.contains(taskType.toUpperCase())) {
+ if (!taskTypes.contains(taskType.toUpperCase(Locale.ENGLISH))) {
System.out.println("Error: Invalid task-type: " + taskType);
displayUsage(cmd);
return exitCode;
}
- if (!taskStates.contains(taskState.toLowerCase())) {
+ if (!taskStates.contains(taskState.toLowerCase(Locale.ENGLISH))) {
System.out.println("Error: Invalid task-state: " + taskState);
displayUsage(cmd);
return exitCode;
@@ -593,7 +594,8 @@ public class CLI extends Configured implements Tool {
throws IOException, InterruptedException {
TaskReport[] reports=null;
- reports = job.getTaskReports(TaskType.valueOf(type.toUpperCase()));
+ reports = job.getTaskReports(
+ TaskType.valueOf(type.toUpperCase(Locale.ENGLISH)));
for (TaskReport report : reports) {
TIPStatus status = report.getCurrentStatus();
if ((state.equalsIgnoreCase("pending") && status ==TIPStatus.PENDING) ||
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java
index d9cd07b..9420497 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java
@@ -29,6 +29,7 @@ import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.PrintStream;
import java.util.Date;
+import java.util.Locale;
import java.util.Random;
import java.util.StringTokenizer;
import org.apache.commons.logging.Log;
@@ -154,16 +155,16 @@ public class TestDFSIO implements Tool {
static ByteMultiple parseString(String sMultiple) {
if(sMultiple == null || sMultiple.isEmpty()) // MB by default
return MB;
- String sMU = sMultiple.toUpperCase();
- if(B.name().toUpperCase().endsWith(sMU))
+ String sMU = sMultiple.toUpperCase(Locale.ENGLISH);
+ if(B.name().toUpperCase(Locale.ENGLISH).endsWith(sMU))
return B;
- if(KB.name().toUpperCase().endsWith(sMU))
+ if(KB.name().toUpperCase(Locale.ENGLISH).endsWith(sMU))
return KB;
- if(MB.name().toUpperCase().endsWith(sMU))
+ if(MB.name().toUpperCase(Locale.ENGLISH).endsWith(sMU))
return MB;
- if(GB.name().toUpperCase().endsWith(sMU))
+ if(GB.name().toUpperCase(Locale.ENGLISH).endsWith(sMU))
return GB;
- if(TB.name().toUpperCase().endsWith(sMU))
+ if(TB.name().toUpperCase(Locale.ENGLISH).endsWith(sMU))
return TB;
throw new IllegalArgumentException("Unsupported ByteMultiple "+sMultiple);
}
@@ -736,7 +737,7 @@ public class TestDFSIO implements Tool {
}
for (int i = 0; i < args.length; i++) { // parse command line
- if (args[i].toLowerCase().startsWith("-read")) {
+ if (args[i].toLowerCase(Locale.ENGLISH).startsWith("-read")) {
testType = TestType.TEST_TYPE_READ;
} else if (args[i].equalsIgnoreCase("-write")) {
testType = TestType.TEST_TYPE_WRITE;
@@ -755,9 +756,10 @@ public class TestDFSIO implements Tool {
testType = TestType.TEST_TYPE_TRUNCATE;
} else if (args[i].equalsIgnoreCase("-clean")) {
testType = TestType.TEST_TYPE_CLEANUP;
- } else if (args[i].toLowerCase().startsWith("-seq")) {
+ } else if (args[i].toLowerCase(Locale.ENGLISH).startsWith("-seq")) {
isSequential = true;
- } else if (args[i].toLowerCase().startsWith("-compression")) {
+ } else if (
+ args[i].toLowerCase(Locale.ENGLISH).startsWith("-compression")) {
compressionClass = args[++i];
} else if (args[i].equalsIgnoreCase("-nrfiles")) {
nrFiles = Integer.parseInt(args[++i]);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestFileSystem.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestFileSystem.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestFileSystem.java
index 13e27cd..e2b8985 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestFileSystem.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestFileSystem.java
@@ -24,6 +24,7 @@ import java.io.OutputStream;
import java.security.PrivilegedExceptionAction;
import java.util.Arrays;
import java.util.Collections;
+import java.util.Locale;
import java.util.Random;
import java.util.List;
import java.util.ArrayList;
@@ -556,7 +557,10 @@ public class TestFileSystem extends TestCase {
static void checkPath(MiniDFSCluster cluster, FileSystem fileSys) throws IOException {
InetSocketAddress add = cluster.getNameNode().getNameNodeAddress();
// Test upper/lower case
- fileSys.checkPath(new Path("hdfs://" + add.getHostName().toUpperCase() + ":" + add.getPort()));
+ fileSys.checkPath(
+ new Path("hdfs://"
+ + add.getHostName().toUpperCase(Locale.ENGLISH)
+ + ":" + add.getPort()));
}
public void testFsClose() throws Exception {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/Constants.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/Constants.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/Constants.java
index 6f4f442..adf5270 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/Constants.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/Constants.java
@@ -18,6 +18,8 @@
package org.apache.hadoop.fs.slive;
+import java.util.Locale;
+
/**
* Constants used in various places in slive
*/
@@ -35,7 +37,7 @@ class Constants {
enum Distribution {
BEG, END, UNIFORM, MID;
String lowerName() {
- return this.name().toLowerCase();
+ return this.name().toLowerCase(Locale.ENGLISH);
}
}
@@ -45,7 +47,7 @@ class Constants {
enum OperationType {
READ, APPEND, RENAME, LS, MKDIR, DELETE, CREATE;
String lowerName() {
- return this.name().toLowerCase();
+ return this.name().toLowerCase(Locale.ENGLISH);
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/OperationData.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/OperationData.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/OperationData.java
index b4c98f7..462db86 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/OperationData.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/OperationData.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.fs.slive;
+import java.util.Locale;
import org.apache.hadoop.fs.slive.Constants.Distribution;
/**
@@ -52,7 +53,8 @@ class OperationData {
percent = (Double.parseDouble(pieces[0]) / 100.0d);
} else if (pieces.length >= 2) {
percent = (Double.parseDouble(pieces[0]) / 100.0d);
- distribution = Distribution.valueOf(pieces[1].toUpperCase());
+ distribution =
+ Distribution.valueOf(pieces[1].toUpperCase(Locale.ENGLISH));
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/OperationOutput.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/OperationOutput.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/OperationOutput.java
index 57ef017..d60a607 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/OperationOutput.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/OperationOutput.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.fs.slive;
+import java.util.Locale;
import org.apache.hadoop.io.Text;
/**
@@ -67,7 +68,8 @@ class OperationOutput {
"Invalid key format - no type seperator - " + TYPE_SEP);
}
try {
- dataType = OutputType.valueOf(key.substring(0, place).toUpperCase());
+ dataType = OutputType.valueOf(
+ key.substring(0, place).toUpperCase(Locale.ENGLISH));
} catch (Exception e) {
throw new IllegalArgumentException(
"Invalid key format - invalid output type", e);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/SliveTest.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/SliveTest.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/SliveTest.java
index ce1837f..443089d 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/SliveTest.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/SliveTest.java
@@ -27,6 +27,7 @@ import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.util.ArrayList;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.TreeMap;
@@ -157,7 +158,7 @@ public class SliveTest implements Tool {
if (val == null) {
return false;
}
- String cleanupOpt = val.toLowerCase().trim();
+ String cleanupOpt = val.toLowerCase(Locale.ENGLISH).trim();
if (cleanupOpt.equals("true") || cleanupOpt.equals("1")) {
return true;
} else {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/io/FileBench.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/io/FileBench.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/io/FileBench.java
index f155dae..0830f37 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/io/FileBench.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/io/FileBench.java
@@ -23,6 +23,7 @@ import java.util.ArrayList;
import java.util.Date;
import java.util.EnumSet;
import java.util.HashMap;
+import java.util.Locale;
import java.util.Map;
import java.util.Random;
@@ -214,23 +215,25 @@ public class FileBench extends Configured implements Tool {
if (!(fmt == Format.txt || cod == CCodec.pln)) {
for (CType typ : ct) {
String fn =
- fmt.name().toUpperCase() + "_" +
- cod.name().toUpperCase() + "_" +
- typ.name().toUpperCase();
+ fmt.name().toUpperCase(Locale.ENGLISH) + "_" +
+ cod.name().toUpperCase(Locale.ENGLISH) + "_" +
+ typ.name().toUpperCase(Locale.ENGLISH);
typ.configure(job);
- System.out.print(rwop.name().toUpperCase() + " " + fn + ": ");
+ System.out.print(
+ rwop.name().toUpperCase(Locale.ENGLISH) + " " + fn + ": ");
System.out.println(rwop.exec(fn, job) / 1000 +
" seconds");
}
} else {
String fn =
- fmt.name().toUpperCase() + "_" +
- cod.name().toUpperCase();
+ fmt.name().toUpperCase(Locale.ENGLISH) + "_" +
+ cod.name().toUpperCase(Locale.ENGLISH);
Path p = new Path(root, fn);
if (rwop == RW.r && !fs.exists(p)) {
fn += cod.getExt();
}
- System.out.print(rwop.name().toUpperCase() + " " + fn + ": ");
+ System.out.print(
+ rwop.name().toUpperCase(Locale.ENGLISH) + " " + fn + ": ");
System.out.println(rwop.exec(fn, job) / 1000 +
" seconds");
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMapRed.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMapRed.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMapRed.java
index 02a083b..b9bbd60 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMapRed.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMapRed.java
@@ -28,6 +28,7 @@ import java.io.File;
import java.util.EnumSet;
import java.util.HashSet;
import java.util.Iterator;
+import java.util.Locale;
import java.util.Random;
import org.apache.hadoop.conf.Configuration;
@@ -280,7 +281,7 @@ public class TestMapRed extends Configured implements Tool {
public void map(WritableComparable key, Text value,
OutputCollector<Text, Text> output,
Reporter reporter) throws IOException {
- String str = value.toString().toLowerCase();
+ String str = value.toString().toLowerCase(Locale.ENGLISH);
output.collect(new Text(str), value);
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/DBCountPageView.java
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/DBCountPageView.java b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/DBCountPageView.java
index 270ddc9..4379cd5 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/DBCountPageView.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/DBCountPageView.java
@@ -27,6 +27,7 @@ import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
+import java.util.Locale;
import java.util.Random;
import org.apache.commons.logging.Log;
@@ -102,7 +103,7 @@ public class DBCountPageView extends Configured implements Tool {
private void createConnection(String driverClassName
, String url) throws Exception {
- if(driverClassName.toLowerCase().contains("oracle")) {
+ if(driverClassName.toLowerCase(Locale.ENGLISH).contains("oracle")) {
isOracle = true;
}
Class.forName(driverClassName);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/versioninfo/VersionInfoMojo.java
----------------------------------------------------------------------
diff --git a/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/versioninfo/VersionInfoMojo.java b/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/versioninfo/VersionInfoMojo.java
index f342463..f0ec59c 100644
--- a/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/versioninfo/VersionInfoMojo.java
+++ b/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/versioninfo/VersionInfoMojo.java
@@ -35,6 +35,7 @@ import java.util.Collections;
import java.util.Comparator;
import java.util.Date;
import java.util.List;
+import java.util.Locale;
import java.util.TimeZone;
/**
@@ -329,7 +330,8 @@ public class VersionInfoMojo extends AbstractMojo {
}
private String normalizePath(File file) {
- return file.getPath().toUpperCase().replaceAll("\\\\", "/");
+ return file.getPath().toUpperCase(Locale.ENGLISH)
+ .replaceAll("\\\\", "/");
}
});
byte[] md5 = computeMD5(files);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
index 2412698..bd8ca6e 100644
--- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
+++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
@@ -979,8 +979,8 @@ public class AzureNativeFileSystemStore implements NativeFileSystemStore {
private String verifyAndConvertToStandardFormat(String rawDir) throws URISyntaxException {
URI asUri = new URI(rawDir);
if (asUri.getAuthority() == null
- || asUri.getAuthority().toLowerCase(Locale.US).equalsIgnoreCase(
- sessionUri.getAuthority().toLowerCase(Locale.US))) {
+ || asUri.getAuthority().toLowerCase(Locale.ENGLISH).equalsIgnoreCase(
+ sessionUri.getAuthority().toLowerCase(Locale.ENGLISH))) {
// Applies to me.
return trim(asUri.getPath(), "/");
} else {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java
index 71e84a1..36873c7 100644
--- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java
+++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java
@@ -121,8 +121,9 @@ public class DistCpUtils {
*/
public static Class<? extends InputFormat> getStrategy(Configuration conf,
DistCpOptions options) {
- String confLabel = "distcp." +
- options.getCopyStrategy().toLowerCase(Locale.getDefault()) + ".strategy.impl";
+ String confLabel = "distcp."
+ + options.getCopyStrategy().toLowerCase(Locale.ENGLISH)
+ + ".strategy.impl";
return conf.getClass(confLabel, UniformSizeInputFormat.class, InputFormat.class);
}
@@ -221,7 +222,8 @@ public class DistCpUtils {
final boolean preserveXAttrs = attributes.contains(FileAttribute.XATTR);
if (preserveXAttrs || preserveRawXattrs) {
- final String rawNS = XAttr.NameSpace.RAW.name().toLowerCase();
+ final String rawNS =
+ XAttr.NameSpace.RAW.name().toLowerCase(Locale.ENGLISH);
Map<String, byte[]> srcXAttrs = srcFileStatus.getXAttrs();
Map<String, byte[]> targetXAttrs = getXAttrs(targetFS, path);
if (srcXAttrs != null && !srcXAttrs.equals(targetXAttrs)) {
@@ -321,7 +323,8 @@ public class DistCpUtils {
copyListingFileStatus.setXAttrs(srcXAttrs);
} else {
Map<String, byte[]> trgXAttrs = Maps.newHashMap();
- final String rawNS = XAttr.NameSpace.RAW.name().toLowerCase();
+ final String rawNS =
+ XAttr.NameSpace.RAW.name().toLowerCase(Locale.ENGLISH);
for (Map.Entry<String, byte[]> ent : srcXAttrs.entrySet()) {
final String xattrName = ent.getKey();
if (xattrName.startsWith(rawNS)) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCpV1.java
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCpV1.java b/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCpV1.java
index f46c421..1db17f7 100644
--- a/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCpV1.java
+++ b/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCpV1.java
@@ -31,6 +31,7 @@ import java.util.HashSet;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
+import java.util.Locale;
import java.util.Random;
import java.util.Stack;
import java.util.StringTokenizer;
@@ -169,7 +170,9 @@ public class DistCpV1 implements Tool {
final char symbol;
- private FileAttribute() {symbol = toString().toLowerCase().charAt(0);}
+ private FileAttribute() {
+ symbol = toString().toLowerCase(Locale.ENGLISH).charAt(0);
+ }
static EnumSet<FileAttribute> parse(String s) {
if (s == null || s.length() == 0) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/GridmixJobSubmissionPolicy.java
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/GridmixJobSubmissionPolicy.java b/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/GridmixJobSubmissionPolicy.java
index 83eb947..b010725 100644
--- a/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/GridmixJobSubmissionPolicy.java
+++ b/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/GridmixJobSubmissionPolicy.java
@@ -24,6 +24,7 @@ import org.apache.hadoop.mapred.gridmix.Statistics.JobStats;
import org.apache.hadoop.mapred.gridmix.Statistics.ClusterStats;
import java.util.concurrent.CountDownLatch;
+import java.util.Locale;
import java.io.IOException;
enum GridmixJobSubmissionPolicy {
@@ -84,6 +85,6 @@ enum GridmixJobSubmissionPolicy {
public static GridmixJobSubmissionPolicy getPolicy(
Configuration conf, GridmixJobSubmissionPolicy defaultPolicy) {
String policy = conf.get(JOB_SUBMISSION_POLICY, defaultPolicy.name());
- return valueOf(policy.toUpperCase());
+ return valueOf(policy.toUpperCase(Locale.ENGLISH));
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/HadoopLogsAnalyzer.java
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/HadoopLogsAnalyzer.java b/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/HadoopLogsAnalyzer.java
index 47fdb1a..776391d 100644
--- a/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/HadoopLogsAnalyzer.java
+++ b/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/HadoopLogsAnalyzer.java
@@ -28,6 +28,7 @@ import java.util.Arrays;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
+import java.util.Locale;
import java.util.Map;
import java.util.StringTokenizer;
import java.util.ArrayList;
@@ -319,42 +320,43 @@ public class HadoopLogsAnalyzer extends Configured implements Tool {
}
for (int i = 0; i < args.length - (inputFilename == null ? 0 : 1); ++i) {
- if ("-h".equals(args[i].toLowerCase())
- || "-help".equals(args[i].toLowerCase())) {
+ if ("-h".equals(args[i].toLowerCase(Locale.ENGLISH))
+ || "-help".equals(args[i].toLowerCase(Locale.ENGLISH))) {
usage();
return 0;
}
- if ("-c".equals(args[i].toLowerCase())
- || "-collect-prefixes".equals(args[i].toLowerCase())) {
+ if ("-c".equals(args[i].toLowerCase(Locale.ENGLISH))
+ || "-collect-prefixes".equals(args[i].toLowerCase(Locale.ENGLISH))) {
collecting = true;
continue;
}
// these control the job digest
- if ("-write-job-trace".equals(args[i].toLowerCase())) {
+ if ("-write-job-trace".equals(args[i].toLowerCase(Locale.ENGLISH))) {
++i;
jobTraceFilename = new Path(args[i]);
continue;
}
- if ("-single-line-job-traces".equals(args[i].toLowerCase())) {
+ if ("-single-line-job-traces".equals(
+ args[i].toLowerCase(Locale.ENGLISH))) {
prettyprintTrace = false;
continue;
}
- if ("-omit-task-details".equals(args[i].toLowerCase())) {
+ if ("-omit-task-details".equals(args[i].toLowerCase(Locale.ENGLISH))) {
omitTaskDetails = true;
continue;
}
- if ("-write-topology".equals(args[i].toLowerCase())) {
+ if ("-write-topology".equals(args[i].toLowerCase(Locale.ENGLISH))) {
++i;
topologyFilename = new Path(args[i]);
continue;
}
- if ("-job-digest-spectra".equals(args[i].toLowerCase())) {
+ if ("-job-digest-spectra".equals(args[i].toLowerCase(Locale.ENGLISH))) {
ArrayList<Integer> values = new ArrayList<Integer>();
++i;
@@ -384,13 +386,13 @@ public class HadoopLogsAnalyzer extends Configured implements Tool {
continue;
}
- if ("-d".equals(args[i].toLowerCase())
- || "-debug".equals(args[i].toLowerCase())) {
+ if ("-d".equals(args[i].toLowerCase(Locale.ENGLISH))
+ || "-debug".equals(args[i].toLowerCase(Locale.ENGLISH))) {
debug = true;
continue;
}
- if ("-spreads".equals(args[i].toLowerCase())) {
+ if ("-spreads".equals(args[i].toLowerCase(Locale.ENGLISH))) {
int min = Integer.parseInt(args[i + 1]);
int max = Integer.parseInt(args[i + 2]);
@@ -404,22 +406,22 @@ public class HadoopLogsAnalyzer extends Configured implements Tool {
}
// These control log-wide CDF outputs
- if ("-delays".equals(args[i].toLowerCase())) {
+ if ("-delays".equals(args[i].toLowerCase(Locale.ENGLISH))) {
delays = true;
continue;
}
- if ("-runtimes".equals(args[i].toLowerCase())) {
+ if ("-runtimes".equals(args[i].toLowerCase(Locale.ENGLISH))) {
runtimes = true;
continue;
}
- if ("-tasktimes".equals(args[i].toLowerCase())) {
+ if ("-tasktimes".equals(args[i].toLowerCase(Locale.ENGLISH))) {
collectTaskTimes = true;
continue;
}
- if ("-v1".equals(args[i].toLowerCase())) {
+ if ("-v1".equals(args[i].toLowerCase(Locale.ENGLISH))) {
version = 1;
continue;
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/JobBuilder.java
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/JobBuilder.java b/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/JobBuilder.java
index eaa9547..4b512d2 100644
--- a/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/JobBuilder.java
+++ b/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/JobBuilder.java
@@ -20,6 +20,7 @@ package org.apache.hadoop.tools.rumen;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.Properties;
import java.util.StringTokenizer;
@@ -433,7 +434,7 @@ public class JobBuilder {
return Values.SUCCESS;
}
- return Values.valueOf(name.toUpperCase());
+ return Values.valueOf(name.toUpperCase(Locale.ENGLISH));
}
private void processTaskUpdatedEvent(TaskUpdatedEvent event) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/LoggedTask.java
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/LoggedTask.java b/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/LoggedTask.java
index 903d5fb..068ac5b 100644
--- a/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/LoggedTask.java
+++ b/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/LoggedTask.java
@@ -20,6 +20,7 @@ package org.apache.hadoop.tools.rumen;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.Locale;
import java.util.Set;
import java.util.TreeSet;
@@ -243,7 +244,7 @@ public class LoggedTask implements DeepCompare {
}
private static String canonicalizeCounterName(String nonCanonicalName) {
- String result = nonCanonicalName.toLowerCase();
+ String result = nonCanonicalName.toLowerCase(Locale.ENGLISH);
result = result.replace(' ', '|');
result = result.replace('-', '|');
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java b/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java
index d1b365e..c7823c4 100644
--- a/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java
+++ b/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java
@@ -20,6 +20,7 @@ package org.apache.hadoop.tools.rumen;
import java.util.ArrayList;
import java.util.List;
+import java.util.Locale;
import java.util.Set;
import java.util.TreeSet;
@@ -611,7 +612,7 @@ public class LoggedTaskAttempt implements DeepCompare {
}
private static String canonicalizeCounterName(String nonCanonicalName) {
- String result = nonCanonicalName.toLowerCase();
+ String result = nonCanonicalName.toLowerCase(Locale.ENGLISH);
result = result.replace(' ', '|');
result = result.replace('-', '|');
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/Environment.java
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/Environment.java b/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/Environment.java
index 98d8aa03..72ff3b0 100644
--- a/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/Environment.java
+++ b/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/Environment.java
@@ -43,7 +43,7 @@ public class Environment extends Properties {
// http://lopica.sourceforge.net/os.html
String command = null;
String OS = System.getProperty("os.name");
- String lowerOs = OS.toLowerCase();
+ String lowerOs = OS.toLowerCase(Locale.ENGLISH);
if (OS.indexOf("Windows") > -1) {
command = "cmd /C set";
} else if (lowerOs.indexOf("ix") > -1 || lowerOs.indexOf("linux") > -1
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
index de8f740..1cb3e58 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
@@ -26,6 +26,7 @@ import java.text.DecimalFormat;
import java.util.EnumSet;
import java.util.HashSet;
import java.util.List;
+import java.util.Locale;
import java.util.Set;
import org.apache.commons.cli.CommandLine;
@@ -173,7 +174,7 @@ public class ApplicationCLI extends YarnCLI {
if (types != null) {
for (String type : types) {
if (!type.trim().isEmpty()) {
- appTypes.add(type.toUpperCase().trim());
+ appTypes.add(type.toUpperCase(Locale.ENGLISH).trim());
}
}
}
@@ -192,7 +193,7 @@ public class ApplicationCLI extends YarnCLI {
}
try {
appStates.add(YarnApplicationState.valueOf(state
- .toUpperCase().trim()));
+ .toUpperCase(Locale.ENGLISH).trim()));
} catch (IllegalArgumentException ex) {
sysout.println("The application state " + state
+ " is invalid.");
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java
index 22c240f..a6ed9b5 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java
@@ -27,6 +27,7 @@ import java.util.Collections;
import java.util.Date;
import java.util.HashSet;
import java.util.List;
+import java.util.Locale;
import java.util.Set;
import org.apache.commons.cli.CommandLine;
@@ -110,7 +111,8 @@ public class NodeCLI extends YarnCLI {
if (types != null) {
for (String type : types) {
if (!type.trim().isEmpty()) {
- nodeStates.add(NodeState.valueOf(type.trim().toUpperCase()));
+ nodeStates.add(
+ NodeState.valueOf(type.trim().toUpperCase(Locale.ENGLISH)));
}
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsRequestPBImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsRequestPBImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsRequestPBImpl.java
index a8996f0..0b39dfe 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsRequestPBImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsRequestPBImpl.java
@@ -21,6 +21,7 @@ package org.apache.hadoop.yarn.api.protocolrecords.impl.pb;
import java.util.EnumSet;
import java.util.HashSet;
import java.util.List;
+import java.util.Locale;
import java.util.Set;
import org.apache.commons.lang.math.LongRange;
@@ -213,7 +214,7 @@ public class GetApplicationsRequestPBImpl extends GetApplicationsRequest {
// Convert applicationTags to lower case and add
this.applicationTags = new HashSet<String>();
for (String tag : tags) {
- this.applicationTags.add(tag.toLowerCase());
+ this.applicationTags.add(tag.toLowerCase(Locale.ENGLISH));
}
}
@@ -258,7 +259,8 @@ public class GetApplicationsRequestPBImpl extends GetApplicationsRequest {
public void setApplicationStates(Set<String> applicationStates) {
EnumSet<YarnApplicationState> appStates = null;
for (YarnApplicationState state : YarnApplicationState.values()) {
- if (applicationStates.contains(state.name().toLowerCase())) {
+ if (applicationStates.contains(
+ state.name().toLowerCase(Locale.ENGLISH))) {
if (appStates == null) {
appStates = EnumSet.of(state);
} else {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java
index 303b437..fe89f81 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java
@@ -19,6 +19,7 @@
package org.apache.hadoop.yarn.api.records.impl.pb;
import java.util.HashSet;
+import java.util.Locale;
import java.util.Set;
import org.apache.hadoop.classification.InterfaceAudience.Private;
@@ -291,7 +292,7 @@ extends ApplicationSubmissionContext {
// Convert applicationTags to lower case and add
this.applicationTags = new HashSet<String>();
for (String tag : tags) {
- this.applicationTags.add(tag.toLowerCase());
+ this.applicationTags.add(tag.toLowerCase(Locale.ENGLISH));
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/hamlet/HamletGen.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/hamlet/HamletGen.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/hamlet/HamletGen.java
index c848828..45d9b61 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/hamlet/HamletGen.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/hamlet/HamletGen.java
@@ -241,7 +241,8 @@ public class HamletGen {
puts(indent, "\n",
"private <T extends _> ", retName, "<T> ", methodName,
"_(T e, boolean inline) {\n",
- " return new ", retName, "<T>(\"", retName.toLowerCase(Locale.US),
+ " return new ", retName, "<T>(\"",
+ retName.toLowerCase(Locale.ENGLISH),
"\", e, opt(", !endTagOptional.contains(retName), ", inline, ",
retName.equals("PRE"), ")); }");
}
@@ -258,7 +259,7 @@ public class HamletGen {
puts(0, ") {");
puts(indent,
topMode ? "" : " closeAttrs();\n",
- " return ", retName.toLowerCase(Locale.US), "_(this, ",
+ " return ", retName.toLowerCase(Locale.ENGLISH), "_(this, ",
isInline(className, retName), ");\n", "}");
} else if (params.length == 1) {
puts(0, "String selector) {");
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebServices.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebServices.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebServices.java
index 2040f57..4140df4 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebServices.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebServices.java
@@ -19,6 +19,7 @@
package org.apache.hadoop.yarn.server.applicationhistoryservice.webapp;
import java.util.Collections;
+import java.util.Locale;
import java.util.Set;
import javax.servlet.http.HttpServletRequest;
@@ -147,7 +148,8 @@ public class AHSWebServices extends WebServices {
}
Set<String> appStates = parseQueries(statesQuery, true);
for (String appState : appStates) {
- switch (YarnApplicationState.valueOf(appState.toUpperCase())) {
+ switch (YarnApplicationState.valueOf(
+ appState.toUpperCase(Locale.ENGLISH))) {
case FINISHED:
case FAILED:
case KILLED:
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/webapp/TimelineWebServices.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/webapp/TimelineWebServices.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/webapp/TimelineWebServices.java
index 0907f2c..44ba2b9 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/webapp/TimelineWebServices.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/webapp/TimelineWebServices.java
@@ -24,6 +24,7 @@ import java.util.Collection;
import java.util.EnumSet;
import java.util.HashSet;
import java.util.List;
+import java.util.Locale;
import java.util.Set;
import java.util.SortedSet;
import java.util.TreeSet;
@@ -417,7 +418,7 @@ public class TimelineWebServices {
String[] strs = str.split(delimiter);
List<Field> fieldList = new ArrayList<Field>();
for (String s : strs) {
- s = s.trim().toUpperCase();
+ s = s.trim().toUpperCase(Locale.ENGLISH);
if (s.equals("EVENTS")) {
fieldList.add(Field.EVENTS);
} else if (s.equals("LASTEVENTONLY")) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java
index 385d10a..cc9000f 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java
@@ -23,6 +23,7 @@ import java.security.PrivilegedExceptionAction;
import java.util.Arrays;
import java.util.Collection;
import java.util.HashSet;
+import java.util.Locale;
import java.util.Set;
import javax.servlet.http.HttpServletRequest;
@@ -164,7 +165,7 @@ public class WebServices {
if (checkAppStates
&& !appStates.contains(appReport.getYarnApplicationState().toString()
- .toLowerCase())) {
+ .toLowerCase(Locale.ENGLISH))) {
continue;
}
if (finalStatusQuery != null && !finalStatusQuery.isEmpty()) {
@@ -186,7 +187,7 @@ public class WebServices {
}
if (checkAppTypes
&& !appTypes.contains(appReport.getApplicationType().trim()
- .toLowerCase())) {
+ .toLowerCase(Locale.ENGLISH))) {
continue;
}
@@ -368,7 +369,8 @@ public class WebServices {
if (isState) {
try {
// enum string is in the uppercase
- YarnApplicationState.valueOf(paramStr.trim().toUpperCase());
+ YarnApplicationState.valueOf(
+ paramStr.trim().toUpperCase(Locale.ENGLISH));
} catch (RuntimeException e) {
YarnApplicationState[] stateArray =
YarnApplicationState.values();
@@ -378,7 +380,7 @@ public class WebServices {
+ allAppStates);
}
}
- params.add(paramStr.trim().toLowerCase());
+ params.add(paramStr.trim().toLowerCase(Locale.ENGLISH));
}
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
index d3ccb91..38f7b93 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
@@ -29,6 +29,7 @@ import java.util.Collections;
import java.util.EnumSet;
import java.util.Iterator;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.atomic.AtomicInteger;
@@ -754,7 +755,7 @@ public class ClientRMService extends AbstractService implements
if (applicationTypes != null && !applicationTypes.isEmpty()) {
String appTypeToMatch = caseSensitive
? application.getApplicationType()
- : application.getApplicationType().toLowerCase();
+ : application.getApplicationType().toLowerCase(Locale.ENGLISH);
if (!applicationTypes.contains(appTypeToMatch)) {
continue;
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/ResourceWeights.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/ResourceWeights.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/ResourceWeights.java
index 230f9a9..e95b725 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/ResourceWeights.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/ResourceWeights.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.yarn.server.resourcemanager.resource;
+import java.util.Locale;
import org.apache.hadoop.classification.InterfaceAudience.Private;
import org.apache.hadoop.classification.InterfaceStability.Evolving;
@@ -61,7 +62,7 @@ public class ResourceWeights {
sb.append(", ");
}
ResourceType resourceType = ResourceType.values()[i];
- sb.append(resourceType.name().toLowerCase());
+ sb.append(resourceType.name().toLowerCase(Locale.ENGLISH));
sb.append(String.format(" weight=%.1f", getWeight(resourceType)));
}
sb.append(">");
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
index 3528c2d..5e6d3eb 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
@@ -24,6 +24,7 @@ import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.StringTokenizer;
@@ -394,7 +395,8 @@ public class CapacitySchedulerConfiguration extends ReservationSchedulerConfigur
public QueueState getState(String queue) {
String state = get(getQueuePrefix(queue) + STATE);
return (state != null) ?
- QueueState.valueOf(state.toUpperCase()) : QueueState.RUNNING;
+ QueueState.valueOf(state.toUpperCase(Locale.ENGLISH)) :
+ QueueState.RUNNING;
}
public void setAccessibleNodeLabels(String queue, Set<String> labels) {
@@ -490,7 +492,7 @@ public class CapacitySchedulerConfiguration extends ReservationSchedulerConfigur
}
private static String getAclKey(QueueACL acl) {
- return "acl_" + acl.toString().toLowerCase();
+ return "acl_" + acl.toString().toLowerCase(Locale.ENGLISH);
}
public AccessControlList getAcl(String queue, QueueACL acl) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
index 32ef906..0922092 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
@@ -20,6 +20,7 @@ package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
+import java.util.Locale;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
@@ -241,7 +242,7 @@ public class FairSchedulerConfiguration extends Configuration {
public static Resource parseResourceConfigValue(String val)
throws AllocationConfigurationException {
try {
- val = val.toLowerCase();
+ val = val.toLowerCase(Locale.ENGLISH);
int memory = findResource(val, "mb");
int vcores = findResource(val, "vcores");
return BuilderUtils.newResource(memory, vcores);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
index cc28afc..23e7b81 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
@@ -17,6 +17,7 @@
*/
package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+import java.util.Locale;
import org.apache.hadoop.classification.InterfaceAudience.Public;
import org.apache.hadoop.classification.InterfaceStability.Evolving;
import org.apache.hadoop.util.ReflectionUtils;
@@ -72,7 +73,7 @@ public abstract class SchedulingPolicy {
throws AllocationConfigurationException {
@SuppressWarnings("rawtypes")
Class clazz;
- String text = policy.toLowerCase();
+ String text = policy.toLowerCase(Locale.ENGLISH);
if (text.equalsIgnoreCase(FairSharePolicy.NAME)) {
clazz = FairSharePolicy.class;
} else if (text.equalsIgnoreCase(FifoPolicy.NAME)) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NodesPage.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NodesPage.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NodesPage.java
index f28a9a8..881ecb7 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NodesPage.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NodesPage.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.yarn.server.resourcemanager.webapp;
+import java.util.Locale;
import static org.apache.hadoop.yarn.webapp.YarnWebParams.NODE_STATE;
import static org.apache.hadoop.yarn.webapp.YarnWebParams.NODE_LABEL;
import static org.apache.hadoop.yarn.webapp.view.JQueryUI.DATATABLES;
@@ -77,7 +78,7 @@ class NodesPage extends RmView {
.th(".nodeManagerVersion", "Version")._()._().tbody();
NodeState stateFilter = null;
if (type != null && !type.isEmpty()) {
- stateFilter = NodeState.valueOf(type.toUpperCase());
+ stateFilter = NodeState.valueOf(type.toUpperCase(Locale.ENGLISH));
}
Collection<RMNode> rmNodes = this.rm.getRMContext().getRMNodes().values();
boolean isInactive = false;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/946456c6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
index 1834b6a..fb40054 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
@@ -30,6 +30,7 @@ import java.util.EnumSet;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentMap;
@@ -257,7 +258,8 @@ public class RMWebServices {
} else {
acceptedStates = EnumSet.noneOf(NodeState.class);
for (String stateStr : states.split(",")) {
- acceptedStates.add(NodeState.valueOf(stateStr.toUpperCase()));
+ acceptedStates.add(NodeState.valueOf(
+ stateStr.toUpperCase(Locale.ENGLISH)));
}
}
@@ -506,7 +508,7 @@ public class RMWebServices {
// if no states, returns the counts of all RMAppStates
if (states.size() == 0) {
for (YarnApplicationState state : YarnApplicationState.values()) {
- states.add(state.toString().toLowerCase());
+ states.add(state.toString().toLowerCase(Locale.ENGLISH));
}
}
// in case we extend to multiple applicationTypes in the future
@@ -518,8 +520,9 @@ public class RMWebServices {
ConcurrentMap<ApplicationId, RMApp> apps = rm.getRMContext().getRMApps();
for (RMApp rmapp : apps.values()) {
YarnApplicationState state = rmapp.createApplicationState();
- String type = rmapp.getApplicationType().trim().toLowerCase();
- if (states.contains(state.toString().toLowerCase())) {
+ String type =
+ rmapp.getApplicationType().trim().toLowerCase(Locale.ENGLISH);
+ if (states.contains(state.toString().toLowerCase(Locale.ENGLISH))) {
if (types.contains(ANY)) {
countApp(scoreboard, state, ANY);
} else if (types.contains(type)) {
@@ -554,7 +557,8 @@ public class RMWebServices {
if (isState) {
try {
// enum string is in the uppercase
- YarnApplicationState.valueOf(paramStr.trim().toUpperCase());
+ YarnApplicationState.valueOf(
+ paramStr.trim().toUpperCase(Locale.ENGLISH));
} catch (RuntimeException e) {
YarnApplicationState[] stateArray =
YarnApplicationState.values();
@@ -564,7 +568,7 @@ public class RMWebServices {
+ " specified. It should be one of " + allAppStates);
}
}
- params.add(paramStr.trim().toLowerCase());
+ params.add(paramStr.trim().toLowerCase(Locale.ENGLISH));
}
}
}
@@ -582,7 +586,8 @@ public class RMWebServices {
for (String state : states) {
Map<String, Long> partScoreboard = new HashMap<String, Long>();
scoreboard.put(
- YarnApplicationState.valueOf(state.toUpperCase()), partScoreboard);
+ YarnApplicationState.valueOf(
+ state.toUpperCase(Locale.ENGLISH)), partScoreboard);
// types is verified no to be empty
for (String type : types) {
partScoreboard.put(type, 0L);
[33/52] [abbrv] hadoop git commit: HDFS-7740. Test truncate with
DataNodes restarting. (yliu)
Posted by zh...@apache.org.
HDFS-7740. Test truncate with DataNodes restarting. (yliu)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/737bad02
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/737bad02
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/737bad02
Branch: refs/heads/HDFS-7285
Commit: 737bad02d4cf879fa7d20b7c0e083d9dc59f604c
Parents: 6f01330
Author: yliu <yl...@apache.org>
Authored: Sat Feb 21 06:32:34 2015 +0800
Committer: yliu <yl...@apache.org>
Committed: Sat Feb 21 06:32:34 2015 +0800
----------------------------------------------------------------------
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 2 +
.../hdfs/server/namenode/TestFileTruncate.java | 221 +++++++++++++++++++
2 files changed, 223 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/737bad02/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index c47b89d..c8b6610 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -666,6 +666,8 @@ Release 2.7.0 - UNRELEASED
HDFS-7773. Additional metrics in HDFS to be accessed via jmx.
(Anu Engineer via cnauroth)
+ HDFS-7740. Test truncate with DataNodes restarting. (yliu)
+
OPTIMIZATIONS
HDFS-7454. Reduce memory footprint for AclEntries in NameNode.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/737bad02/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
index 253727d..19b5cde 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
@@ -53,6 +53,7 @@ import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.HdfsConstants;
import org.apache.hadoop.hdfs.protocol.HdfsConstants.SafeModeAction;
+import org.apache.hadoop.hdfs.protocol.LocatedBlock;
import org.apache.hadoop.hdfs.protocol.LocatedBlocks;
import org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction;
import org.apache.hadoop.hdfs.server.common.HdfsServerConstants;
@@ -93,6 +94,8 @@ public class TestFileTruncate {
conf.setLong(DFSConfigKeys.DFS_NAMENODE_MIN_BLOCK_SIZE_KEY, BLOCK_SIZE);
conf.setInt(DFSConfigKeys.DFS_BYTES_PER_CHECKSUM_KEY, BLOCK_SIZE);
conf.setInt(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, SHORT_HEARTBEAT);
+ conf.setLong(
+ DFSConfigKeys.DFS_NAMENODE_REPLICATION_PENDING_TIMEOUT_SEC_KEY, 1);
cluster = new MiniDFSCluster.Builder(conf)
.format(true)
.numDataNodes(DATANODE_NUM)
@@ -623,6 +626,224 @@ public class TestFileTruncate {
}
/**
+ * The last block is truncated at mid. (non copy-on-truncate)
+ * dn0 is shutdown before truncate and restart after truncate successful.
+ */
+ @Test(timeout=60000)
+ public void testTruncateWithDataNodesRestart() throws Exception {
+ int startingFileSize = 3 * BLOCK_SIZE;
+ byte[] contents = AppendTestUtil.initBuffer(startingFileSize);
+ final Path parent = new Path("/test");
+ final Path p = new Path(parent, "testTruncateWithDataNodesRestart");
+
+ writeContents(contents, startingFileSize, p);
+ LocatedBlock oldBlock = getLocatedBlocks(p).getLastLocatedBlock();
+
+ int dn = 0;
+ int toTruncateLength = 1;
+ int newLength = startingFileSize - toTruncateLength;
+ cluster.getDataNodes().get(dn).shutdown();
+ try {
+ boolean isReady = fs.truncate(p, newLength);
+ assertFalse(isReady);
+ } finally {
+ cluster.restartDataNode(dn);
+ cluster.waitActive();
+ cluster.triggerBlockReports();
+ }
+
+ LocatedBlock newBlock = getLocatedBlocks(p).getLastLocatedBlock();
+ /*
+ * For non copy-on-truncate, the truncated block id is the same, but the
+ * GS should increase.
+ * We trigger block report for dn0 after it restarts, since the GS
+ * of replica for the last block on it is old, so the reported last block
+ * from dn0 should be marked corrupt on nn and the replicas of last block
+ * on nn should decrease 1, then the truncated block will be replicated
+ * to dn0.
+ */
+ assertEquals(newBlock.getBlock().getBlockId(),
+ oldBlock.getBlock().getBlockId());
+ assertEquals(newBlock.getBlock().getGenerationStamp(),
+ oldBlock.getBlock().getGenerationStamp() + 1);
+
+ checkBlockRecovery(p);
+ // Wait replicas come to 3
+ DFSTestUtil.waitReplication(fs, p, REPLICATION);
+ // Old replica is disregarded and replaced with the truncated one
+ assertEquals(cluster.getBlockFile(dn, newBlock.getBlock()).length(),
+ newBlock.getBlockSize());
+ assertTrue(cluster.getBlockMetadataFile(dn,
+ newBlock.getBlock()).getName().endsWith(
+ newBlock.getBlock().getGenerationStamp() + ".meta"));
+
+ // Validate the file
+ FileStatus fileStatus = fs.getFileStatus(p);
+ assertThat(fileStatus.getLen(), is((long) newLength));
+ checkFullFile(p, newLength, contents);
+
+ fs.delete(parent, true);
+ }
+
+ /**
+ * The last block is truncated at mid. (copy-on-truncate)
+ * dn1 is shutdown before truncate and restart after truncate successful.
+ */
+ @Test(timeout=60000)
+ public void testCopyOnTruncateWithDataNodesRestart() throws Exception {
+ int startingFileSize = 3 * BLOCK_SIZE;
+ byte[] contents = AppendTestUtil.initBuffer(startingFileSize);
+ final Path parent = new Path("/test");
+ final Path p = new Path(parent, "testCopyOnTruncateWithDataNodesRestart");
+
+ writeContents(contents, startingFileSize, p);
+ LocatedBlock oldBlock = getLocatedBlocks(p).getLastLocatedBlock();
+ fs.allowSnapshot(parent);
+ fs.createSnapshot(parent, "ss0");
+
+ int dn = 1;
+ int toTruncateLength = 1;
+ int newLength = startingFileSize - toTruncateLength;
+ cluster.getDataNodes().get(dn).shutdown();
+ try {
+ boolean isReady = fs.truncate(p, newLength);
+ assertFalse(isReady);
+ } finally {
+ cluster.restartDataNode(dn);
+ cluster.waitActive();
+ cluster.triggerBlockReports();
+ }
+
+ LocatedBlock newBlock = getLocatedBlocks(p).getLastLocatedBlock();
+ /*
+ * For copy-on-truncate, new block is made with new block id and new GS.
+ * We trigger block report for dn1 after it restarts. The replicas of
+ * the new block is 2, and then it will be replicated to dn1.
+ */
+ assertNotEquals(newBlock.getBlock().getBlockId(),
+ oldBlock.getBlock().getBlockId());
+ assertEquals(newBlock.getBlock().getGenerationStamp(),
+ oldBlock.getBlock().getGenerationStamp() + 1);
+
+ checkBlockRecovery(p);
+ // Wait replicas come to 3
+ DFSTestUtil.waitReplication(fs, p, REPLICATION);
+ // New block is replicated to dn1
+ assertEquals(cluster.getBlockFile(dn, newBlock.getBlock()).length(),
+ newBlock.getBlockSize());
+ // Old replica exists too since there is snapshot
+ assertEquals(cluster.getBlockFile(dn, oldBlock.getBlock()).length(),
+ oldBlock.getBlockSize());
+ assertTrue(cluster.getBlockMetadataFile(dn,
+ oldBlock.getBlock()).getName().endsWith(
+ oldBlock.getBlock().getGenerationStamp() + ".meta"));
+
+ // Validate the file
+ FileStatus fileStatus = fs.getFileStatus(p);
+ assertThat(fileStatus.getLen(), is((long) newLength));
+ checkFullFile(p, newLength, contents);
+
+ fs.deleteSnapshot(parent, "ss0");
+ fs.delete(parent, true);
+ }
+
+ /**
+ * The last block is truncated at mid. (non copy-on-truncate)
+ * dn0, dn1 are restarted immediately after truncate.
+ */
+ @Test(timeout=60000)
+ public void testTruncateWithDataNodesRestartImmediately() throws Exception {
+ int startingFileSize = 3 * BLOCK_SIZE;
+ byte[] contents = AppendTestUtil.initBuffer(startingFileSize);
+ final Path parent = new Path("/test");
+ final Path p = new Path(parent, "testTruncateWithDataNodesRestartImmediately");
+
+ writeContents(contents, startingFileSize, p);
+ LocatedBlock oldBlock = getLocatedBlocks(p).getLastLocatedBlock();
+
+ int dn0 = 0;
+ int dn1 = 1;
+ int toTruncateLength = 1;
+ int newLength = startingFileSize - toTruncateLength;
+ boolean isReady = fs.truncate(p, newLength);
+ assertFalse(isReady);
+
+ cluster.restartDataNode(dn0);
+ cluster.restartDataNode(dn1);
+ cluster.waitActive();
+ cluster.triggerBlockReports();
+
+ LocatedBlock newBlock = getLocatedBlocks(p).getLastLocatedBlock();
+ /*
+ * For non copy-on-truncate, the truncated block id is the same, but the
+ * GS should increase.
+ */
+ assertEquals(newBlock.getBlock().getBlockId(),
+ oldBlock.getBlock().getBlockId());
+ assertEquals(newBlock.getBlock().getGenerationStamp(),
+ oldBlock.getBlock().getGenerationStamp() + 1);
+
+ checkBlockRecovery(p);
+ // Wait replicas come to 3
+ DFSTestUtil.waitReplication(fs, p, REPLICATION);
+ // Old replica is disregarded and replaced with the truncated one on dn0
+ assertEquals(cluster.getBlockFile(dn0, newBlock.getBlock()).length(),
+ newBlock.getBlockSize());
+ assertTrue(cluster.getBlockMetadataFile(dn0,
+ newBlock.getBlock()).getName().endsWith(
+ newBlock.getBlock().getGenerationStamp() + ".meta"));
+
+ // Old replica is disregarded and replaced with the truncated one on dn1
+ assertEquals(cluster.getBlockFile(dn1, newBlock.getBlock()).length(),
+ newBlock.getBlockSize());
+ assertTrue(cluster.getBlockMetadataFile(dn1,
+ newBlock.getBlock()).getName().endsWith(
+ newBlock.getBlock().getGenerationStamp() + ".meta"));
+
+ // Validate the file
+ FileStatus fileStatus = fs.getFileStatus(p);
+ assertThat(fileStatus.getLen(), is((long) newLength));
+ checkFullFile(p, newLength, contents);
+
+ fs.delete(parent, true);
+ }
+
+ /**
+ * The last block is truncated at mid. (non copy-on-truncate)
+ * shutdown the datanodes immediately after truncate.
+ */
+ @Test(timeout=60000)
+ public void testTruncateWithDataNodesShutdownImmediately() throws Exception {
+ int startingFileSize = 3 * BLOCK_SIZE;
+ byte[] contents = AppendTestUtil.initBuffer(startingFileSize);
+ final Path parent = new Path("/test");
+ final Path p = new Path(parent, "testTruncateWithDataNodesShutdownImmediately");
+
+ writeContents(contents, startingFileSize, p);
+
+ int toTruncateLength = 1;
+ int newLength = startingFileSize - toTruncateLength;
+ boolean isReady = fs.truncate(p, newLength);
+ assertFalse(isReady);
+
+ cluster.shutdownDataNodes();
+ try {
+ for(int i = 0; i < SUCCESS_ATTEMPTS && cluster.isDataNodeUp(); i++) {
+ Thread.sleep(SLEEP);
+ }
+ assertFalse("All DataNodes should be down.", cluster.isDataNodeUp());
+ LocatedBlocks blocks = getLocatedBlocks(p);
+ assertTrue(blocks.isUnderConstruction());
+ } finally {
+ cluster.startDataNodes(conf, DATANODE_NUM, true,
+ StartupOption.REGULAR, null);
+ cluster.waitActive();
+ }
+
+ fs.delete(parent, true);
+ }
+
+ /**
* EditLogOp load test for Truncate.
*/
@Test
[28/52] [abbrv] hadoop git commit: HDFS-7814. Fix usage string of
storageType parameter for "dfsadmin -setSpaceQuota/clrSpaceQuota".
Contributed by Xiaoyu Yao.
Posted by zh...@apache.org.
HDFS-7814. Fix usage string of storageType parameter for "dfsadmin -setSpaceQuota/clrSpaceQuota". Contributed by Xiaoyu Yao.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/8c6ae0d6
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/8c6ae0d6
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/8c6ae0d6
Branch: refs/heads/HDFS-7285
Commit: 8c6ae0d6199efa327d8f2761f2ad2163a60e5508
Parents: ce5bf92
Author: cnauroth <cn...@apache.org>
Authored: Fri Feb 20 12:21:46 2015 -0800
Committer: cnauroth <cn...@apache.org>
Committed: Fri Feb 20 12:21:46 2015 -0800
----------------------------------------------------------------------
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 +++
.../src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java | 8 ++++----
.../hadoop-hdfs/src/test/resources/testHDFSConf.xml | 4 ++--
3 files changed, 9 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8c6ae0d6/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index 71ce48f..7d9d0ea 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -995,6 +995,9 @@ Release 2.7.0 - UNRELEASED
HDFS-7788. Post-2.6 namenode may not start up with an image containing
inodes created with an old release. (Rushabh Shah via kihwal)
+ HDFS-7814. Fix usage string of storageType parameter for
+ "dfsadmin -setSpaceQuota/clrSpaceQuota". (Xiaoyu Yao via cnauroth)
+
BREAKDOWN OF HDFS-7584 SUBTASKS AND RELATED JIRAS
HDFS-7720. Quota by Storage Type API, tools and ClientNameNode
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8c6ae0d6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
index ae9088c..6ba778f 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
@@ -207,9 +207,9 @@ public class DFSAdmin extends FsShell {
/** A class that supports command clearSpaceQuota */
private static class ClearSpaceQuotaCommand extends DFSAdminCommand {
private static final String NAME = "clrSpaceQuota";
- private static final String USAGE = "-"+NAME+" <dirname>...<dirname> -storageType <storagetype>";
+ private static final String USAGE = "-"+NAME+" [-storageType <storagetype>] <dirname>...<dirname>";
private static final String DESCRIPTION = USAGE + ": " +
- "Clear the disk space quota for each directory <dirName>.\n" +
+ "Clear the space quota for each directory <dirName>.\n" +
"\t\tFor each directory, attempt to clear the quota. An error will be reported if\n" +
"\t\t1. the directory does not exist or is a file, or\n" +
"\t\t2. user is not an administrator.\n" +
@@ -259,9 +259,9 @@ public class DFSAdmin extends FsShell {
private static class SetSpaceQuotaCommand extends DFSAdminCommand {
private static final String NAME = "setSpaceQuota";
private static final String USAGE =
- "-"+NAME+" <quota> <dirname>...<dirname> -storageType <storagetype>";
+ "-"+NAME+" <quota> [-storageType <storagetype>] <dirname>...<dirname>";
private static final String DESCRIPTION = USAGE + ": " +
- "Set the disk space quota <quota> for each directory <dirName>.\n" +
+ "Set the space quota <quota> for each directory <dirName>.\n" +
"\t\tThe space quota is a long integer that puts a hard limit\n" +
"\t\ton the total size of all the files under the directory tree.\n" +
"\t\tThe extra space required for replication is also counted. E.g.\n" +
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8c6ae0d6/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml b/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
index 935cd58..2d3de1f 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
@@ -15711,7 +15711,7 @@
<comparators>
<comparator>
<type>RegexpComparator</type>
- <expected-output>^-setSpaceQuota <quota> <dirname>...<dirname> -storageType <storagetype>: Set the disk space quota <quota> for each directory <dirName>.( )*</expected-output>
+ <expected-output>^-setSpaceQuota <quota> \[-storageType <storagetype>\] <dirname>...<dirname>: Set the space quota <quota> for each directory <dirName>.( )*</expected-output>
</comparator>
<comparator>
<type>RegexpComparator</type>
@@ -15746,7 +15746,7 @@
<comparators>
<comparator>
<type>RegexpComparator</type>
- <expected-output>^-clrSpaceQuota <dirname>...<dirname> -storageType <storagetype>: Clear the disk space quota for each directory <dirName>.( )*</expected-output>
+ <expected-output>^-clrSpaceQuota \[-storageType <storagetype>\] <dirname>...<dirname>: Clear the space quota for each directory <dirName>.( )*</expected-output>
</comparator>
<comparator>
<type>RegexpComparator</type>
[26/52] [abbrv] hadoop git commit: YARN-2799. Cleanup
TestLogAggregationService based on the change in YARN-90. Contributed by
Zhihai Xu
Posted by zh...@apache.org.
YARN-2799. Cleanup TestLogAggregationService based on the change in YARN-90. Contributed by Zhihai Xu
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/c33ae271
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/c33ae271
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/c33ae271
Branch: refs/heads/HDFS-7285
Commit: c33ae271c24f0770c9735ccd2086cafda4f4e0b2
Parents: a64dd3d
Author: Junping Du <ju...@apache.org>
Authored: Fri Feb 20 09:43:39 2015 -0800
Committer: Junping Du <ju...@apache.org>
Committed: Fri Feb 20 09:43:39 2015 -0800
----------------------------------------------------------------------
hadoop-yarn-project/CHANGES.txt | 3 +++
.../TestLogAggregationService.java | 25 +++-----------------
2 files changed, 6 insertions(+), 22 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c33ae271/hadoop-yarn-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 8ec2409..e71da2d 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -307,6 +307,9 @@ Release 2.7.0 - UNRELEASED
YARN-3076. Add API/Implementation to YarnClient to retrieve label-to-node
mapping. (Varun Saxena via wangda)
+ YARN-2799. Cleanup TestLogAggregationService based on the change in YARN-90.
+ (Zhihai Xu via junping_du)
+
OPTIMIZATIONS
YARN-2990. FairScheduler's delay-scheduling always waits for node-local and
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c33ae271/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
index 7d911e9..901e45a 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
@@ -248,7 +248,6 @@ public class TestLogAggregationService extends BaseContainerManagerTest {
};
checkEvents(appEventHandler, expectedEvents, true, "getType", "getApplicationID");
- dispatcher.stop();
}
@Test
@@ -295,7 +294,6 @@ public class TestLogAggregationService extends BaseContainerManagerTest {
ApplicationEventType.APPLICATION_LOG_HANDLING_FINISHED)
};
checkEvents(appEventHandler, expectedEvents, true, "getType", "getApplicationID");
- dispatcher.stop();
logAggregationService.close();
}
@@ -308,10 +306,7 @@ public class TestLogAggregationService extends BaseContainerManagerTest {
this.remoteRootLogDir.getAbsolutePath());
String[] fileNames = new String[] { "stdout", "stderr", "syslog" };
- DrainDispatcher dispatcher = createDispatcher();
- EventHandler<ApplicationEvent> appEventHandler = mock(EventHandler.class);
- dispatcher.register(ApplicationEventType.class, appEventHandler);
-
+
LogAggregationService logAggregationService =
new LogAggregationService(dispatcher, this.context, this.delSrvc,
super.dirsHandler);
@@ -441,7 +436,6 @@ public class TestLogAggregationService extends BaseContainerManagerTest {
ApplicationEventType.APPLICATION_LOG_HANDLING_FINISHED)
};
checkEvents(appEventHandler, expectedFinishedEvents, false, "getType", "getApplicationID");
- dispatcher.stop();
}
@Test
@@ -518,8 +512,7 @@ public class TestLogAggregationService extends BaseContainerManagerTest {
File aNewFile = new File(String.valueOf("tmp"+System.currentTimeMillis()));
this.conf.set(YarnConfiguration.NM_REMOTE_APP_LOG_DIR,
aNewFile.getAbsolutePath());
-
- DrainDispatcher dispatcher = createDispatcher();
+
LogAggregationService logAggregationService = spy(
new LogAggregationService(dispatcher, this.context, this.delSrvc,
super.dirsHandler));
@@ -590,6 +583,7 @@ public class TestLogAggregationService extends BaseContainerManagerTest {
verify(spyFs, never()).mkdirs(eq(appDir3), isA(FsPermission.class));
aggSvc.stop();
aggSvc.close();
+ dispatcher.stop();
}
@Test
@@ -1122,10 +1116,6 @@ public class TestLogAggregationService extends BaseContainerManagerTest {
this.conf.set(YarnConfiguration.NM_REMOTE_APP_LOG_DIR,
this.remoteRootLogDir.getAbsolutePath());
- DrainDispatcher dispatcher = createDispatcher();
- EventHandler<ApplicationEvent> appEventHandler = mock(EventHandler.class);
- dispatcher.register(ApplicationEventType.class, appEventHandler);
-
ApplicationId application1 = BuilderUtils.newApplicationId(1234, 1);
ApplicationId application2 = BuilderUtils.newApplicationId(1234, 2);
ApplicationId application3 = BuilderUtils.newApplicationId(1234, 3);
@@ -1282,7 +1272,6 @@ public class TestLogAggregationService extends BaseContainerManagerTest {
ApplicationEventType.APPLICATION_LOG_HANDLING_FINISHED) };
checkEvents(appEventHandler, expectedFinishedEvents, false, "getType",
"getApplicationID");
- dispatcher.stop();
}
@Test (timeout = 50000)
@@ -1319,10 +1308,6 @@ public class TestLogAggregationService extends BaseContainerManagerTest {
// again in next cycle.
this.conf.setLong(YarnConfiguration.DEBUG_NM_DELETE_DELAY_SEC, 3600);
- DrainDispatcher dispatcher = createDispatcher();
- EventHandler<ApplicationEvent> appEventHandler = mock(EventHandler.class);
- dispatcher.register(ApplicationEventType.class, appEventHandler);
-
ApplicationId application = BuilderUtils.newApplicationId(123456, 1);
ApplicationAttemptId appAttemptId =
BuilderUtils.newApplicationAttemptId(application, 1);
@@ -1426,7 +1411,6 @@ public class TestLogAggregationService extends BaseContainerManagerTest {
new ContainerId[] { container }, logFiles3, 3, true);
logAggregationService.stop();
assertEquals(0, logAggregationService.getNumAggregators());
- dispatcher.stop();
}
@@ -1436,8 +1420,6 @@ public class TestLogAggregationService extends BaseContainerManagerTest {
conf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
"kerberos");
UserGroupInformation.setConfiguration(conf);
- DrainDispatcher dispatcher = createDispatcher();
- dispatcher.register(ApplicationEventType.class, appEventHandler);
ApplicationId application1 = BuilderUtils.newApplicationId(1234, 1);
Application mockApp = mock(Application.class);
@@ -1484,7 +1466,6 @@ public class TestLogAggregationService extends BaseContainerManagerTest {
}
}, 1000, 20000);
logAggregationService.stop();
- dispatcher.stop();
}
private int numOfLogsAvailable(LogAggregationService logAggregationService,
[35/52] [abbrv] hadoop git commit: HADOOP-11612. Workaround for
Curator's ChildReaper requiring Guava 15+. (rkanter)
Posted by zh...@apache.org.
HADOOP-11612. Workaround for Curator's ChildReaper requiring Guava 15+. (rkanter)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/6f013303
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/6f013303
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/6f013303
Branch: refs/heads/HDFS-7285
Commit: 6f0133039a064ca82363ac6f29fb255506f31b8a
Parents: 0d6af57
Author: Robert Kanter <rk...@apache.org>
Authored: Fri Feb 20 19:47:28 2015 -0800
Committer: Robert Kanter <rk...@apache.org>
Committed: Fri Feb 20 19:47:28 2015 -0800
----------------------------------------------------------------------
hadoop-common-project/hadoop-common/CHANGES.txt | 3 +
.../apache/hadoop/util/curator/ChildReaper.java | 234 +++++++++++++++++++
.../hadoop/util/curator/TestChildReaper.java | 208 +++++++++++++++++
3 files changed, 445 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6f013303/hadoop-common-project/hadoop-common/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt b/hadoop-common-project/hadoop-common/CHANGES.txt
index b09868a..143692e 100644
--- a/hadoop-common-project/hadoop-common/CHANGES.txt
+++ b/hadoop-common-project/hadoop-common/CHANGES.txt
@@ -981,6 +981,9 @@ Release 2.7.0 - UNRELEASED
HADOOP-11604. Prevent ConcurrentModificationException while closing domain
sockets during shutdown of DomainSocketWatcher thread. (cnauroth)
+ HADOOP-11612. Workaround for Curator's ChildReaper requiring Guava 15+.
+ (rkanter)
+
Release 2.6.1 - UNRELEASED
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6f013303/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ChildReaper.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ChildReaper.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ChildReaper.java
new file mode 100644
index 0000000..3bff187
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ChildReaper.java
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Sets;
+import org.apache.curator.framework.recipes.locks.Reaper;
+import org.apache.curator.utils.CloseableUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.utils.CloseableScheduledExecutorService;
+import org.apache.curator.utils.ThreadUtils;
+import org.apache.curator.utils.ZKPaths;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.zookeeper.data.Stat;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.curator.utils.PathUtils;
+
+/**
+ * This is a copy of Curator 2.7.1's ChildReaper class, modified to work with
+ * Guava 11.0.2. The problem is the 'paths' Collection, which calls Guava's
+ * Sets.newConcurrentHashSet(), which was added in Guava 15.0.
+ * <p>
+ * Utility to reap empty child nodes of a parent node. Periodically calls getChildren on
+ * the node and adds empty nodes to an internally managed {@link Reaper}
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+public class ChildReaper implements Closeable
+{
+ private final Logger log = LoggerFactory.getLogger(getClass());
+ private final Reaper reaper;
+ private final AtomicReference<State> state = new AtomicReference<State>(State.LATENT);
+ private final CuratorFramework client;
+ private final Collection<String> paths = newConcurrentHashSet();
+ private final Reaper.Mode mode;
+ private final CloseableScheduledExecutorService executor;
+ private final int reapingThresholdMs;
+
+ private volatile Future<?> task;
+
+ // This is copied from Curator's Reaper class
+ static final int DEFAULT_REAPING_THRESHOLD_MS = (int)TimeUnit.MILLISECONDS.convert(5, TimeUnit.MINUTES);
+
+ // This is copied from Guava
+ /**
+ * Creates a thread-safe set backed by a hash map. The set is backed by a
+ * {@link ConcurrentHashMap} instance, and thus carries the same concurrency
+ * guarantees.
+ *
+ * <p>Unlike {@code HashSet}, this class does NOT allow {@code null} to be
+ * used as an element. The set is serializable.
+ *
+ * @return a new, empty thread-safe {@code Set}
+ * @since 15.0
+ */
+ public static <E> Set<E> newConcurrentHashSet() {
+ return Sets.newSetFromMap(new ConcurrentHashMap<E, Boolean>());
+ }
+
+ private enum State
+ {
+ LATENT,
+ STARTED,
+ CLOSED
+ }
+
+ /**
+ * @param client the client
+ * @param path path to reap children from
+ * @param mode reaping mode
+ */
+ public ChildReaper(CuratorFramework client, String path, Reaper.Mode mode)
+ {
+ this(client, path, mode, newExecutorService(), DEFAULT_REAPING_THRESHOLD_MS, null);
+ }
+
+ /**
+ * @param client the client
+ * @param path path to reap children from
+ * @param reapingThresholdMs threshold in milliseconds that determines that a path can be deleted
+ * @param mode reaping mode
+ */
+ public ChildReaper(CuratorFramework client, String path, Reaper.Mode mode, int reapingThresholdMs)
+ {
+ this(client, path, mode, newExecutorService(), reapingThresholdMs, null);
+ }
+
+ /**
+ * @param client the client
+ * @param path path to reap children from
+ * @param executor executor to use for background tasks
+ * @param reapingThresholdMs threshold in milliseconds that determines that a path can be deleted
+ * @param mode reaping mode
+ */
+ public ChildReaper(CuratorFramework client, String path, Reaper.Mode mode, ScheduledExecutorService executor, int reapingThresholdMs)
+ {
+ this(client, path, mode, executor, reapingThresholdMs, null);
+ }
+
+ /**
+ * @param client the client
+ * @param path path to reap children from
+ * @param executor executor to use for background tasks
+ * @param reapingThresholdMs threshold in milliseconds that determines that a path can be deleted
+ * @param mode reaping mode
+ * @param leaderPath if not null, uses a leader selection so that only 1 reaper is active in the cluster
+ */
+ public ChildReaper(CuratorFramework client, String path, Reaper.Mode mode, ScheduledExecutorService executor, int reapingThresholdMs, String leaderPath)
+ {
+ this.client = client;
+ this.mode = mode;
+ this.executor = new CloseableScheduledExecutorService(executor);
+ this.reapingThresholdMs = reapingThresholdMs;
+ this.reaper = new Reaper(client, executor, reapingThresholdMs, leaderPath);
+ addPath(path);
+ }
+
+ /**
+ * The reaper must be started
+ *
+ * @throws Exception errors
+ */
+ public void start() throws Exception
+ {
+ Preconditions.checkState(state.compareAndSet(State.LATENT, State.STARTED), "Cannot be started more than once");
+
+ task = executor.scheduleWithFixedDelay
+ (
+ new Runnable()
+ {
+ @Override
+ public void run()
+ {
+ doWork();
+ }
+ },
+ reapingThresholdMs,
+ reapingThresholdMs,
+ TimeUnit.MILLISECONDS
+ );
+
+ reaper.start();
+ }
+
+ @Override
+ public void close() throws IOException
+ {
+ if ( state.compareAndSet(State.STARTED, State.CLOSED) )
+ {
+ CloseableUtils.closeQuietly(reaper);
+ task.cancel(true);
+ }
+ }
+
+ /**
+ * Add a path to reap children from
+ *
+ * @param path the path
+ * @return this for chaining
+ */
+ public ChildReaper addPath(String path)
+ {
+ paths.add(PathUtils.validatePath(path));
+ return this;
+ }
+
+ /**
+ * Remove a path from reaping
+ *
+ * @param path the path
+ * @return true if the path existed and was removed
+ */
+ public boolean removePath(String path)
+ {
+ return paths.remove(PathUtils.validatePath(path));
+ }
+
+ private static ScheduledExecutorService newExecutorService()
+ {
+ return ThreadUtils.newFixedThreadScheduledPool(2, "ChildReaper");
+ }
+
+ private void doWork()
+ {
+ for ( String path : paths )
+ {
+ try
+ {
+ List<String> children = client.getChildren().forPath(path);
+ for ( String name : children )
+ {
+ String thisPath = ZKPaths.makePath(path, name);
+ Stat stat = client.checkExists().forPath(thisPath);
+ if ( (stat != null) && (stat.getNumChildren() == 0) )
+ {
+ reaper.addPath(thisPath, mode);
+ }
+ }
+ }
+ catch ( Exception e )
+ {
+ log.error("Could not get children for path: " + path, e);
+ }
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6f013303/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestChildReaper.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestChildReaper.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestChildReaper.java
new file mode 100644
index 0000000..11b254f
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestChildReaper.java
@@ -0,0 +1,208 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util.curator;
+
+import org.apache.curator.framework.recipes.locks.Reaper;
+import org.apache.curator.test.TestingServer;
+import org.apache.curator.utils.CloseableUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.retry.RetryOneTime;
+import org.apache.curator.test.Timing;
+import org.apache.zookeeper.data.Stat;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.net.BindException;
+import java.util.Random;
+
+/**
+ * This is a copy of Curator 2.7.1's TestChildReaper class, with minor
+ * modifications to make it work with JUnit (some setup code taken from
+ * Curator's BaseClassForTests). This is to ensure that the ChildReaper
+ * class we modified is still correct.
+ */
+public class TestChildReaper
+{
+ protected TestingServer server;
+
+ @Before
+ public void setup() throws Exception {
+ while(this.server == null) {
+ try {
+ this.server = new TestingServer();
+ } catch (BindException var2) {
+ System.err.println("Getting bind exception - retrying to allocate server");
+ this.server = null;
+ }
+ }
+ }
+
+ @After
+ public void teardown() throws Exception {
+ this.server.close();
+ this.server = null;
+ }
+
+ @Test
+ public void testSomeNodes() throws Exception
+ {
+
+ Timing timing = new Timing();
+ ChildReaper reaper = null;
+ CuratorFramework client = CuratorFrameworkFactory.newClient(server.getConnectString(), timing.session(), timing.connection(), new RetryOneTime(1));
+ try
+ {
+ client.start();
+
+ Random r = new Random();
+ int nonEmptyNodes = 0;
+ for ( int i = 0; i < 10; ++i )
+ {
+ client.create().creatingParentsIfNeeded().forPath("/test/" + Integer.toString(i));
+ if ( r.nextBoolean() )
+ {
+ client.create().forPath("/test/" + Integer.toString(i) + "/foo");
+ ++nonEmptyNodes;
+ }
+ }
+
+ reaper = new ChildReaper(client, "/test", Reaper.Mode.REAP_UNTIL_DELETE, 1);
+ reaper.start();
+
+ timing.forWaiting().sleepABit();
+
+ Stat stat = client.checkExists().forPath("/test");
+ Assert.assertEquals(stat.getNumChildren(), nonEmptyNodes);
+ }
+ finally
+ {
+ CloseableUtils.closeQuietly(reaper);
+ CloseableUtils.closeQuietly(client);
+ }
+ }
+
+ @Test
+ public void testSimple() throws Exception
+ {
+ Timing timing = new Timing();
+ ChildReaper reaper = null;
+ CuratorFramework client = CuratorFrameworkFactory.newClient(server.getConnectString(), timing.session(), timing.connection(), new RetryOneTime(1));
+ try
+ {
+ client.start();
+
+ for ( int i = 0; i < 10; ++i )
+ {
+ client.create().creatingParentsIfNeeded().forPath("/test/" + Integer.toString(i));
+ }
+
+ reaper = new ChildReaper(client, "/test", Reaper.Mode.REAP_UNTIL_DELETE, 1);
+ reaper.start();
+
+ timing.forWaiting().sleepABit();
+
+ Stat stat = client.checkExists().forPath("/test");
+ Assert.assertEquals(stat.getNumChildren(), 0);
+ }
+ finally
+ {
+ CloseableUtils.closeQuietly(reaper);
+ CloseableUtils.closeQuietly(client);
+ }
+ }
+
+ @Test
+ public void testMultiPath() throws Exception
+ {
+ Timing timing = new Timing();
+ ChildReaper reaper = null;
+ CuratorFramework client = CuratorFrameworkFactory.newClient(server.getConnectString(), timing.session(), timing.connection(), new RetryOneTime(1));
+ try
+ {
+ client.start();
+
+ for ( int i = 0; i < 10; ++i )
+ {
+ client.create().creatingParentsIfNeeded().forPath("/test1/" + Integer.toString(i));
+ client.create().creatingParentsIfNeeded().forPath("/test2/" + Integer.toString(i));
+ client.create().creatingParentsIfNeeded().forPath("/test3/" + Integer.toString(i));
+ }
+
+ reaper = new ChildReaper(client, "/test2", Reaper.Mode.REAP_UNTIL_DELETE, 1);
+ reaper.start();
+ reaper.addPath("/test1");
+
+ timing.forWaiting().sleepABit();
+
+ Stat stat = client.checkExists().forPath("/test1");
+ Assert.assertEquals(stat.getNumChildren(), 0);
+ stat = client.checkExists().forPath("/test2");
+ Assert.assertEquals(stat.getNumChildren(), 0);
+ stat = client.checkExists().forPath("/test3");
+ Assert.assertEquals(stat.getNumChildren(), 10);
+ }
+ finally
+ {
+ CloseableUtils.closeQuietly(reaper);
+ CloseableUtils.closeQuietly(client);
+ }
+ }
+
+ @Test
+ public void testNamespace() throws Exception
+ {
+ Timing timing = new Timing();
+ ChildReaper reaper = null;
+ CuratorFramework client = CuratorFrameworkFactory.builder()
+ .connectString(server.getConnectString())
+ .sessionTimeoutMs(timing.session())
+ .connectionTimeoutMs(timing.connection())
+ .retryPolicy(new RetryOneTime(1))
+ .namespace("foo")
+ .build();
+ try
+ {
+ client.start();
+
+ for ( int i = 0; i < 10; ++i )
+ {
+ client.create().creatingParentsIfNeeded().forPath("/test/" + Integer.toString(i));
+ }
+
+ reaper = new ChildReaper(client, "/test", Reaper.Mode.REAP_UNTIL_DELETE, 1);
+ reaper.start();
+
+ timing.forWaiting().sleepABit();
+
+ Stat stat = client.checkExists().forPath("/test");
+ Assert.assertEquals(stat.getNumChildren(), 0);
+
+ stat = client.usingNamespace(null).checkExists().forPath("/foo/test");
+ Assert.assertNotNull(stat);
+ Assert.assertEquals(stat.getNumChildren(), 0);
+ }
+ finally
+ {
+ CloseableUtils.closeQuietly(reaper);
+ CloseableUtils.closeQuietly(client);
+ }
+ }
+}
[39/52] [abbrv] hadoop git commit: YARN-3238. Connection timeouts to
nodemanagers are retried at multiple levels. Contributed by Jason Lowe
Posted by zh...@apache.org.
YARN-3238. Connection timeouts to nodemanagers are retried at multiple
levels. Contributed by Jason Lowe
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/92d67ace
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/92d67ace
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/92d67ace
Branch: refs/heads/HDFS-7285
Commit: 92d67ace3248930c0c0335070cc71a480c566a36
Parents: 8b465b4
Author: Xuan <xg...@apache.org>
Authored: Sat Feb 21 16:06:12 2015 -0800
Committer: Xuan <xg...@apache.org>
Committed: Sat Feb 21 16:06:12 2015 -0800
----------------------------------------------------------------------
hadoop-yarn-project/CHANGES.txt | 3 +++
.../src/main/java/org/apache/hadoop/yarn/client/ServerProxy.java | 1 -
2 files changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/92d67ace/hadoop-yarn-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 359e647..1650a20 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -637,6 +637,9 @@ Release 2.7.0 - UNRELEASED
YARN-3194. RM should handle NMContainerStatuses sent by NM while
registering if NM is Reconnected node (Rohith via jlowe)
+ YARN-3238. Connection timeouts to nodemanagers are retried at
+ multiple levels (Jason Lowe via xgong)
+
Release 2.6.0 - 2014-11-18
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/92d67ace/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/ServerProxy.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/ServerProxy.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/ServerProxy.java
index b6fea62..6024560 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/ServerProxy.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/ServerProxy.java
@@ -72,7 +72,6 @@ public class ServerProxy {
exceptionToPolicyMap.put(ConnectException.class, retryPolicy);
exceptionToPolicyMap.put(NoRouteToHostException.class, retryPolicy);
exceptionToPolicyMap.put(UnknownHostException.class, retryPolicy);
- exceptionToPolicyMap.put(ConnectTimeoutException.class, retryPolicy);
exceptionToPolicyMap.put(RetriableException.class, retryPolicy);
exceptionToPolicyMap.put(SocketException.class, retryPolicy);
[50/52] [abbrv] hadoop git commit: HDFS-7716. Erasure Coding: extend
BlockInfo to handle EC info. Contributed by Jing Zhao.
Posted by zh...@apache.org.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
index c4612a3..3a5e66e 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
@@ -24,6 +24,7 @@ import java.util.List;
import com.google.common.annotations.VisibleForTesting;
import org.apache.hadoop.fs.StorageType;
+import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.hdfs.server.protocol.DatanodeStorage;
import org.apache.hadoop.hdfs.server.protocol.DatanodeStorage.State;
@@ -80,10 +81,10 @@ public class DatanodeStorageInfo {
/**
* Iterates over the list of blocks belonging to the data-node.
*/
- class BlockIterator implements Iterator<BlockInfoContiguous> {
- private BlockInfoContiguous current;
+ class BlockIterator implements Iterator<BlockInfo> {
+ private BlockInfo current;
- BlockIterator(BlockInfoContiguous head) {
+ BlockIterator(BlockInfo head) {
this.current = head;
}
@@ -91,8 +92,8 @@ public class DatanodeStorageInfo {
return current != null;
}
- public BlockInfoContiguous next() {
- BlockInfoContiguous res = current;
+ public BlockInfo next() {
+ BlockInfo res = current;
current = current.getNext(current.findStorageInfo(DatanodeStorageInfo.this));
return res;
}
@@ -112,7 +113,7 @@ public class DatanodeStorageInfo {
private volatile long remaining;
private long blockPoolUsed;
- private volatile BlockInfoContiguous blockList = null;
+ private volatile BlockInfo blockList = null;
private int numBlocks = 0;
/** The number of block reports received */
@@ -215,7 +216,7 @@ public class DatanodeStorageInfo {
return blockPoolUsed;
}
- public AddBlockResult addBlock(BlockInfoContiguous b) {
+ public AddBlockResult addBlock(BlockInfo b, Block reportedBlock) {
// First check whether the block belongs to a different storage
// on the same DN.
AddBlockResult result = AddBlockResult.ADDED;
@@ -234,13 +235,21 @@ public class DatanodeStorageInfo {
}
// add to the head of the data-node list
- b.addStorage(this);
+ b.addStorage(this, reportedBlock);
+ insertToList(b);
+ return result;
+ }
+
+ AddBlockResult addBlock(BlockInfoContiguous b) {
+ return addBlock(b, b);
+ }
+
+ public void insertToList(BlockInfo b) {
blockList = b.listInsert(blockList, this);
numBlocks++;
- return result;
}
- public boolean removeBlock(BlockInfoContiguous b) {
+ public boolean removeBlock(BlockInfo b) {
blockList = b.listRemove(blockList, this);
if (b.removeStorage(this)) {
numBlocks--;
@@ -254,16 +263,15 @@ public class DatanodeStorageInfo {
return numBlocks;
}
- Iterator<BlockInfoContiguous> getBlockIterator() {
+ Iterator<BlockInfo> getBlockIterator() {
return new BlockIterator(blockList);
-
}
/**
* Move block to the head of the list of blocks belonging to the data-node.
* @return the index of the head of the blockList
*/
- int moveBlockToHead(BlockInfoContiguous b, int curIndex, int headIndex) {
+ int moveBlockToHead(BlockInfo b, int curIndex, int headIndex) {
blockList = b.moveBlockToHead(blockList, this, curIndex, headIndex);
return curIndex;
}
@@ -273,7 +281,7 @@ public class DatanodeStorageInfo {
* @return the head of the blockList
*/
@VisibleForTesting
- BlockInfoContiguous getBlockListHeadForTesting(){
+ BlockInfo getBlockListHeadForTesting(){
return blockList;
}
@@ -360,6 +368,6 @@ public class DatanodeStorageInfo {
}
static enum AddBlockResult {
- ADDED, REPLACED, ALREADY_EXIST;
+ ADDED, REPLACED, ALREADY_EXIST
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java
new file mode 100644
index 0000000..f4600cb7
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java
@@ -0,0 +1,119 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.hdfs.server.common.HdfsServerConstants;
+
+/**
+ * ReplicaUnderConstruction contains information about replicas (or blocks
+ * belonging to a block group) while they are under construction.
+ *
+ * The GS, the length and the state of the replica is as reported by the
+ * datanode.
+ *
+ * It is not guaranteed, but expected, that datanodes actually have
+ * corresponding replicas.
+ */
+class ReplicaUnderConstruction extends Block {
+ private final DatanodeStorageInfo expectedLocation;
+ private HdfsServerConstants.ReplicaState state;
+ private boolean chosenAsPrimary;
+
+ ReplicaUnderConstruction(Block block,
+ DatanodeStorageInfo target,
+ HdfsServerConstants.ReplicaState state) {
+ super(block);
+ this.expectedLocation = target;
+ this.state = state;
+ this.chosenAsPrimary = false;
+ }
+
+ /**
+ * Expected block replica location as assigned when the block was allocated.
+ * This defines the pipeline order.
+ * It is not guaranteed, but expected, that the data-node actually has
+ * the replica.
+ */
+ DatanodeStorageInfo getExpectedStorageLocation() {
+ return expectedLocation;
+ }
+
+ /**
+ * Get replica state as reported by the data-node.
+ */
+ HdfsServerConstants.ReplicaState getState() {
+ return state;
+ }
+
+ /**
+ * Whether the replica was chosen for recovery.
+ */
+ boolean getChosenAsPrimary() {
+ return chosenAsPrimary;
+ }
+
+ /**
+ * Set replica state.
+ */
+ void setState(HdfsServerConstants.ReplicaState s) {
+ state = s;
+ }
+
+ /**
+ * Set whether this replica was chosen for recovery.
+ */
+ void setChosenAsPrimary(boolean chosenAsPrimary) {
+ this.chosenAsPrimary = chosenAsPrimary;
+ }
+
+ /**
+ * Is data-node the replica belongs to alive.
+ */
+ boolean isAlive() {
+ return expectedLocation.getDatanodeDescriptor().isAlive;
+ }
+
+ @Override // Block
+ public int hashCode() {
+ return super.hashCode();
+ }
+
+ @Override // Block
+ public boolean equals(Object obj) {
+ // Sufficient to rely on super's implementation
+ return (this == obj) || super.equals(obj);
+ }
+
+ @Override
+ public String toString() {
+ final StringBuilder b = new StringBuilder(50);
+ appendStringTo(b);
+ return b.toString();
+ }
+
+ @Override
+ public void appendStringTo(StringBuilder sb) {
+ sb.append("ReplicaUC[")
+ .append(expectedLocation)
+ .append("|")
+ .append(state)
+ .append("]");
+ }
+}
+
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
index 627abfa..4e1f63a 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
@@ -466,8 +466,8 @@ public class FSDirectory implements Closeable {
* Add a block to the file. Returns a reference to the added block.
*/
BlockInfoContiguous addBlock(String path, INodesInPath inodesInPath,
- Block block, DatanodeStorageInfo[] targets,
- boolean isStriped) throws IOException {
+ Block block, DatanodeStorageInfo[] targets, boolean isStriped)
+ throws IOException {
writeLock();
try {
final INodeFile fileINode = inodesInPath.getLastINode().asFile();
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
index 5b01ebc..db7cb64 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
@@ -3143,8 +3143,8 @@ public class FSNamesystem implements Namesystem, FSNamesystemMBean,
src + ". Returning previously allocated block " + lastBlockInFile);
long offset = pendingFile.computeFileSize();
onRetryBlock[0] = makeLocatedBlock(lastBlockInFile,
- ((BlockInfoContiguousUnderConstruction)lastBlockInFile).getExpectedStorageLocations(),
- offset);
+ ((BlockInfoContiguousUnderConstruction)lastBlockInFile)
+ .getExpectedStorageLocations(), offset);
return new FileState(pendingFile, src, iip);
} else {
// Case 3
@@ -3487,9 +3487,11 @@ public class FSNamesystem implements Namesystem, FSNamesystemMBean,
}
}
- private static boolean isCompleteBlock(String src, BlockInfoContiguous b, int minRepl) {
+ private static boolean isCompleteBlock(String src, BlockInfoContiguous b,
+ int minRepl) {
if (!b.isComplete()) {
- final BlockInfoContiguousUnderConstruction uc = (BlockInfoContiguousUnderConstruction)b;
+ final BlockInfoContiguousUnderConstruction uc =
+ (BlockInfoContiguousUnderConstruction) b;
final int numNodes = b.numNodes();
LOG.info("BLOCK* " + b + " is not COMPLETE (ucState = "
+ uc.getBlockUCState() + ", replication# = " + numNodes
@@ -3976,7 +3978,8 @@ public class FSNamesystem implements Namesystem, FSNamesystemMBean,
throw new AlreadyBeingCreatedException(message);
case UNDER_CONSTRUCTION:
case UNDER_RECOVERY:
- final BlockInfoContiguousUnderConstruction uc = (BlockInfoContiguousUnderConstruction)lastBlock;
+ final BlockInfoContiguousUnderConstruction uc =
+ (BlockInfoContiguousUnderConstruction)lastBlock;
// determine if last block was intended to be truncated
Block recoveryBlock = uc.getTruncateBlock();
boolean truncateRecovery = recoveryBlock != null;
@@ -4086,9 +4089,8 @@ public class FSNamesystem implements Namesystem, FSNamesystemMBean,
blockManager.checkReplication(pendingFile);
}
- @VisibleForTesting
- BlockInfoContiguous getStoredBlock(Block block) {
- return blockManager.getStoredBlock(block);
+ public BlockInfoContiguous getStoredBlock(Block block) {
+ return (BlockInfoContiguous) blockManager.getStoredBlock(block);
}
@Override
@@ -4247,9 +4249,9 @@ public class FSNamesystem implements Namesystem, FSNamesystemMBean,
trimmedTargets.get(i).getStorageInfo(trimmedStorages.get(i));
if (storageInfo != null) {
if(copyTruncate) {
- storageInfo.addBlock(truncatedBlock);
+ storageInfo.addBlock(truncatedBlock, truncatedBlock);
} else {
- storageInfo.addBlock(storedBlock);
+ storageInfo.addBlock(storedBlock, storedBlock);
}
}
}
@@ -4611,7 +4613,7 @@ public class FSNamesystem implements Namesystem, FSNamesystemMBean,
while (it.hasNext()) {
Block b = it.next();
- BlockInfoContiguous blockInfo = blockManager.getStoredBlock(b);
+ BlockInfoContiguous blockInfo = getStoredBlock(b);
if (blockInfo.getBlockCollection().getStoragePolicyID() == lpPolicy.getId()) {
filesToDelete.add(blockInfo.getBlockCollection());
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
index dc9494d..bca2c06 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
@@ -230,7 +230,8 @@ public class NamenodeFsck implements DataEncryptionKeyFactory {
//get blockInfo
Block block = new Block(Block.getBlockId(blockId));
//find which file this block belongs to
- BlockInfoContiguous blockInfo = bm.getStoredBlock(block);
+ BlockInfoContiguous blockInfo = namenode.getNamesystem()
+ .getStoredBlock(block);
if(blockInfo == null) {
out.println("Block "+ blockId +" " + NONEXISTENT_STATUS);
LOG.warn("Block "+ blockId + " " + NONEXISTENT_STATUS);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
index c4cbbc1..87b370a 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
@@ -239,10 +239,12 @@ public class FSImageFormatPBSnapshot {
FileDiff diff = new FileDiff(pbf.getSnapshotId(), copy, null,
pbf.getFileSize());
List<BlockProto> bpl = pbf.getBlocksList();
+ // TODO: also persist striped blocks
BlockInfoContiguous[] blocks = new BlockInfoContiguous[bpl.size()];
for(int j = 0, e = bpl.size(); j < e; ++j) {
Block blk = PBHelper.convert(bpl.get(j));
- BlockInfoContiguous storedBlock = fsn.getBlockManager().getStoredBlock(blk);
+ BlockInfoContiguous storedBlock =
+ (BlockInfoContiguous) fsn.getBlockManager().getStoredBlock(blk);
if(storedBlock == null) {
storedBlock = fsn.getBlockManager().addBlockCollection(
new BlockInfoContiguous(blk, copy.getFileReplication()), file);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
index 5b391c5..9adf639 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
@@ -1548,8 +1548,8 @@ public class DFSTestUtil {
*/
public static DatanodeDescriptor getExpectedPrimaryNode(NameNode nn,
ExtendedBlock blk) {
- BlockManager bm0 = nn.getNamesystem().getBlockManager();
- BlockInfoContiguous storedBlock = bm0.getStoredBlock(blk.getLocalBlock());
+ FSNamesystem fsn = nn.getNamesystem();
+ BlockInfoContiguous storedBlock = fsn.getStoredBlock(blk.getLocalBlock());
assertTrue("Block " + blk + " should be under construction, " +
"got: " + storedBlock,
storedBlock instanceof BlockInfoContiguousUnderConstruction);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java
index 7425c6a..89fd6db 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java
@@ -53,7 +53,7 @@ public class TestBlockInfo {
final DatanodeStorageInfo storage = DFSTestUtil.createDatanodeStorageInfo("storageID", "127.0.0.1");
- boolean added = blockInfo.addStorage(storage);
+ boolean added = blockInfo.addStorage(storage, blockInfo);
Assert.assertTrue(added);
Assert.assertEquals(storage, blockInfo.getStorageInfo(0));
@@ -108,7 +108,7 @@ public class TestBlockInfo {
// list length should be equal to the number of blocks we inserted
LOG.info("Checking list length...");
assertEquals("Length should be MAX_BLOCK", MAX_BLOCKS, dd.numBlocks());
- Iterator<BlockInfoContiguous> it = dd.getBlockIterator();
+ Iterator<BlockInfo> it = dd.getBlockIterator();
int len = 0;
while (it.hasNext()) {
it.next();
@@ -130,7 +130,7 @@ public class TestBlockInfo {
// move head of the list to the head - this should not change the list
LOG.info("Moving head to the head...");
- BlockInfoContiguous temp = dd.getBlockListHeadForTesting();
+ BlockInfo temp = dd.getBlockListHeadForTesting();
curIndex = 0;
headIndex = 0;
dd.moveBlockToHead(temp, curIndex, headIndex);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoStriped.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoStriped.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoStriped.java
new file mode 100644
index 0000000..74ddac0
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoStriped.java
@@ -0,0 +1,219 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.hdfs.DFSTestUtil;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo.AddBlockResult;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.internal.util.reflection.Whitebox;
+
+import static org.apache.hadoop.hdfs.protocol.HdfsConstants.NUM_DATA_BLOCKS;
+import static org.apache.hadoop.hdfs.protocol.HdfsConstants.NUM_PARITY_BLOCKS;
+
+/**
+ * Test {@link BlockInfoStriped}
+ */
+public class TestBlockInfoStriped {
+ private static final int TOTAL_NUM_BLOCKS = NUM_DATA_BLOCKS + NUM_PARITY_BLOCKS;
+ private static final long BASE_ID = -1600;
+ private static final Block baseBlock = new Block(BASE_ID);
+ private BlockInfoStriped info;
+
+ @Before
+ public void setup() {
+ info = new BlockInfoStriped(baseBlock, NUM_DATA_BLOCKS, NUM_PARITY_BLOCKS);
+ }
+
+ private Block[] createReportedBlocks(int num) {
+ Block[] blocks = new Block[num];
+ for (int i = 0; i < num; i++) {
+ blocks[i] = new Block(BASE_ID + i);
+ }
+ return blocks;
+ }
+
+ /**
+ * Test adding storage and reported block
+ */
+ @Test
+ public void testAddStorage() {
+ // first add NUM_DATA_BLOCKS + NUM_PARITY_BLOCKS storages, i.e., a complete
+ // group of blocks/storages
+ DatanodeStorageInfo[] storageInfos = DFSTestUtil.createDatanodeStorageInfos(
+ TOTAL_NUM_BLOCKS);
+ Block[] blocks = createReportedBlocks(TOTAL_NUM_BLOCKS);
+ int i = 0;
+ for (; i < storageInfos.length; i += 2) {
+ info.addStorage(storageInfos[i], blocks[i]);
+ Assert.assertEquals(i/2 + 1, info.numNodes());
+ }
+ i /= 2;
+ for (int j = 1; j < storageInfos.length; j += 2) {
+ Assert.assertTrue(info.addStorage(storageInfos[j], blocks[j]));
+ Assert.assertEquals(i + (j+1)/2, info.numNodes());
+ }
+
+ // check
+ byte[] indices = (byte[]) Whitebox.getInternalState(info, "indices");
+ Assert.assertEquals(TOTAL_NUM_BLOCKS, info.getCapacity());
+ Assert.assertEquals(TOTAL_NUM_BLOCKS, indices.length);
+ i = 0;
+ for (DatanodeStorageInfo storage : storageInfos) {
+ int index = info.findStorageInfo(storage);
+ Assert.assertEquals(i++, index);
+ Assert.assertEquals(index, indices[index]);
+ }
+
+ // the same block is reported from the same storage twice
+ i = 0;
+ for (DatanodeStorageInfo storage : storageInfos) {
+ Assert.assertTrue(info.addStorage(storage, blocks[i++]));
+ }
+ Assert.assertEquals(TOTAL_NUM_BLOCKS, info.getCapacity());
+ Assert.assertEquals(TOTAL_NUM_BLOCKS, info.numNodes());
+ Assert.assertEquals(TOTAL_NUM_BLOCKS, indices.length);
+ i = 0;
+ for (DatanodeStorageInfo storage : storageInfos) {
+ int index = info.findStorageInfo(storage);
+ Assert.assertEquals(i++, index);
+ Assert.assertEquals(index, indices[index]);
+ }
+
+ // the same block is reported from another storage
+ DatanodeStorageInfo[] storageInfos2 = DFSTestUtil.createDatanodeStorageInfos(
+ TOTAL_NUM_BLOCKS * 2);
+ // only add the second half of info2
+ for (i = TOTAL_NUM_BLOCKS; i < storageInfos2.length; i++) {
+ info.addStorage(storageInfos2[i], blocks[i % TOTAL_NUM_BLOCKS]);
+ Assert.assertEquals(i + 1, info.getCapacity());
+ Assert.assertEquals(i + 1, info.numNodes());
+ indices = (byte[]) Whitebox.getInternalState(info, "indices");
+ Assert.assertEquals(i + 1, indices.length);
+ }
+ for (i = TOTAL_NUM_BLOCKS; i < storageInfos2.length; i++) {
+ int index = info.findStorageInfo(storageInfos2[i]);
+ Assert.assertEquals(i++, index);
+ Assert.assertEquals(index - TOTAL_NUM_BLOCKS, indices[index]);
+ }
+ }
+
+ @Test
+ public void testRemoveStorage() {
+ // first add TOTAL_NUM_BLOCKS into the BlockInfoStriped
+ DatanodeStorageInfo[] storages = DFSTestUtil.createDatanodeStorageInfos(
+ TOTAL_NUM_BLOCKS);
+ Block[] blocks = createReportedBlocks(TOTAL_NUM_BLOCKS);
+ for (int i = 0; i < storages.length; i++) {
+ info.addStorage(storages[i], blocks[i]);
+ }
+
+ // remove two storages
+ info.removeStorage(storages[0]);
+ info.removeStorage(storages[2]);
+
+ // check
+ Assert.assertEquals(TOTAL_NUM_BLOCKS, info.getCapacity());
+ Assert.assertEquals(TOTAL_NUM_BLOCKS - 2, info.numNodes());
+ byte[] indices = (byte[]) Whitebox.getInternalState(info, "indices");
+ for (int i = 0; i < storages.length; i++) {
+ int index = info.findStorageInfo(storages[i]);
+ if (i != 0 && i != 2) {
+ Assert.assertEquals(i, index);
+ Assert.assertEquals(index, indices[index]);
+ } else {
+ Assert.assertEquals(-1, index);
+ Assert.assertEquals(-1, indices[i]);
+ }
+ }
+
+ // the same block is reported from another storage
+ DatanodeStorageInfo[] storages2 = DFSTestUtil.createDatanodeStorageInfos(
+ TOTAL_NUM_BLOCKS * 2);
+ for (int i = TOTAL_NUM_BLOCKS; i < storages2.length; i++) {
+ info.addStorage(storages2[i], blocks[i % TOTAL_NUM_BLOCKS]);
+ }
+ // now we should have 8 storages
+ Assert.assertEquals(TOTAL_NUM_BLOCKS * 2 - 2, info.numNodes());
+ Assert.assertEquals(TOTAL_NUM_BLOCKS * 2 - 2, info.getCapacity());
+ indices = (byte[]) Whitebox.getInternalState(info, "indices");
+ Assert.assertEquals(TOTAL_NUM_BLOCKS * 2 - 2, indices.length);
+ int j = TOTAL_NUM_BLOCKS;
+ for (int i = TOTAL_NUM_BLOCKS; i < storages2.length; i++) {
+ int index = info.findStorageInfo(storages2[i]);
+ if (i == TOTAL_NUM_BLOCKS || i == TOTAL_NUM_BLOCKS + 2) {
+ Assert.assertEquals(i - TOTAL_NUM_BLOCKS, index);
+ } else {
+ Assert.assertEquals(j++, index);
+ }
+ }
+
+ // remove the storages from storages2
+ for (int i = 0; i < TOTAL_NUM_BLOCKS; i++) {
+ info.removeStorage(storages2[i + TOTAL_NUM_BLOCKS]);
+ }
+ // now we should have 3 storages
+ Assert.assertEquals(TOTAL_NUM_BLOCKS - 2, info.numNodes());
+ Assert.assertEquals(TOTAL_NUM_BLOCKS * 2 - 2, info.getCapacity());
+ indices = (byte[]) Whitebox.getInternalState(info, "indices");
+ Assert.assertEquals(TOTAL_NUM_BLOCKS * 2 - 2, indices.length);
+ for (int i = 0; i < TOTAL_NUM_BLOCKS; i++) {
+ if (i == 0 || i == 2) {
+ int index = info.findStorageInfo(storages2[i + TOTAL_NUM_BLOCKS]);
+ Assert.assertEquals(-1, index);
+ } else {
+ int index = info.findStorageInfo(storages[i]);
+ Assert.assertEquals(i, index);
+ }
+ }
+ for (int i = TOTAL_NUM_BLOCKS; i < TOTAL_NUM_BLOCKS * 2 - 2; i++) {
+ Assert.assertEquals(-1, indices[i]);
+ Assert.assertNull(info.getDatanode(i));
+ }
+ }
+
+ @Test
+ public void testReplaceBlock() {
+ DatanodeStorageInfo[] storages = DFSTestUtil.createDatanodeStorageInfos(
+ TOTAL_NUM_BLOCKS);
+ Block[] blocks = createReportedBlocks(TOTAL_NUM_BLOCKS);
+ // add block/storage 0, 2, 4 into the BlockInfoStriped
+ for (int i = 0; i < storages.length; i += 2) {
+ Assert.assertEquals(AddBlockResult.ADDED,
+ storages[i].addBlock(info, blocks[i]));
+ }
+
+ BlockInfoStriped newBlockInfo = new BlockInfoStriped(info);
+ info.replaceBlock(newBlockInfo);
+
+ // make sure the newBlockInfo is correct
+ byte[] indices = (byte[]) Whitebox.getInternalState(newBlockInfo, "indices");
+ for (int i = 0; i < storages.length; i += 2) {
+ int index = newBlockInfo.findStorageInfo(storages[i]);
+ Assert.assertEquals(i, index);
+ Assert.assertEquals(index, indices[i]);
+
+ // make sure the newBlockInfo is added to the linked list of the storage
+ Assert.assertSame(newBlockInfo, storages[i].getBlockListHeadForTesting());
+ Assert.assertEquals(1, storages[i].numBlocks());
+ Assert.assertNull(newBlockInfo.getNext());
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
index f9c2bdc..236a583 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
@@ -378,7 +378,7 @@ public class TestBlockManager {
for (int i = 1; i < pipeline.length; i++) {
DatanodeStorageInfo storage = pipeline[i];
bm.addBlock(storage, blockInfo, null);
- blockInfo.addStorage(storage);
+ blockInfo.addStorage(storage, blockInfo);
}
}
@@ -388,7 +388,7 @@ public class TestBlockManager {
for (DatanodeDescriptor dn : nodes) {
for (DatanodeStorageInfo storage : dn.getStorageInfos()) {
- blockInfo.addStorage(storage);
+ blockInfo.addStorage(storage, blockInfo);
}
}
return blockInfo;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d1ec74d/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
index 24fd81d..1a2a9f0 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
@@ -1234,7 +1234,7 @@ public class TestReplicationPolicy {
when(storage.removeBlock(any(BlockInfoContiguous.class))).thenReturn(true);
when(storage.addBlock(any(BlockInfoContiguous.class))).thenReturn
(DatanodeStorageInfo.AddBlockResult.ADDED);
- ucBlock.addStorage(storage);
+ ucBlock.addStorage(storage, ucBlock);
when(mbc.setLastBlock((BlockInfoContiguous) any(), (DatanodeStorageInfo[]) any()))
.thenReturn(ucBlock);
[13/52] [abbrv] hadoop git commit: HADOOP-11595. Add default
implementation for AbstractFileSystem#truncate. (yliu)
Posted by zh...@apache.org.
HADOOP-11595. Add default implementation for AbstractFileSystem#truncate. (yliu)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/64a83756
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/64a83756
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/64a83756
Branch: refs/heads/HDFS-7285
Commit: 64a83756350d9d0f07b72c84f2719e82cf78ee49
Parents: a19820f
Author: yliu <yl...@apache.org>
Authored: Thu Feb 19 08:26:42 2015 +0800
Committer: yliu <yl...@apache.org>
Committed: Thu Feb 19 08:26:42 2015 +0800
----------------------------------------------------------------------
hadoop-common-project/hadoop-common/CHANGES.txt | 3 +++
.../main/java/org/apache/hadoop/fs/AbstractFileSystem.java | 7 +++++--
.../src/test/java/org/apache/hadoop/fs/TestAfsCheckPath.java | 6 ------
3 files changed, 8 insertions(+), 8 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/64a83756/hadoop-common-project/hadoop-common/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt b/hadoop-common-project/hadoop-common/CHANGES.txt
index e6d560a..c01e3d6 100644
--- a/hadoop-common-project/hadoop-common/CHANGES.txt
+++ b/hadoop-common-project/hadoop-common/CHANGES.txt
@@ -970,6 +970,9 @@ Release 2.7.0 - UNRELEASED
HADOOP-11545. ArrayIndexOutOfBoundsException is thrown with "hadoop
credential list -provider". (Brahma Reddy Battula via aajisaka)
+ HADOOP-11595. Add default implementation for AbstractFileSystem#truncate.
+ (yliu)
+
Release 2.6.1 - UNRELEASED
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/64a83756/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java
index 975cc3c..959d9d5 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java
@@ -642,9 +642,12 @@ public abstract class AbstractFileSystem {
* {@link FileContext#truncate(Path, long)} except that Path f must be for
* this file system.
*/
- public abstract boolean truncate(Path f, long newLength)
+ public boolean truncate(Path f, long newLength)
throws AccessControlException, FileNotFoundException,
- UnresolvedLinkException, IOException;
+ UnresolvedLinkException, IOException {
+ throw new UnsupportedOperationException(getClass().getSimpleName()
+ + " doesn't support truncate");
+ }
/**
* The specification of this method matches that of
http://git-wip-us.apache.org/repos/asf/hadoop/blob/64a83756/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestAfsCheckPath.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestAfsCheckPath.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestAfsCheckPath.java
index 6b9378d..3bd14f1 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestAfsCheckPath.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestAfsCheckPath.java
@@ -141,12 +141,6 @@ public class TestAfsCheckPath {
}
@Override
- public boolean truncate(Path f, long newLength) throws IOException {
- // deliberately empty
- return false;
- }
-
- @Override
public void renameInternal(Path src, Path dst) throws IOException {
// deliberately empty
}
[25/52] [abbrv] hadoop git commit: YARN-3194. RM should handle
NMContainerStatuses sent by NM while registering if NM is Reconnected node.
Contributed by Rohith
Posted by zh...@apache.org.
YARN-3194. RM should handle NMContainerStatuses sent by NM while registering if NM is Reconnected node. Contributed by Rohith
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/a64dd3d2
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/a64dd3d2
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/a64dd3d2
Branch: refs/heads/HDFS-7285
Commit: a64dd3d24bfcb9af21eb63869924f6482b147fd3
Parents: 7ae5255
Author: Jason Lowe <jl...@apache.org>
Authored: Fri Feb 20 15:08:48 2015 +0000
Committer: Jason Lowe <jl...@apache.org>
Committed: Fri Feb 20 15:10:10 2015 +0000
----------------------------------------------------------------------
hadoop-yarn-project/CHANGES.txt | 3 +
.../resourcemanager/ResourceTrackerService.java | 9 +-
.../resourcemanager/rmnode/RMNodeImpl.java | 111 ++++++++++-------
.../rmnode/RMNodeReconnectEvent.java | 9 +-
.../resourcemanager/TestApplicationCleanup.java | 121 +++++++++++++++++++
.../resourcemanager/TestRMNodeTransitions.java | 4 +-
6 files changed, 209 insertions(+), 48 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a64dd3d2/hadoop-yarn-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index cac6680..8ec2409 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -626,6 +626,9 @@ Release 2.7.0 - UNRELEASED
YARN-933. Fixed InvalidStateTransitonException at FINAL_SAVING state in
RMApp. (Rohith Sharmaks via jianhe)
+ YARN-3194. RM should handle NMContainerStatuses sent by NM while
+ registering if NM is Reconnected node (Rohith via jlowe)
+
Release 2.6.0 - 2014-11-18
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a64dd3d2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
index 61a0349..0de556b 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
@@ -312,9 +312,12 @@ public class ResourceTrackerService extends AbstractService implements
} else {
LOG.info("Reconnect from the node at: " + host);
this.nmLivelinessMonitor.unregister(nodeId);
- this.rmContext.getDispatcher().getEventHandler().handle(
- new RMNodeReconnectEvent(nodeId, rmNode,
- request.getRunningApplications()));
+ this.rmContext
+ .getDispatcher()
+ .getEventHandler()
+ .handle(
+ new RMNodeReconnectEvent(nodeId, rmNode, request
+ .getRunningApplications(), request.getNMContainerStatuses()));
}
// On every node manager register we will be clearing NMToken keys if
// present for any running application.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a64dd3d2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
index 1bc98b2..9701775 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
@@ -601,6 +601,8 @@ public class RMNodeImpl implements RMNode, EventHandler<RMNodeEvent> {
rmNode.httpAddress = newNode.getHttpAddress();
rmNode.totalCapability = newNode.getTotalCapability();
+ handleNMContainerStatus(reconnectEvent.getNMContainerStatuses(), rmNode);
+
// Reset heartbeat ID since node just restarted.
rmNode.getLastNodeHeartBeatResponse().setResponseId(0);
}
@@ -622,6 +624,26 @@ public class RMNodeImpl implements RMNode, EventHandler<RMNodeEvent> {
}
}
+
+ private void handleNMContainerStatus(
+ List<NMContainerStatus> nmContainerStatuses, RMNodeImpl rmnode) {
+ List<ContainerStatus> containerStatuses =
+ new ArrayList<ContainerStatus>();
+ for (NMContainerStatus nmContainerStatus : nmContainerStatuses) {
+ containerStatuses.add(createContainerStatus(nmContainerStatus));
+ }
+ rmnode.handleContainerStatus(containerStatuses);
+ }
+
+ private ContainerStatus createContainerStatus(
+ NMContainerStatus remoteContainer) {
+ ContainerStatus cStatus =
+ ContainerStatus.newInstance(remoteContainer.getContainerId(),
+ remoteContainer.getContainerState(),
+ remoteContainer.getDiagnostics(),
+ remoteContainer.getContainerExitStatus());
+ return cStatus;
+ }
}
public static class UpdateNodeResourceWhenRunningTransition
@@ -747,49 +769,8 @@ public class RMNodeImpl implements RMNode, EventHandler<RMNodeEvent> {
return NodeState.UNHEALTHY;
}
- // Filter the map to only obtain just launched containers and finished
- // containers.
- List<ContainerStatus> newlyLaunchedContainers =
- new ArrayList<ContainerStatus>();
- List<ContainerStatus> completedContainers =
- new ArrayList<ContainerStatus>();
- for (ContainerStatus remoteContainer : statusEvent.getContainers()) {
- ContainerId containerId = remoteContainer.getContainerId();
-
- // Don't bother with containers already scheduled for cleanup, or for
- // applications already killed. The scheduler doens't need to know any
- // more about this container
- if (rmNode.containersToClean.contains(containerId)) {
- LOG.info("Container " + containerId + " already scheduled for " +
- "cleanup, no further processing");
- continue;
- }
- if (rmNode.finishedApplications.contains(containerId
- .getApplicationAttemptId().getApplicationId())) {
- LOG.info("Container " + containerId
- + " belongs to an application that is already killed,"
- + " no further processing");
- continue;
- }
+ rmNode.handleContainerStatus(statusEvent.getContainers());
- // Process running containers
- if (remoteContainer.getState() == ContainerState.RUNNING) {
- if (!rmNode.launchedContainers.contains(containerId)) {
- // Just launched container. RM knows about it the first time.
- rmNode.launchedContainers.add(containerId);
- newlyLaunchedContainers.add(remoteContainer);
- }
- } else {
- // A finished container
- rmNode.launchedContainers.remove(containerId);
- completedContainers.add(remoteContainer);
- }
- }
- if(newlyLaunchedContainers.size() != 0
- || completedContainers.size() != 0) {
- rmNode.nodeUpdateQueue.add(new UpdatedContainerInfo
- (newlyLaunchedContainers, completedContainers));
- }
if(rmNode.nextHeartBeat) {
rmNode.nextHeartBeat = false;
rmNode.context.getDispatcher().getEventHandler().handle(
@@ -874,4 +855,50 @@ public class RMNodeImpl implements RMNode, EventHandler<RMNodeEvent> {
}
return nlm.getLabelsOnNode(nodeId);
}
+
+ private void handleContainerStatus(List<ContainerStatus> containerStatuses) {
+ // Filter the map to only obtain just launched containers and finished
+ // containers.
+ List<ContainerStatus> newlyLaunchedContainers =
+ new ArrayList<ContainerStatus>();
+ List<ContainerStatus> completedContainers =
+ new ArrayList<ContainerStatus>();
+ for (ContainerStatus remoteContainer : containerStatuses) {
+ ContainerId containerId = remoteContainer.getContainerId();
+
+ // Don't bother with containers already scheduled for cleanup, or for
+ // applications already killed. The scheduler doens't need to know any
+ // more about this container
+ if (containersToClean.contains(containerId)) {
+ LOG.info("Container " + containerId + " already scheduled for "
+ + "cleanup, no further processing");
+ continue;
+ }
+ if (finishedApplications.contains(containerId.getApplicationAttemptId()
+ .getApplicationId())) {
+ LOG.info("Container " + containerId
+ + " belongs to an application that is already killed,"
+ + " no further processing");
+ continue;
+ }
+
+ // Process running containers
+ if (remoteContainer.getState() == ContainerState.RUNNING) {
+ if (!launchedContainers.contains(containerId)) {
+ // Just launched container. RM knows about it the first time.
+ launchedContainers.add(containerId);
+ newlyLaunchedContainers.add(remoteContainer);
+ }
+ } else {
+ // A finished container
+ launchedContainers.remove(containerId);
+ completedContainers.add(remoteContainer);
+ }
+ }
+ if (newlyLaunchedContainers.size() != 0 || completedContainers.size() != 0) {
+ nodeUpdateQueue.add(new UpdatedContainerInfo(newlyLaunchedContainers,
+ completedContainers));
+ }
+ }
+
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a64dd3d2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeReconnectEvent.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeReconnectEvent.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeReconnectEvent.java
index ebbac9a..0bea44b 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeReconnectEvent.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeReconnectEvent.java
@@ -22,16 +22,19 @@ import java.util.List;
import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.api.records.NodeId;
+import org.apache.hadoop.yarn.server.api.protocolrecords.NMContainerStatus;
public class RMNodeReconnectEvent extends RMNodeEvent {
private RMNode reconnectedNode;
private List<ApplicationId> runningApplications;
+ private List<NMContainerStatus> containerStatuses;
public RMNodeReconnectEvent(NodeId nodeId, RMNode newNode,
- List<ApplicationId> runningApps) {
+ List<ApplicationId> runningApps, List<NMContainerStatus> containerReports) {
super(nodeId, RMNodeEventType.RECONNECTED);
reconnectedNode = newNode;
runningApplications = runningApps;
+ containerStatuses = containerReports;
}
public RMNode getReconnectedNode() {
@@ -41,4 +44,8 @@ public class RMNodeReconnectEvent extends RMNodeEvent {
public List<ApplicationId> getRunningApplications() {
return runningApplications;
}
+
+ public List<NMContainerStatus> getNMContainerStatuses() {
+ return containerStatuses;
+ }
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a64dd3d2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
index 891130f..6e08aeb 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
@@ -28,7 +28,9 @@ import java.util.Map;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest;
import org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.api.records.Container;
import org.apache.hadoop.yarn.api.records.ContainerId;
@@ -48,6 +50,7 @@ import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp;
import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppState;
import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttempt;
import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptState;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler;
import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.SchedulerEvent;
import org.apache.hadoop.yarn.server.utils.BuilderUtils;
import org.apache.log4j.Level;
@@ -478,6 +481,124 @@ public class TestApplicationCleanup {
rm1.stop();
}
+ // The test verifies processing of NMContainerStatuses which are sent during
+ // NM registration.
+ // 1. Start the cluster-RM,NM,Submit app with 1024MB,Launch & register AM
+ // 2. AM sends ResourceRequest for 1 container with memory 2048MB.
+ // 3. Verify for number of container allocated by RM
+ // 4. Verify Memory Usage by cluster, it should be 3072. AM memory + requested
+ // memory. 1024 + 2048=3072
+ // 5. Re-register NM by sending completed container status
+ // 6. Verify for Memory Used, it should be 1024
+ // 7. Send AM heatbeat to RM. Allocated response should contain completed
+ // container.
+ @Test(timeout = 60000)
+ public void testProcessingNMContainerStatusesOnNMRestart() throws Exception {
+ conf.setInt(YarnConfiguration.RM_AM_MAX_ATTEMPTS, 1);
+ MemoryRMStateStore memStore = new MemoryRMStateStore();
+ memStore.init(conf);
+
+ // 1. Start the cluster-RM,NM,Submit app with 1024MB,Launch & register AM
+ MockRM rm1 = new MockRM(conf, memStore);
+ rm1.start();
+ int nmMemory = 8192;
+ int amMemory = 1024;
+ int containerMemory = 2048;
+ MockNM nm1 =
+ new MockNM("127.0.0.1:1234", nmMemory, rm1.getResourceTrackerService());
+ nm1.registerNode();
+
+ RMApp app0 = rm1.submitApp(amMemory);
+ MockAM am0 = MockRM.launchAndRegisterAM(app0, rm1, nm1);
+
+ // 2. AM sends ResourceRequest for 1 container with memory 2048MB.
+ int noOfContainers = 1;
+ List<Container> allocateContainers =
+ am0.allocateAndWaitForContainers(noOfContainers, containerMemory, nm1);
+
+ // 3. Verify for number of container allocated by RM
+ Assert.assertEquals(noOfContainers, allocateContainers.size());
+ Container container = allocateContainers.get(0);
+
+ nm1.nodeHeartbeat(am0.getApplicationAttemptId(), 1, ContainerState.RUNNING);
+ nm1.nodeHeartbeat(am0.getApplicationAttemptId(), container.getId()
+ .getContainerId(), ContainerState.RUNNING);
+
+ rm1.waitForState(app0.getApplicationId(), RMAppState.RUNNING);
+
+ // 4. Verify Memory Usage by cluster, it should be 3072. AM memory +
+ // requested memory. 1024 + 2048=3072
+ ResourceScheduler rs = rm1.getRMContext().getScheduler();
+ int allocatedMB = rs.getRootQueueMetrics().getAllocatedMB();
+ Assert.assertEquals(amMemory + containerMemory, allocatedMB);
+
+ // 5. Re-register NM by sending completed container status
+ List<NMContainerStatus> nMContainerStatusForApp =
+ createNMContainerStatusForApp(am0);
+ nm1.registerNode(nMContainerStatusForApp,
+ Arrays.asList(app0.getApplicationId()));
+
+ waitForClusterMemory(nm1, rs, amMemory);
+
+ // 6. Verify for Memory Used, it should be 1024
+ Assert.assertEquals(amMemory, rs.getRootQueueMetrics().getAllocatedMB());
+
+ // 7. Send AM heatbeat to RM. Allocated response should contain completed
+ // container
+ AllocateRequest req =
+ AllocateRequest.newInstance(0, 0F, new ArrayList<ResourceRequest>(),
+ new ArrayList<ContainerId>(), null);
+ AllocateResponse allocate = am0.allocate(req);
+ List<ContainerStatus> completedContainersStatuses =
+ allocate.getCompletedContainersStatuses();
+ Assert.assertEquals(noOfContainers, completedContainersStatuses.size());
+
+ // Application clean up should happen Cluster memory used is 0
+ nm1.nodeHeartbeat(am0.getApplicationAttemptId(), 1, ContainerState.COMPLETE);
+ waitForClusterMemory(nm1, rs, 0);
+
+ rm1.stop();
+ }
+
+ private void waitForClusterMemory(MockNM nm1, ResourceScheduler rs,
+ int clusterMemory) throws Exception, InterruptedException {
+ int counter = 0;
+ while (rs.getRootQueueMetrics().getAllocatedMB() != clusterMemory) {
+ nm1.nodeHeartbeat(true);
+
+ Thread.sleep(100);
+ if (counter++ == 50) {
+ Assert.fail("Wait for cluster memory is timed out.Expected="
+ + clusterMemory + " Actual="
+ + rs.getRootQueueMetrics().getAllocatedMB());
+ }
+ }
+ }
+
+ public static List<NMContainerStatus> createNMContainerStatusForApp(MockAM am) {
+ List<NMContainerStatus> list = new ArrayList<NMContainerStatus>();
+ NMContainerStatus amContainer =
+ createNMContainerStatus(am.getApplicationAttemptId(), 1,
+ ContainerState.RUNNING, 1024);
+ NMContainerStatus completedContainer =
+ createNMContainerStatus(am.getApplicationAttemptId(), 2,
+ ContainerState.COMPLETE, 2048);
+ list.add(amContainer);
+ list.add(completedContainer);
+ return list;
+ }
+
+ public static NMContainerStatus createNMContainerStatus(
+ ApplicationAttemptId appAttemptId, int id, ContainerState containerState,
+ int memory) {
+ ContainerId containerId = ContainerId.newContainerId(appAttemptId, id);
+ NMContainerStatus containerReport =
+ NMContainerStatus.newInstance(containerId, containerState,
+ Resource.newInstance(memory, 1), "recover container", 0,
+ Priority.newInstance(0), 0);
+ return containerReport;
+ }
+
public static void main(String[] args) throws Exception {
TestApplicationCleanup t = new TestApplicationCleanup();
t.testAppCleanup();
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a64dd3d2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
index d877e25..c6da3fd 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java
@@ -540,7 +540,7 @@ public class TestRMNodeTransitions {
int initialUnhealthy = cm.getUnhealthyNMs();
int initialDecommissioned = cm.getNumDecommisionedNMs();
int initialRebooted = cm.getNumRebootedNMs();
- node.handle(new RMNodeReconnectEvent(node.getNodeID(), node, null));
+ node.handle(new RMNodeReconnectEvent(node.getNodeID(), node, null, null));
Assert.assertEquals("Active Nodes", initialActive, cm.getNumActiveNMs());
Assert.assertEquals("Lost Nodes", initialLost, cm.getNumLostNMs());
Assert.assertEquals("Unhealthy Nodes",
@@ -614,7 +614,7 @@ public class TestRMNodeTransitions {
Assert.assertEquals(nmVersion1, node.getNodeManagerVersion());
RMNodeImpl reconnectingNode = getRunningNode(nmVersion2);
node.handle(new RMNodeReconnectEvent(node.getNodeID(), reconnectingNode,
- null));
+ null, null));
Assert.assertEquals(nmVersion2, node.getNodeManagerVersion());
}
}
[48/52] [abbrv] hadoop git commit: HADOOP-11541. Raw XOR coder
Posted by zh...@apache.org.
HADOOP-11541. Raw XOR coder
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/14248add
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/14248add
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/14248add
Branch: refs/heads/HDFS-7285
Commit: 14248add59531523481b66f4b1b8ee456117b937
Parents: 2dbe4c5
Author: Kai Zheng <dr...@apache.org>
Authored: Sun Feb 8 01:40:27 2015 +0800
Committer: Zhe Zhang <zh...@apache.org>
Committed: Mon Feb 23 11:20:31 2015 -0800
----------------------------------------------------------------------
.../io/erasurecode/rawcoder/XorRawDecoder.java | 81 ++++++
.../io/erasurecode/rawcoder/XorRawEncoder.java | 61 +++++
.../hadoop/io/erasurecode/TestCoderBase.java | 262 +++++++++++++++++++
.../erasurecode/rawcoder/TestRawCoderBase.java | 96 +++++++
.../erasurecode/rawcoder/TestXorRawCoder.java | 52 ++++
5 files changed, 552 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/14248add/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/XorRawDecoder.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/XorRawDecoder.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/XorRawDecoder.java
new file mode 100644
index 0000000..98307a7
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/XorRawDecoder.java
@@ -0,0 +1,81 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.io.erasurecode.rawcoder;
+
+import java.nio.ByteBuffer;
+
+/**
+ * A raw decoder in XOR code scheme in pure Java, adapted from HDFS-RAID.
+ */
+public class XorRawDecoder extends AbstractRawErasureDecoder {
+
+ @Override
+ protected void doDecode(ByteBuffer[] inputs, int[] erasedIndexes,
+ ByteBuffer[] outputs) {
+ assert(erasedIndexes.length == outputs.length);
+ assert(erasedIndexes.length <= 1);
+
+ int bufSize = inputs[0].remaining();
+ int erasedIdx = erasedIndexes[0];
+
+ // Set the output to zeros.
+ for (int j = 0; j < bufSize; j++) {
+ outputs[0].put(j, (byte) 0);
+ }
+
+ // Process the inputs.
+ for (int i = 0; i < inputs.length; i++) {
+ // Skip the erased location.
+ if (i == erasedIdx) {
+ continue;
+ }
+
+ for (int j = 0; j < bufSize; j++) {
+ outputs[0].put(j, (byte) (outputs[0].get(j) ^ inputs[i].get(j)));
+ }
+ }
+ }
+
+ @Override
+ protected void doDecode(byte[][] inputs, int[] erasedIndexes,
+ byte[][] outputs) {
+ assert(erasedIndexes.length == outputs.length);
+ assert(erasedIndexes.length <= 1);
+
+ int bufSize = inputs[0].length;
+ int erasedIdx = erasedIndexes[0];
+
+ // Set the output to zeros.
+ for (int j = 0; j < bufSize; j++) {
+ outputs[0][j] = 0;
+ }
+
+ // Process the inputs.
+ for (int i = 0; i < inputs.length; i++) {
+ // Skip the erased location.
+ if (i == erasedIdx) {
+ continue;
+ }
+
+ for (int j = 0; j < bufSize; j++) {
+ outputs[0][j] ^= inputs[i][j];
+ }
+ }
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/14248add/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/XorRawEncoder.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/XorRawEncoder.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/XorRawEncoder.java
new file mode 100644
index 0000000..99b20b9
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/XorRawEncoder.java
@@ -0,0 +1,61 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.io.erasurecode.rawcoder;
+
+import java.nio.ByteBuffer;
+
+/**
+ * A raw encoder in XOR code scheme in pure Java, adapted from HDFS-RAID.
+ */
+public class XorRawEncoder extends AbstractRawErasureEncoder {
+
+ @Override
+ protected void doEncode(ByteBuffer[] inputs, ByteBuffer[] outputs) {
+ int bufSize = inputs[0].remaining();
+
+ // Get the first buffer's data.
+ for (int j = 0; j < bufSize; j++) {
+ outputs[0].put(j, inputs[0].get(j));
+ }
+
+ // XOR with everything else.
+ for (int i = 1; i < inputs.length; i++) {
+ for (int j = 0; j < bufSize; j++) {
+ outputs[0].put(j, (byte) (outputs[0].get(j) ^ inputs[i].get(j)));
+ }
+ }
+ }
+
+ @Override
+ protected void doEncode(byte[][] inputs, byte[][] outputs) {
+ int bufSize = inputs[0].length;
+
+ // Get the first buffer's data.
+ for (int j = 0; j < bufSize; j++) {
+ outputs[0][j] = inputs[0][j];
+ }
+
+ // XOR with everything else.
+ for (int i = 1; i < inputs.length; i++) {
+ for (int j = 0; j < bufSize; j++) {
+ outputs[0][j] ^= inputs[i][j];
+ }
+ }
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/14248add/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestCoderBase.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestCoderBase.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestCoderBase.java
new file mode 100644
index 0000000..9482b43
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/TestCoderBase.java
@@ -0,0 +1,262 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.io.erasurecode;
+
+import java.nio.ByteBuffer;
+import java.util.Random;
+
+import static org.junit.Assert.assertArrayEquals;
+
+/**
+ * Test base of common utilities for tests not only raw coders but also block
+ * coders.
+ */
+public abstract class TestCoderBase {
+ protected static Random RAND = new Random();
+
+ protected int numDataUnits;
+ protected int numParityUnits;
+ protected int chunkSize = 16 * 1024;
+
+ // Indexes of erased data units. Will also support test of erasing
+ // parity units
+ protected int[] erasedDataIndexes = new int[] {0};
+
+ // Data buffers are either direct or on-heap, for performance the two cases
+ // may go to different coding implementations.
+ protected boolean usingDirectBuffer = true;
+
+ /**
+ * Compare and verify if erased chunks are equal to recovered chunks
+ * @param erasedChunks
+ * @param recoveredChunks
+ */
+ protected void compareAndVerify(ECChunk[] erasedChunks,
+ ECChunk[] recoveredChunks) {
+ byte[][] erased = ECChunk.toArray(erasedChunks);
+ byte[][] recovered = ECChunk.toArray(recoveredChunks);
+ for (int i = 0; i < erasedChunks.length; ++i) {
+ assertArrayEquals("Decoding and comparing failed.", erased[i],
+ recovered[i]);
+ }
+ }
+
+ /**
+ * Adjust and return erased indexes based on the array of the input chunks (
+ * parity chunks + data chunks).
+ * @return
+ */
+ protected int[] getErasedIndexesForDecoding() {
+ int[] erasedIndexesForDecoding = new int[erasedDataIndexes.length];
+ for (int i = 0; i < erasedDataIndexes.length; ++i) {
+ erasedIndexesForDecoding[i] = erasedDataIndexes[i] + numParityUnits;
+ }
+ return erasedIndexesForDecoding;
+ }
+
+ /**
+ * Return input chunks for decoding, which is parityChunks + dataChunks.
+ * @param dataChunks
+ * @param parityChunks
+ * @return
+ */
+ protected ECChunk[] prepareInputChunksForDecoding(ECChunk[] dataChunks,
+ ECChunk[] parityChunks) {
+ ECChunk[] inputChunks = new ECChunk[numParityUnits + numDataUnits];
+
+ int idx = 0;
+ for (int i = 0; i < numParityUnits; i++) {
+ inputChunks[idx ++] = parityChunks[i];
+ }
+ for (int i = 0; i < numDataUnits; i++) {
+ inputChunks[idx ++] = dataChunks[i];
+ }
+
+ return inputChunks;
+ }
+
+ /**
+ * Have a copy of the data chunks that's to be erased thereafter. The copy
+ * will be used to compare and verify with the to be recovered chunks.
+ * @param dataChunks
+ * @return
+ */
+ protected ECChunk[] copyDataChunksToErase(ECChunk[] dataChunks) {
+ ECChunk[] copiedChunks = new ECChunk[erasedDataIndexes.length];
+
+ int j = 0;
+ for (int i = 0; i < erasedDataIndexes.length; ++i) {
+ copiedChunks[j ++] = cloneChunkWithData(dataChunks[erasedDataIndexes[i]]);
+ }
+
+ return copiedChunks;
+ }
+
+ /**
+ * Erase some data chunks to test the recovering of them
+ * @param dataChunks
+ */
+ protected void eraseSomeDataBlocks(ECChunk[] dataChunks) {
+ for (int i = 0; i < erasedDataIndexes.length; ++i) {
+ eraseDataFromChunk(dataChunks[erasedDataIndexes[i]]);
+ }
+ }
+
+ /**
+ * Erase data from the specified chunks, putting ZERO bytes to the buffers.
+ * @param chunks
+ */
+ protected void eraseDataFromChunks(ECChunk[] chunks) {
+ for (int i = 0; i < chunks.length; ++i) {
+ eraseDataFromChunk(chunks[i]);
+ }
+ }
+
+ /**
+ * Erase data from the specified chunk, putting ZERO bytes to the buffer.
+ * @param chunk
+ */
+ protected void eraseDataFromChunk(ECChunk chunk) {
+ ByteBuffer chunkBuffer = chunk.getBuffer();
+ // erase the data
+ chunkBuffer.position(0);
+ for (int i = 0; i < chunkSize; ++i) {
+ chunkBuffer.put((byte) 0);
+ }
+ chunkBuffer.flip();
+ }
+
+ /**
+ * Clone chunks along with copying the associated data. It respects how the
+ * chunk buffer is allocated, direct or non-direct. It avoids affecting the
+ * original chunk buffers.
+ * @param chunks
+ * @return
+ */
+ protected static ECChunk[] cloneChunksWithData(ECChunk[] chunks) {
+ ECChunk[] results = new ECChunk[chunks.length];
+ for (int i = 0; i < chunks.length; ++i) {
+ results[i] = cloneChunkWithData(chunks[i]);
+ }
+
+ return results;
+ }
+
+ /**
+ * Clone chunk along with copying the associated data. It respects how the
+ * chunk buffer is allocated, direct or non-direct. It avoids affecting the
+ * original chunk.
+ * @param chunk
+ * @return a new chunk
+ */
+ protected static ECChunk cloneChunkWithData(ECChunk chunk) {
+ ByteBuffer srcBuffer = chunk.getBuffer();
+ ByteBuffer destBuffer;
+
+ byte[] bytesArr = new byte[srcBuffer.remaining()];
+ srcBuffer.mark();
+ srcBuffer.get(bytesArr);
+ srcBuffer.reset();
+
+ if (srcBuffer.hasArray()) {
+ destBuffer = ByteBuffer.wrap(bytesArr);
+ } else {
+ destBuffer = ByteBuffer.allocateDirect(srcBuffer.remaining());
+ destBuffer.put(bytesArr);
+ destBuffer.flip();
+ }
+
+ return new ECChunk(destBuffer);
+ }
+
+ /**
+ * Allocate a chunk for output or writing.
+ * @return
+ */
+ protected ECChunk allocateOutputChunk() {
+ ByteBuffer buffer = allocateOutputBuffer();
+
+ return new ECChunk(buffer);
+ }
+
+ /**
+ * Allocate a buffer for output or writing.
+ * @return
+ */
+ protected ByteBuffer allocateOutputBuffer() {
+ ByteBuffer buffer = usingDirectBuffer ?
+ ByteBuffer.allocateDirect(chunkSize) : ByteBuffer.allocate(chunkSize);
+
+ return buffer;
+ }
+
+ /**
+ * Prepare data chunks for each data unit, by generating random data.
+ * @return
+ */
+ protected ECChunk[] prepareDataChunksForEncoding() {
+ ECChunk[] chunks = new ECChunk[numDataUnits];
+ for (int i = 0; i < chunks.length; i++) {
+ chunks[i] = generateDataChunk();
+ }
+
+ return chunks;
+ }
+
+ /**
+ * Generate data chunk by making random data.
+ * @return
+ */
+ protected ECChunk generateDataChunk() {
+ ByteBuffer buffer = allocateOutputBuffer();
+ for (int i = 0; i < chunkSize; i++) {
+ buffer.put((byte) RAND.nextInt(256));
+ }
+ buffer.flip();
+
+ return new ECChunk(buffer);
+ }
+
+ /**
+ * Prepare parity chunks for encoding, each chunk for each parity unit.
+ * @return
+ */
+ protected ECChunk[] prepareParityChunksForEncoding() {
+ ECChunk[] chunks = new ECChunk[numParityUnits];
+ for (int i = 0; i < chunks.length; i++) {
+ chunks[i] = allocateOutputChunk();
+ }
+
+ return chunks;
+ }
+
+ /**
+ * Prepare output chunks for decoding, each output chunk for each erased
+ * chunk.
+ * @return
+ */
+ protected ECChunk[] prepareOutputChunksForDecoding() {
+ ECChunk[] chunks = new ECChunk[erasedDataIndexes.length];
+ for (int i = 0; i < chunks.length; i++) {
+ chunks[i] = allocateOutputChunk();
+ }
+
+ return chunks;
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/14248add/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestRawCoderBase.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestRawCoderBase.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestRawCoderBase.java
new file mode 100644
index 0000000..9119211
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestRawCoderBase.java
@@ -0,0 +1,96 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.io.erasurecode.rawcoder;
+
+import org.apache.hadoop.io.erasurecode.ECChunk;
+import org.apache.hadoop.io.erasurecode.TestCoderBase;
+
+/**
+ * Raw coder test base with utilities.
+ */
+public abstract class TestRawCoderBase extends TestCoderBase {
+ protected Class<? extends RawErasureEncoder> encoderClass;
+ protected Class<? extends RawErasureDecoder> decoderClass;
+
+ /**
+ * Generating source data, encoding, recovering and then verifying.
+ * RawErasureCoder mainly uses ECChunk to pass input and output data buffers,
+ * it supports two kinds of ByteBuffers, one is array backed, the other is
+ * direct ByteBuffer. Have usingDirectBuffer to indicate which case to test.
+ * @param usingDirectBuffer
+ */
+ protected void testCoding(boolean usingDirectBuffer) {
+ // Generate data and encode
+ ECChunk[] dataChunks = prepareDataChunksForEncoding();
+ ECChunk[] parityChunks = prepareParityChunksForEncoding();
+ RawErasureEncoder encoder = createEncoder();
+
+ // Backup all the source chunks for later recovering because some coders
+ // may affect the source data.
+ ECChunk[] clonedDataChunks = cloneChunksWithData(dataChunks);
+ // Make a copy of a strip for later comparing
+ ECChunk[] toEraseDataChunks = copyDataChunksToErase(clonedDataChunks);
+
+ encoder.encode(dataChunks, parityChunks);
+ // Erase the copied sources
+ eraseSomeDataBlocks(clonedDataChunks);
+
+ //Decode
+ ECChunk[] inputChunks = prepareInputChunksForDecoding(clonedDataChunks,
+ parityChunks);
+ ECChunk[] recoveredChunks = prepareOutputChunksForDecoding();
+ RawErasureDecoder decoder = createDecoder();
+ decoder.decode(inputChunks, getErasedIndexesForDecoding(), recoveredChunks);
+
+ //Compare
+ compareAndVerify(toEraseDataChunks, recoveredChunks);
+ }
+
+ /**
+ * Create the raw erasure encoder to test
+ * @return
+ */
+ protected RawErasureEncoder createEncoder() {
+ RawErasureEncoder encoder;
+ try {
+ encoder = encoderClass.newInstance();
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to create encoder", e);
+ }
+
+ encoder.initialize(numDataUnits, numParityUnits, chunkSize);
+ return encoder;
+ }
+
+ /**
+ * create the raw erasure decoder to test
+ * @return
+ */
+ protected RawErasureDecoder createDecoder() {
+ RawErasureDecoder decoder;
+ try {
+ decoder = decoderClass.newInstance();
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to create decoder", e);
+ }
+
+ decoder.initialize(numDataUnits, numParityUnits, chunkSize);
+ return decoder;
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/14248add/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestXorRawCoder.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestXorRawCoder.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestXorRawCoder.java
new file mode 100644
index 0000000..8e59b8a
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestXorRawCoder.java
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.io.erasurecode.rawcoder;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Random;
+
+/**
+ * Test XOR encoding and decoding.
+ */
+public class TestXorRawCoder extends TestRawCoderBase {
+ private static Random RAND = new Random();
+
+ @Before
+ public void setup() {
+ this.encoderClass = XorRawEncoder.class;
+ this.decoderClass = XorRawDecoder.class;
+
+ this.numDataUnits = 10;
+ this.numParityUnits = 1;
+
+ this.erasedDataIndexes = new int[] {0};
+ }
+
+ @Test
+ public void testCodingNoDirectBuffer() {
+ testCoding(false);
+ }
+
+ @Test
+ public void testCodingDirectBuffer() {
+ testCoding(true);
+ }
+
+}
[10/52] [abbrv] hadoop git commit: HDFS-7772. Document hdfs balancer
-exclude/-include option in HDFSCommands.html. Contributed by Xiaoyu Yao.
Posted by zh...@apache.org.
HDFS-7772. Document hdfs balancer -exclude/-include option in HDFSCommands.html. Contributed by Xiaoyu Yao.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/2aa9979a
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/2aa9979a
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/2aa9979a
Branch: refs/heads/HDFS-7285
Commit: 2aa9979a713ab79853885264ad7739c48226aaa4
Parents: f5da556
Author: cnauroth <cn...@apache.org>
Authored: Wed Feb 18 11:46:57 2015 -0800
Committer: cnauroth <cn...@apache.org>
Committed: Wed Feb 18 12:03:07 2015 -0800
----------------------------------------------------------------------
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 +++
.../hadoop/hdfs/server/balancer/Balancer.java | 7 +++----
.../hadoop-hdfs/src/site/markdown/HDFSCommands.md | 17 +++++++++++++----
3 files changed, 19 insertions(+), 8 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2aa9979a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index ec1c837..70eae1c 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -653,6 +653,9 @@ Release 2.7.0 - UNRELEASED
HDFS-7804. correct the haadmin command usage in #HDFSHighAvailabilityWithQJM.html
(Brahma Reddy Battula via umamahesh)
+ HDFS-7772. Document hdfs balancer -exclude/-include option in
+ HDFSCommands.html (Xiaoyu Yao via cnauroth)
+
OPTIMIZATIONS
HDFS-7454. Reduce memory footprint for AclEntries in NameNode.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2aa9979a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
index 5b87cb5..71338e6 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
@@ -170,15 +170,14 @@ public class Balancer {
private static final long GB = 1L << 30; //1GB
private static final long MAX_SIZE_TO_MOVE = 10*GB;
- private static final String USAGE = "Usage: java "
- + Balancer.class.getSimpleName()
+ private static final String USAGE = "Usage: hdfs balancer"
+ "\n\t[-policy <policy>]\tthe balancing policy: "
+ BalancingPolicy.Node.INSTANCE.getName() + " or "
+ BalancingPolicy.Pool.INSTANCE.getName()
+ "\n\t[-threshold <threshold>]\tPercentage of disk capacity"
- + "\n\t[-exclude [-f <hosts-file> | comma-sperated list of hosts]]"
+ + "\n\t[-exclude [-f <hosts-file> | <comma-separated list of hosts>]]"
+ "\tExcludes the specified datanodes."
- + "\n\t[-include [-f <hosts-file> | comma-sperated list of hosts]]"
+ + "\n\t[-include [-f <hosts-file> | <comma-separated list of hosts>]]"
+ "\tIncludes only the specified datanodes."
+ "\n\t[-idleiterations <idleiterations>]"
+ "\tNumber of consecutive idle iterations (-1 for Infinite) before exit.";
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2aa9979a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
index 6a7f34c..0573158 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
@@ -245,13 +245,22 @@ Commands useful for administrators of a hadoop cluster.
### `balancer`
-Usage: `hdfs balancer [-threshold <threshold>] [-policy <policy>] [-idleiterations <idleiterations>]`
+Usage:
+
+ hdfs balancer
+ [-threshold <threshold>]
+ [-policy <policy>]
+ [-exclude [-f <hosts-file> | <comma-separated list of hosts>]]
+ [-include [-f <hosts-file> | <comma-separated list of hosts>]]
+ [-idleiterations <idleiterations>]
| COMMAND\_OPTION | Description |
|:---- |:---- |
-| `-policy` *policy* | `datanode` (default): Cluster is balanced if each datanode is balanced.<br/> `blockpool`: Cluster is balanced if each block pool in each datanode is balanced. |
-| `-threshold` *threshold* | Percentage of disk capacity. This overwrites the default threshold. |
-| `-idleiterations` *iterations* | Maximum number of idle iterations before exit. This overwrites the default idleiterations(5). |
+| `-policy` \<policy\> | `datanode` (default): Cluster is balanced if each datanode is balanced.<br/> `blockpool`: Cluster is balanced if each block pool in each datanode is balanced. |
+| `-threshold` \<threshold\> | Percentage of disk capacity. This overwrites the default threshold. |
+| `-exclude -f` \<hosts-file\> \| \<comma-separated list of hosts\> | Excludes the specified datanodes from being balanced by the balancer. |
+| `-include -f` \<hosts-file\> \| \<comma-separated list of hosts\> | Includes only the specified datanodes to be balanced by the balancer. |
+| `-idleiterations` \<iterations\> | Maximum number of idle iterations before exit. This overwrites the default idleiterations(5). |
Runs a cluster balancing utility. An administrator can simply press Ctrl-C to stop the rebalancing process. See [Balancer](./HdfsUserGuide.html#Balancer) for more details.
[14/52] [abbrv] hadoop git commit: HDFS-7656. Expose truncate API for
HDFS httpfs. (yliu)
Posted by zh...@apache.org.
HDFS-7656. Expose truncate API for HDFS httpfs. (yliu)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/2fd02afe
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/2fd02afe
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/2fd02afe
Branch: refs/heads/HDFS-7285
Commit: 2fd02afeca3710f487b6a039a65c1a666322b229
Parents: 64a8375
Author: yliu <yl...@apache.org>
Authored: Thu Feb 19 08:36:31 2015 +0800
Committer: yliu <yl...@apache.org>
Committed: Thu Feb 19 08:36:31 2015 +0800
----------------------------------------------------------------------
.../hadoop/fs/http/client/HttpFSFileSystem.java | 24 ++++++++++-
.../hadoop/fs/http/server/FSOperations.java | 43 +++++++++++++++++++-
.../http/server/HttpFSParametersProvider.java | 20 +++++++++
.../hadoop/fs/http/server/HttpFSServer.java | 10 +++++
.../fs/http/client/BaseTestHttpFSWith.java | 40 ++++++++++++++++--
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 2 +
6 files changed, 133 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2fd02afe/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java
index 5b079e9..20b212e 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java
@@ -109,12 +109,15 @@ public class HttpFSFileSystem extends FileSystem
public static final String XATTR_VALUE_PARAM = "xattr.value";
public static final String XATTR_SET_FLAG_PARAM = "flag";
public static final String XATTR_ENCODING_PARAM = "encoding";
+ public static final String NEW_LENGTH_PARAM = "newlength";
public static final Short DEFAULT_PERMISSION = 0755;
public static final String ACLSPEC_DEFAULT = "";
public static final String RENAME_JSON = "boolean";
+ public static final String TRUNCATE_JSON = "boolean";
+
public static final String DELETE_JSON = "boolean";
public static final String MKDIRS_JSON = "boolean";
@@ -191,7 +194,7 @@ public class HttpFSFileSystem extends FileSystem
GETHOMEDIRECTORY(HTTP_GET), GETCONTENTSUMMARY(HTTP_GET),
GETFILECHECKSUM(HTTP_GET), GETFILEBLOCKLOCATIONS(HTTP_GET),
INSTRUMENTATION(HTTP_GET), GETACLSTATUS(HTTP_GET),
- APPEND(HTTP_POST), CONCAT(HTTP_POST),
+ APPEND(HTTP_POST), CONCAT(HTTP_POST), TRUNCATE(HTTP_POST),
CREATE(HTTP_PUT), MKDIRS(HTTP_PUT), RENAME(HTTP_PUT), SETOWNER(HTTP_PUT),
SETPERMISSION(HTTP_PUT), SETREPLICATION(HTTP_PUT), SETTIMES(HTTP_PUT),
MODIFYACLENTRIES(HTTP_PUT), REMOVEACLENTRIES(HTTP_PUT),
@@ -568,6 +571,25 @@ public class HttpFSFileSystem extends FileSystem
}
/**
+ * Truncate a file.
+ *
+ * @param f the file to be truncated.
+ * @param newLength The size the file is to be truncated to.
+ *
+ * @throws IOException
+ */
+ @Override
+ public boolean truncate(Path f, long newLength) throws IOException {
+ Map<String, String> params = new HashMap<String, String>();
+ params.put(OP_PARAM, Operation.TRUNCATE.toString());
+ params.put(NEW_LENGTH_PARAM, Long.toString(newLength));
+ HttpURLConnection conn = getConnection(Operation.TRUNCATE.getMethod(),
+ params, f, true);
+ JSONObject json = (JSONObject) HttpFSUtils.jsonParse(conn);
+ return (Boolean) json.get(TRUNCATE_JSON);
+ }
+
+ /**
* Concat existing files together.
* @param f the path to the target destination.
* @param psrcs the paths to the sources to use for the concatenation.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2fd02afe/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
index 4b72a51..bc290a2 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
@@ -364,7 +364,7 @@ public class FSOperations {
}
/**
- * Executor that performs an append FileSystemAccess files system operation.
+ * Executor that performs a concat FileSystemAccess files system operation.
*/
@InterfaceAudience.Private
public static class FSConcat implements FileSystemAccess.FileSystemExecutor<Void> {
@@ -405,6 +405,47 @@ public class FSOperations {
}
/**
+ * Executor that performs a truncate FileSystemAccess files system operation.
+ */
+ @InterfaceAudience.Private
+ public static class FSTruncate implements
+ FileSystemAccess.FileSystemExecutor<JSONObject> {
+ private Path path;
+ private long newLength;
+
+ /**
+ * Creates a Truncate executor.
+ *
+ * @param path target path to truncate to.
+ * @param newLength The size the file is to be truncated to.
+ */
+ public FSTruncate(String path, long newLength) {
+ this.path = new Path(path);
+ this.newLength = newLength;
+ }
+
+ /**
+ * Executes the filesystem operation.
+ *
+ * @param fs filesystem instance to use.
+ *
+ * @return <code>true</code> if the file has been truncated to the desired,
+ * <code>false</code> if a background process of adjusting the
+ * length of the last block has been started, and clients should
+ * wait for it to complete before proceeding with further file
+ * updates.
+ *
+ * @throws IOException thrown if an IO error occured.
+ */
+ @Override
+ public JSONObject execute(FileSystem fs) throws IOException {
+ boolean result = fs.truncate(path, newLength);
+ return toJSON(HttpFSFileSystem.TRUNCATE_JSON.toLowerCase(), result);
+ }
+
+ }
+
+ /**
* Executor that performs a content-summary FileSystemAccess files system operation.
*/
@InterfaceAudience.Private
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2fd02afe/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSParametersProvider.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSParametersProvider.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSParametersProvider.java
index fb06667..73853c4 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSParametersProvider.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSParametersProvider.java
@@ -63,6 +63,7 @@ public class HttpFSParametersProvider extends ParametersProvider {
PARAMS_DEF.put(Operation.INSTRUMENTATION, new Class[]{});
PARAMS_DEF.put(Operation.APPEND, new Class[]{DataParam.class});
PARAMS_DEF.put(Operation.CONCAT, new Class[]{SourcesParam.class});
+ PARAMS_DEF.put(Operation.TRUNCATE, new Class[]{NewLengthParam.class});
PARAMS_DEF.put(Operation.CREATE,
new Class[]{PermissionParam.class, OverwriteParam.class,
ReplicationParam.class, BlockSizeParam.class, DataParam.class});
@@ -290,6 +291,25 @@ public class HttpFSParametersProvider extends ParametersProvider {
}
/**
+ * Class for newlength parameter.
+ */
+ @InterfaceAudience.Private
+ public static class NewLengthParam extends LongParam {
+
+ /**
+ * Parameter name.
+ */
+ public static final String NAME = HttpFSFileSystem.NEW_LENGTH_PARAM;
+
+ /**
+ * Constructor.
+ */
+ public NewLengthParam() {
+ super(NAME, 0l);
+ }
+ }
+
+ /**
* Class for overwrite parameter.
*/
@InterfaceAudience.Private
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2fd02afe/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java
index 9103718..1f903ba 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java
@@ -33,6 +33,7 @@ import org.apache.hadoop.fs.http.server.HttpFSParametersProvider.FilterParam;
import org.apache.hadoop.fs.http.server.HttpFSParametersProvider.GroupParam;
import org.apache.hadoop.fs.http.server.HttpFSParametersProvider.LenParam;
import org.apache.hadoop.fs.http.server.HttpFSParametersProvider.ModifiedTimeParam;
+import org.apache.hadoop.fs.http.server.HttpFSParametersProvider.NewLengthParam;
import org.apache.hadoop.fs.http.server.HttpFSParametersProvider.OffsetParam;
import org.apache.hadoop.fs.http.server.HttpFSParametersProvider.OperationParam;
import org.apache.hadoop.fs.http.server.HttpFSParametersProvider.OverwriteParam;
@@ -427,6 +428,15 @@ public class HttpFSServer {
response = Response.ok().build();
break;
}
+ case TRUNCATE: {
+ Long newLength = params.get(NewLengthParam.NAME, NewLengthParam.class);
+ FSOperations.FSTruncate command =
+ new FSOperations.FSTruncate(path, newLength);
+ JSONObject json = fsExecute(user, command);
+ AUDIT_LOG.info("Truncate [{}] to length [{}]", path, newLength);
+ response = Response.ok(json).type(MediaType.APPLICATION_JSON).build();
+ break;
+ }
default: {
throw new IOException(
MessageFormat.format("Invalid HTTP POST operation [{0}]",
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2fd02afe/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
index f063e33..2cc67d4 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
@@ -24,12 +24,14 @@ import org.apache.hadoop.fs.ContentSummary;
import org.apache.hadoop.fs.FileChecksum;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileSystemTestHelper;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.http.server.HttpFSServerWebApp;
import org.apache.hadoop.fs.permission.AclEntry;
import org.apache.hadoop.fs.permission.AclStatus;
import org.apache.hadoop.fs.permission.FsAction;
import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.hdfs.AppendTestUtil;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSTestUtil;
import org.apache.hadoop.security.UserGroupInformation;
@@ -192,7 +194,7 @@ public abstract class BaseTestHttpFSWith extends HFSTestCase {
Assert.fail("the create should have failed because the file exists " +
"and override is FALSE");
} catch (IOException ex) {
-System.out.println("#");
+ System.out.println("#");
} catch (Exception ex) {
Assert.fail(ex.toString());
}
@@ -222,6 +224,31 @@ System.out.println("#");
}
}
+ private void testTruncate() throws Exception {
+ if (!isLocalFS()) {
+ final short repl = 3;
+ final int blockSize = 1024;
+ final int numOfBlocks = 2;
+ FileSystem fs = FileSystem.get(getProxiedFSConf());
+ fs.mkdirs(getProxiedFSTestDir());
+ Path file = new Path(getProxiedFSTestDir(), "foo.txt");
+ final byte[] data = FileSystemTestHelper.getFileData(
+ numOfBlocks, blockSize);
+ FileSystemTestHelper.createFile(fs, file, data, blockSize, repl);
+
+ final int newLength = blockSize;
+
+ boolean isReady = fs.truncate(file, newLength);
+ Assert.assertTrue("Recovery is not expected.", isReady);
+
+ FileStatus fileStatus = fs.getFileStatus(file);
+ Assert.assertEquals(fileStatus.getLen(), newLength);
+ AppendTestUtil.checkFullFile(fs, file, newLength, data, file.toString());
+
+ fs.close();
+ }
+ }
+
private void testConcat() throws Exception {
Configuration config = getProxiedFSConf();
config.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 1024);
@@ -784,9 +811,10 @@ System.out.println("#");
}
protected enum Operation {
- GET, OPEN, CREATE, APPEND, CONCAT, RENAME, DELETE, LIST_STATUS, WORKING_DIRECTORY, MKDIRS,
- SET_TIMES, SET_PERMISSION, SET_OWNER, SET_REPLICATION, CHECKSUM, CONTENT_SUMMARY,
- FILEACLS, DIRACLS, SET_XATTR, GET_XATTRS, REMOVE_XATTR, LIST_XATTRS
+ GET, OPEN, CREATE, APPEND, TRUNCATE, CONCAT, RENAME, DELETE, LIST_STATUS,
+ WORKING_DIRECTORY, MKDIRS, SET_TIMES, SET_PERMISSION, SET_OWNER,
+ SET_REPLICATION, CHECKSUM, CONTENT_SUMMARY, FILEACLS, DIRACLS, SET_XATTR,
+ GET_XATTRS, REMOVE_XATTR, LIST_XATTRS
}
private void operation(Operation op) throws Exception {
@@ -803,8 +831,12 @@ System.out.println("#");
case APPEND:
testAppend();
break;
+ case TRUNCATE:
+ testTruncate();
+ break;
case CONCAT:
testConcat();
+ break;
case RENAME:
testRename();
break;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2fd02afe/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index 3735e90..80a086a 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -337,6 +337,8 @@ Release 2.7.0 - UNRELEASED
HDFS-7584. Enable Quota Support for Storage Types (See breakdown of
tasks below)
+ HDFS-7656. Expose truncate API for HDFS httpfs. (yliu)
+
IMPROVEMENTS
HDFS-7055. Add tracing to DFSInputStream (cmccabe)
[38/52] [abbrv] hadoop git commit: HDFS-7806. Refactor: move
StorageType from hadoop-hdfs to hadoop-common. (Contributed by Xiaoyu Yao)
Posted by zh...@apache.org.
HDFS-7806. Refactor: move StorageType from hadoop-hdfs to hadoop-common. (Contributed by Xiaoyu Yao)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/8b465b4b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/8b465b4b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/8b465b4b
Branch: refs/heads/HDFS-7285
Commit: 8b465b4b8caed31ca9daeaae108f9a868a30a455
Parents: 709ff99
Author: Arpit Agarwal <ar...@apache.org>
Authored: Sat Feb 21 15:38:35 2015 -0800
Committer: Arpit Agarwal <ar...@apache.org>
Committed: Sat Feb 21 15:38:35 2015 -0800
----------------------------------------------------------------------
.../java/org/apache/hadoop/fs/StorageType.java | 94 ++++++++++++++++++++
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 +
.../apache/hadoop/hdfs/BlockReaderFactory.java | 1 +
.../apache/hadoop/hdfs/BlockReaderLocal.java | 1 +
.../hadoop/hdfs/BlockReaderLocalLegacy.java | 1 +
.../java/org/apache/hadoop/hdfs/DFSClient.java | 1 +
.../org/apache/hadoop/hdfs/DFSInputStream.java | 1 +
.../org/apache/hadoop/hdfs/DFSOutputStream.java | 3 +-
.../hadoop/hdfs/DistributedFileSystem.java | 1 +
.../org/apache/hadoop/hdfs/StorageType.java | 94 --------------------
.../apache/hadoop/hdfs/client/HdfsAdmin.java | 2 +-
.../hdfs/protocol/BlockStoragePolicy.java | 2 +-
.../hadoop/hdfs/protocol/ClientProtocol.java | 2 +-
.../hdfs/protocol/DatanodeInfoWithStorage.java | 2 +-
.../hadoop/hdfs/protocol/LocatedBlock.java | 2 +-
.../QuotaByStorageTypeExceededException.java | 2 +-
.../datatransfer/DataTransferProtocol.java | 2 +-
.../hdfs/protocol/datatransfer/Sender.java | 2 +-
.../ClientNamenodeProtocolTranslatorPB.java | 2 +-
.../apache/hadoop/hdfs/protocolPB/PBHelper.java | 2 +-
.../hadoop/hdfs/server/balancer/Balancer.java | 2 +-
.../hdfs/server/balancer/BalancingPolicy.java | 2 +-
.../hadoop/hdfs/server/balancer/Dispatcher.java | 2 +-
.../server/blockmanagement/BlockManager.java | 2 +-
.../blockmanagement/BlockPlacementPolicy.java | 2 +-
.../BlockPlacementPolicyDefault.java | 2 +-
.../BlockPlacementPolicyWithNodeGroup.java | 3 +-
.../BlockStoragePolicySuite.java | 2 +-
.../blockmanagement/DatanodeDescriptor.java | 2 +-
.../blockmanagement/DatanodeStorageInfo.java | 2 +-
.../hdfs/server/datanode/BPOfferService.java | 2 +-
.../hdfs/server/datanode/BlockReceiver.java | 2 +-
.../hadoop/hdfs/server/datanode/DataNode.java | 2 +-
.../hdfs/server/datanode/DataXceiver.java | 2 +-
.../server/datanode/ReportBadBlockAction.java | 2 +-
.../hdfs/server/datanode/StorageLocation.java | 2 +-
.../server/datanode/fsdataset/FsDatasetSpi.java | 2 +-
.../server/datanode/fsdataset/FsVolumeSpi.java | 2 +-
.../RoundRobinVolumeChoosingPolicy.java | 1 -
.../datanode/fsdataset/impl/FsDatasetImpl.java | 2 +-
.../datanode/fsdataset/impl/FsVolumeImpl.java | 2 +-
.../datanode/fsdataset/impl/FsVolumeList.java | 2 +-
.../apache/hadoop/hdfs/server/mover/Mover.java | 6 +-
.../namenode/DirectoryWithQuotaFeature.java | 2 +-
.../hdfs/server/namenode/FSDirAttrOp.java | 2 +-
.../hdfs/server/namenode/FSDirConcatOp.java | 2 +-
.../hdfs/server/namenode/FSDirectory.java | 2 +-
.../hadoop/hdfs/server/namenode/FSEditLog.java | 2 +-
.../hdfs/server/namenode/FSEditLogOp.java | 2 +-
.../hadoop/hdfs/server/namenode/FSImage.java | 2 +-
.../server/namenode/FSImageFormatPBINode.java | 2 +-
.../hdfs/server/namenode/FSNamesystem.java | 2 +-
.../hdfs/server/namenode/INodeDirectory.java | 4 +-
.../namenode/INodeDirectoryAttributes.java | 2 +-
.../hadoop/hdfs/server/namenode/INodeFile.java | 2 +-
.../hdfs/server/namenode/NameNodeRpcServer.java | 2 +-
.../namenode/QuotaByStorageTypeEntry.java | 2 +-
.../hdfs/server/namenode/QuotaCounts.java | 2 +-
.../snapshot/FSImageFormatPBSnapshot.java | 2 +-
.../snapshot/FileWithSnapshotFeature.java | 2 +-
.../hdfs/server/protocol/BlockCommand.java | 2 +-
.../server/protocol/BlocksWithLocations.java | 2 +-
.../hdfs/server/protocol/DatanodeStorage.java | 2 +-
.../org/apache/hadoop/hdfs/tools/DFSAdmin.java | 2 +-
.../org/apache/hadoop/hdfs/DFSTestUtil.java | 11 ++-
.../org/apache/hadoop/hdfs/MiniDFSCluster.java | 1 +
.../hdfs/MiniDFSClusterWithNodeGroup.java | 4 +-
.../hadoop/hdfs/TestBlockStoragePolicy.java | 1 +
.../hadoop/hdfs/TestDataTransferProtocol.java | 1 +
.../hdfs/TestWriteBlockGetsBlockLengthHint.java | 1 +
.../hadoop/hdfs/protocolPB/TestPBHelper.java | 2 +-
.../hdfs/server/balancer/TestBalancer.java | 6 +-
.../blockmanagement/TestBlockManager.java | 2 +-
.../blockmanagement/TestDatanodeManager.java | 2 +-
.../blockmanagement/TestReplicationPolicy.java | 2 +-
.../TestReplicationPolicyWithNodeGroup.java | 2 +-
.../server/datanode/BlockReportTestBase.java | 3 +-
.../server/datanode/SimulatedFSDataset.java | 2 +-
.../hdfs/server/datanode/TestBlockRecovery.java | 2 +-
.../server/datanode/TestBlockReplacement.java | 2 +-
.../hdfs/server/datanode/TestDataDirs.java | 2 +-
.../server/datanode/TestDirectoryScanner.java | 7 +-
.../hdfs/server/datanode/TestDiskError.java | 2 +-
.../server/datanode/TestSimulatedFSDataset.java | 2 +-
.../hdfs/server/datanode/TestStorageReport.java | 5 +-
.../extdataset/ExternalDatasetImpl.java | 2 +-
.../datanode/extdataset/ExternalVolumeImpl.java | 2 +-
.../fsdataset/impl/LazyPersistTestCase.java | 6 +-
.../fsdataset/impl/TestFsDatasetImpl.java | 2 +-
.../fsdataset/impl/TestFsVolumeList.java | 2 +-
.../fsdataset/impl/TestLazyPersistFiles.java | 5 +-
.../fsdataset/impl/TestScrLazyPersistFiles.java | 6 +-
.../fsdataset/impl/TestWriteToReplica.java | 2 +-
.../hadoop/hdfs/server/mover/TestMover.java | 8 +-
.../hdfs/server/mover/TestStorageMover.java | 2 +-
.../hdfs/server/namenode/TestAddBlockRetry.java | 1 -
.../server/namenode/TestQuotaByStorageType.java | 9 +-
.../hdfs/server/namenode/ha/TestDNFencing.java | 2 +-
98 files changed, 233 insertions(+), 192 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
new file mode 100644
index 0000000..24055d7
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
@@ -0,0 +1,94 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs;
+
+import java.util.Arrays;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+
+/**
+ * Defines the types of supported storage media. The default storage
+ * medium is assumed to be DISK.
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Unstable
+public enum StorageType {
+ DISK(false),
+ SSD(false),
+ ARCHIVE(false),
+ RAM_DISK(true);
+
+ private final boolean isTransient;
+
+ public static final StorageType DEFAULT = DISK;
+
+ public static final StorageType[] EMPTY_ARRAY = {};
+
+ private static final StorageType[] VALUES = values();
+
+ StorageType(boolean isTransient) {
+ this.isTransient = isTransient;
+ }
+
+ public boolean isTransient() {
+ return isTransient;
+ }
+
+ public boolean supportTypeQuota() {
+ return !isTransient;
+ }
+
+ public boolean isMovable() {
+ return !isTransient;
+ }
+
+ public static List<StorageType> asList() {
+ return Arrays.asList(VALUES);
+ }
+
+ public static List<StorageType> getMovableTypes() {
+ return getNonTransientTypes();
+ }
+
+ public static List<StorageType> getTypesSupportingQuota() {
+ return getNonTransientTypes();
+ }
+
+ public static StorageType parseStorageType(int i) {
+ return VALUES[i];
+ }
+
+ public static StorageType parseStorageType(String s) {
+ return StorageType.valueOf(s.toUpperCase(Locale.ENGLISH));
+ }
+
+ private static List<StorageType> getNonTransientTypes() {
+ List<StorageType> nonTransientTypes = new ArrayList<>();
+ for (StorageType t : VALUES) {
+ if ( t.isTransient == false ) {
+ nonTransientTypes.add(t);
+ }
+ }
+ return nonTransientTypes;
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index c8b6610..1ffaec6 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -1023,6 +1023,9 @@ Release 2.7.0 - UNRELEASED
HDFS-7775. Use consistent naming for NN-internal quota related types
and functions. (Xiaoyu Yao via Arpit Agarwal)
+ HDFS-7806. Refactor: move StorageType from hadoop-hdfs to
+ hadoop-common. (Xiaoyu Yao via Arpit Agarwal)
+
Release 2.6.1 - UNRELEASED
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
index 7e40917..ba48c79 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
@@ -29,6 +29,7 @@ import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.net.DomainPeer;
import org.apache.hadoop.hdfs.net.Peer;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
index 5b697e0..8073ea0 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
@@ -25,6 +25,7 @@ import java.util.EnumSet;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.fs.ReadOption;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSClient.Conf;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
import org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocalLegacy.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocalLegacy.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocalLegacy.java
index 3582f67..8e190e7 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocalLegacy.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocalLegacy.java
@@ -33,6 +33,7 @@ import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.ReadOption;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.BlockLocalPathInfo;
import org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
index 3fd4e12..792c2dd 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
@@ -127,6 +127,7 @@ import org.apache.hadoop.fs.Options;
import org.apache.hadoop.fs.Options.ChecksumOpt;
import org.apache.hadoop.fs.ParentNotDirectoryException;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.fs.RemoteIterator;
import org.apache.hadoop.fs.UnresolvedLinkException;
import org.apache.hadoop.fs.VolumeId;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
index 09d6513..c408524 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
@@ -54,6 +54,7 @@ import org.apache.hadoop.fs.ChecksumException;
import org.apache.hadoop.fs.FSInputStream;
import org.apache.hadoop.fs.HasEnhancedByteBufferAccess;
import org.apache.hadoop.fs.ReadOption;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.fs.UnresolvedLinkException;
import org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
index 85d3410..9d7dca9 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
@@ -52,8 +52,9 @@ import org.apache.hadoop.fs.FSOutputSummer;
import org.apache.hadoop.fs.FileAlreadyExistsException;
import org.apache.hadoop.fs.FileEncryptionInfo;
import org.apache.hadoop.fs.ParentNotDirectoryException;
-import org.apache.hadoop.fs.Syncable;
import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.fs.StorageType;
+import org.apache.hadoop.fs.Syncable;
import org.apache.hadoop.hdfs.client.HdfsDataOutputStream;
import org.apache.hadoop.hdfs.client.HdfsDataOutputStream.SyncFlag;
import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
index df286f5..e6bbd92 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
@@ -60,6 +60,7 @@ import org.apache.hadoop.fs.permission.AclEntry;
import org.apache.hadoop.fs.permission.AclStatus;
import org.apache.hadoop.fs.permission.FsPermission;
import org.apache.hadoop.fs.permission.FsAction;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.client.HdfsAdmin;
import org.apache.hadoop.hdfs.client.HdfsDataOutputStream;
import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StorageType.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StorageType.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StorageType.java
deleted file mode 100644
index 745e44b..0000000
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StorageType.java
+++ /dev/null
@@ -1,94 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hadoop.hdfs;
-
-import java.util.Arrays;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Locale;
-
-import org.apache.hadoop.classification.InterfaceAudience;
-import org.apache.hadoop.classification.InterfaceStability;
-
-/**
- * Defines the types of supported storage media. The default storage
- * medium is assumed to be DISK.
- */
-@InterfaceAudience.Public
-@InterfaceStability.Unstable
-public enum StorageType {
- DISK(false),
- SSD(false),
- ARCHIVE(false),
- RAM_DISK(true);
-
- private final boolean isTransient;
-
- public static final StorageType DEFAULT = DISK;
-
- public static final StorageType[] EMPTY_ARRAY = {};
-
- private static final StorageType[] VALUES = values();
-
- StorageType(boolean isTransient) {
- this.isTransient = isTransient;
- }
-
- public boolean isTransient() {
- return isTransient;
- }
-
- public boolean supportTypeQuota() {
- return !isTransient;
- }
-
- public boolean isMovable() {
- return !isTransient;
- }
-
- public static List<StorageType> asList() {
- return Arrays.asList(VALUES);
- }
-
- public static List<StorageType> getMovableTypes() {
- return getNonTransientTypes();
- }
-
- public static List<StorageType> getTypesSupportingQuota() {
- return getNonTransientTypes();
- }
-
- public static StorageType parseStorageType(int i) {
- return VALUES[i];
- }
-
- public static StorageType parseStorageType(String s) {
- return StorageType.valueOf(s.toUpperCase(Locale.ENGLISH));
- }
-
- private static List<StorageType> getNonTransientTypes() {
- List<StorageType> nonTransientTypes = new ArrayList<>();
- for (StorageType t : VALUES) {
- if ( t.isTransient == false ) {
- nonTransientTypes.add(t);
- }
- }
- return nonTransientTypes;
- }
-}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java
index 4db19ca..84499bb 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java
@@ -29,9 +29,9 @@ import org.apache.hadoop.fs.CacheFlag;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSInotifyEventInputStream;
import org.apache.hadoop.hdfs.DistributedFileSystem;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.CacheDirectiveEntry;
import org.apache.hadoop.hdfs.protocol.CacheDirectiveInfo;
import org.apache.hadoop.hdfs.protocol.CachePoolEntry;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockStoragePolicy.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockStoragePolicy.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockStoragePolicy.java
index ef13e0c..4bfb33b 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockStoragePolicy.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockStoragePolicy.java
@@ -25,7 +25,7 @@ import java.util.List;
import com.google.common.annotations.VisibleForTesting;
import org.apache.hadoop.classification.InterfaceAudience;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
index 799e7f7..d4fe903 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
@@ -35,6 +35,7 @@ import org.apache.hadoop.fs.InvalidPathException;
import org.apache.hadoop.fs.Options;
import org.apache.hadoop.fs.Options.Rename;
import org.apache.hadoop.fs.ParentNotDirectoryException;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.fs.UnresolvedLinkException;
import org.apache.hadoop.fs.XAttr;
import org.apache.hadoop.fs.XAttrSetFlag;
@@ -51,7 +52,6 @@ import org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSelector;
import org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException;
import org.apache.hadoop.hdfs.server.namenode.SafeModeException;
import org.apache.hadoop.hdfs.server.protocol.DatanodeStorageReport;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.io.EnumSetWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.retry.AtMostOnce;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfoWithStorage.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfoWithStorage.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfoWithStorage.java
index 54ffec65..18c940b 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfoWithStorage.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfoWithStorage.java
@@ -19,7 +19,7 @@ package org.apache.hadoop.hdfs.protocol;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
@InterfaceAudience.Private
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java
index f14c8da..0d52191 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java
@@ -22,7 +22,7 @@ import java.util.List;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier;
import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo;
import org.apache.hadoop.hdfs.protocol.DatanodeInfoWithStorage;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/QuotaByStorageTypeExceededException.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/QuotaByStorageTypeExceededException.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/QuotaByStorageTypeExceededException.java
index 1de0a30..25084c7 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/QuotaByStorageTypeExceededException.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/QuotaByStorageTypeExceededException.java
@@ -20,7 +20,7 @@ package org.apache.hadoop.hdfs.protocol;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import static org.apache.hadoop.util.StringUtils.TraditionalBinaryPrefix.long2String;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
index 5fc263e..4be42a8 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java
@@ -23,7 +23,7 @@ import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
import org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
index eb30afb..7fea33e 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
@@ -25,7 +25,7 @@ import java.io.IOException;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
import org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.CachingStrategyProto;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
index 771582d..e970293 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
@@ -37,6 +37,7 @@ import org.apache.hadoop.fs.FileAlreadyExistsException;
import org.apache.hadoop.fs.FsServerDefaults;
import org.apache.hadoop.fs.Options.Rename;
import org.apache.hadoop.fs.ParentNotDirectoryException;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.fs.UnresolvedLinkException;
import org.apache.hadoop.fs.XAttr;
import org.apache.hadoop.fs.XAttrSetFlag;
@@ -172,7 +173,6 @@ import org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifie
import org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException;
import org.apache.hadoop.hdfs.server.namenode.SafeModeException;
import org.apache.hadoop.hdfs.server.protocol.DatanodeStorageReport;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.io.EnumSetWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.ipc.ProtobufHelper;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
index 76db47b..ee1603c 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
@@ -36,6 +36,7 @@ import org.apache.hadoop.fs.ContentSummary;
import org.apache.hadoop.fs.CreateFlag;
import org.apache.hadoop.fs.FsServerDefaults;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.fs.XAttr;
import org.apache.hadoop.fs.XAttrSetFlag;
import org.apache.hadoop.fs.permission.AclEntry;
@@ -49,7 +50,6 @@ import org.apache.hadoop.ha.proto.HAServiceProtocolProtos;
import org.apache.hadoop.hdfs.inotify.EventBatch;
import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy;
import org.apache.hadoop.hdfs.DFSUtil;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.inotify.Event;
import org.apache.hadoop.hdfs.inotify.EventBatchList;
import org.apache.hadoop.hdfs.protocol.Block;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
index 71338e6..90212a3 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
@@ -38,10 +38,10 @@ import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSUtil;
import org.apache.hadoop.hdfs.HdfsConfiguration;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.server.balancer.Dispatcher.DDatanode;
import org.apache.hadoop.hdfs.server.balancer.Dispatcher.DDatanode.StorageGroup;
import org.apache.hadoop.hdfs.server.balancer.Dispatcher.Source;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/BalancingPolicy.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/BalancingPolicy.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/BalancingPolicy.java
index 646abd4..a63dacf 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/BalancingPolicy.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/BalancingPolicy.java
@@ -18,7 +18,7 @@
package org.apache.hadoop.hdfs.server.balancer;
import org.apache.hadoop.classification.InterfaceAudience;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.server.protocol.DatanodeStorageReport;
import org.apache.hadoop.hdfs.server.protocol.StorageReport;
import org.apache.hadoop.hdfs.util.EnumCounters;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
index 63e151c..fa17cac 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
@@ -46,9 +46,9 @@ import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSUtil;
import org.apache.hadoop.hdfs.DistributedFileSystem;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
index b05d6a2..58a8b94 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
@@ -40,11 +40,11 @@ import java.util.concurrent.atomic.AtomicLong;
import org.apache.hadoop.HadoopIllegalArgumentException;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSUtil;
import org.apache.hadoop.hdfs.HAUtil;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.BlockListAsLongs;
import org.apache.hadoop.hdfs.protocol.BlockListAsLongs.BlockReportIterator;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
index caeb6ea..9696179 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
@@ -27,9 +27,9 @@ import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy;
import org.apache.hadoop.hdfs.DFSConfigKeys;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.hdfs.protocol.LocatedBlock;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
index 30ab5a6..cb17596 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
@@ -23,10 +23,10 @@ import java.util.*;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSUtil;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.hdfs.protocol.HdfsConstants;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java
index 19fcb14..28a1e56 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyWithNodeGroup.java
@@ -20,8 +20,9 @@ package org.apache.hadoop.hdfs.server.blockmanagement;
import java.util.*;
import org.apache.hadoop.conf.Configuration;
+
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSUtil;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.net.NetworkTopology;
import org.apache.hadoop.net.NetworkTopologyWithNodeGroup;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
index a0eddad..9d8bdb5 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
@@ -21,8 +21,8 @@ import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.fs.XAttr;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.XAttrHelper;
import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy;
import org.apache.hadoop.hdfs.protocol.HdfsConstants;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
index b898013..c0a17b1 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
@@ -35,7 +35,7 @@ import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.DatanodeID;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
index d5ad5fe..c4612a3 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
@@ -23,7 +23,7 @@ import java.util.List;
import com.google.common.annotations.VisibleForTesting;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.hdfs.server.protocol.DatanodeStorage;
import org.apache.hadoop.hdfs.server.protocol.DatanodeStorage.State;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
index 0289167..86c8816 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
@@ -24,8 +24,8 @@ import com.google.common.collect.Sets;
import org.apache.commons.logging.Log;
import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.ha.HAServiceProtocol.HAServiceState;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
index 1db2c78..0a2b650 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
@@ -38,7 +38,7 @@ import java.util.zip.Checksum;
import org.apache.commons.logging.Log;
import org.apache.hadoop.fs.ChecksumException;
import org.apache.hadoop.fs.FSOutputSummer;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
import org.apache.hadoop.hdfs.protocol.HdfsConstants;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
index 4428408..f233e02 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
@@ -103,12 +103,12 @@ import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.LocalFileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.client.BlockReportOptions;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSUtil;
import org.apache.hadoop.hdfs.HDFSPolicyProvider;
import org.apache.hadoop.hdfs.HdfsConfiguration;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.net.DomainPeerServer;
import org.apache.hadoop.hdfs.net.TcpPeerServer;
import org.apache.hadoop.hdfs.protocol.Block;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
index 704993a..6a2250f 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
@@ -45,8 +45,8 @@ import java.security.MessageDigest;
import java.util.Arrays;
import org.apache.commons.logging.Log;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.ExtendedBlockId;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.net.Peer;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
index 07f99f6..fd01a01 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
@@ -20,7 +20,7 @@ package org.apache.hadoop.hdfs.server.datanode;
import java.io.IOException;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
import org.apache.hadoop.hdfs.protocol.LocatedBlock;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
index 2b2da5c..1a553c9 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
@@ -27,7 +27,7 @@ import java.net.URI;
import java.util.regex.Matcher;
import org.apache.hadoop.classification.InterfaceAudience;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.server.common.Util;
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
index 437b795..10c8369 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
@@ -30,8 +30,8 @@ import java.util.Map;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSConfigKeys;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.BlockListAsLongs;
import org.apache.hadoop.hdfs.protocol.BlockLocalPathInfo;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
index 1355e31..2a8f31b 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
@@ -22,7 +22,7 @@ import java.io.File;
import java.io.IOException;
import java.nio.channels.ClosedChannelException;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RoundRobinVolumeChoosingPolicy.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RoundRobinVolumeChoosingPolicy.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RoundRobinVolumeChoosingPolicy.java
index 55a3560..19452d8 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RoundRobinVolumeChoosingPolicy.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RoundRobinVolumeChoosingPolicy.java
@@ -22,7 +22,6 @@ import java.util.List;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.util.DiskChecker.DiskOutOfSpaceException;
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
index 2b9feb5..cc6220a 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
@@ -55,9 +55,9 @@ import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.LocalFileSystem;
import org.apache.hadoop.fs.permission.FsPermission;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.ExtendedBlockId;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.BlockListAsLongs;
import org.apache.hadoop.hdfs.protocol.BlockLocalPathInfo;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
index 5ce2710..297a47d 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
@@ -47,8 +47,8 @@ import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.DF;
import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSConfigKeys;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
import org.apache.hadoop.hdfs.server.datanode.DataStorage;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
index 4573172..b38d41f 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
@@ -32,7 +32,7 @@ import java.util.concurrent.atomic.AtomicReference;
import com.google.common.collect.Lists;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeReference;
import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeSpi;
import org.apache.hadoop.hdfs.server.datanode.fsdataset.VolumeChoosingPolicy;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
index 346d511..89487aa 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java
@@ -28,7 +28,11 @@ import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hdfs.*;
+import org.apache.hadoop.fs.StorageType;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
+import org.apache.hadoop.hdfs.DFSClient;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.DFSUtil;
import org.apache.hadoop.hdfs.protocol.*;
import org.apache.hadoop.hdfs.server.balancer.Dispatcher;
import org.apache.hadoop.hdfs.server.balancer.Dispatcher.*;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java
index 73473c5..01eb22f 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java
@@ -17,13 +17,13 @@
*/
package org.apache.hadoop.hdfs.server.namenode;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.DSQuotaExceededException;
import org.apache.hadoop.hdfs.protocol.HdfsConstants;
import org.apache.hadoop.hdfs.protocol.NSQuotaExceededException;
import org.apache.hadoop.hdfs.protocol.QuotaExceededException;
import org.apache.hadoop.hdfs.protocol.QuotaByStorageTypeExceededException;
import org.apache.hadoop.hdfs.server.namenode.snapshot.Snapshot;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.util.EnumCounters;
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java
index 1fb2688..a3881b8 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java
@@ -19,6 +19,7 @@ package org.apache.hadoop.hdfs.server.namenode;
import org.apache.hadoop.HadoopIllegalArgumentException;
import org.apache.hadoop.fs.PathIsNotDirectoryException;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.fs.UnresolvedLinkException;
import org.apache.hadoop.fs.XAttr;
import org.apache.hadoop.fs.XAttrSetFlag;
@@ -32,7 +33,6 @@ import org.apache.hadoop.hdfs.protocol.QuotaExceededException;
import org.apache.hadoop.hdfs.protocol.SnapshotAccessControlException;
import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager;
import org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.util.EnumCounters;
import org.apache.hadoop.security.AccessControlException;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java
index 7ba8c4e..5ccd3ea 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java
@@ -20,11 +20,11 @@ package org.apache.hadoop.hdfs.server.namenode;
import com.google.common.base.Preconditions;
import org.apache.hadoop.HadoopIllegalArgumentException;
import org.apache.hadoop.fs.permission.FsAction;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy;
import org.apache.hadoop.hdfs.protocol.HdfsFileStatus;
import org.apache.hadoop.hdfs.protocol.QuotaExceededException;
import org.apache.hadoop.hdfs.protocol.SnapshotException;
-import org.apache.hadoop.hdfs.StorageType;
import java.io.IOException;
import java.util.Arrays;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
index ba18a40..f6ab077 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
@@ -31,6 +31,7 @@ import org.apache.hadoop.fs.FileAlreadyExistsException;
import org.apache.hadoop.fs.FileEncryptionInfo;
import org.apache.hadoop.fs.ParentNotDirectoryException;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.fs.UnresolvedLinkException;
import org.apache.hadoop.fs.XAttr;
import org.apache.hadoop.fs.XAttrSetFlag;
@@ -60,7 +61,6 @@ import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo;
import org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite;
import org.apache.hadoop.hdfs.server.common.HdfsServerConstants.BlockUCState;
import org.apache.hadoop.hdfs.server.namenode.INode.BlocksMapUpdateInfo;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.util.ByteArray;
import org.apache.hadoop.hdfs.util.EnumCounters;
import org.apache.hadoop.security.AccessControlException;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
index 7675703..9d487e5 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
@@ -38,6 +38,7 @@ import org.apache.hadoop.fs.XAttr;
import org.apache.hadoop.fs.permission.AclEntry;
import org.apache.hadoop.fs.permission.FsPermission;
import org.apache.hadoop.fs.permission.PermissionStatus;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.CacheDirectiveInfo;
@@ -98,7 +99,6 @@ import org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics;
import org.apache.hadoop.hdfs.server.protocol.NamenodeRegistration;
import org.apache.hadoop.hdfs.server.protocol.NamespaceInfo;
import org.apache.hadoop.hdfs.server.protocol.RemoteEditLogManifest;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.ipc.Server;
import org.apache.hadoop.security.token.delegation.DelegationKey;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
index 3dcce5f..d9ec543 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
@@ -94,6 +94,7 @@ import org.apache.hadoop.fs.permission.AclEntryType;
import org.apache.hadoop.fs.permission.FsAction;
import org.apache.hadoop.fs.permission.FsPermission;
import org.apache.hadoop.fs.permission.PermissionStatus;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DeprecatedUTF8;
import org.apache.hadoop.hdfs.protocol.Block;
@@ -108,7 +109,6 @@ import org.apache.hadoop.hdfs.protocol.proto.XAttrProtos.XAttrEditLogProto;
import org.apache.hadoop.hdfs.protocolPB.PBHelper;
import org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier;
import org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.util.XMLUtils;
import org.apache.hadoop.hdfs.util.XMLUtils.InvalidXmlException;
import org.apache.hadoop.hdfs.util.XMLUtils.Stanza;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
index 32e7486..1aeb0b8 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
@@ -40,6 +40,7 @@ import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSUtil;
import org.apache.hadoop.hdfs.HAUtil;
@@ -66,7 +67,6 @@ import org.apache.hadoop.hdfs.server.protocol.NamenodeCommand;
import org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol;
import org.apache.hadoop.hdfs.server.protocol.NamenodeRegistration;
import org.apache.hadoop.hdfs.server.protocol.NamespaceInfo;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.util.Canceler;
import org.apache.hadoop.hdfs.util.EnumCounters;
import org.apache.hadoop.hdfs.util.MD5FileUtils;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
index 2c0f728..b758458 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
@@ -36,8 +36,8 @@ import org.apache.hadoop.fs.permission.AclEntryType;
import org.apache.hadoop.fs.permission.FsAction;
import org.apache.hadoop.fs.permission.FsPermission;
import org.apache.hadoop.fs.permission.PermissionStatus;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.fs.XAttr;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.HdfsConstants;
import org.apache.hadoop.hdfs.protocol.proto.HdfsProtos.BlockProto;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
index 06d7bd0..120a597 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
@@ -158,6 +158,7 @@ import org.apache.hadoop.fs.permission.AclStatus;
import org.apache.hadoop.fs.permission.FsAction;
import org.apache.hadoop.fs.permission.FsPermission;
import org.apache.hadoop.fs.permission.PermissionStatus;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.ha.HAServiceProtocol.HAServiceState;
import org.apache.hadoop.ha.ServiceFailedException;
import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy;
@@ -251,7 +252,6 @@ import org.apache.hadoop.hdfs.server.protocol.NamespaceInfo;
import org.apache.hadoop.hdfs.server.protocol.StorageReceivedDeletedBlocks;
import org.apache.hadoop.hdfs.server.protocol.StorageReport;
import org.apache.hadoop.hdfs.server.protocol.VolumeFailureSummary;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.io.EnumSetWritable;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.io.Text;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
index 75a7349..ebb8ae4 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
@@ -26,8 +26,9 @@ import java.util.List;
import java.util.Map;
import org.apache.hadoop.fs.PathIsNotDirectoryException;
-import org.apache.hadoop.fs.XAttr;
import org.apache.hadoop.fs.permission.PermissionStatus;
+import org.apache.hadoop.fs.StorageType;
+import org.apache.hadoop.fs.XAttr;
import org.apache.hadoop.hdfs.DFSUtil;
import org.apache.hadoop.hdfs.protocol.QuotaExceededException;
import org.apache.hadoop.hdfs.protocol.SnapshotException;
@@ -37,7 +38,6 @@ import org.apache.hadoop.hdfs.server.namenode.snapshot.DirectorySnapshottableFea
import org.apache.hadoop.hdfs.server.namenode.snapshot.DirectoryWithSnapshotFeature;
import org.apache.hadoop.hdfs.server.namenode.snapshot.DirectoryWithSnapshotFeature.DirectoryDiffList;
import org.apache.hadoop.hdfs.server.namenode.snapshot.Snapshot;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.util.Diff.ListType;
import org.apache.hadoop.hdfs.util.ReadOnlyList;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectoryAttributes.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectoryAttributes.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectoryAttributes.java
index 44d5581..956deae 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectoryAttributes.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectoryAttributes.java
@@ -19,7 +19,7 @@ package org.apache.hadoop.hdfs.server.namenode;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.fs.permission.PermissionStatus;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.util.EnumCounters;
import com.google.common.base.Preconditions;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
index 3743bf0..887a259 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
@@ -30,6 +30,7 @@ import java.util.Set;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.fs.permission.PermissionStatus;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy;
import org.apache.hadoop.hdfs.protocol.QuotaExceededException;
@@ -43,7 +44,6 @@ import org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff;
import org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList;
import org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature;
import org.apache.hadoop.hdfs.server.namenode.snapshot.Snapshot;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.util.LongBitFormat;
import com.google.common.annotations.VisibleForTesting;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
index da34bd2..9ccdb40 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
@@ -51,6 +51,7 @@ import org.apache.hadoop.fs.InvalidPathException;
import org.apache.hadoop.fs.Options;
import org.apache.hadoop.fs.ParentNotDirectoryException;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.fs.UnresolvedLinkException;
import org.apache.hadoop.fs.XAttr;
import org.apache.hadoop.fs.XAttrSetFlag;
@@ -137,7 +138,6 @@ import org.apache.hadoop.hdfs.server.protocol.StorageBlockReport;
import org.apache.hadoop.hdfs.server.protocol.StorageReceivedDeletedBlocks;
import org.apache.hadoop.hdfs.server.protocol.StorageReport;
import org.apache.hadoop.hdfs.server.protocol.VolumeFailureSummary;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.io.EnumSetWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.ipc.ProtobufRpcEngine;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaByStorageTypeEntry.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaByStorageTypeEntry.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaByStorageTypeEntry.java
index fe185f6..7059c64 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaByStorageTypeEntry.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaByStorageTypeEntry.java
@@ -18,7 +18,7 @@
package org.apache.hadoop.hdfs.server.namenode;
import com.google.common.base.Objects;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import java.util.Locale;
public class QuotaByStorageTypeEntry {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java
index 033a5ed..a3b0448 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java
@@ -18,7 +18,7 @@
package org.apache.hadoop.hdfs.server.namenode;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.util.EnumCounters;
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
index f9fc91c..c4cbbc1 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
@@ -37,6 +37,7 @@ import java.util.Map;
import com.google.common.collect.ImmutableList;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.fs.permission.PermissionStatus;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.HdfsConstants;
import org.apache.hadoop.hdfs.protocol.proto.HdfsProtos.BlockProto;
@@ -74,7 +75,6 @@ import org.apache.hadoop.hdfs.server.namenode.snapshot.DirectoryWithSnapshotFeat
import org.apache.hadoop.hdfs.server.namenode.snapshot.DirectoryWithSnapshotFeature.DirectoryDiffList;
import org.apache.hadoop.hdfs.server.namenode.snapshot.Snapshot.Root;
import org.apache.hadoop.hdfs.server.namenode.XAttrFeature;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.util.Diff.ListType;
import org.apache.hadoop.hdfs.util.EnumCounters;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
index 7d1edbe..abd54a2 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
@@ -19,6 +19,7 @@ package org.apache.hadoop.hdfs.server.namenode.snapshot;
import java.util.List;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguous;
import org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite;
@@ -29,7 +30,6 @@ import org.apache.hadoop.hdfs.server.namenode.AclStorage;
import org.apache.hadoop.hdfs.server.namenode.INodeFile;
import org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes;
import org.apache.hadoop.hdfs.server.namenode.QuotaCounts;
-import org.apache.hadoop.hdfs.StorageType;
import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy;
import org.apache.hadoop.hdfs.util.EnumCounters;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockCommand.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockCommand.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockCommand.java
index f17d702..c886bea 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockCommand.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockCommand.java
@@ -21,7 +21,7 @@ import java.util.List;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor.BlockTargetPair;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlocksWithLocations.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlocksWithLocations.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlocksWithLocations.java
index c907f3b..a985dbd 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlocksWithLocations.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlocksWithLocations.java
@@ -19,7 +19,7 @@ package org.apache.hadoop.hdfs.server.protocol;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8b465b4b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeStorage.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeStorage.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeStorage.java
index 4d224d5..0c8b6c93 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeStorage.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeStorage.java
@@ -17,7 +17,7 @@
*/
package org.apache.hadoop.hdfs.server.protocol;
-import org.apache.hadoop.hdfs.StorageType;
+import org.apache.hadoop.fs.StorageType;
import java.util.UUID;
[29/52] [abbrv] hadoop git commit: HDFS-7773. Additional metrics in
HDFS to be accessed via jmx. Contributed by Anu Engineer.
Posted by zh...@apache.org.
HDFS-7773. Additional metrics in HDFS to be accessed via jmx. Contributed by Anu Engineer.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/02e7dec7
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/02e7dec7
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/02e7dec7
Branch: refs/heads/HDFS-7285
Commit: 02e7dec79d2d4f2b801435343219d8fb53ec931f
Parents: 8c6ae0d
Author: cnauroth <cn...@apache.org>
Authored: Fri Feb 20 12:37:48 2015 -0800
Committer: cnauroth <cn...@apache.org>
Committed: Fri Feb 20 12:37:48 2015 -0800
----------------------------------------------------------------------
.../hadoop-common/src/site/markdown/Metrics.md | 5 +++
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 ++
.../hdfs/server/datanode/BlockReceiver.java | 1 +
.../hdfs/server/datanode/DataXceiver.java | 26 +++++++----
.../datanode/metrics/DataNodeMetrics.java | 36 ++++++++++++++--
.../namenode/metrics/NameNodeMetrics.java | 25 +++++++++++
.../server/datanode/TestDataNodeMetrics.java | 45 ++++++++++++++++++++
.../namenode/metrics/TestNameNodeMetrics.java | 20 +++++++++
8 files changed, 148 insertions(+), 13 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/02e7dec7/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md b/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
index 6953c3b..0e0fc09 100644
--- a/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
+++ b/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
@@ -191,6 +191,7 @@ Each metrics record contains tags such as ProcessName, SessionId, and Hostname a
| `GetImageAvgTime` | Average fsimage download time in milliseconds |
| `PutImageNumOps` | Total number of fsimage uploads to SecondaryNameNode |
| `PutImageAvgTime` | Average fsimage upload time in milliseconds |
+| `TotalFileOps`| Total number of file operations performed |
FSNamesystem
------------
@@ -314,6 +315,10 @@ Each metrics record contains tags such as SessionId and Hostname as additional i
| `SendDataPacketBlockedOnNetworkNanosAvgTime` | Average waiting time of sending packets in nanoseconds |
| `SendDataPacketTransferNanosNumOps` | Total number of sending packets |
| `SendDataPacketTransferNanosAvgTime` | Average transfer time of sending packets in nanoseconds |
+| `TotalWriteTime`| Total number of milliseconds spent on write operation |
+| `TotalReadTime` | Total number of milliseconds spent on read operation |
+| `RemoteBytesRead` | Number of bytes read by remote clients |
+| `RemoteBytesWritten` | Number of bytes written by remote clients |
yarn context
============
http://git-wip-us.apache.org/repos/asf/hadoop/blob/02e7dec7/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index 7d9d0ea..5c472a8 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -663,6 +663,9 @@ Release 2.7.0 - UNRELEASED
HDFS-7772. Document hdfs balancer -exclude/-include option in
HDFSCommands.html (Xiaoyu Yao via cnauroth)
+ HDFS-7773. Additional metrics in HDFS to be accessed via jmx.
+ (Anu Engineer via cnauroth)
+
OPTIMIZATIONS
HDFS-7454. Reduce memory footprint for AclEntries in NameNode.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/02e7dec7/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
index 368d80d..1db2c78 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
@@ -658,6 +658,7 @@ class BlockReceiver implements Closeable {
replicaInfo.setLastChecksumAndDataLen(offsetInBlock, lastCrc);
datanode.metrics.incrBytesWritten(len);
+ datanode.metrics.incrTotalWriteTime(duration);
manageWriterOsCache(offsetInBlock);
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/02e7dec7/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
index bb5323a..704993a 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
@@ -86,6 +86,7 @@ import org.apache.hadoop.util.DataChecksum;
import com.google.common.base.Preconditions;
import com.google.protobuf.ByteString;
+import org.apache.hadoop.util.Time;
/**
@@ -480,7 +481,7 @@ class DataXceiver extends Receiver implements Runnable {
final boolean sendChecksum,
final CachingStrategy cachingStrategy) throws IOException {
previousOpClientName = clientName;
-
+ long read = 0;
OutputStream baseStream = getOutputStream();
DataOutputStream out = new DataOutputStream(new BufferedOutputStream(
baseStream, HdfsConstants.SMALL_BUFFER_SIZE));
@@ -515,8 +516,9 @@ class DataXceiver extends Receiver implements Runnable {
// send op status
writeSuccessWithChecksumInfo(blockSender, new DataOutputStream(getOutputStream()));
- long read = blockSender.sendBlock(out, baseStream, null); // send data
-
+ long beginRead = Time.monotonicNow();
+ read = blockSender.sendBlock(out, baseStream, null); // send data
+ long duration = Time.monotonicNow() - beginRead;
if (blockSender.didSendEntireByteRange()) {
// If we sent the entire range, then we should expect the client
// to respond with a Status enum.
@@ -539,6 +541,7 @@ class DataXceiver extends Receiver implements Runnable {
}
datanode.metrics.incrBytesRead((int) read);
datanode.metrics.incrBlocksRead();
+ datanode.metrics.incrTotalReadTime(duration);
} catch ( SocketException ignored ) {
if (LOG.isTraceEnabled()) {
LOG.trace(dnR + ":Ignoring exception while serving " + block + " to " +
@@ -563,7 +566,7 @@ class DataXceiver extends Receiver implements Runnable {
//update metrics
datanode.metrics.addReadBlockOp(elapsed());
- datanode.metrics.incrReadsFromClient(peer.isLocal());
+ datanode.metrics.incrReadsFromClient(peer.isLocal(), read);
}
@Override
@@ -590,7 +593,7 @@ class DataXceiver extends Receiver implements Runnable {
final boolean isClient = !isDatanode;
final boolean isTransfer = stage == BlockConstructionStage.TRANSFER_RBW
|| stage == BlockConstructionStage.TRANSFER_FINALIZED;
-
+ long size = 0;
// check single target for transfer-RBW/Finalized
if (isTransfer && targets.length > 0) {
throw new IOException(stage + " does not support multiple targets "
@@ -796,7 +799,9 @@ class DataXceiver extends Receiver implements Runnable {
+ localAddress + " of size " + block.getNumBytes());
}
-
+ if(isClient) {
+ size = block.getNumBytes();
+ }
} catch (IOException ioe) {
LOG.info("opWriteBlock " + block + " received exception " + ioe);
incrDatanodeNetworkErrors();
@@ -813,7 +818,7 @@ class DataXceiver extends Receiver implements Runnable {
//update metrics
datanode.metrics.addWriteBlockOp(elapsed());
- datanode.metrics.incrWritesFromClient(peer.isLocal());
+ datanode.metrics.incrWritesFromClient(peer.isLocal(), size);
}
@Override
@@ -993,12 +998,15 @@ class DataXceiver extends Receiver implements Runnable {
// send status first
writeSuccessWithChecksumInfo(blockSender, reply);
+
+ long beginRead = Time.monotonicNow();
// send block content to the target
- long read = blockSender.sendBlock(reply, baseStream,
+ long read = blockSender.sendBlock(reply, baseStream,
dataXceiverServer.balanceThrottler);
-
+ long duration = Time.monotonicNow() - beginRead;
datanode.metrics.incrBytesRead((int) read);
datanode.metrics.incrBlocksRead();
+ datanode.metrics.incrTotalReadTime(duration);
LOG.info("Copied " + block + " to " + peer.getRemoteAddressString());
} catch (IOException ioe) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/02e7dec7/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
index 09ad3da..0fbc2ee 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
@@ -50,7 +50,11 @@ import org.apache.hadoop.metrics2.source.JvmMetrics;
public class DataNodeMetrics {
@Metric MutableCounterLong bytesWritten;
+ @Metric("Milliseconds spent writing")
+ MutableCounterLong totalWriteTime;
@Metric MutableCounterLong bytesRead;
+ @Metric("Milliseconds spent reading")
+ MutableCounterLong totalReadTime;
@Metric MutableCounterLong blocksWritten;
@Metric MutableCounterLong blocksRead;
@Metric MutableCounterLong blocksReplicated;
@@ -64,6 +68,10 @@ public class DataNodeMetrics {
@Metric MutableCounterLong writesFromLocalClient;
@Metric MutableCounterLong writesFromRemoteClient;
@Metric MutableCounterLong blocksGetLocalPathInfo;
+ @Metric("Bytes read by remote client")
+ MutableCounterLong remoteBytesRead;
+ @Metric("Bytes written by remote client")
+ MutableCounterLong remoteBytesWritten;
// RamDisk metrics on read/write
@Metric MutableCounterLong ramDiskBlocksWrite;
@@ -262,6 +270,15 @@ public class DataNodeMetrics {
fsyncCount.incr();
}
+ public void incrTotalWriteTime(long timeTaken) {
+ totalWriteTime.incr(timeTaken);
+ }
+
+ public void incrTotalReadTime(long timeTaken) {
+ totalReadTime.incr(timeTaken);
+ }
+
+
public void addPacketAckRoundTripTimeNanos(long latencyNanos) {
packetAckRoundTripTimeNanos.add(latencyNanos);
for (MutableQuantiles q : packetAckRoundTripTimeNanosQuantiles) {
@@ -287,12 +304,23 @@ public class DataNodeMetrics {
DefaultMetricsSystem.shutdown();
}
- public void incrWritesFromClient(boolean local) {
- (local ? writesFromLocalClient : writesFromRemoteClient).incr();
+ public void incrWritesFromClient(boolean local, long size) {
+ if(local) {
+ writesFromLocalClient.incr();
+ } else {
+ writesFromRemoteClient.incr();
+ remoteBytesWritten.incr(size);
+ }
}
- public void incrReadsFromClient(boolean local) {
- (local ? readsFromLocalClient : readsFromRemoteClient).incr();
+ public void incrReadsFromClient(boolean local, long size) {
+
+ if (local) {
+ readsFromLocalClient.incr();
+ } else {
+ readsFromRemoteClient.incr();
+ remoteBytesRead.incr(size);
+ }
}
public void incrVolumeFailures() {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/02e7dec7/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
index 94e845b..31bc164 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
@@ -77,6 +77,31 @@ public class NameNodeMetrics {
@Metric("Number of blockReports from individual storages")
MutableCounterLong storageBlockReportOps;
+ @Metric("Number of file system operations")
+ public long totalFileOps(){
+ return
+ getBlockLocations.value() +
+ createFileOps.value() +
+ filesAppended.value() +
+ addBlockOps.value() +
+ getAdditionalDatanodeOps.value() +
+ filesRenamed.value() +
+ filesTruncated.value() +
+ deleteFileOps.value() +
+ getListingOps.value() +
+ fileInfoOps.value() +
+ getLinkTargetOps.value() +
+ createSnapshotOps.value() +
+ deleteSnapshotOps.value() +
+ allowSnapshotOps.value() +
+ disallowSnapshotOps.value() +
+ renameSnapshotOps.value() +
+ listSnapshottableDirOps.value() +
+ createSymlinkOps.value() +
+ snapshotDiffReportOps.value();
+ }
+
+
@Metric("Journal transactions") MutableRate transactions;
@Metric("Journal syncs") MutableRate syncs;
final MutableQuantiles[] syncsQuantiles;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/02e7dec7/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
index 0b85d35..8a2bacf 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
@@ -47,6 +47,7 @@ import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.util.Time;
import org.junit.Test;
import org.mockito.Mockito;
@@ -246,4 +247,48 @@ public class TestDataNodeMetrics {
DataNodeFaultInjector.instance = new DataNodeFaultInjector();
}
}
+
+ /**
+ * This function ensures that writing causes TotalWritetime to increment
+ * and reading causes totalReadTime to move.
+ * @throws Exception
+ */
+ @Test
+ public void testDataNodeTimeSpend() throws Exception {
+ Configuration conf = new HdfsConfiguration();
+ SimulatedFSDataset.setFactory(conf);
+ MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).build();
+ try {
+ FileSystem fs = cluster.getFileSystem();
+ List<DataNode> datanodes = cluster.getDataNodes();
+ assertEquals(datanodes.size(), 1);
+ DataNode datanode = datanodes.get(0);
+ MetricsRecordBuilder rb = getMetrics(datanode.getMetrics().name());
+ final long LONG_FILE_LEN = 1024 * 1024 * 10;
+
+ long startWriteValue = getLongCounter("TotalWriteTime", rb);
+ long startReadValue = getLongCounter("TotalReadTime", rb);
+
+ for (int x =0; x < 50; x++) {
+ DFSTestUtil.createFile(fs, new Path("/time.txt."+ x),
+ LONG_FILE_LEN, (short) 1, Time.monotonicNow());
+ }
+
+ for (int x =0; x < 50; x++) {
+ String s = DFSTestUtil.readFile(fs, new Path("/time.txt." + x));
+ }
+
+ MetricsRecordBuilder rbNew = getMetrics(datanode.getMetrics().name());
+ long endWriteValue = getLongCounter("TotalWriteTime", rbNew);
+ long endReadValue = getLongCounter("TotalReadTime", rbNew);
+
+ assertTrue(endReadValue > startReadValue);
+ assertTrue(endWriteValue > startWriteValue);
+ } finally {
+ if (cluster != null) {
+ cluster.shutdown();
+ }
+ }
+ }
+
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/02e7dec7/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
index 6c37822..6771ad8 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
@@ -455,4 +455,24 @@ public class TestNameNodeMetrics {
assertQuantileGauges("Syncs1s", rb);
assertQuantileGauges("BlockReport1s", rb);
}
+
+ /**
+ * Test NN ReadOps Count and WriteOps Count
+ */
+ @Test
+ public void testReadWriteOps() throws Exception {
+ MetricsRecordBuilder rb = getMetrics(NN_METRICS);
+ long startWriteCounter = MetricsAsserts.getLongCounter("TransactionsNumOps",
+ rb);
+ Path file1_Path = new Path(TEST_ROOT_DIR_PATH, "ReadData.dat");
+
+ //Perform create file operation
+ createFile(file1_Path, 1024 * 1024,(short)2);
+
+ // Perform read file operation on earlier created file
+ readFile(fs, file1_Path);
+ MetricsRecordBuilder rbNew = getMetrics(NN_METRICS);
+ assertTrue(MetricsAsserts.getLongCounter("TransactionsNumOps", rbNew) >
+ startWriteCounter);
+ }
}
[06/52] [abbrv] hadoop git commit: HDFS-7804. correct the haadmin
command usage in #HDFSHighAvailabilityWithQJM.html (Brahma Reddy Battula via
umamahesh)
Posted by zh...@apache.org.
HDFS-7804. correct the haadmin command usage in #HDFSHighAvailabilityWithQJM.html (Brahma Reddy Battula via umamahesh)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/2ecea5ab
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/2ecea5ab
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/2ecea5ab
Branch: refs/heads/HDFS-7285
Commit: 2ecea5ab741f62e8fd0449251f2ea4a5759f4e77
Parents: 3f56a4c
Author: Uma Maheswara Rao G <um...@apache.org>
Authored: Wed Feb 18 19:24:45 2015 +0530
Committer: Uma Maheswara Rao G <um...@apache.org>
Committed: Wed Feb 18 19:24:45 2015 +0530
----------------------------------------------------------------------
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 +++
.../hadoop-hdfs/src/site/markdown/HDFSHighAvailabilityWithQJM.md | 2 +-
2 files changed, 4 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2ecea5ab/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index 308b61f..ec1c837 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -650,6 +650,9 @@ Release 2.7.0 - UNRELEASED
HDFS-7780. Update use of Iterator to Iterable in DataXceiverServer and
SnapshotDiffInfo. (Ray Chiang via aajisaka)
+ HDFS-7804. correct the haadmin command usage in #HDFSHighAvailabilityWithQJM.html
+ (Brahma Reddy Battula via umamahesh)
+
OPTIMIZATIONS
HDFS-7454. Reduce memory footprint for AclEntries in NameNode.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2ecea5ab/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSHighAvailabilityWithQJM.md
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSHighAvailabilityWithQJM.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSHighAvailabilityWithQJM.md
index a285fde..63813d9 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSHighAvailabilityWithQJM.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSHighAvailabilityWithQJM.md
@@ -382,7 +382,7 @@ You can visit each of the NameNodes' web pages separately by browsing to their c
Now that your HA NameNodes are configured and started, you will have access to some additional commands to administer your HA HDFS cluster. Specifically, you should familiarize yourself with all of the subcommands of the "*hdfs haadmin*" command. Running this command without any additional arguments will display the following usage information:
- Usage: DFSHAAdmin [-ns <nameserviceId>]
+ Usage: haadmin
[-transitionToActive <serviceId>]
[-transitionToStandby <serviceId>]
[-failover [--forcefence] [--forceactive] <serviceId> <serviceId>]
[02/52] [abbrv] hadoop git commit: HADOOP-11593. Convert site
documentation from apt to markdown (stragglers) (Masatake Iwasaki via aw)
Posted by zh...@apache.org.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-tools/hadoop-sls/src/site/apt/SchedulerLoadSimulator.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-sls/src/site/apt/SchedulerLoadSimulator.apt.vm b/hadoop-tools/hadoop-sls/src/site/apt/SchedulerLoadSimulator.apt.vm
deleted file mode 100644
index a8b408c..0000000
--- a/hadoop-tools/hadoop-sls/src/site/apt/SchedulerLoadSimulator.apt.vm
+++ /dev/null
@@ -1,439 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License.
-
- ---
- Yarn Scheduler Load Simulator (SLS)
- ---
- ---
- ${maven.build.timestamp}
-
-Yarn Scheduler Load Simulator (SLS)
-
-%{toc|section=1|fromDepth=0}
-
-* Overview
-
-** Overview
-
- The Yarn scheduler is a fertile area of interest with different
- implementations, e.g., Fifo, Capacity and Fair schedulers. Meanwhile, several
- optimizations are also made to improve scheduler performance for different
- scenarios and workload. Each scheduler algorithm has its own set of features,
- and drives scheduling decisions by many factors, such as fairness, capacity
- guarantee, resource availability, etc. It is very important to evaluate a
- scheduler algorithm very well before we deploy in a production cluster.
- Unfortunately, currently it is non-trivial to evaluate a scheduler algorithm.
- Evaluating in a real cluster is always time and cost consuming, and it is
- also very hard to find a large-enough cluster. Hence, a simulator which can
- predict how well a scheduler algorithm for some specific workload would be
- quite useful.
-
- The Yarn Scheduler Load Simulator (SLS) is such a tool, which can simulate
- large-scale Yarn clusters and application loads in a single machine.This
- simulator would be invaluable in furthering Yarn by providing a tool for
- researchers and developers to prototype new scheduler features and predict
- their behavior and performance with reasonable amount of confidence,
- thereby aiding rapid innovation.
-
- The simulator will exercise the real Yarn <<<ResourceManager>>> removing the
- network factor by simulating <<<NodeManagers>>> and <<<ApplicationMasters>>>
- via handling and dispatching <<<NM>>>/<<<AMs>>> heartbeat events from within
- the same JVM. To keep tracking of scheduler behavior and performance, a
- scheduler wrapper will wrap the real scheduler.
-
- The size of the cluster and the application load can be loaded from
- configuration files, which are generated from job history files directly by
- adopting {{{https://hadoop.apache.org/docs/stable/rumen.html}Apache Rumen}}.
-
- The simulator will produce real time metrics while executing, including:
-
- * Resource usages for whole cluster and each queue, which can be utilized to
- configure cluster and queue's capacity.
-
- * The detailed application execution trace (recorded in relation to simulated
- time), which can be analyzed to understand/validate the scheduler behavior
- (individual jobs turn around time, throughput, fairness, capacity guarantee,
- etc.).
-
- * Several key metrics of scheduler algorithm, such as time cost of each
- scheduler operation (allocate, handle, etc.), which can be utilized by Hadoop
- developers to find the code spots and scalability limits.
-
-** Goals
-
- * Exercise the scheduler at scale without a real cluster using real job
- traces.
-
- * Being able to simulate real workloads.
-
-** Architecture
-
- The following figure illustrates the implementation architecture of the
- simulator.
-
-[images/sls_arch.png] The architecture of the simulator
-
- The simulator takes input of workload traces, and fetches the cluster and
- applications information. For each NM and AM, the simulator builds a simulator
- to simulate their running. All NM/AM simulators run in a thread pool. The
- simulator reuses Yarn Resource Manager, and builds a wrapper out of the
- scheduler. The Scheduler Wrapper can track the scheduler behaviors and
- generates several logs, which are the outputs of the simulator and can be
- further analyzed.
-
-** Usecases
-
- * Engineering
-
- * Verify correctness of scheduler algorithm under load
-
- * Cheap/practical way for finding code hotspots/critical-path.
-
- * Validate the impact of changes and new features.
-
- * Determine what drives the scheduler scalability limits.
-
- []
-
- * QA
-
- * Validate scheduler behavior for "large" clusters and several workload
- profiles.
-
- * Solutions/Sales.
-
- * Sizing model for predefined/typical workloads.
-
- * Cluster sizing tool using real customer data (job traces).
-
- * Determine minimum SLAs under a particular workload.
-
-* Usage
-
- This section will show how to use the simulator. Here let <<<$HADOOP_ROOT>>>
- represent the Hadoop install directory. If you build Hadoop yourself,
- <<<$HADOOP_ROOT>>> is <<<hadoop-dist/target/hadoop-$VERSION>>>. The simulator
- is located at <<<$HADOOP_ROOT/share/hadoop/tools/sls>>>. The fold <<<sls>>>
- containers four directories: <<<bin>>>, <<<html>>>, <<<sample-conf>>>, and
- <<<sample-data>>>
-
- * <<<bin>>>: contains running scripts for the simulator.
-
- * <<<html>>>: contains several html/css/js files we needed for real-time
- tracking.
-
- * <<<sample-conf>>>: specifies the simulator configurations.
-
- * <<<sample-data>>>: provides an example rumen trace, which can be used to
- generate inputs of the simulator.
-
- []
-
- The following sections will describe how to use the simulator step by step.
- Before start, make sure that command <<<hadoop>>> is included in your
- <<<$PATH>>> environment parameter.
-
-** Step 1: Configure Hadoop and the simulator
-
- Before we start, make sure Hadoop and the simulator are configured well.
- All configuration files for Hadoop and the simulator should be placed in
- directory <<<$HADOOP_ROOT/etc/hadoop>>>, where the <<<ResourceManager>>>
- and Yarn scheduler load their configurations. Directory
- <<<$HADOOP_ROOT/share/hadoop/tools/sls/sample-conf/>>> provides several
- example configurations, that can be used to start a demo.
-
- For configuration of Hadoop and Yarn scheduler, users can refer to Yarn’s
- website ({{{http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/}
- http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/}}).
-
- For the simulator, it loads configuration information from file
- <<<$HADOOP_ROOT/etc/hadoop/sls-runner.xml>>>.
-
- Here we illustrate each configuration parameter in <<<sls-runner.xml>>>.
- Note that <<<$HADOOP_ROOT/share/hadoop/tools/sls/sample-conf/sls-runner.xml>>>
- contains all the default values for these configuration parameters.
-
- * <<<yarn.sls.runner.pool.size>>>
-
- The simulator uses a thread pool to simulate the <<<NM>>> and <<<AM>>> running
- , and this parameter specifies the number of threads in the pool.
-
- * <<<yarn.sls.nm.memory.mb>>>
-
- The total memory for each <<<NMSimulator>>>.
-
- * <<<yarn.sls.nm.vcores>>>
-
- The total vCores for each <<<NMSimulator>>>.
-
- * <<<yarn.sls.nm.heartbeat.interval.ms>>>
-
- The heartbeat interval for each <<<NMSimulator>>>.
-
- * <<<yarn.sls.am.heartbeat.interval.ms>>>
-
- The heartbeat interval for each <<<AMSimulator>>>.
-
- * <<<yarn.sls.am.type.mapreduce>>>
-
- The <<<AMSimulator>>> implementation for MapReduce-like applications.
- Users can specify implementations for other type of applications.
-
- * <<<yarn.sls.container.memory.mb>>>
-
- The memory required for each container simulator.
-
- * <<<yarn.sls.container.vcores>>>
-
- The vCores required for each container simulator.
-
- * <<<yarn.sls.runner.metrics.switch>>>
-
- The simulator introduces {{{http://metrics.codahale.com/}Metrics}} to measure
- the behaviors of critical components and operations. This field specifies
- whether we open (<<<ON>>>) or close (<<<OFF>>>) the Metrics running.
-
- * <<<yarn.sls.metrics.web.address.port>>>
-
- The port used by simulator to provide real-time tracking. The default value is
- 10001.
-
- * <<<org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler>>>
-
- The implementation of scheduler metrics of Fifo Scheduler.
-
- * <<<org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler>>>
-
- The implementation of scheduler metrics of Fair Scheduler.
-
- * <<<org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler>>>
-
- The implementation of scheduler metrics of Capacity Scheduler.
-
-** Step 2: Run the simulator
-
- The simulator supports two types of input files: the rumen traces and its own
- input traces. The script to start the simulator is <<<slsrun.sh>>>.
-
-+----+
-$ cd $HADOOP_ROOT/share/hadoop/tools/sls
-$ bin/slsrun.sh
- --input-rumen|--input-sls=<TRACE_FILE1,TRACE_FILE2,...>
- --output-dir=<SLS_SIMULATION_OUTPUT_DIRECTORY> [--nodes=<SLS_NODES_FILE>]
- [--track-jobs=<JOBID1,JOBID2,...>] [--print-simulation]
-+----+
-
- * <<<--input-rumen>>>: The input rumen trace files. Users can input multiple
- files, separated by comma. One example trace is provided in
- <<<$HADOOP_ROOT/share/hadoop/tools/sls/sample-data/2jobs2min-rumen-jh.json>>>.
-
- * <<<--input-sls>>>: Simulator its own file format. The simulator also
- provides a tool to convert rumen traces to sls traces (<<<rumen2sls.sh>>>).
- Refer to appendix for an example of sls input json file.
-
- * <<<--output-dir>>>: The output directory for generated running logs and
- metrics.
-
- * <<<--nodes>>>: The cluster topology. By default, the simulator will use the
- topology fetched from the input json files. Users can specifies a new topology
- by setting this parameter. Refer to the appendix for the topology file format.
-
- * <<<--track-jobs>>>: The particular jobs that will be tracked during
- simulator running, spearated by comma.
-
- * <<<--print-simulation>>>: Whether to print out simulation information
- before simulator running, including number of nodes, applications, tasks,
- and information for each application.
-
- In comparison to rumen format, here the sls format is much simpler and users
- can easily generate various workload. The simulator also provides a tool to
- convert rumen traces to sls traces.
-
-+----+
-$ bin/rumen2sls.sh
- --rumen-file=<RUMEN_FILE>
- --output-dir=<SLS_OUTPUT_DIRECTORY>
- [--output-prefix=<SLS_FILE_PREFIX>]
-+----+
-
- * <<<--rumen-file>>>: The rumen format file. One example trace is provided
- in directory <<<sample-data>>>.
-
- * <<<--output-dir>>>: The output directory of generated simulation traces.
- Two files will be generated in this output directory, including one trace
- file including all job and task information, and another file showing the
- topology information.
-
- * <<<--output-prefix>>>: The prefix of the generated files. The default value
- is ”sls”, and the two generated files are <<<sls-jobs.json>>> and
- <<<sls-nodes.json>>>.
-
-* Metrics
-
- The Yarn Scheduler Load Simulator has integrated
- {{{http://metrics.codahale.com/}Metrics}} to measure the behaviors of critical
- components and operations, including running applications and containers,
- cluster available resources, scheduler operation timecost, et al. If the
- switch <<<yarn.sls.runner.metrics.switch>>> is set <<<ON>>>, <<<Metrics>>>
- will run and output it logs in <<<--output-dir>>> directory specified by users.
- Users can track these information during simulator running, and can also
- analyze these logs after running to evaluate the scheduler performance.
-
-** Real-time Tracking
-
- The simulator provides an interface for tracking its running in real-time.
- Users can go to <<<http://host:port/simulate>>> to track whole running,
- and <<<http://host:port/track>>> to track a particular job or queue. Here
- the <<<host>>> is the place when we run the simulator, and <<<port>>> is
- the value configured by <<<yarn.sls.metrics.web.address.port>>> (default value
- is 10001).
-
- Here we'll illustrate each chart shown in the webpage.
-
- The first figure describes the number of running applications and containers.
-
-[images/sls_running_apps_containers.png] Number of running applications/containers
-
- The second figure describes the allocated and available resources (memory)
- in the cluster.
-
-[images/sls_cluster_memory.png] Cluster Resource (Memory)
-
- The third figure describes the allocated resource for each queue. Here we have
- three queues: sls_queue_1, sls_queue_2, and sls_queue_3.The first two queues
- are configured with 25% share, while the last one has 50% share.
-
-[images/sls_queue_allocated_memory.png] Queue Allocated Resource (Memory)
-
- The fourth figure describes the timecost for each scheduler operation.
-
-[images/sls_scheduler_operation_timecost.png] Scheduler Opertion Timecost
-
- Finally, we measure the memory used by the simulator.
-
-[images/sls_JVM.png] JVM Memory
-
- The simulator also provides an interface for tracking some particular
- jobs and queues. Go to <<<http://<Host>:<Port>/track>>> to get these
- information.
-
- Here the first figure illustrates the resource usage information for queue
- <<<SLS_Queue_1>>>.
-
-[images/sls_track_queue.png] Tracking Queue <<<sls_queue_3>>>
-
- The second figure illustrates the resource usage information for job
- <<<job_1369942127770_0653>>>.
-
-[images/sls_track_job.png] Tracking Job <<<job_1369942127770_0653>>>
-
-** Offline Analysis
-
- After the simulator finishes, all logs are saved in the output directory
- specified by <<<--output-dir>>> in
- <<<$HADOOP_ROOT/share/hadoop/tools/sls/bin/slsrun.sh>>>.
-
- * File <<<realtimetrack.json>>>: records all real-time tracking logs every 1
- second.
-
- * File <<<jobruntime.csv>>>: records all jobs’ start and end time in the
- simulator.
-
- * Folder <<<metrics>>>: logs generated by the Metrics.
-
- []
-
- Users can also reproduce those real-time tracking charts in offline mode.
- Just upload the <<<realtimetrack.json>>> to
- <<<$HADOOP_ROOT/share/hadoop/tools/sls/html/showSimulationTrace.html>>>.
- For browser security problem, need to put files <<<realtimetrack.json>>> and
- <<<showSimulationTrace.html>>> in the same directory.
-
-* Appendix
-
-** Resources
-
- {{{https://issues.apache.org/jira/browse/YARN-1021}YARN-1021}} is the main
- JIRA that introduces Yarn Scheduler Load Simulator to Hadoop Yarn project.
-
-** SLS JSON input file format
-
- Here we provide an example format of the sls json file, which contains 2 jobs.
- The first job has 3 map tasks and the second one has 2 map tasks.
-
-+----+
-{
- "am.type" : "mapreduce",
- "job.start.ms" : 0,
- "job.end.ms" : 95375,
- "job.queue.name" : "sls_queue_1",
- "job.id" : "job_1",
- "job.user" : "default",
- "job.tasks" : [ {
- "container.host" : "/default-rack/node1",
- "container.start.ms" : 6664,
- "container.end.ms" : 23707,
- "container.priority" : 20,
- "container.type" : "map"
- }, {
- "container.host" : "/default-rack/node3",
- "container.start.ms" : 6665,
- "container.end.ms" : 21593,
- "container.priority" : 20,
- "container.type" : "map"
- }, {
- "container.host" : "/default-rack/node2",
- "container.start.ms" : 68770,
- "container.end.ms" : 86613,
- "container.priority" : 20,
- "container.type" : "map"
- } ]
-}
-{
- "am.type" : "mapreduce",
- "job.start.ms" : 105204,
- "job.end.ms" : 197256,
- "job.queue.name" : "sls_queue_2",
- "job.id" : "job_2",
- "job.user" : "default",
- "job.tasks" : [ {
- "container.host" : "/default-rack/node1",
- "container.start.ms" : 111822,
- "container.end.ms" : 133985,
- "container.priority" : 20,
- "container.type" : "map"
- }, {
- "container.host" : "/default-rack/node2",
- "container.start.ms" : 111788,
- "container.end.ms" : 131377,
- "container.priority" : 20,
- "container.type" : "map"
- } ]
-}
-+----+
-
-** Simulator input topology file format
-
- Here is an example input topology file which has 3 nodes organized in 1 rack.
-
-+----+
-{
- "rack" : "default-rack",
- "nodes" : [ {
- "node" : "node1"
- }, {
- "node" : "node2"
- }, {
- "node" : "node3"
- }]
-}
-+----+
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-tools/hadoop-sls/src/site/markdown/SchedulerLoadSimulator.md
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-sls/src/site/markdown/SchedulerLoadSimulator.md b/hadoop-tools/hadoop-sls/src/site/markdown/SchedulerLoadSimulator.md
new file mode 100644
index 0000000..ca179ee
--- /dev/null
+++ b/hadoop-tools/hadoop-sls/src/site/markdown/SchedulerLoadSimulator.md
@@ -0,0 +1,357 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+Yarn Scheduler Load Simulator (SLS)
+===================================
+
+* [Yarn Scheduler Load Simulator (SLS)](#Yarn_Scheduler_Load_Simulator_SLS)
+ * [Overview](#Overview)
+ * [Overview](#Overview)
+ * [Goals](#Goals)
+ * [Architecture](#Architecture)
+ * [Usecases](#Usecases)
+ * [Usage](#Usage)
+ * [Step 1: Configure Hadoop and the simulator](#Step_1:_Configure_Hadoop_and_the_simulator)
+ * [Step 2: Run the simulator](#Step_2:_Run_the_simulator)
+ * [Metrics](#Metrics)
+ * [Real-time Tracking](#Real-time_Tracking)
+ * [Offline Analysis](#Offline_Analysis)
+ * [Appendix](#Appendix)
+ * [Resources](#Resources)
+ * [SLS JSON input file format](#SLS_JSON_input_file_format)
+ * [Simulator input topology file format](#Simulator_input_topology_file_format)
+
+Overview
+--------
+
+### Overview
+
+The Yarn scheduler is a fertile area of interest with different implementations, e.g., Fifo, Capacity and Fair schedulers. Meanwhile, several optimizations are also made to improve scheduler performance for different scenarios and workload. Each scheduler algorithm has its own set of features, and drives scheduling decisions by many factors, such as fairness, capacity guarantee, resource availability, etc. It is very important to evaluate a scheduler algorithm very well before we deploy in a production cluster. Unfortunately, currently it is non-trivial to evaluate a scheduler algorithm. Evaluating in a real cluster is always time and cost consuming, and it is also very hard to find a large-enough cluster. Hence, a simulator which can predict how well a scheduler algorithm for some specific workload would be quite useful.
+
+The Yarn Scheduler Load Simulator (SLS) is such a tool, which can simulate large-scale Yarn clusters and application loads in a single machine.This simulator would be invaluable in furthering Yarn by providing a tool for researchers and developers to prototype new scheduler features and predict their behavior and performance with reasonable amount of confidence, thereby aiding rapid innovation.
+o
+The simulator will exercise the real Yarn `ResourceManager` removing the network factor by simulating `NodeManagers` and `ApplicationMasters` via handling and dispatching `NM`/`AMs` heartbeat events from within the same JVM. To keep tracking of scheduler behavior and performance, a scheduler wrapper will wrap the real scheduler.
+
+The size of the cluster and the application load can be loaded from configuration files, which are generated from job history files directly by adopting [Apache Rumen](https://hadoop.apache.org/docs/stable/rumen.html).
+
+The simulator will produce real time metrics while executing, including:
+
+* Resource usages for whole cluster and each queue, which can be utilized to
+ configure cluster and queue's capacity.
+
+* The detailed application execution trace (recorded in relation to simulated
+ time), which can be analyzed to understand/validate the scheduler behavior
+ (individual jobs turn around time, throughput, fairness, capacity guarantee,
+ etc.).
+
+* Several key metrics of scheduler algorithm, such as time cost of each
+ scheduler operation (allocate, handle, etc.), which can be utilized by Hadoop
+ developers to find the code spots and scalability limits.
+
+### Goals
+
+* Exercise the scheduler at scale without a real cluster using real job
+ traces.
+
+* Being able to simulate real workloads.
+
+### Architecture
+
+The following figure illustrates the implementation architecture of the simulator.
+
+
+
+The simulator takes input of workload traces, and fetches the cluster and applications information. For each NM and AM, the simulator builds a simulator to simulate their running. All NM/AM simulators run in a thread pool. The simulator reuses Yarn Resource Manager, and builds a wrapper out of the scheduler. The Scheduler Wrapper can track the scheduler behaviors and generates several logs, which are the outputs of the simulator and can be further analyzed.
+
+### Usecases
+
+* Engineering
+ * Verify correctness of scheduler algorithm under load
+ * Cheap/practical way for finding code hotspots/critical-path.
+ * Validate the impact of changes and new features.
+ * Determine what drives the scheduler scalability limits.
+
+* QA
+ * Validate scheduler behavior for "large" clusters and several workload profiles.
+
+* Solutions/Sales.
+ * Sizing model for predefined/typical workloads.
+ * Cluster sizing tool using real customer data (job traces).
+ * Determine minimum SLAs under a particular workload.
+
+Usage
+-----
+
+This section will show how to use the simulator. Here let `$HADOOP_ROOT` represent the Hadoop install directory. If you build Hadoop yourself, `$HADOOP_ROOT` is `hadoop-dist/target/hadoop-$VERSION`. The simulator is located at `$HADOOP_ROOT/share/hadoop/tools/sls`. The fold `sls` containers four directories: `bin`, `html`, `sample-conf`, and `sample-data`
+
+* `bin`: contains running scripts for the simulator.
+
+* `html`: contains several html/css/js files we needed for real-time tracking.
+
+* `sample-conf`: specifies the simulator configurations.
+
+* `sample-data`: provides an example rumen trace, which can be used to
+ generate inputs of the simulator.
+
+The following sections will describe how to use the simulator step by step. Before start, make sure that command `hadoop` is included in your `$PATH` environment parameter.
+
+### Step 1: Configure Hadoop and the simulator
+
+Before we start, make sure Hadoop and the simulator are configured well. All configuration files for Hadoop and the simulator should be placed in directory `$HADOOP_ROOT/etc/hadoop`, where the `ResourceManager` and Yarn scheduler load their configurations. Directory `$HADOOP_ROOT/share/hadoop/tools/sls/sample-conf/` provides several example configurations, that can be used to start a demo.
+
+For configuration of Hadoop and Yarn scheduler, users can refer to Yarn’s website (<http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/>).
+
+For the simulator, it loads configuration information from file `$HADOOP_ROOT/etc/hadoop/sls-runner.xml`.
+
+Here we illustrate each configuration parameter in `sls-runner.xml`. Note that `$HADOOP_ROOT/share/hadoop/tools/sls/sample-conf/sls-runner.xml` contains all the default values for these configuration parameters.
+
+* `yarn.sls.runner.pool.size`
+
+ The simulator uses a thread pool to simulate the `NM` and `AM` running,
+ and this parameter specifies the number of threads in the pool.
+
+* `yarn.sls.nm.memory.mb`
+
+ The total memory for each `NMSimulator`.
+
+* `yarn.sls.nm.vcores`
+
+ The total vCores for each `NMSimulator`.
+
+* `yarn.sls.nm.heartbeat.interval.ms`
+
+ The heartbeat interval for each `NMSimulator`.
+
+* `yarn.sls.am.heartbeat.interval.ms`
+
+ The heartbeat interval for each `AMSimulator`.
+
+* `yarn.sls.am.type.mapreduce`
+
+ The `AMSimulator` implementation for MapReduce-like applications.
+ Users can specify implementations for other type of applications.
+
+* `yarn.sls.container.memory.mb`
+
+ The memory required for each container simulator.
+
+* `yarn.sls.container.vcores`
+
+ The vCores required for each container simulator.
+
+* `yarn.sls.runner.metrics.switch`
+
+ The simulator introduces [Metrics](http://metrics.codahale.com/) to measure
+ the behaviors of critical components and operations. This field specifies
+ whether we open (`ON`) or close (`OFF`) the Metrics running.
+
+* `yarn.sls.metrics.web.address.port`
+
+ The port used by simulator to provide real-time tracking. The default value is
+ 10001.
+
+* `org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler`
+
+ The implementation of scheduler metrics of Fifo Scheduler.
+
+* `org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler`
+
+ The implementation of scheduler metrics of Fair Scheduler.
+
+* `org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler`
+
+ The implementation of scheduler metrics of Capacity Scheduler.
+
+### Step 2: Run the simulator
+
+The simulator supports two types of input files: the rumen traces and its own input traces. The script to start the simulator is `slsrun.sh`.
+
+ $ cd $HADOOP_ROOT/share/hadoop/tools/sls
+ $ bin/slsrun.sh
+ --input-rumen |--input-sls=<TRACE_FILE1,TRACE_FILE2,...>
+ --output-dir=<SLS_SIMULATION_OUTPUT_DIRECTORY> [--nodes=<SLS_NODES_FILE>]
+ [--track-jobs=<JOBID1,JOBID2,...>] [--print-simulation]
+
+* `--input-rumen`: The input rumen trace files. Users can input multiple
+ files, separated by comma. One example trace is provided in
+ `$HADOOP_ROOT/share/hadoop/tools/sls/sample-data/2jobs2min-rumen-jh.json`.
+
+* `--input-sls`: Simulator its own file format. The simulator also
+ provides a tool to convert rumen traces to sls traces (`rumen2sls.sh`).
+ Refer to appendix for an example of sls input json file.
+
+* `--output-dir`: The output directory for generated running logs and
+ metrics.
+
+* `--nodes`: The cluster topology. By default, the simulator will use the
+ topology fetched from the input json files. Users can specifies a new topology
+ by setting this parameter. Refer to the appendix for the topology file format.
+
+* `--track-jobs`: The particular jobs that will be tracked during
+ simulator running, spearated by comma.
+
+* `--print-simulation`: Whether to print out simulation information
+ before simulator running, including number of nodes, applications, tasks,
+ and information for each application.
+
+ In comparison to rumen format, here the sls format is much simpler and users
+ can easily generate various workload. The simulator also provides a tool to
+ convert rumen traces to sls traces.
+
+ $ bin/rumen2sls.sh
+ --rumen-file=<RUMEN_FILE>
+ --output-dir=<SLS_OUTPUT_DIRECTORY>
+ [--output-prefix=<SLS_FILE_PREFIX>]
+
+* `--rumen-file`: The rumen format file. One example trace is provided
+ in directory `sample-data`.
+
+* `--output-dir`: The output directory of generated simulation traces.
+ Two files will be generated in this output directory, including one trace
+ file including all job and task information, and another file showing the
+ topology information.
+
+* `--output-prefix`: The prefix of the generated files. The default value
+ is "sls", and the two generated files are `sls-jobs.json` and
+ `sls-nodes.json`.
+
+Metrics
+-------
+
+The Yarn Scheduler Load Simulator has integrated [Metrics](http://metrics.codahale.com/) to measure the behaviors of critical components and operations, including running applications and containers, cluster available resources, scheduler operation timecost, et al. If the switch `yarn.sls.runner.metrics.switch` is set `ON`, `Metrics` will run and output it logs in `--output-dir` directory specified by users. Users can track these information during simulator running, and can also analyze these logs after running to evaluate the scheduler performance.
+
+### Real-time Tracking
+
+The simulator provides an interface for tracking its running in real-time. Users can go to `http://host:port/simulate` to track whole running, and `http://host:port/track` to track a particular job or queue. Here the `host` is the place when we run the simulator, and `port` is the value configured by `yarn.sls.metrics.web.address.port` (default value is 10001).
+
+Here we'll illustrate each chart shown in the webpage.
+
+The first figure describes the number of running applications and containers.
+
+
+
+The second figure describes the allocated and available resources (memory) in the cluster.
+
+
+
+The third figure describes the allocated resource for each queue. Here we have three queues: sls\_queue\_1, sls\_queue\_2, and sls\_queue\_3.The first two queues are configured with 25% share, while the last one has 50% share.
+
+
+
+The fourth figure describes the timecost for each scheduler operation.
+
+
+
+Finally, we measure the memory used by the simulator.
+
+
+
+The simulator also provides an interface for tracking some particular jobs and queues. Go to `http://<Host>:<Port>/track` to get these information.
+
+Here the first figure illustrates the resource usage information for queue `SLS_Queue_1`.
+
+
+
+The second figure illustrates the resource usage information for job `job_1369942127770_0653`.
+
+
+
+### Offline Analysis
+
+After the simulator finishes, all logs are saved in the output directory specified by `--output-dir` in `$HADOOP_ROOT/share/hadoop/tools/sls/bin/slsrun.sh`.
+
+* File `realtimetrack.json`: records all real-time tracking logs every 1
+ second.
+
+* File `jobruntime.csv`: records all jobs’ start and end time in the
+ simulator.
+
+* Folder `metrics`: logs generated by the Metrics.
+
+Users can also reproduce those real-time tracking charts in offline mode. Just upload the `realtimetrack.json` to `$HADOOP_ROOT/share/hadoop/tools/sls/html/showSimulationTrace.html`. For browser security problem, need to put files `realtimetrack.json` and `showSimulationTrace.html` in the same directory.
+
+Appendix
+--------
+
+### Resources
+
+[YARN-1021](https://issues.apache.org/jira/browse/YARN-1021) is the main JIRA that introduces Yarn Scheduler Load Simulator to Hadoop Yarn project.
+
+### SLS JSON input file format
+
+Here we provide an example format of the sls json file, which contains 2 jobs. The first job has 3 map tasks and the second one has 2 map tasks.
+
+ {
+ "am.type" : "mapreduce",
+ "job.start.ms" : 0,
+ "job.end.ms" : 95375,
+ "job.queue.name" : "sls_queue_1",
+ "job.id" : "job_1",
+ "job.user" : "default",
+ "job.tasks" : [ {
+ "container.host" : "/default-rack/node1",
+ "container.start.ms" : 6664,
+ "container.end.ms" : 23707,
+ "container.priority" : 20,
+ "container.type" : "map"
+ }, {
+ "container.host" : "/default-rack/node3",
+ "container.start.ms" : 6665,
+ "container.end.ms" : 21593,
+ "container.priority" : 20,
+ "container.type" : "map"
+ }, {
+ "container.host" : "/default-rack/node2",
+ "container.start.ms" : 68770,
+ "container.end.ms" : 86613,
+ "container.priority" : 20,
+ "container.type" : "map"
+ } ]
+ }
+ {
+ "am.type" : "mapreduce",
+ "job.start.ms" : 105204,
+ "job.end.ms" : 197256,
+ "job.queue.name" : "sls_queue_2",
+ "job.id" : "job_2",
+ "job.user" : "default",
+ "job.tasks" : [ {
+ "container.host" : "/default-rack/node1",
+ "container.start.ms" : 111822,
+ "container.end.ms" : 133985,
+ "container.priority" : 20,
+ "container.type" : "map"
+ }, {
+ "container.host" : "/default-rack/node2",
+ "container.start.ms" : 111788,
+ "container.end.ms" : 131377,
+ "container.priority" : 20,
+ "container.type" : "map"
+ } ]
+ }
+
+### Simulator input topology file format
+
+Here is an example input topology file which has 3 nodes organized in 1 rack.
+
+ {
+ "rack" : "default-rack",
+ "nodes" : [ {
+ "node" : "node1"
+ }, {
+ "node" : "node2"
+ }, {
+ "node" : "node3"
+ }]
+ }
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-tools/hadoop-streaming/src/site/apt/HadoopStreaming.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-streaming/src/site/apt/HadoopStreaming.apt.vm b/hadoop-tools/hadoop-streaming/src/site/apt/HadoopStreaming.apt.vm
deleted file mode 100644
index 8be92b5..0000000
--- a/hadoop-tools/hadoop-streaming/src/site/apt/HadoopStreaming.apt.vm
+++ /dev/null
@@ -1,792 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- Hadoop Streaming
- ---
- ---
- ${maven.build.timestamp}
-
-Hadoop Streaming
-
-%{toc|section=1|fromDepth=0|toDepth=4}
-
-* Hadoop Streaming
-
- Hadoop streaming is a utility that comes with the Hadoop distribution. The
- utility allows you to create and run Map/Reduce jobs with any executable or
- script as the mapper and/or the reducer. For example:
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -input myInputDirs \
- -output myOutputDir \
- -mapper /bin/cat \
- -reducer /usr/bin/wc
-+---+
-
-* How Streaming Works
-
- In the above example, both the mapper and the reducer are executables that
- read the input from stdin (line by line) and emit the output to stdout. The
- utility will create a Map/Reduce job, submit the job to an appropriate
- cluster, and monitor the progress of the job until it completes.
-
- When an executable is specified for mappers, each mapper task will launch the
- executable as a separate process when the mapper is initialized. As the
- mapper task runs, it converts its inputs into lines and feed the lines to the
- stdin of the process. In the meantime, the mapper collects the line oriented
- outputs from the stdout of the process and converts each line into a
- key/value pair, which is collected as the output of the mapper. By default,
- the <prefix of a line up to the first tab character> is the <<<key>>> and the
- rest of the line (excluding the tab character) will be the <<<value>>>. If
- there is no tab character in the line, then entire line is considered as key
- and the value is null. However, this can be customized by setting
- <<<-inputformat>>> command option, as discussed later.
-
- When an executable is specified for reducers, each reducer task will launch
- the executable as a separate process then the reducer is initialized. As the
- reducer task runs, it converts its input key/values pairs into lines and
- feeds the lines to the stdin of the process. In the meantime, the reducer
- collects the line oriented outputs from the stdout of the process, converts
- each line into a key/value pair, which is collected as the output of the
- reducer. By default, the prefix of a line up to the first tab character is
- the key and the rest of the line (excluding the tab character) is the value.
- However, this can be customized by setting <<<-outputformat>>> command
- option, as discussed later.
-
- This is the basis for the communication protocol between the Map/Reduce
- framework and the streaming mapper/reducer.
-
- User can specify <<<stream.non.zero.exit.is.failure>>> as <<<true>>> or
- <<<false>>> to make a streaming task that exits with a non-zero status to be
- <<<Failure>>> or <<<Success>>> respectively. By default, streaming tasks
- exiting with non-zero status are considered to be failed tasks.
-
-* Streaming Command Options
-
- Streaming supports streaming command options as well as
- {{{Generic_Command_Options}generic command options}}. The general command
- line syntax is shown below.
-
- <<Note:>> Be sure to place the generic options before the streaming options,
- otherwise the command will fail. For an example, see
- {{{Making_Archives_Available_to_Tasks}Making Archives Available to Tasks}}.
-
-+---+
-hadoop command [genericOptions] [streamingOptions]
-+---+
-
- The Hadoop streaming command options are listed here:
-
-*-------------*--------------------*------------------------------------------*
-|| Parameter || Optional/Required || Description |
-*-------------+--------------------+------------------------------------------+
-| -input directoryname or filename | Required | Input location for mapper
-*-------------+--------------------+------------------------------------------+
-| -output directoryname | Required | Output location for reducer
-*-------------+--------------------+------------------------------------------+
-| -mapper executable or JavaClassName | Required | Mapper executable
-*-------------+--------------------+------------------------------------------+
-| -reducer executable or JavaClassName | Required | Reducer executable
-*-------------+--------------------+------------------------------------------+
-| -file filename | Optional | Make the mapper, reducer, or combiner executable
-| | | available locally on the compute nodes
-*-------------+--------------------+------------------------------------------+
-| -inputformat JavaClassName | Optional | Class you supply should return
-| | | key/value pairs of Text class. If not
-| | | specified, TextInputFormat is used as
-| | | the default
-*-------------+--------------------+------------------------------------------+
-| -outputformat JavaClassName | Optional | Class you supply should take
-| | | key/value pairs of Text class. If
-| | | not specified, TextOutputformat is
-| | | used as the default
-*-------------+--------------------+------------------------------------------+
-| -partitioner JavaClassName | Optional | Class that determines which reduce a
-| | | key is sent to
-*-------------+--------------------+------------------------------------------+
-| -combiner streamingCommand | Optional | Combiner executable for map output
-| or JavaClassName | |
-*-------------+--------------------+------------------------------------------+
-| -cmdenv name=value | Optional | Pass environment variable to streaming
-| | | commands
-*-------------+--------------------+------------------------------------------+
-| -inputreader | Optional | For backwards-compatibility: specifies a record
-| | | reader class (instead of an input format class)
-*-------------+--------------------+------------------------------------------+
-| -verbose | Optional | Verbose output
-*-------------+--------------------+------------------------------------------+
-| -lazyOutput | Optional | Create output lazily. For example, if the output
-| | | format is based on FileOutputFormat, the output file
-| | | is created only on the first call to Context.write
-*-------------+--------------------+------------------------------------------+
-| -numReduceTasks | Optional | Specify the number of reducers
-*-------------+--------------------+------------------------------------------+
-| -mapdebug | Optional | Script to call when map task fails
-*-------------+--------------------+------------------------------------------+
-| -reducedebug | Optional | Script to call when reduce task fails
-*-------------+--------------------+------------------------------------------+
-
-** Specifying a Java Class as the Mapper/Reducer
-
- You can supply a Java class as the mapper and/or the reducer.
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -input myInputDirs \
- -output myOutputDir \
- -inputformat org.apache.hadoop.mapred.KeyValueTextInputFormat \
- -mapper org.apache.hadoop.mapred.lib.IdentityMapper \
- -reducer /usr/bin/wc
-+---+
-
- You can specify <<<stream.non.zero.exit.is.failure>>> as <<<true>>> or
- <<<false>>> to make a streaming task that exits with a non-zero status to be
- <<<Failure>>> or <<<Success>>> respectively. By default, streaming tasks
- exiting with non-zero status are considered to be failed tasks.
-
-** Packaging Files With Job Submissions
-
- You can specify any executable as the mapper and/or the reducer. The
- executables do not need to pre-exist on the machines in the cluster; however,
- if they don't, you will need to use "-file" option to tell the framework to
- pack your executable files as a part of job submission. For example:
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -input myInputDirs \
- -output myOutputDir \
- -mapper myPythonScript.py \
- -reducer /usr/bin/wc \
- -file myPythonScript.py
-+---+
-
- The above example specifies a user defined Python executable as the mapper.
- The option "-file myPythonScript.py" causes the python executable shipped
- to the cluster machines as a part of job submission.
-
- In addition to executable files, you can also package other auxiliary files
- (such as dictionaries, configuration files, etc) that may be used by the
- mapper and/or the reducer. For example:
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -input myInputDirs \
- -output myOutputDir \
- -mapper myPythonScript.py \
- -reducer /usr/bin/wc \
- -file myPythonScript.py \
- -file myDictionary.txt
-+---+
-
-** Specifying Other Plugins for Jobs
-
- Just as with a normal Map/Reduce job, you can specify other plugins for a
- streaming job:
-
-+---+
- -inputformat JavaClassName
- -outputformat JavaClassName
- -partitioner JavaClassName
- -combiner streamingCommand or JavaClassName
-+---+
-
- The class you supply for the input format should return key/value pairs of
- Text class. If you do not specify an input format class, the TextInputFormat
- is used as the default. Since the TextInputFormat returns keys of
- LongWritable class, which are actually not part of the input data, the keys
- will be discarded; only the values will be piped to the streaming mapper.
-
- The class you supply for the output format is expected to take key/value
- pairs of Text class. If you do not specify an output format class, the
- TextOutputFormat is used as the default.
-
-** Setting Environment Variables
-
- To set an environment variable in a streaming command use:
-
-+---+
- -cmdenv EXAMPLE_DIR=/home/example/dictionaries/
-+---+
-
-* Generic Command Options
-
- Streaming supports {{{Streaming_Command_Options}streaming command options}}
- as well as generic command options. The general command line syntax is shown
- below.
-
- <<Note:>> Be sure to place the generic options before the streaming options,
- otherwise the command will fail. For an example, see
- {{{Making_Archives_Available_to_Tasks}Making Archives Available to Tasks}}.
-
-+---+
-hadoop command [genericOptions] [streamingOptions]
-+---+
-
- The Hadoop generic command options you can use with streaming are listed
- here:
-
-*-------------*--------------------*------------------------------------------*
-|| Parameter || Optional/Required || Description |
-*-------------+--------------------+------------------------------------------+
-| -conf configuration_file | Optional | Specify an application configuration
-| | | file
-*-------------+--------------------+------------------------------------------+
-| -D property=value | Optional | Use value for given property
-*-------------+--------------------+------------------------------------------+
-| -fs host:port or local | Optional | Specify a namenode
-*-------------+--------------------+------------------------------------------+
-| -files | Optional | Specify comma-separated files to be copied to the
-| | | Map/Reduce cluster
-*-------------+--------------------+------------------------------------------+
-| -libjars | Optional | Specify comma-separated jar files to include in the
-| | | classpath
-*-------------+--------------------+------------------------------------------+
-| -archives | Optional | Specify comma-separated archives to be unarchived on
-| | | the compute machines
-*-------------+--------------------+------------------------------------------+
-
-** Specifying Configuration Variables with the -D Option
-
- You can specify additional configuration variables by using
- "-D \<property\>=\<value\>".
-
-*** Specifying Directories
-
- To change the local temp directory use:
-
-+---+
- -D dfs.data.dir=/tmp
-+---+
-
- To specify additional local temp directories use:
-
-+---+
- -D mapred.local.dir=/tmp/local
- -D mapred.system.dir=/tmp/system
- -D mapred.temp.dir=/tmp/temp
-+---+
-
- <<Note:>> For more details on job configuration parameters see:
- {{{./mapred-default.xml}mapred-default.xml}}
-
-*** Specifying Map-Only Jobs
-
- Often, you may want to process input data using a map function only. To do
- this, simply set <<<mapreduce.job.reduces>>> to zero. The Map/Reduce
- framework will not create any reducer tasks. Rather, the outputs of the
- mapper tasks will be the final output of the job.
-
-+---+
- -D mapreduce.job.reduces=0
-+---+
-
- To be backward compatible, Hadoop Streaming also supports the "-reducer NONE"
- option, which is equivalent to "-D mapreduce.job.reduces=0".
-
-*** Specifying the Number of Reducers
-
- To specify the number of reducers, for example two, use:
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -D mapreduce.job.reduces=2 \
- -input myInputDirs \
- -output myOutputDir \
- -mapper /bin/cat \
- -reducer /usr/bin/wc
-+---+
-
-*** Customizing How Lines are Split into Key/Value Pairs
-
- As noted earlier, when the Map/Reduce framework reads a line from the stdout
- of the mapper, it splits the line into a key/value pair. By default, the
- prefix of the line up to the first tab character is the key and the rest of
- the line (excluding the tab character) is the value.
-
- However, you can customize this default. You can specify a field separator
- other than the tab character (the default), and you can specify the nth
- (n >= 1) character rather than the first character in a line (the default) as
- the separator between the key and value. For example:
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -D stream.map.output.field.separator=. \
- -D stream.num.map.output.key.fields=4 \
- -input myInputDirs \
- -output myOutputDir \
- -mapper /bin/cat \
- -reducer /bin/cat
-+---+
-
- In the above example, "-D stream.map.output.field.separator=." specifies "."
- as the field separator for the map outputs, and the prefix up to the fourth
- "." in a line will be the key and the rest of the line (excluding the fourth
- ".") will be the value. If a line has less than four "."s, then the whole
- line will be the key and the value will be an empty Text object (like the one
- created by new Text("")).
-
- Similarly, you can use "-D stream.reduce.output.field.separator=SEP" and
- "-D stream.num.reduce.output.fields=NUM" to specify the nth field separator
- in a line of the reduce outputs as the separator between the key and the
- value.
-
- Similarly, you can specify "stream.map.input.field.separator" and
- "stream.reduce.input.field.separator" as the input separator for Map/Reduce
- inputs. By default the separator is the tab character.
-
-** Working with Large Files and Archives
-
- The -files and -archives options allow you to make files and archives
- available to the tasks. The argument is a URI to the file or archive that you
- have already uploaded to HDFS. These files and archives are cached across
- jobs. You can retrieve the host and fs_port values from the fs.default.name
- config variable.
-
- <<Note:>> The -files and -archives options are generic options. Be sure to
- place the generic options before the command options, otherwise the command
- will fail.
-
-*** Making Files Available to Tasks
-
- The -files option creates a symlink in the current working directory of the
- tasks that points to the local copy of the file.
-
- In this example, Hadoop automatically creates a symlink named testfile.txt in
- the current working directory of the tasks. This symlink points to the local
- copy of testfile.txt.
-
-+---+
--files hdfs://host:fs_port/user/testfile.txt
-+---+
-
- User can specify a different symlink name for -files using #.
-
-+---+
--files hdfs://host:fs_port/user/testfile.txt#testfile
-+---+
-
- Multiple entries can be specified like this:
-
-+---+
--files hdfs://host:fs_port/user/testfile1.txt,hdfs://host:fs_port/user/testfile2.txt
-+---+
-
-*** Making Archives Available to Tasks
-
- The -archives option allows you to copy jars locally to the current working
- directory of tasks and automatically unjar the files.
-
- In this example, Hadoop automatically creates a symlink named testfile.jar in
- the current working directory of tasks. This symlink points to the directory
- that stores the unjarred contents of the uploaded jar file.
-
-+---+
--archives hdfs://host:fs_port/user/testfile.jar
-+---+
-
- User can specify a different symlink name for -archives using #.
-
-+---+
--archives hdfs://host:fs_port/user/testfile.tgz#tgzdir
-+---+
-
- In this example, the input.txt file has two lines specifying the names of the
- two files: cachedir.jar/cache.txt and cachedir.jar/cache2.txt. "cachedir.jar"
- is a symlink to the archived directory, which has the files "cache.txt" and
- "cache2.txt".
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -archives 'hdfs://hadoop-nn1.example.com/user/me/samples/cachefile/cachedir.jar' \
- -D mapreduce.job.maps=1 \
- -D mapreduce.job.reduces=1 \
- -D mapreduce.job.name="Experiment" \
- -input "/user/me/samples/cachefile/input.txt" \
- -output "/user/me/samples/cachefile/out" \
- -mapper "xargs cat" \
- -reducer "cat"
-
-$ ls test_jar/
-cache.txt cache2.txt
-
-$ jar cvf cachedir.jar -C test_jar/ .
-added manifest
-adding: cache.txt(in = 30) (out= 29)(deflated 3%)
-adding: cache2.txt(in = 37) (out= 35)(deflated 5%)
-
-$ hdfs dfs -put cachedir.jar samples/cachefile
-
-$ hdfs dfs -cat /user/me/samples/cachefile/input.txt
-cachedir.jar/cache.txt
-cachedir.jar/cache2.txt
-
-$ cat test_jar/cache.txt
-This is just the cache string
-
-$ cat test_jar/cache2.txt
-This is just the second cache string
-
-$ hdfs dfs -ls /user/me/samples/cachefile/out
-Found 2 items
--rw-r--r-- 1 me supergroup 0 2013-11-14 17:00 /user/me/samples/cachefile/out/_SUCCESS
--rw-r--r-- 1 me supergroup 69 2013-11-14 17:00 /user/me/samples/cachefile/out/part-00000
-
-$ hdfs dfs -cat /user/me/samples/cachefile/out/part-00000
-This is just the cache string
-This is just the second cache string
-+---+
-
-* More Usage Examples
-
-** Hadoop Partitioner Class
-
- Hadoop has a library class,
- {{{../../api/org/apache/hadoop/mapred/lib/KeyFieldBasedPartitioner.html}
- KeyFieldBasedPartitioner}}, that is useful for many applications. This class
- allows the Map/Reduce framework to partition the map outputs based on certain
- key fields, not the whole keys. For example:
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -D stream.map.output.field.separator=. \
- -D stream.num.map.output.key.fields=4 \
- -D map.output.key.field.separator=. \
- -D mapreduce.partition.keypartitioner.options=-k1,2 \
- -D mapreduce.job.reduces=12 \
- -input myInputDirs \
- -output myOutputDir \
- -mapper /bin/cat \
- -reducer /bin/cat \
- -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner
-+---+
-
- Here, <-D stream.map.output.field.separator=.> and
- <-D stream.num.map.output.key.fields=4> are as explained in previous example.
- The two variables are used by streaming to identify the key/value pair of
- mapper.
-
- The map output keys of the above Map/Reduce job normally have four fields
- separated by ".". However, the Map/Reduce framework will partition the map
- outputs by the first two fields of the keys using the
- <-D mapred.text.key.partitioner.options=-k1,2> option. Here,
- <-D map.output.key.field.separator=.> specifies the separator for the
- partition. This guarantees that all the key/value pairs with the same first
- two fields in the keys will be partitioned into the same reducer.
-
- <This is effectively equivalent to specifying the first two fields as the
- primary key and the next two fields as the secondary. The primary key is used
- for partitioning, and the combination of the primary and secondary keys is
- used for sorting.> A simple illustration is shown here:
-
- Output of map (the keys)
-
-+---+
-11.12.1.2
-11.14.2.3
-11.11.4.1
-11.12.1.1
-11.14.2.2
-+---+
-
- Partition into 3 reducers (the first 2 fields are used as keys for partition)
-
-+---+
-11.11.4.1
------------
-11.12.1.2
-11.12.1.1
------------
-11.14.2.3
-11.14.2.2
-+---+
-
- Sorting within each partition for the reducer(all 4 fields used for sorting)
-
-+---+
-11.11.4.1
------------
-11.12.1.1
-11.12.1.2
------------
-11.14.2.2
-11.14.2.3
-+---+
-
-** Hadoop Comparator Class
-
- Hadoop has a library class,
- {{{../../api/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedComparator.html}
- KeyFieldBasedComparator}}, that is useful for many applications. This class
- provides a subset of features provided by the Unix/GNU Sort. For example:
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -D mapreduce.job.output.key.comparator.class=org.apache.hadoop.mapreduce.lib.partition.KeyFieldBasedComparator \
- -D stream.map.output.field.separator=. \
- -D stream.num.map.output.key.fields=4 \
- -D mapreduce.map.output.key.field.separator=. \
- -D mapreduce.partition.keycomparator.options=-k2,2nr \
- -D mapreduce.job.reduces=1 \
- -input myInputDirs \
- -output myOutputDir \
- -mapper /bin/cat \
- -reducer /bin/cat
-+---+
-
- The map output keys of the above Map/Reduce job normally have four fields
- separated by ".". However, the Map/Reduce framework will sort the outputs by
- the second field of the keys using the
- <-D mapreduce.partition.keycomparator.options=-k2,2nr> option. Here, <-n>
- specifies that the sorting is numerical sorting and <-r> specifies that the
- result should be reversed. A simple illustration is shown below:
-
- Output of map (the keys)
-
-+---+
-11.12.1.2
-11.14.2.3
-11.11.4.1
-11.12.1.1
-11.14.2.2
-+---+
-
- Sorting output for the reducer (where second field used for sorting)
-
-+---+
-11.14.2.3
-11.14.2.2
-11.12.1.2
-11.12.1.1
-11.11.4.1
-+---+
-
-** Hadoop Aggregate Package
-
- Hadoop has a library package called
- {{{../../org/apache/hadoop/mapred/lib/aggregate/package-summary.html}
- Aggregate}}. Aggregate provides a special reducer class and a special
- combiner class, and a list of simple aggregators that perform aggregations
- such as "sum", "max", "min" and so on over a sequence of values. Aggregate
- allows you to define a mapper plugin class that is expected to generate
- "aggregatable items" for each input key/value pair of the mappers. The
- combiner/reducer will aggregate those aggregatable items by invoking the
- appropriate aggregators.
-
- To use Aggregate, simply specify "-reducer aggregate":
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -input myInputDirs \
- -output myOutputDir \
- -mapper myAggregatorForKeyCount.py \
- -reducer aggregate \
- -file myAggregatorForKeyCount.py \
-+---+
-
- The python program myAggregatorForKeyCount.py looks like:
-
-+---+
-#!/usr/bin/python
-
-import sys;
-
-def generateLongCountToken(id):
- return "LongValueSum:" + id + "\t" + "1"
-
-def main(argv):
- line = sys.stdin.readline();
- try:
- while line:
- line = line[:-1];
- fields = line.split("\t");
- print generateLongCountToken(fields[0]);
- line = sys.stdin.readline();
- except "end of file":
- return None
-if __name__ == "__main__":
- main(sys.argv)
-+---+
-
-** Hadoop Field Selection Class
-
- Hadoop has a library class,
- {{{../../api/org/apache/hadoop/mapred/lib/FieldSelectionMapReduce.html}
- FieldSelectionMapReduce}}, that effectively allows you to process text data
- like the unix "cut" utility. The map function defined in the class treats
- each input key/value pair as a list of fields. You can specify the field
- separator (the default is the tab character). You can select an arbitrary
- list of fields as the map output key, and an arbitrary list of fields as the
- map output value. Similarly, the reduce function defined in the class treats
- each input key/value pair as a list of fields. You can select an arbitrary
- list of fields as the reduce output key, and an arbitrary list of fields as
- the reduce output value. For example:
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -D mapreduce.map.output.key.field.separator=. \
- -D mapreduce.partition.keypartitioner.options=-k1,2 \
- -D mapreduce.fieldsel.data.field.separator=. \
- -D mapreduce.fieldsel.map.output.key.value.fields.spec=6,5,1-3:0- \
- -D mapreduce.fieldsel.reduce.output.key.value.fields.spec=0-2:5- \
- -D mapreduce.map.output.key.class=org.apache.hadoop.io.Text \
- -D mapreduce.job.reduces=12 \
- -input myInputDirs \
- -output myOutputDir \
- -mapper org.apache.hadoop.mapred.lib.FieldSelectionMapReduce \
- -reducer org.apache.hadoop.mapred.lib.FieldSelectionMapReduce \
- -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner
-+---+
-
- The option "-D
- mapreduce.fieldsel.map.output.key.value.fields.spec=6,5,1-3:0-" specifies
- key/value selection for the map outputs. Key selection spec and value
- selection spec are separated by ":". In this case, the map output key will
- consist of fields 6, 5, 1, 2, and 3. The map output value will consist of all
- fields (0- means field 0 and all the subsequent fields).
-
- The option "-D mapreduce.fieldsel.reduce.output.key.value.fields.spec=0-2:5-"
- specifies key/value selection for the reduce outputs. In this case, the
- reduce output key will consist of fields 0, 1, 2 (corresponding to the
- original fields 6, 5, 1). The reduce output value will consist of all fields
- starting from field 5 (corresponding to all the original fields).
-
-* Frequently Asked Questions
-
-** How do I use Hadoop Streaming to run an arbitrary set of (semi) independent
- tasks?
-
- Often you do not need the full power of Map Reduce, but only need to run
- multiple instances of the same program - either on different parts of the
- data, or on the same data, but with different parameters. You can use Hadoop
- Streaming to do this.
-
-** How do I process files, one per map?
-
- As an example, consider the problem of zipping (compressing) a set of files
- across the hadoop cluster. You can achieve this by using Hadoop Streaming
- and custom mapper script:
-
- * Generate a file containing the full HDFS path of the input files. Each map
- task would get one file name as input.
-
- * Create a mapper script which, given a filename, will get the file to local
- disk, gzip the file and put it back in the desired output directory.
-
-** How many reducers should I use?
-
- See MapReduce Tutorial for details: {{{./MapReduceTutorial.html#Reducer}
- Reducer}}
-
-** If I set up an alias in my shell script, will that work after -mapper?
-
- For example, say I do: alias c1='cut -f1'. Will -mapper "c1" work?
-
- Using an alias will not work, but variable substitution is allowed as shown
- in this example:
-
-+---+
-$ hdfs dfs -cat /user/me/samples/student_marks
-alice 50
-bruce 70
-charlie 80
-dan 75
-
-$ c2='cut -f2'; hadoop jar hadoop-streaming-${project.version}.jar \
- -D mapreduce.job.name='Experiment' \
- -input /user/me/samples/student_marks \
- -output /user/me/samples/student_out \
- -mapper "$c2" -reducer 'cat'
-
-$ hdfs dfs -cat /user/me/samples/student_out/part-00000
-50
-70
-75
-80
-+---+
-
-** Can I use UNIX pipes?
-
- For example, will -mapper "cut -f1 | sed s/foo/bar/g" work?
-
- Currently this does not work and gives an "java.io.IOException: Broken pipe"
- error. This is probably a bug that needs to be investigated.
-
-** What do I do if I get the "No space left on device" error?
-
- For example, when I run a streaming job by distributing large executables
- (for example, 3.6G) through the -file option, I get a "No space left on
- device" error.
-
- The jar packaging happens in a directory pointed to by the configuration
- variable stream.tmpdir. The default value of stream.tmpdir is /tmp. Set the
- value to a directory with more space:
-
-+---+
--D stream.tmpdir=/export/bigspace/...
-+---+
-
-** How do I specify multiple input directories?
-
- You can specify multiple input directories with multiple '-input' options:
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -input '/user/foo/dir1' -input '/user/foo/dir2' \
- (rest of the command)
-+---+
-
-** How do I generate output files with gzip format?
-
- Instead of plain text files, you can generate gzip files as your generated
- output. Pass '-D mapreduce.output.fileoutputformat.compress=true -D
- mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec'
- as option to your streaming job.
-
-** How do I provide my own input/output format with streaming?
-
- You can specify your own custom class by packing them and putting the custom
- jar to \$\{HADOOP_CLASSPATH\}.
-
-** How do I parse XML documents using streaming?
-
- You can use the record reader StreamXmlRecordReader to process XML documents.
-
-+---+
-hadoop jar hadoop-streaming-${project.version}.jar \
- -inputreader "StreamXmlRecord,begin=BEGIN_STRING,end=END_STRING" \
- (rest of the command)
-+---+
-
- Anything found between BEGIN_STRING and END_STRING would be treated as one
- record for map tasks.
-
-** How do I update counters in streaming applications?
-
- A streaming process can use the stderr to emit counter information.
- <<<reporter:counter:\<group\>,\<counter\>,\<amount\>>>> should be sent to
- stderr to update the counter.
-
-** How do I update status in streaming applications?
-
- A streaming process can use the stderr to emit status information. To set a
- status, <<<reporter:status:\<message\>>>> should be sent to stderr.
-
-** How do I get the Job variables in a streaming job's mapper/reducer?
-
- See {{{./MapReduceTutorial.html#Configured_Parameters}
- Configured Parameters}}. During the execution of a streaming job, the names
- of the "mapred" parameters are transformed. The dots ( . ) become underscores
- ( _ ). For example, mapreduce.job.id becomes mapreduce_job_id and
- mapreduce.job.jar becomes mapreduce_job_jar. In your code, use the parameter
- names with the underscores.
[32/52] [abbrv] hadoop git commit: YARN-3237. AppLogAggregatorImpl
fails to log error cause. Contributed by Rushabh S Shah
Posted by zh...@apache.org.
YARN-3237. AppLogAggregatorImpl fails to log error cause. Contributed by
Rushabh S Shah
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f56c65bb
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f56c65bb
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f56c65bb
Branch: refs/heads/HDFS-7285
Commit: f56c65bb3eb9436b67de2df63098e26589e70e56
Parents: 3c5ff07
Author: Xuan <xg...@apache.org>
Authored: Fri Feb 20 14:02:40 2015 -0800
Committer: Xuan <xg...@apache.org>
Committed: Fri Feb 20 14:02:40 2015 -0800
----------------------------------------------------------------------
hadoop-yarn-project/CHANGES.txt | 3 +++
.../containermanager/logaggregation/AppLogAggregatorImpl.java | 4 ++--
2 files changed, 5 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f56c65bb/hadoop-yarn-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index c028043..359e647 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -312,6 +312,9 @@ Release 2.7.0 - UNRELEASED
YARN-3230. Clarify application states on the web UI. (Jian He via wangda)
+ YARN-3237. AppLogAggregatorImpl fails to log error cause.
+ (Rushabh S Shah via xgong)
+
OPTIMIZATIONS
YARN-2990. FairScheduler's delay-scheduling always waits for node-local and
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f56c65bb/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java
index 8eb00f4..787422b 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java
@@ -248,7 +248,7 @@ public class AppLogAggregatorImpl implements AppLogAggregator {
} catch (IOException e1) {
LOG.error("Cannot create writer for app " + this.applicationId
- + ". Skip log upload this time. ");
+ + ". Skip log upload this time. ", e1);
return;
}
@@ -549,7 +549,7 @@ public class AppLogAggregatorImpl implements AppLogAggregator {
writer.append(logKey, logValue);
} catch (Exception e) {
LOG.error("Couldn't upload logs for " + containerId
- + ". Skipping this container.");
+ + ". Skipping this container.", e);
return new HashSet<Path>();
}
this.uploadedFileMeta.addAll(logValue
[41/52] [abbrv] hadoop git commit: YARN-2797.
TestWorkPreservingRMRestart should use ParametrizedSchedulerTestBase.
Contributed by Karthik Kambatla
Posted by zh...@apache.org.
YARN-2797. TestWorkPreservingRMRestart should use
ParametrizedSchedulerTestBase. Contributed by Karthik Kambatla
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/fe7a3024
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/fe7a3024
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/fe7a3024
Branch: refs/heads/HDFS-7285
Commit: fe7a302473251b7310105a936edf220e401c613f
Parents: e3d2902
Author: Xuan <xg...@apache.org>
Authored: Sat Feb 21 19:17:29 2015 -0800
Committer: Xuan <xg...@apache.org>
Committed: Sat Feb 21 19:17:29 2015 -0800
----------------------------------------------------------------------
hadoop-yarn-project/CHANGES.txt | 3 +
.../ParameterizedSchedulerTestBase.java | 12 +++
.../TestWorkPreservingRMRestart.java | 79 +++-----------------
3 files changed, 25 insertions(+), 69 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/fe7a3024/hadoop-yarn-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 1982688..3507420 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -318,6 +318,9 @@ Release 2.7.0 - UNRELEASED
YARN-3236. Cleanup RMAuthenticationFilter#AUTH_HANDLER_PROPERTY.
(zhihai xu via xgong)
+ YARN-2797. TestWorkPreservingRMRestart should use ParametrizedSchedulerTestBase
+ (Karthik Kambatla via xgong)
+
OPTIMIZATIONS
YARN-2990. FairScheduler's delay-scheduling always waits for node-local and
http://git-wip-us.apache.org/repos/asf/hadoop/blob/fe7a3024/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ParameterizedSchedulerTestBase.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ParameterizedSchedulerTestBase.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ParameterizedSchedulerTestBase.java
index cfd1600..b099836 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ParameterizedSchedulerTestBase.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ParameterizedSchedulerTestBase.java
@@ -83,10 +83,22 @@ public abstract class ParameterizedSchedulerTestBase {
out.println("<?xml version=\"1.0\"?>");
out.println("<allocations>");
out.println("<queueMaxAMShareDefault>-1.0</queueMaxAMShareDefault>");
+ out.println("<defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy>");
+ out.println("<queue name=\"root\">");
+ out.println(" <schedulingPolicy>drf</schedulingPolicy>");
+ out.println(" <weight>1.0</weight>");
+ out.println(" <fairSharePreemptionTimeout>100</fairSharePreemptionTimeout>");
+ out.println(" <minSharePreemptionTimeout>120</minSharePreemptionTimeout>");
+ out.println(" <fairSharePreemptionThreshold>.5</fairSharePreemptionThreshold>");
+ out.println("</queue>");
out.println("</allocations>");
out.close();
conf.set(YarnConfiguration.RM_SCHEDULER, FairScheduler.class.getName());
conf.set(FairSchedulerConfiguration.ALLOCATION_FILE, FS_ALLOC_FILE);
}
+
+ public SchedulerType getSchedulerType() {
+ return schedulerType;
+ }
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/fe7a3024/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
index a9caf77..3033496 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
@@ -97,23 +97,24 @@ import com.google.common.base.Supplier;
@SuppressWarnings({"rawtypes", "unchecked"})
@RunWith(value = Parameterized.class)
-public class TestWorkPreservingRMRestart {
+public class TestWorkPreservingRMRestart extends ParameterizedSchedulerTestBase {
private YarnConfiguration conf;
- private Class<?> schedulerClass;
MockRM rm1 = null;
MockRM rm2 = null;
+ public TestWorkPreservingRMRestart(SchedulerType type) {
+ super(type);
+ }
+
@Before
public void setup() throws UnknownHostException {
Logger rootLogger = LogManager.getRootLogger();
rootLogger.setLevel(Level.DEBUG);
- conf = new YarnConfiguration();
+ conf = getConf();
UserGroupInformation.setConfiguration(conf);
conf.set(YarnConfiguration.RECOVERY_ENABLED, "true");
conf.set(YarnConfiguration.RM_STORE, MemoryRMStateStore.class.getName());
- conf.setClass(YarnConfiguration.RM_SCHEDULER, schedulerClass,
- ResourceScheduler.class);
conf.setBoolean(YarnConfiguration.RM_WORK_PRESERVING_RECOVERY_ENABLED, true);
conf.setLong(YarnConfiguration.RM_WORK_PRESERVING_RECOVERY_SCHEDULING_WAIT_MS, 0);
DefaultMetricsSystem.setMiniClusterMode(true);
@@ -129,16 +130,6 @@ public class TestWorkPreservingRMRestart {
}
}
- @Parameterized.Parameters
- public static Collection<Object[]> getTestParameters() {
- return Arrays.asList(new Object[][] { { CapacityScheduler.class },
- { FifoScheduler.class }, {FairScheduler.class } });
- }
-
- public TestWorkPreservingRMRestart(Class<?> schedulerClass) {
- this.schedulerClass = schedulerClass;
- }
-
// Test common scheduler state including SchedulerAttempt, SchedulerNode,
// AppSchedulingInfo can be reconstructed via the container recovery reports
// on NM re-registration.
@@ -159,9 +150,6 @@ public class TestWorkPreservingRMRestart {
MemoryRMStateStore memStore = new MemoryRMStateStore();
memStore.init(conf);
rm1 = new MockRM(conf, memStore);
- if (schedulerClass.equals(FairScheduler.class)) {
- initFairScheduler(rm1);
- }
rm1.start();
MockNM nm1 =
new MockNM("127.0.0.1:1234", 8192, rm1.getResourceTrackerService());
@@ -174,9 +162,6 @@ public class TestWorkPreservingRMRestart {
// Re-start RM
rm2 = new MockRM(conf, memStore);
- if (schedulerClass.equals(FairScheduler.class)) {
- initFairScheduler(rm2);
- }
rm2.start();
nm1.setResourceTrackerService(rm2.getResourceTrackerService());
// recover app
@@ -249,11 +234,9 @@ public class TestWorkPreservingRMRestart {
SchedulerApplication schedulerApp =
schedulerApps.get(recoveredApp1.getApplicationId());
- if (schedulerClass.equals(CapacityScheduler.class)) {
+ if (getSchedulerType() == SchedulerType.CAPACITY) {
checkCSQueue(rm2, schedulerApp, nmResource, nmResource, usedResources, 2);
- } else if (schedulerClass.equals(FifoScheduler.class)) {
- checkFifoQueue(rm2, schedulerApp, usedResources, availableResources);
- } else if (schedulerClass.equals(FairScheduler.class)) {
+ } else {
checkFSQueue(rm2, schedulerApp, usedResources, availableResources);
}
@@ -324,25 +307,6 @@ public class TestWorkPreservingRMRestart {
.getUsed());
}
- private void checkFifoQueue(ResourceManager rm,
- SchedulerApplication schedulerApp, Resource usedResources,
- Resource availableResources) throws Exception {
- FifoScheduler scheduler = (FifoScheduler) rm.getResourceScheduler();
- // ************ check cluster used Resources ********
- assertEquals(usedResources, scheduler.getUsedResource());
-
- // ************ check app headroom ****************
- SchedulerApplicationAttempt schedulerAttempt =
- schedulerApp.getCurrentAppAttempt();
- assertEquals(availableResources, schedulerAttempt.getHeadroom());
-
- // ************ check queue metrics ****************
- QueueMetrics queueMetrics = scheduler.getRootQueueMetrics();
- assertMetrics(queueMetrics, 1, 0, 1, 0, 2, availableResources.getMemory(),
- availableResources.getVirtualCores(), usedResources.getMemory(),
- usedResources.getVirtualCores());
- }
-
private void checkFSQueue(ResourceManager rm,
SchedulerApplication schedulerApp, Resource usedResources,
Resource availableResources) throws Exception {
@@ -379,29 +343,6 @@ public class TestWorkPreservingRMRestart {
usedResources.getVirtualCores());
}
- private void initFairScheduler(ResourceManager rm) throws IOException {
- FairScheduler scheduler = (FairScheduler) rm.getResourceScheduler();
- String testDir =
- new File(
- System.getProperty("test.build.data", "/tmp")).getAbsolutePath();
- String allocFile = new File(testDir, "test-queues").getAbsolutePath();
- conf.set(FairSchedulerConfiguration.ALLOCATION_FILE, allocFile);
-
- PrintWriter out = new PrintWriter(new FileWriter(allocFile));
- out.println("<?xml version=\"1.0\"?>");
- out.println("<allocations>");
- out.println("<defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy>");
- out.println("<queue name=\"root\">");
- out.println(" <schedulingPolicy>drf</schedulingPolicy>");
- out.println(" <weight>1.0</weight>");
- out.println(" <fairSharePreemptionTimeout>100</fairSharePreemptionTimeout>");
- out.println(" <minSharePreemptionTimeout>120</minSharePreemptionTimeout>");
- out.println(" <fairSharePreemptionThreshold>.5</fairSharePreemptionThreshold>");
- out.println("</queue>");
- out.println("</allocations>");
- out.close();
- }
-
// create 3 container reports for AM
public static List<NMContainerStatus>
createNMContainerStatusForApp(MockAM am) {
@@ -468,7 +409,7 @@ public class TestWorkPreservingRMRestart {
// 10. Assert each user's consumption inside the queue.
@Test (timeout = 30000)
public void testCapacitySchedulerRecovery() throws Exception {
- if (!schedulerClass.equals(CapacityScheduler.class)) {
+ if (getSchedulerType() != SchedulerType.CAPACITY) {
return;
}
conf.setBoolean(CapacitySchedulerConfiguration.ENABLE_USER_METRICS, true);
@@ -587,7 +528,7 @@ public class TestWorkPreservingRMRestart {
//3. Verify that the expected exception was thrown
@Test (timeout = 30000, expected = QueueNotFoundException.class)
public void testCapacitySchedulerQueueRemovedRecovery() throws Exception {
- if (!schedulerClass.equals(CapacityScheduler.class)) {
+ if (getSchedulerType() != SchedulerType.CAPACITY) {
throw new QueueNotFoundException("Dummy");
}
conf.setBoolean(CapacitySchedulerConfiguration.ENABLE_USER_METRICS, true);
[49/52] [abbrv] hadoop git commit: Added the missed entry for commit
of HADOOP-11541
Posted by zh...@apache.org.
Added the missed entry for commit of HADOOP-11541
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/0d3b4623
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/0d3b4623
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/0d3b4623
Branch: refs/heads/HDFS-7285
Commit: 0d3b4623aa1c61da7b20c438bd214f859b4a3906
Parents: 14248ad
Author: drankye <dr...@gmail.com>
Authored: Mon Feb 9 22:04:08 2015 +0800
Committer: Zhe Zhang <zh...@apache.org>
Committed: Mon Feb 23 11:20:37 2015 -0800
----------------------------------------------------------------------
hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/0d3b4623/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt b/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt
index 2124800..9728f97 100644
--- a/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt
+++ b/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt
@@ -4,4 +4,7 @@
(Kai Zheng via umamahesh)
HADOOP-11534. Minor improvements for raw erasure coders
- ( Kai Zheng via vinayakumarb )
\ No newline at end of file
+ ( Kai Zheng via vinayakumarb )
+
+ HADOOP-11541. Raw XOR coder
+ ( Kai Zheng )
[46/52] [abbrv] hadoop git commit: HADOOP-11514. Raw Erasure Coder
API for concrete encoding and decoding (Kai Zheng via umamahesh)
Posted by zh...@apache.org.
HADOOP-11514. Raw Erasure Coder API for concrete encoding and decoding (Kai Zheng via umamahesh)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/65dca224
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/65dca224
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/65dca224
Branch: refs/heads/HDFS-7285
Commit: 65dca22456925a0516f9e515ee5fe901f918fae6
Parents: 08a9ac3
Author: Uma Maheswara Rao G <um...@apache.org>
Authored: Thu Jan 29 14:15:13 2015 +0530
Committer: Zhe Zhang <zh...@apache.org>
Committed: Mon Feb 23 11:20:19 2015 -0800
----------------------------------------------------------------------
.../hadoop-common/CHANGES-HDFS-EC-7285.txt | 4 +
.../apache/hadoop/io/erasurecode/ECChunk.java | 82 +++++++++++++++++
.../rawcoder/AbstractRawErasureCoder.java | 63 +++++++++++++
.../rawcoder/AbstractRawErasureDecoder.java | 93 ++++++++++++++++++++
.../rawcoder/AbstractRawErasureEncoder.java | 93 ++++++++++++++++++++
.../erasurecode/rawcoder/RawErasureCoder.java | 78 ++++++++++++++++
.../erasurecode/rawcoder/RawErasureDecoder.java | 55 ++++++++++++
.../erasurecode/rawcoder/RawErasureEncoder.java | 54 ++++++++++++
8 files changed, 522 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/65dca224/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt b/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt
new file mode 100644
index 0000000..8ce5a89
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/CHANGES-HDFS-EC-7285.txt
@@ -0,0 +1,4 @@
+ BREAKDOWN OF HADOOP-11264 SUBTASKS AND RELATED JIRAS (Common part of HDFS-7285)
+
+ HADOOP-11514. Raw Erasure Coder API for concrete encoding and decoding
+ (Kai Zheng via umamahesh)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/65dca224/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECChunk.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECChunk.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECChunk.java
new file mode 100644
index 0000000..f84eb11
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECChunk.java
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.io.erasurecode;
+
+import java.nio.ByteBuffer;
+
+/**
+ * A wrapper for ByteBuffer or bytes array for an erasure code chunk.
+ */
+public class ECChunk {
+
+ private ByteBuffer chunkBuffer;
+
+ /**
+ * Wrapping a ByteBuffer
+ * @param buffer
+ */
+ public ECChunk(ByteBuffer buffer) {
+ this.chunkBuffer = buffer;
+ }
+
+ /**
+ * Wrapping a bytes array
+ * @param buffer
+ */
+ public ECChunk(byte[] buffer) {
+ this.chunkBuffer = ByteBuffer.wrap(buffer);
+ }
+
+ /**
+ * Convert to ByteBuffer
+ * @return ByteBuffer
+ */
+ public ByteBuffer getBuffer() {
+ return chunkBuffer;
+ }
+
+ /**
+ * Convert an array of this chunks to an array of ByteBuffers
+ * @param chunks
+ * @return an array of ByteBuffers
+ */
+ public static ByteBuffer[] toBuffers(ECChunk[] chunks) {
+ ByteBuffer[] buffers = new ByteBuffer[chunks.length];
+
+ for (int i = 0; i < chunks.length; i++) {
+ buffers[i] = chunks[i].getBuffer();
+ }
+
+ return buffers;
+ }
+
+ /**
+ * Convert an array of this chunks to an array of byte array
+ * @param chunks
+ * @return an array of byte array
+ */
+ public static byte[][] toArray(ECChunk[] chunks) {
+ byte[][] bytesArr = new byte[chunks.length][];
+
+ for (int i = 0; i < chunks.length; i++) {
+ bytesArr[i] = chunks[i].getBuffer().array();
+ }
+
+ return bytesArr;
+ }
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/65dca224/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureCoder.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureCoder.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureCoder.java
new file mode 100644
index 0000000..474542b
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureCoder.java
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.io.erasurecode.rawcoder;
+
+/**
+ * A common class of basic facilities to be shared by encoder and decoder
+ *
+ * It implements the {@link RawErasureCoder} interface.
+ */
+public abstract class AbstractRawErasureCoder implements RawErasureCoder {
+
+ private int dataSize;
+ private int paritySize;
+ private int chunkSize;
+
+ @Override
+ public void initialize(int numDataUnits, int numParityUnits,
+ int chunkSize) {
+ this.dataSize = numDataUnits;
+ this.paritySize = numParityUnits;
+ this.chunkSize = chunkSize;
+ }
+
+ @Override
+ public int getNumDataUnits() {
+ return dataSize;
+ }
+
+ @Override
+ public int getNumParityUnits() {
+ return paritySize;
+ }
+
+ @Override
+ public int getChunkSize() {
+ return chunkSize;
+ }
+
+ @Override
+ public boolean preferNativeBuffer() {
+ return false;
+ }
+
+ @Override
+ public void release() {
+ // Nothing to do by default
+ }
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/65dca224/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureDecoder.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureDecoder.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureDecoder.java
new file mode 100644
index 0000000..4613b25
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureDecoder.java
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.io.erasurecode.rawcoder;
+
+import org.apache.hadoop.io.erasurecode.ECChunk;
+
+import java.nio.ByteBuffer;
+
+/**
+ * An abstract raw erasure decoder that's to be inherited by new decoders.
+ *
+ * It implements the {@link RawErasureDecoder} interface.
+ */
+public abstract class AbstractRawErasureDecoder extends AbstractRawErasureCoder
+ implements RawErasureDecoder {
+
+ @Override
+ public void decode(ByteBuffer[] inputs, int[] erasedIndexes,
+ ByteBuffer[] outputs) {
+ if (erasedIndexes.length == 0) {
+ return;
+ }
+
+ doDecode(inputs, erasedIndexes, outputs);
+ }
+
+ /**
+ * Perform the real decoding using ByteBuffer
+ * @param inputs
+ * @param erasedIndexes
+ * @param outputs
+ */
+ protected abstract void doDecode(ByteBuffer[] inputs, int[] erasedIndexes,
+ ByteBuffer[] outputs);
+
+ @Override
+ public void decode(byte[][] inputs, int[] erasedIndexes, byte[][] outputs) {
+ if (erasedIndexes.length == 0) {
+ return;
+ }
+
+ doDecode(inputs, erasedIndexes, outputs);
+ }
+
+ /**
+ * Perform the real decoding using bytes array
+ * @param inputs
+ * @param erasedIndexes
+ * @param outputs
+ */
+ protected abstract void doDecode(byte[][] inputs, int[] erasedIndexes,
+ byte[][] outputs);
+
+ @Override
+ public void decode(ECChunk[] inputs, int[] erasedIndexes,
+ ECChunk[] outputs) {
+ doDecode(inputs, erasedIndexes, outputs);
+ }
+
+ /**
+ * Perform the real decoding using chunks
+ * @param inputs
+ * @param erasedIndexes
+ * @param outputs
+ */
+ protected void doDecode(ECChunk[] inputs, int[] erasedIndexes,
+ ECChunk[] outputs) {
+ if (inputs[0].getBuffer().hasArray()) {
+ byte[][] inputBytesArr = ECChunk.toArray(inputs);
+ byte[][] outputBytesArr = ECChunk.toArray(outputs);
+ doDecode(inputBytesArr, erasedIndexes, outputBytesArr);
+ } else {
+ ByteBuffer[] inputBuffers = ECChunk.toBuffers(inputs);
+ ByteBuffer[] outputBuffers = ECChunk.toBuffers(outputs);
+ doDecode(inputBuffers, erasedIndexes, outputBuffers);
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/65dca224/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureEncoder.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureEncoder.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureEncoder.java
new file mode 100644
index 0000000..4feaf39
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureEncoder.java
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.io.erasurecode.rawcoder;
+
+import org.apache.hadoop.io.erasurecode.ECChunk;
+
+import java.nio.ByteBuffer;
+
+/**
+ * An abstract raw erasure encoder that's to be inherited by new encoders.
+ *
+ * It implements the {@link RawErasureEncoder} interface.
+ */
+public abstract class AbstractRawErasureEncoder extends AbstractRawErasureCoder
+ implements RawErasureEncoder {
+
+ @Override
+ public void encode(ByteBuffer[] inputs, ByteBuffer[] outputs) {
+ assert (inputs.length == getNumDataUnits());
+ assert (outputs.length == getNumParityUnits());
+
+ doEncode(inputs, outputs);
+ }
+
+ /**
+ * Perform the real encoding work using ByteBuffer
+ * @param inputs
+ * @param outputs
+ */
+ protected abstract void doEncode(ByteBuffer[] inputs, ByteBuffer[] outputs);
+
+ @Override
+ public void encode(byte[][] inputs, byte[][] outputs) {
+ assert (inputs.length == getNumDataUnits());
+ assert (outputs.length == getNumParityUnits());
+
+ doEncode(inputs, outputs);
+ }
+
+ /**
+ * Perform the real encoding work using bytes array
+ * @param inputs
+ * @param outputs
+ */
+ protected abstract void doEncode(byte[][] inputs, byte[][] outputs);
+
+ @Override
+ public void encode(ECChunk[] inputs, ECChunk[] outputs) {
+ assert (inputs.length == getNumDataUnits());
+ assert (outputs.length == getNumParityUnits());
+
+ doEncode(inputs, outputs);
+ }
+
+ /**
+ * Perform the real encoding work using chunks.
+ * @param inputs
+ * @param outputs
+ */
+ protected void doEncode(ECChunk[] inputs, ECChunk[] outputs) {
+ /**
+ * Note callers may pass byte array, or ByteBuffer via ECChunk according
+ * to how ECChunk is created. Some implementations of coder use byte array
+ * (ex: pure Java), some use native ByteBuffer (ex: ISA-L), all for the
+ * better performance.
+ */
+ if (inputs[0].getBuffer().hasArray()) {
+ byte[][] inputBytesArr = ECChunk.toArray(inputs);
+ byte[][] outputBytesArr = ECChunk.toArray(outputs);
+ doEncode(inputBytesArr, outputBytesArr);
+ } else {
+ ByteBuffer[] inputBuffers = ECChunk.toBuffers(inputs);
+ ByteBuffer[] outputBuffers = ECChunk.toBuffers(outputs);
+ doEncode(inputBuffers, outputBuffers);
+ }
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/65dca224/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureCoder.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureCoder.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureCoder.java
new file mode 100644
index 0000000..91a9abf
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureCoder.java
@@ -0,0 +1,78 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.io.erasurecode.rawcoder;
+
+/**
+ * RawErasureCoder is a common interface for {@link RawErasureEncoder} and
+ * {@link RawErasureDecoder} as both encoder and decoder share some properties.
+ *
+ * RawErasureCoder is part of ErasureCodec framework, where ErasureCoder is
+ * used to encode/decode a group of blocks (BlockGroup) according to the codec
+ * specific BlockGroup layout and logic. An ErasureCoder extracts chunks of
+ * data from the blocks and can employ various low level RawErasureCoders to
+ * perform encoding/decoding against the chunks.
+ *
+ * To distinguish from ErasureCoder, here RawErasureCoder is used to mean the
+ * low level constructs, since it only takes care of the math calculation with
+ * a group of byte buffers.
+ */
+public interface RawErasureCoder {
+
+ /**
+ * Initialize with the important parameters for the code.
+ * @param numDataUnits how many data inputs for the coding
+ * @param numParityUnits how many parity outputs the coding generates
+ * @param chunkSize the size of the input/output buffer
+ */
+ public void initialize(int numDataUnits, int numParityUnits, int chunkSize);
+
+ /**
+ * The number of data input units for the coding. A unit can be a byte,
+ * chunk or buffer or even a block.
+ * @return count of data input units
+ */
+ public int getNumDataUnits();
+
+ /**
+ * The number of parity output units for the coding. A unit can be a byte,
+ * chunk, buffer or even a block.
+ * @return count of parity output units
+ */
+ public int getNumParityUnits();
+
+ /**
+ * Chunk buffer size for the input/output
+ * @return chunk buffer size
+ */
+ public int getChunkSize();
+
+ /**
+ * Tell if native or off-heap buffer is preferred or not. It's for callers to
+ * decide how to allocate coding chunk buffers, either on heap or off heap.
+ * It will return false by default.
+ * @return true if native buffer is preferred for performance consideration,
+ * otherwise false.
+ */
+ public boolean preferNativeBuffer();
+
+ /**
+ * Should be called when release this coder. Good chance to release encoding
+ * or decoding buffers
+ */
+ public void release();
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/65dca224/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureDecoder.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureDecoder.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureDecoder.java
new file mode 100644
index 0000000..1358b7d
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureDecoder.java
@@ -0,0 +1,55 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.io.erasurecode.rawcoder;
+
+import org.apache.hadoop.io.erasurecode.ECChunk;
+
+import java.nio.ByteBuffer;
+
+/**
+ * RawErasureDecoder performs decoding given chunks of input data and generates
+ * missing data that corresponds to an erasure code scheme, like XOR and
+ * Reed-Solomon.
+ *
+ * It extends the {@link RawErasureCoder} interface.
+ */
+public interface RawErasureDecoder extends RawErasureCoder {
+
+ /**
+ * Decode with inputs and erasedIndexes, generates outputs
+ * @param inputs
+ * @param outputs
+ */
+ public void decode(ByteBuffer[] inputs, int[] erasedIndexes,
+ ByteBuffer[] outputs);
+
+ /**
+ * Decode with inputs and erasedIndexes, generates outputs
+ * @param inputs
+ * @param outputs
+ */
+ public void decode(byte[][] inputs, int[] erasedIndexes, byte[][] outputs);
+
+ /**
+ * Decode with inputs and erasedIndexes, generates outputs
+ * @param inputs
+ * @param outputs
+ */
+ public void decode(ECChunk[] inputs, int[] erasedIndexes, ECChunk[] outputs);
+
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/65dca224/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureEncoder.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureEncoder.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureEncoder.java
new file mode 100644
index 0000000..974f86c
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureEncoder.java
@@ -0,0 +1,54 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.io.erasurecode.rawcoder;
+
+import org.apache.hadoop.io.erasurecode.ECChunk;
+
+import java.nio.ByteBuffer;
+
+/**
+ * RawErasureEncoder performs encoding given chunks of input data and generates
+ * parity outputs that corresponds to an erasure code scheme, like XOR and
+ * Reed-Solomon.
+ *
+ * It extends the {@link RawErasureCoder} interface.
+ */
+public interface RawErasureEncoder extends RawErasureCoder {
+
+ /**
+ * Encode with inputs and generates outputs
+ * @param inputs
+ * @param outputs
+ */
+ public void encode(ByteBuffer[] inputs, ByteBuffer[] outputs);
+
+ /**
+ * Encode with inputs and generates outputs
+ * @param inputs
+ * @param outputs
+ */
+ public void encode(byte[][] inputs, byte[][] outputs);
+
+ /**
+ * Encode with inputs and generates outputs
+ * @param inputs
+ * @param outputs
+ */
+ public void encode(ECChunk[] inputs, ECChunk[] outputs);
+
+}
[43/52] [abbrv] hadoop git commit: HDFS-7339. Allocating and
persisting block groups in NameNode. Contributed by Zhe Zhang
Posted by zh...@apache.org.
HDFS-7339. Allocating and persisting block groups in NameNode. Contributed by Zhe Zhang
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/6d6fffc2
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/6d6fffc2
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/6d6fffc2
Branch: refs/heads/HDFS-7285
Commit: 6d6fffc229adc487f6f3f70fb70bff0ad71b5f60
Parents: 9815e23
Author: Zhe Zhang <zh...@apache.org>
Authored: Fri Jan 30 16:16:26 2015 -0800
Committer: Zhe Zhang <zh...@apache.org>
Committed: Mon Feb 23 11:19:31 2015 -0800
----------------------------------------------------------------------
.../org/apache/hadoop/hdfs/DFSConfigKeys.java | 2 +
.../hadoop/hdfs/protocol/HdfsConstants.java | 4 +
.../server/blockmanagement/BlockIdManager.java | 8 +-
.../SequentialBlockGroupIdGenerator.java | 82 +++++++++++++++++++
.../SequentialBlockIdGenerator.java | 6 +-
.../hdfs/server/namenode/FSDirectory.java | 8 +-
.../hdfs/server/namenode/FSNamesystem.java | 34 +++++---
.../hadoop/hdfs/server/namenode/INodeFile.java | 11 +++
.../hdfs/server/namenode/TestAddBlockgroup.java | 84 ++++++++++++++++++++
9 files changed, 223 insertions(+), 16 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6d6fffc2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
index 975f023..9e9cd40 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
@@ -219,6 +219,8 @@ public class DFSConfigKeys extends CommonConfigurationKeys {
public static final int DFS_NAMENODE_REPLICATION_INTERVAL_DEFAULT = 3;
public static final String DFS_NAMENODE_REPLICATION_MIN_KEY = "dfs.namenode.replication.min";
public static final int DFS_NAMENODE_REPLICATION_MIN_DEFAULT = 1;
+ public static final String DFS_NAMENODE_STRIPE_MIN_KEY = "dfs.namenode.stripe.min";
+ public static final int DFS_NAMENODE_STRIPE_MIN_DEFAULT = 1;
public static final String DFS_NAMENODE_REPLICATION_PENDING_TIMEOUT_SEC_KEY = "dfs.namenode.replication.pending.timeout-sec";
public static final int DFS_NAMENODE_REPLICATION_PENDING_TIMEOUT_SEC_DEFAULT = -1;
public static final String DFS_NAMENODE_REPLICATION_MAX_STREAMS_KEY = "dfs.namenode.replication.max-streams";
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6d6fffc2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
index 6945074..e3e3f37 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
@@ -181,4 +181,8 @@ public class HdfsConstants {
public static final byte WARM_STORAGE_POLICY_ID = 5;
public static final byte EC_STORAGE_POLICY_ID = 4;
public static final byte COLD_STORAGE_POLICY_ID = 2;
+
+ public static final byte NUM_DATA_BLOCKS = 3;
+ public static final byte NUM_PARITY_BLOCKS = 2;
+ public static final byte MAX_BLOCKS_IN_GROUP = 16;
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6d6fffc2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
index 1c69203..c8b9d20 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockIdManager.java
@@ -53,10 +53,12 @@ public class BlockIdManager {
* The global block ID space for this file system.
*/
private final SequentialBlockIdGenerator blockIdGenerator;
+ private final SequentialBlockGroupIdGenerator blockGroupIdGenerator;
public BlockIdManager(BlockManager blockManager) {
this.generationStampV1Limit = GenerationStamp.GRANDFATHER_GENERATION_STAMP;
this.blockIdGenerator = new SequentialBlockIdGenerator(blockManager);
+ this.blockGroupIdGenerator = new SequentialBlockGroupIdGenerator(blockManager);
}
/**
@@ -190,6 +192,10 @@ public class BlockIdManager {
return blockIdGenerator.nextValue();
}
+ public long nextBlockGroupId() {
+ return blockGroupIdGenerator.nextValue();
+ }
+
public boolean isGenStampInFuture(Block block) {
if (isLegacyBlock(block)) {
return block.getGenerationStamp() > getGenerationStampV1();
@@ -205,4 +211,4 @@ public class BlockIdManager {
.LAST_RESERVED_BLOCK_ID);
generationStampV1Limit = GenerationStamp.GRANDFATHER_GENERATION_STAMP;
}
-}
\ No newline at end of file
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6d6fffc2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockGroupIdGenerator.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockGroupIdGenerator.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockGroupIdGenerator.java
new file mode 100644
index 0000000..e9e22ee
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockGroupIdGenerator.java
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.hdfs.protocol.HdfsConstants;
+import org.apache.hadoop.util.SequentialNumber;
+
+/**
+ * Generate the next valid block group ID by incrementing the maximum block
+ * group ID allocated so far, with the first 2^10 block group IDs reserved.
+ * HDFS-EC introduces a hierarchical protocol to name blocks and groups:
+ * Contiguous: {reserved block IDs | flag | block ID}
+ * Striped: {reserved block IDs | flag | block group ID | index in group}
+ *
+ * Following n bits of reserved block IDs, The (n+1)th bit in an ID
+ * distinguishes contiguous (0) and striped (1) blocks. For a striped block,
+ * bits (n+2) to (64-m) represent the ID of its block group, while the last m
+ * bits represent its index of the group. The value m is determined by the
+ * maximum number of blocks in a group (MAX_BLOCKS_IN_GROUP).
+ */
+@InterfaceAudience.Private
+public class SequentialBlockGroupIdGenerator extends SequentialNumber {
+
+ private final BlockManager blockManager;
+
+ SequentialBlockGroupIdGenerator(BlockManager blockManagerRef) {
+ super(Long.MIN_VALUE);
+ this.blockManager = blockManagerRef;
+ }
+
+ @Override // NumberGenerator
+ public long nextValue() {
+ // Skip to next legitimate block group ID based on the naming protocol
+ while (super.getCurrentValue() % HdfsConstants.MAX_BLOCKS_IN_GROUP > 0) {
+ super.nextValue();
+ }
+ // Make sure there's no conflict with existing random block IDs
+ while (hasValidBlockInRange(super.getCurrentValue())) {
+ super.skipTo(super.getCurrentValue() +
+ HdfsConstants.MAX_BLOCKS_IN_GROUP);
+ }
+ if (super.getCurrentValue() >= 0) {
+ BlockManager.LOG.warn("All negative block group IDs are used, " +
+ "growing into positive IDs, " +
+ "which might conflict with non-erasure coded blocks.");
+ }
+ return super.getCurrentValue();
+ }
+
+ /**
+ *
+ * @param id The starting ID of the range
+ * @return true if any ID in the range
+ * {id, id+HdfsConstants.MAX_BLOCKS_IN_GROUP} is pointed-to by a file
+ */
+ private boolean hasValidBlockInRange(long id) {
+ for (int i = 0; i < HdfsConstants.MAX_BLOCKS_IN_GROUP; i++) {
+ Block b = new Block(id + i);
+ if (blockManager.getBlockCollection(b) != null) {
+ return true;
+ }
+ }
+ return false;
+ }
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6d6fffc2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockIdGenerator.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockIdGenerator.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockIdGenerator.java
index eef8857..c97de4b 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockIdGenerator.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockIdGenerator.java
@@ -19,7 +19,6 @@ package org.apache.hadoop.hdfs.server.blockmanagement;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.hdfs.protocol.Block;
-import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager;
import org.apache.hadoop.util.SequentialNumber;
/**
@@ -54,6 +53,11 @@ public class SequentialBlockIdGenerator extends SequentialNumber {
while(isValidBlock(b)) {
b.setBlockId(super.nextValue());
}
+ if (b.getBlockId() < 0) {
+ BlockManager.LOG.warn("All positive block IDs are used, " +
+ "wrapping to negative IDs, " +
+ "which might conflict with erasure coded block groups.");
+ }
return b.getBlockId();
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6d6fffc2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
index f6ab077..627abfa 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
@@ -466,10 +466,14 @@ public class FSDirectory implements Closeable {
* Add a block to the file. Returns a reference to the added block.
*/
BlockInfoContiguous addBlock(String path, INodesInPath inodesInPath,
- Block block, DatanodeStorageInfo[] targets) throws IOException {
+ Block block, DatanodeStorageInfo[] targets,
+ boolean isStriped) throws IOException {
writeLock();
try {
final INodeFile fileINode = inodesInPath.getLastINode().asFile();
+ short numLocations = isStriped ?
+ HdfsConstants.NUM_DATA_BLOCKS + HdfsConstants.NUM_PARITY_BLOCKS :
+ fileINode.getFileReplication();
Preconditions.checkState(fileINode.isUnderConstruction());
// check quota limits and updated space consumed
@@ -480,7 +484,7 @@ public class FSDirectory implements Closeable {
BlockInfoContiguousUnderConstruction blockInfo =
new BlockInfoContiguousUnderConstruction(
block,
- fileINode.getFileReplication(),
+ numLocations,
BlockUCState.UNDER_CONSTRUCTION,
targets);
getBlockManager().addBlockCollection(blockInfo, fileINode);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6d6fffc2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
index 120a597..5b01ebc 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
@@ -2022,7 +2022,7 @@ public class FSNamesystem implements Namesystem, FSNamesystemMBean,
BlockInfoContiguous oldBlock = file.getLastBlock();
boolean shouldCopyOnTruncate = shouldCopyOnTruncate(file, oldBlock);
if(newBlock == null) {
- newBlock = (shouldCopyOnTruncate) ? createNewBlock() :
+ newBlock = (shouldCopyOnTruncate) ? createNewBlock(file.isStriped()) :
new Block(oldBlock.getBlockId(), oldBlock.getNumBytes(),
nextGenerationStamp(blockIdManager.isLegacyBlock(oldBlock)));
}
@@ -2914,8 +2914,9 @@ public class FSNamesystem implements Namesystem, FSNamesystemMBean,
ExtendedBlock previous, Set<Node> excludedNodes,
List<String> favoredNodes) throws IOException {
final long blockSize;
- final int replication;
+ final short numTargets;
final byte storagePolicyID;
+ final boolean isStriped;
Node clientNode = null;
String clientMachine = null;
@@ -2953,7 +2954,11 @@ public class FSNamesystem implements Namesystem, FSNamesystemMBean,
.getClientMachine();
clientNode = blockManager.getDatanodeManager().getDatanodeByHost(
clientMachine);
- replication = pendingFile.getFileReplication();
+ // TODO: make block group size configurable (HDFS-7337)
+ isStriped = pendingFile.isStriped();
+ numTargets = isStriped ?
+ HdfsConstants.NUM_DATA_BLOCKS + HdfsConstants.NUM_PARITY_BLOCKS :
+ pendingFile.getFileReplication();
storagePolicyID = pendingFile.getStoragePolicyID();
} finally {
readUnlock();
@@ -2965,7 +2970,7 @@ public class FSNamesystem implements Namesystem, FSNamesystemMBean,
// choose targets for the new block to be allocated.
final DatanodeStorageInfo targets[] = getBlockManager().chooseTarget4NewBlock(
- src, replication, clientNode, excludedNodes, blockSize, favoredNodes,
+ src, numTargets, clientNode, excludedNodes, blockSize, favoredNodes,
storagePolicyID);
// Part II.
@@ -3004,9 +3009,9 @@ public class FSNamesystem implements Namesystem, FSNamesystemMBean,
ExtendedBlock.getLocalBlock(previous));
// allocate new block, record block locations in INode.
- newBlock = createNewBlock();
+ newBlock = createNewBlock(isStriped);
INodesInPath inodesInPath = INodesInPath.fromINode(pendingFile);
- saveAllocatedBlock(src, inodesInPath, newBlock, targets);
+ saveAllocatedBlock(src, inodesInPath, newBlock, targets, isStriped);
persistNewBlock(src, pendingFile);
offset = pendingFile.computeFileSize();
@@ -3427,13 +3432,15 @@ public class FSNamesystem implements Namesystem, FSNamesystemMBean,
* The last INode is the INode for {@code src} file.
* @param newBlock newly allocated block to be save
* @param targets target datanodes where replicas of the new block is placed
+ * @param isStriped is the file under striping or contigunous layout?
* @throws QuotaExceededException If addition of block exceeds space quota
*/
BlockInfoContiguous saveAllocatedBlock(String src, INodesInPath inodesInPath,
- Block newBlock, DatanodeStorageInfo[] targets)
+ Block newBlock, DatanodeStorageInfo[] targets, boolean isStriped)
throws IOException {
assert hasWriteLock();
- BlockInfoContiguous b = dir.addBlock(src, inodesInPath, newBlock, targets);
+ BlockInfoContiguous b = dir.addBlock(src, inodesInPath, newBlock, targets,
+ isStriped);
NameNode.stateChangeLog.info("BLOCK* allocate " + b + " for " + src);
DatanodeStorageInfo.incrementBlocksScheduled(targets);
return b;
@@ -3441,10 +3448,11 @@ public class FSNamesystem implements Namesystem, FSNamesystemMBean,
/**
* Create new block with a unique block id and a new generation stamp.
+ * @param isStriped is the file under striping or contiguous layout?
*/
- Block createNewBlock() throws IOException {
+ Block createNewBlock(boolean isStriped) throws IOException {
assert hasWriteLock();
- Block b = new Block(nextBlockId(), 0, 0);
+ Block b = new Block(nextBlockId(isStriped), 0, 0);
// Increment the generation stamp for every new block.
b.setGenerationStamp(nextGenerationStamp(false));
return b;
@@ -6038,11 +6046,13 @@ public class FSNamesystem implements Namesystem, FSNamesystemMBean,
/**
* Increments, logs and then returns the block ID
+ * @param isStriped is the file under striping or contiguous layout?
*/
- private long nextBlockId() throws IOException {
+ private long nextBlockId(boolean isStriped) throws IOException {
assert hasWriteLock();
checkNameNodeSafeMode("Cannot get next block ID");
- final long blockId = blockIdManager.nextBlockId();
+ final long blockId = isStriped ?
+ blockIdManager.nextBlockGroupId() : blockIdManager.nextBlockId();
getEditLog().logAllocateBlockId(blockId);
// NB: callers sync the log
return blockId;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6d6fffc2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
index 887a259..9259d2c 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
@@ -33,12 +33,14 @@ import org.apache.hadoop.fs.permission.PermissionStatus;
import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy;
+import org.apache.hadoop.hdfs.protocol.HdfsConstants;
import org.apache.hadoop.hdfs.protocol.QuotaExceededException;
import org.apache.hadoop.hdfs.server.blockmanagement.BlockCollection;
import org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguous;
import org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction;
import org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite;
import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo;
+import org.apache.hadoop.hdfs.server.common.HdfsServerConstants;
import org.apache.hadoop.hdfs.server.common.HdfsServerConstants.BlockUCState;
import org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff;
import org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList;
@@ -866,4 +868,13 @@ public class INodeFile extends INodeWithAdditionalFields
return snapshotBlocks != null &&
Arrays.asList(snapshotBlocks).contains(block);
}
+
+ @VisibleForTesting
+ /**
+ * @return true if the file is in the striping layout.
+ */
+ // TODO: move erasure coding policy to file XAttr (HDFS-7337)
+ public boolean isStriped() {
+ return getStoragePolicyID() == HdfsConstants.EC_STORAGE_POLICY_ID;
+ }
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6d6fffc2/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockgroup.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockgroup.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockgroup.java
new file mode 100644
index 0000000..95133ce
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddBlockgroup.java
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.namenode;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.DFSTestUtil;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.protocol.HdfsConstants;
+import org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+
+import static org.junit.Assert.assertEquals;
+
+public class TestAddBlockgroup {
+
+ public static final Log LOG = LogFactory.getLog(TestAddBlockgroup.class);
+
+ private final short GROUP_SIZE = HdfsConstants.NUM_DATA_BLOCKS +
+ HdfsConstants.NUM_PARITY_BLOCKS;
+ private final short NUM_DATANODES = GROUP_SIZE;
+
+ private static final int BLOCKSIZE = 1024;
+ private static final short REPLICATION = 3;
+
+ private MiniDFSCluster cluster;
+ private Configuration conf;
+
+ @Before
+ public void setup() throws IOException {
+ conf = new Configuration();
+ conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, BLOCKSIZE);
+ cluster = new MiniDFSCluster.Builder(conf).numDataNodes(NUM_DATANODES)
+ .build();
+ cluster.waitActive();
+ cluster.getFileSystem().setStoragePolicy(new Path("/"),
+ HdfsConstants.EC_STORAGE_POLICY_NAME);
+ }
+
+ @After
+ public void tearDown() {
+ if (cluster != null) {
+ cluster.shutdown();
+ }
+ }
+
+ @Test
+ public void testAddBlockGroup() throws Exception {
+ DistributedFileSystem fs = cluster.getFileSystem();
+ FSDirectory fsdir = cluster.getNamesystem().getFSDirectory();
+
+ final Path file1 = new Path("/file1");
+ DFSTestUtil.createFile(fs, file1, BLOCKSIZE * 2, REPLICATION, 0L);
+ INodeFile file1Node = fsdir.getINode4Write(file1.toString()).asFile();
+ BlockInfo[] file1Blocks = file1Node.getBlocks();
+ assertEquals(2, file1Blocks.length);
+ assertEquals(GROUP_SIZE, file1Blocks[0].numNodes());
+ assertEquals(HdfsConstants.MAX_BLOCKS_IN_GROUP,
+ file1Blocks[1].getBlockId() - file1Blocks[0].getBlockId());
+ }
+}
[05/52] [abbrv] hadoop git commit: HADOOP-11599. Client#getTimeout
should use IPC_CLIENT_PING_DEFAULT when IPC_CLIENT_PING_KEY is not
configured. Contributed by zhihai xu.
Posted by zh...@apache.org.
HADOOP-11599. Client#getTimeout should use IPC_CLIENT_PING_DEFAULT when IPC_CLIENT_PING_KEY is not configured. Contributed by zhihai xu.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/3f56a4cb
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/3f56a4cb
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/3f56a4cb
Branch: refs/heads/HDFS-7285
Commit: 3f56a4cb0c57583e285e85a4d0c1584c4de9f1f1
Parents: b6fc1f3
Author: Tsuyoshi Ozawa <oz...@apache.org>
Authored: Wed Feb 18 17:32:50 2015 +0900
Committer: Tsuyoshi Ozawa <oz...@apache.org>
Committed: Wed Feb 18 17:35:11 2015 +0900
----------------------------------------------------------------------
hadoop-common-project/hadoop-common/CHANGES.txt | 3 +++
.../src/main/java/org/apache/hadoop/ipc/Client.java | 3 ++-
.../src/test/java/org/apache/hadoop/ipc/TestIPC.java | 6 ++++++
3 files changed, 11 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3f56a4cb/hadoop-common-project/hadoop-common/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt b/hadoop-common-project/hadoop-common/CHANGES.txt
index b3b2c95..f248555 100644
--- a/hadoop-common-project/hadoop-common/CHANGES.txt
+++ b/hadoop-common-project/hadoop-common/CHANGES.txt
@@ -959,6 +959,9 @@ Release 2.7.0 - UNRELEASED
HADOOP-11295. RPC Server Reader thread can't shutdown if RPCCallQueue is
full. (Ming Ma via kihwal)
+ HADOOP-11599. Client#getTimeout should use IPC_CLIENT_PING_DEFAULT when
+ IPC_CLIENT_PING_KEY is not configured. (zhihai xu via ozawa)
+
Release 2.6.1 - UNRELEASED
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3f56a4cb/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
index bdcb96c..3f93c42 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
@@ -236,7 +236,8 @@ public class Client {
* @return the timeout period in milliseconds. -1 if no timeout value is set
*/
final public static int getTimeout(Configuration conf) {
- if (!conf.getBoolean(CommonConfigurationKeys.IPC_CLIENT_PING_KEY, true)) {
+ if (!conf.getBoolean(CommonConfigurationKeys.IPC_CLIENT_PING_KEY,
+ CommonConfigurationKeys.IPC_CLIENT_PING_DEFAULT)) {
return getPingInterval(conf);
}
return -1;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3f56a4cb/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
index 04a7412..eb19f48 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
@@ -1235,6 +1235,12 @@ public class TestIPC {
}
}
+ @Test
+ public void testClientGetTimeout() throws IOException {
+ Configuration config = new Configuration();
+ assertEquals(Client.getTimeout(config), -1);
+ }
+
private void assertRetriesOnSocketTimeouts(Configuration conf,
int maxTimeoutRetries) throws IOException {
SocketFactory mockFactory = Mockito.mock(SocketFactory.class);
[09/52] [abbrv] hadoop git commit: YARN-3132. RMNodeLabelsManager
should remove node from node-to-label mapping when node becomes deactivated.
Contributed by Wangda Tan
Posted by zh...@apache.org.
YARN-3132. RMNodeLabelsManager should remove node from node-to-label mapping when node becomes deactivated. Contributed by Wangda Tan
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f5da5566
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f5da5566
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f5da5566
Branch: refs/heads/HDFS-7285
Commit: f5da5566d9c392a5df71a2dce4c2d0d50eea51ee
Parents: 4981d08
Author: Jian He <ji...@apache.org>
Authored: Wed Feb 18 11:51:51 2015 -0800
Committer: Jian He <ji...@apache.org>
Committed: Wed Feb 18 11:51:51 2015 -0800
----------------------------------------------------------------------
hadoop-yarn-project/CHANGES.txt | 3 +
.../nodelabels/CommonNodeLabelsManager.java | 3 +-
.../nodelabels/RMNodeLabelsManager.java | 20 ++++++-
.../nodelabels/TestRMNodeLabelsManager.java | 63 ++++++++++++++++++--
4 files changed, 78 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f5da5566/hadoop-yarn-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index cbba046..884b506 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -611,6 +611,9 @@ Release 2.7.0 - UNRELEASED
YARN-3207. Secondary filter matches entites which do not have the key being
filtered for. (Zhijie Shen via xgong)
+ YARN-3132. RMNodeLabelsManager should remove node from node-to-label mapping
+ when node becomes deactivated. (Wangda Tan via jianhe)
+
Release 2.6.0 - 2014-11-18
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f5da5566/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java
index cb6f1f3..e2da664 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java
@@ -45,7 +45,6 @@ import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.event.AsyncDispatcher;
import org.apache.hadoop.yarn.event.Dispatcher;
import org.apache.hadoop.yarn.event.EventHandler;
-import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
import org.apache.hadoop.yarn.nodelabels.event.NodeLabelsStoreEvent;
import org.apache.hadoop.yarn.nodelabels.event.NodeLabelsStoreEventType;
@@ -496,7 +495,7 @@ public class CommonNodeLabelsManager extends AbstractService {
}
}
- private void removeNodeFromLabels(NodeId node, Set<String> labels) {
+ protected void removeNodeFromLabels(NodeId node, Set<String> labels) {
for(String l : labels) {
labelCollections.get(l).removeNodeId(node);
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f5da5566/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsManager.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsManager.java
index 9942d80..e5abdc9 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsManager.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsManager.java
@@ -228,9 +228,23 @@ public class RMNodeLabelsManager extends CommonNodeLabelsManager {
Map<String, Host> before = cloneNodeMap(ImmutableSet.of(nodeId));
Node nm = getNMInNodeSet(nodeId);
if (null != nm) {
- // set nm is not running, and its resource = 0
- nm.running = false;
- nm.resource = Resource.newInstance(0, 0);
+ if (null == nm.labels) {
+ // When node deactivated, remove the nm from node collection if no
+ // labels explicitly set for this particular nm
+
+ // Save labels first, we need to remove label->nodes relation later
+ Set<String> savedNodeLabels = getLabelsOnNode(nodeId);
+
+ // Remove this node in nodes collection
+ nodeCollections.get(nodeId.getHost()).nms.remove(nodeId);
+
+ // Remove this node in labels->node
+ removeNodeFromLabels(nodeId, savedNodeLabels);
+ } else {
+ // set nm is not running, and its resource = 0
+ nm.running = false;
+ nm.resource = Resource.newInstance(0, 0);
+ }
}
// get the node after edition
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f5da5566/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMNodeLabelsManager.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMNodeLabelsManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMNodeLabelsManager.java
index a91947f..8a37c24 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMNodeLabelsManager.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMNodeLabelsManager.java
@@ -62,7 +62,7 @@ public class TestRMNodeLabelsManager extends NodeLabelTestBase {
}
@Test(timeout = 5000)
- public void testNodeActiveDeactiveUpdate() throws Exception {
+ public void testGetLabelResourceWhenNodeActiveDeactive() throws Exception {
mgr.addToCluserNodeLabels(toSet("p1", "p2", "p3"));
mgr.replaceLabelsOnNode(ImmutableMap.of(toNodeId("n1"), toSet("p1"),
toNodeId("n2"), toSet("p2"), toNodeId("n3"), toSet("p3")));
@@ -119,7 +119,7 @@ public class TestRMNodeLabelsManager extends NodeLabelTestBase {
@SuppressWarnings({ "unchecked", "rawtypes" })
@Test(timeout = 5000)
- public void testUpdateNodeLabelWithActiveNode() throws Exception {
+ public void testGetLabelResource() throws Exception {
mgr.addToCluserNodeLabels(toSet("p1", "p2", "p3"));
mgr.replaceLabelsOnNode(ImmutableMap.of(toNodeId("n1"), toSet("p1"),
toNodeId("n2"), toSet("p2"), toNodeId("n3"), toSet("p3")));
@@ -430,6 +430,52 @@ public class TestRMNodeLabelsManager extends NodeLabelTestBase {
}
}
+ @Test(timeout = 5000)
+ public void testGetLabelsOnNodesWhenNodeActiveDeactive() throws Exception {
+ mgr.addToCluserNodeLabels(toSet("p1", "p2", "p3"));
+ mgr.replaceLabelsOnNode(ImmutableMap.of(
+ toNodeId("n1"), toSet("p2")));
+ mgr.replaceLabelsOnNode(ImmutableMap.of(toNodeId("n1:1"), toSet("p1")));
+
+ // Active/Deactive a node directly assigned label, should not remove from
+ // node->label map
+ mgr.activateNode(toNodeId("n1:1"), SMALL_RESOURCE);
+ assertCollectionEquals(mgr.getNodeLabels().get(toNodeId("n1:1")),
+ toSet("p1"));
+ mgr.deactivateNode(toNodeId("n1:1"));
+ assertCollectionEquals(mgr.getNodeLabels().get(toNodeId("n1:1")),
+ toSet("p1"));
+ // Host will not affected
+ assertCollectionEquals(mgr.getNodeLabels().get(toNodeId("n1")),
+ toSet("p2"));
+
+ // Active/Deactive a node doesn't directly assigned label, should remove
+ // from node->label map
+ mgr.activateNode(toNodeId("n1:2"), SMALL_RESOURCE);
+ assertCollectionEquals(mgr.getNodeLabels().get(toNodeId("n1:2")),
+ toSet("p2"));
+ mgr.deactivateNode(toNodeId("n1:2"));
+ Assert.assertNull(mgr.getNodeLabels().get(toNodeId("n1:2")));
+ // Host will not affected too
+ assertCollectionEquals(mgr.getNodeLabels().get(toNodeId("n1")),
+ toSet("p2"));
+
+ // When we change label on the host after active a node without directly
+ // assigned label, such node will still be removed after deactive
+ // Active/Deactive a node doesn't directly assigned label, should remove
+ // from node->label map
+ mgr.activateNode(toNodeId("n1:2"), SMALL_RESOURCE);
+ mgr.replaceLabelsOnNode(ImmutableMap.of(toNodeId("n1"), toSet("p3")));
+ assertCollectionEquals(mgr.getNodeLabels().get(toNodeId("n1:2")),
+ toSet("p3"));
+ mgr.deactivateNode(toNodeId("n1:2"));
+ Assert.assertNull(mgr.getNodeLabels().get(toNodeId("n1:2")));
+ // Host will not affected too
+ assertCollectionEquals(mgr.getNodeLabels().get(toNodeId("n1")),
+ toSet("p3"));
+
+ }
+
private void checkNodeLabelInfo(List<NodeLabel> infos, String labelName, int activeNMs, int memory) {
for (NodeLabel info : infos) {
if (info.getLabelName().equals(labelName)) {
@@ -470,19 +516,24 @@ public class TestRMNodeLabelsManager extends NodeLabelTestBase {
mgr.getLabelsToNodes(), transposeNodeToLabels(mgr.getNodeLabels()));
// Add labels and replace labels on node
- mgr.addToCluserNodeLabels(toSet("p1", "p2", "p3"));
- mgr.replaceLabelsOnNode(ImmutableMap.of(toNodeId("n1"), toSet("p1"),
- toNodeId("n2"), toSet("p2"), toNodeId("n3"), toSet("p3")));
+ mgr.addToCluserNodeLabels(toSet("p1"));
+ mgr.replaceLabelsOnNode(ImmutableMap.of(toNodeId("n1"), toSet("p1")));
+ // p1 -> n1, n1:1
+ Assert.assertEquals(2, mgr.getLabelsToNodes().get("p1").size());
assertLabelsToNodesEquals(
mgr.getLabelsToNodes(), transposeNodeToLabels(mgr.getNodeLabels()));
// Activate a node for which host to label mapping exists
mgr.activateNode(NodeId.newInstance("n1", 2), Resource.newInstance(10, 0));
+ // p1 -> n1, n1:1, n1:2
+ Assert.assertEquals(3, mgr.getLabelsToNodes().get("p1").size());
assertLabelsToNodesEquals(
mgr.getLabelsToNodes(), transposeNodeToLabels(mgr.getNodeLabels()));
- // Deactivate a node. Label mapping should still exist.
+ // Deactivate a node. n1:1 will be removed from the map
mgr.deactivateNode(NodeId.newInstance("n1", 1));
+ // p1 -> n1, n1:2
+ Assert.assertEquals(2, mgr.getLabelsToNodes().get("p1").size());
assertLabelsToNodesEquals(
mgr.getLabelsToNodes(), transposeNodeToLabels(mgr.getNodeLabels()));
}
[36/52] [abbrv] hadoop git commit: HADOOP-11584 s3a file block size
set to 0 in getFileStatus. (Brahma Reddy Battula via stevel)
Posted by zh...@apache.org.
HADOOP-11584 s3a file block size set to 0 in getFileStatus. (Brahma Reddy Battula via stevel)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/709ff99c
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/709ff99c
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/709ff99c
Branch: refs/heads/HDFS-7285
Commit: 709ff99cff4124823bde631e272af7be9a22f83b
Parents: 737bad0
Author: Steve Loughran <st...@apache.org>
Authored: Sat Feb 21 12:02:41 2015 +0000
Committer: Steve Loughran <st...@apache.org>
Committed: Sat Feb 21 12:03:03 2015 +0000
----------------------------------------------------------------------
hadoop-common-project/hadoop-common/CHANGES.txt | 3 +
.../org/apache/hadoop/fs/s3a/S3AFileStatus.java | 5 +-
.../org/apache/hadoop/fs/s3a/S3AFileSystem.java | 41 +++++----
.../apache/hadoop/fs/s3a/TestS3ABlocksize.java | 93 ++++++++++++++++++++
.../src/test/resources/log4j.properties | 18 ++++
5 files changed, 143 insertions(+), 17 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/709ff99c/hadoop-common-project/hadoop-common/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt b/hadoop-common-project/hadoop-common/CHANGES.txt
index 143692e..373a485 100644
--- a/hadoop-common-project/hadoop-common/CHANGES.txt
+++ b/hadoop-common-project/hadoop-common/CHANGES.txt
@@ -984,6 +984,9 @@ Release 2.7.0 - UNRELEASED
HADOOP-11612. Workaround for Curator's ChildReaper requiring Guava 15+.
(rkanter)
+ HADOOP-11584 s3a file block size set to 0 in getFileStatus.
+ (Brahma Reddy Battula via stevel)
+
Release 2.6.1 - UNRELEASED
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/709ff99c/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java
index eb64492..85e0ef7 100644
--- a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java
+++ b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java
@@ -31,8 +31,9 @@ public class S3AFileStatus extends FileStatus {
}
// Files
- public S3AFileStatus(long length, long modification_time, Path path) {
- super(length, false, 1, 0, modification_time, path);
+ public S3AFileStatus(long length, long modification_time, Path path,
+ long blockSize) {
+ super(length, false, 1, blockSize, modification_time, path);
isEmptyDirectory = false;
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/709ff99c/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
index eaa5f2d..2373e7e 100644
--- a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
+++ b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
@@ -77,6 +77,10 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class S3AFileSystem extends FileSystem {
+ /**
+ * Default blocksize as used in blocksize and FS status queries
+ */
+ public static final int DEFAULT_BLOCKSIZE = 32 * 1024 * 1024;
private URI uri;
private Path workingDir;
private AmazonS3Client s3;
@@ -256,7 +260,7 @@ public class S3AFileSystem extends FileSystem {
}
long keepAliveTime = conf.getLong(KEEPALIVE_TIME, DEFAULT_KEEPALIVE_TIME);
LinkedBlockingQueue<Runnable> workQueue =
- new LinkedBlockingQueue<Runnable>(maxThreads *
+ new LinkedBlockingQueue<>(maxThreads *
conf.getInt(MAX_TOTAL_TASKS, DEFAULT_MAX_TOTAL_TASKS));
ThreadPoolExecutor tpe = new ThreadPoolExecutor(
coreThreads,
@@ -434,7 +438,7 @@ public class S3AFileSystem extends FileSystem {
String srcKey = pathToKey(src);
String dstKey = pathToKey(dst);
- if (srcKey.length() == 0 || dstKey.length() == 0) {
+ if (srcKey.isEmpty() || dstKey.isEmpty()) {
if (LOG.isDebugEnabled()) {
LOG.debug("rename: src or dst are empty");
}
@@ -526,7 +530,7 @@ public class S3AFileSystem extends FileSystem {
}
List<DeleteObjectsRequest.KeyVersion> keysToDelete =
- new ArrayList<DeleteObjectsRequest.KeyVersion>();
+ new ArrayList<>();
if (dstStatus != null && dstStatus.isEmptyDirectory()) {
// delete unnecessary fake directory.
keysToDelete.add(new DeleteObjectsRequest.KeyVersion(dstKey));
@@ -640,7 +644,7 @@ public class S3AFileSystem extends FileSystem {
request.setMaxKeys(maxKeys);
List<DeleteObjectsRequest.KeyVersion> keys =
- new ArrayList<DeleteObjectsRequest.KeyVersion>();
+ new ArrayList<>();
ObjectListing objects = s3.listObjects(request);
statistics.incrementReadOps(1);
while (true) {
@@ -663,7 +667,7 @@ public class S3AFileSystem extends FileSystem {
objects = s3.listNextBatchOfObjects(objects);
statistics.incrementReadOps(1);
} else {
- if (keys.size() > 0) {
+ if (!keys.isEmpty()) {
DeleteObjectsRequest deleteRequest =
new DeleteObjectsRequest(bucket).withKeys(keys);
s3.deleteObjects(deleteRequest);
@@ -751,7 +755,8 @@ public class S3AFileSystem extends FileSystem {
}
} else {
result.add(new S3AFileStatus(summary.getSize(),
- dateToLong(summary.getLastModified()), keyPath));
+ dateToLong(summary.getLastModified()), keyPath,
+ getDefaultBlockSize(f.makeQualified(uri, workingDir))));
if (LOG.isDebugEnabled()) {
LOG.debug("Adding: fi: " + keyPath);
}
@@ -877,13 +882,16 @@ public class S3AFileSystem extends FileSystem {
if (LOG.isDebugEnabled()) {
LOG.debug("Found exact file: fake directory");
}
- return new S3AFileStatus(true, true, f.makeQualified(uri, workingDir));
+ return new S3AFileStatus(true, true,
+ f.makeQualified(uri, workingDir));
} else {
if (LOG.isDebugEnabled()) {
LOG.debug("Found exact file: normal file");
}
- return new S3AFileStatus(meta.getContentLength(), dateToLong(meta.getLastModified()),
- f.makeQualified(uri, workingDir));
+ return new S3AFileStatus(meta.getContentLength(),
+ dateToLong(meta.getLastModified()),
+ f.makeQualified(uri, workingDir),
+ getDefaultBlockSize(f.makeQualified(uri, workingDir)));
}
} catch (AmazonServiceException e) {
if (e.getStatusCode() != 404) {
@@ -910,8 +918,10 @@ public class S3AFileSystem extends FileSystem {
} else {
LOG.warn("Found file (with /): real file? should not happen: {}", key);
- return new S3AFileStatus(meta.getContentLength(), dateToLong(meta.getLastModified()),
- f.makeQualified(uri, workingDir));
+ return new S3AFileStatus(meta.getContentLength(),
+ dateToLong(meta.getLastModified()),
+ f.makeQualified(uri, workingDir),
+ getDefaultBlockSize(f.makeQualified(uri, workingDir)));
}
} catch (AmazonServiceException e) {
if (e.getStatusCode() != 404) {
@@ -953,7 +963,8 @@ public class S3AFileSystem extends FileSystem {
}
}
- return new S3AFileStatus(true, false, f.makeQualified(uri, workingDir));
+ return new S3AFileStatus(true, false,
+ f.makeQualified(uri, workingDir));
}
} catch (AmazonServiceException e) {
if (e.getStatusCode() != 404) {
@@ -1128,8 +1139,8 @@ public class S3AFileSystem extends FileSystem {
s3.deleteObject(bucket, key + "/");
statistics.incrementWriteOps(1);
}
- } catch (FileNotFoundException e) {
- } catch (AmazonServiceException e) {}
+ } catch (FileNotFoundException | AmazonServiceException e) {
+ }
if (f.isRoot()) {
break;
@@ -1178,7 +1189,7 @@ public class S3AFileSystem extends FileSystem {
@Deprecated
public long getDefaultBlockSize() {
// default to 32MB: large enough to minimize the impact of seeks
- return getConf().getLong(FS_S3A_BLOCK_SIZE, 32 * 1024 * 1024);
+ return getConf().getLong(FS_S3A_BLOCK_SIZE, DEFAULT_BLOCKSIZE);
}
private void printAmazonServiceException(AmazonServiceException ase) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/709ff99c/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlocksize.java
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlocksize.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlocksize.java
new file mode 100644
index 0000000..76fbf99
--- /dev/null
+++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlocksize.java
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.AbstractFSContract;
+import org.apache.hadoop.fs.contract.AbstractFSContractTestBase;
+import org.apache.hadoop.fs.contract.s3a.S3AContract;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.Timeout;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.createFile;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.fileStatsToString;
+
+public class TestS3ABlocksize extends AbstractFSContractTestBase {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(TestS3ABlocksize.class);
+
+ @Override
+ protected AbstractFSContract createContract(Configuration conf) {
+ return new S3AContract(conf);
+ }
+
+ @Rule
+ public Timeout testTimeout = new Timeout(30 * 60 * 1000);
+
+ @Test
+ @SuppressWarnings("deprecation")
+ public void testBlockSize() throws Exception {
+ FileSystem fs = getFileSystem();
+ long defaultBlockSize = fs.getDefaultBlockSize();
+ assertEquals("incorrect blocksize",
+ S3AFileSystem.DEFAULT_BLOCKSIZE, defaultBlockSize);
+ long newBlockSize = defaultBlockSize * 2;
+ fs.getConf().setLong(Constants.FS_S3A_BLOCK_SIZE, newBlockSize);
+
+ Path dir = path("testBlockSize");
+ Path file = new Path(dir, "file");
+ createFile(fs, file, true, dataset(1024, 'a', 'z' - 'a'));
+ FileStatus fileStatus = fs.getFileStatus(file);
+ assertEquals("Double default block size in stat(): " + fileStatus,
+ newBlockSize,
+ fileStatus.getBlockSize());
+
+ // check the listing & assert that the block size is picked up by
+ // this route too.
+ boolean found = false;
+ FileStatus[] listing = fs.listStatus(dir);
+ for (FileStatus stat : listing) {
+ LOG.info("entry: {}", stat);
+ if (file.equals(stat.getPath())) {
+ found = true;
+ assertEquals("Double default block size in ls(): " + stat,
+ newBlockSize,
+ stat.getBlockSize());
+ }
+ }
+ assertTrue("Did not find " + fileStatsToString(listing, ", "), found);
+ }
+
+ @Test
+ public void testRootFileStatusHasBlocksize() throws Throwable {
+ FileSystem fs = getFileSystem();
+ FileStatus status = fs.getFileStatus(new Path("/"));
+ assertTrue("Invalid root blocksize",
+ status.getBlockSize() >= 0);
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/709ff99c/hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-aws/src/test/resources/log4j.properties b/hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
new file mode 100644
index 0000000..1a6baae
--- /dev/null
+++ b/hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
@@ -0,0 +1,18 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# log4j configuration used during build and unit tests
+
+log4j.rootLogger=info,stdout
+log4j.threshhold=ALL
+log4j.appender.stdout=org.apache.log4j.ConsoleAppender
+log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
+log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
[27/52] [abbrv] hadoop git commit: YARN-3230. Clarify application
states on the web UI. (Jian He via wangda)
Posted by zh...@apache.org.
YARN-3230. Clarify application states on the web UI. (Jian He via wangda)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ce5bf927
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ce5bf927
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ce5bf927
Branch: refs/heads/HDFS-7285
Commit: ce5bf927c3d9f212798de1bf8706e5e9def235a1
Parents: c33ae27
Author: Wangda Tan <wa...@apache.org>
Authored: Fri Feb 20 10:39:28 2015 -0800
Committer: Wangda Tan <wa...@apache.org>
Committed: Fri Feb 20 10:39:28 2015 -0800
----------------------------------------------------------------------
hadoop-yarn-project/CHANGES.txt | 2 ++
.../server/resourcemanager/webapp/AppBlock.java | 33 ++++++++++++++++++--
.../resourcemanager/webapp/AppsBlock.java | 6 ++--
.../resourcemanager/webapp/dao/AppInfo.java | 8 ++---
4 files changed, 41 insertions(+), 8 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ce5bf927/hadoop-yarn-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index e71da2d..c028043 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -310,6 +310,8 @@ Release 2.7.0 - UNRELEASED
YARN-2799. Cleanup TestLogAggregationService based on the change in YARN-90.
(Zhihai Xu via junping_du)
+ YARN-3230. Clarify application states on the web UI. (Jian He via wangda)
+
OPTIMIZATIONS
YARN-2990. FairScheduler's delay-scheduling always waits for node-local and
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ce5bf927/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java
index 1856d75..c2b376e 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java
@@ -32,8 +32,10 @@ import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.util.StringUtils;
import org.apache.hadoop.yarn.api.records.ApplicationAccessType;
import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
import org.apache.hadoop.yarn.api.records.QueueACL;
import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
import org.apache.hadoop.yarn.server.resourcemanager.RMContext;
import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager;
import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp;
@@ -131,8 +133,9 @@ public class AppBlock extends HtmlBlock {
._("Name:", app.getName())
._("Application Type:", app.getApplicationType())
._("Application Tags:", app.getApplicationTags())
- ._("State:", app.getState())
- ._("FinalStatus:", app.getFinalStatus())
+ ._("YarnApplicationState:", clarifyAppState(app.getState()))
+ ._("FinalStatus reported by AM:",
+ clairfyAppFinalStatus(app.getFinalStatus()))
._("Started:", Times.format(app.getStartTime()))
._("Elapsed:",
StringUtils.formatTime(Times.elapsed(app.getStartTime(),
@@ -198,4 +201,30 @@ public class AppBlock extends HtmlBlock {
table._();
div._();
}
+
+ private String clarifyAppState(YarnApplicationState state) {
+ String ret = state.toString();
+ switch (state) {
+ case NEW:
+ return ret + ": waiting for application to be initialized";
+ case NEW_SAVING:
+ return ret + ": waiting for application to be persisted in state-store.";
+ case SUBMITTED:
+ return ret + ": waiting for application to be accepted by scheduler.";
+ case ACCEPTED:
+ return ret + ": waiting for AM container to be allocated, launched and"
+ + " register with RM.";
+ case RUNNING:
+ return ret + ": AM has registered with RM and started running.";
+ default:
+ return ret;
+ }
+ }
+
+ private String clairfyAppFinalStatus(FinalApplicationStatus status) {
+ if (status == FinalApplicationStatus.UNDEFINED) {
+ return "Application has not completed yet.";
+ }
+ return status.toString();
+ }
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ce5bf927/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java
index 096a2b6..054a1a7 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java
@@ -30,6 +30,7 @@ import java.util.concurrent.ConcurrentMap;
import org.apache.commons.lang.StringEscapeUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
import org.apache.hadoop.yarn.api.records.YarnApplicationState;
import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager;
import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp;
@@ -65,7 +66,7 @@ class AppsBlock extends HtmlBlock {
th(".queue", "Queue").
th(".starttime", "StartTime").
th(".finishtime", "FinishTime").
- th(".state", "State").
+ th(".state", "YarnApplicationState").
th(".finalstatus", "FinalStatus").
th(".progress", "Progress").
th(".ui", "Tracking UI")._()._().
@@ -101,7 +102,8 @@ class AppsBlock extends HtmlBlock {
.append(appInfo.getStartTime()).append("\",\"")
.append(appInfo.getFinishTime()).append("\",\"")
.append(appInfo.getState()).append("\",\"")
- .append(appInfo.getFinalStatus()).append("\",\"")
+ .append(appInfo.getFinalStatus() == FinalApplicationStatus.UNDEFINED ?
+ "N/A" : appInfo.getFinalStatus()).append("\",\"")
// Progress bar
.append("<br title='").append(percent)
.append("'> <div class='").append(C_PROGRESSBAR).append("' title='")
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ce5bf927/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java
index 2b0dedc..66940cb 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java
@@ -200,8 +200,8 @@ public class AppInfo {
return this.name;
}
- public String getState() {
- return this.state.toString();
+ public YarnApplicationState getState() {
+ return this.state;
}
public float getProgress() {
@@ -216,8 +216,8 @@ public class AppInfo {
return this.diagnostics;
}
- public String getFinalStatus() {
- return this.finalStatus.toString();
+ public FinalApplicationStatus getFinalStatus() {
+ return this.finalStatus;
}
public String getTrackingUrl() {
[40/52] [abbrv] hadoop git commit: YARN-3236. Cleanup
RMAuthenticationFilter#AUTH_HANDLER_PROPERTY. Contributed by zhihai xu
Posted by zh...@apache.org.
YARN-3236. Cleanup RMAuthenticationFilter#AUTH_HANDLER_PROPERTY.
Contributed by zhihai xu
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/e3d29024
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/e3d29024
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/e3d29024
Branch: refs/heads/HDFS-7285
Commit: e3d290244c8a39edc37146d992cf34e6963b6851
Parents: 92d67ac
Author: Xuan <xg...@apache.org>
Authored: Sat Feb 21 16:18:40 2015 -0800
Committer: Xuan <xg...@apache.org>
Committed: Sat Feb 21 16:18:40 2015 -0800
----------------------------------------------------------------------
hadoop-yarn-project/CHANGES.txt | 3 +++
.../hadoop/yarn/server/security/http/RMAuthenticationFilter.java | 2 --
2 files changed, 3 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e3d29024/hadoop-yarn-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 1650a20..1982688 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -315,6 +315,9 @@ Release 2.7.0 - UNRELEASED
YARN-3237. AppLogAggregatorImpl fails to log error cause.
(Rushabh S Shah via xgong)
+ YARN-3236. Cleanup RMAuthenticationFilter#AUTH_HANDLER_PROPERTY.
+ (zhihai xu via xgong)
+
OPTIMIZATIONS
YARN-2990. FairScheduler's delay-scheduling always waits for node-local and
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e3d29024/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilter.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilter.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilter.java
index 3eeb620..c3497a9 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilter.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilter.java
@@ -40,8 +40,6 @@ public class RMAuthenticationFilter extends
DelegationTokenAuthenticationFilter {
static private AbstractDelegationTokenSecretManager<?> manager;
- public static final String AUTH_HANDLER_PROPERTY =
- "yarn.resourcemanager.authentication-handler";
private static final String OLD_HEADER = "Hadoop-YARN-Auth-Delegation-Token";
public RMAuthenticationFilter() {