You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mrql.apache.org by fe...@apache.org on 2014/06/15 07:53:50 UTC
git commit: MRQL-39: Prepare for 0.9.2 release
Repository: incubator-mrql
Updated Branches:
refs/heads/master 8305578c2 -> a03407c49
MRQL-39: Prepare for 0.9.2 release
Project: http://git-wip-us.apache.org/repos/asf/incubator-mrql/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-mrql/commit/a03407c4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-mrql/tree/a03407c4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-mrql/diff/a03407c4
Branch: refs/heads/master
Commit: a03407c497bd7735ecb123e7165a942f88aab2a5
Parents: 8305578
Author: fegaras <fe...@cse.uta.edu>
Authored: Sun Jun 15 08:52:58 2014 +0300
Committer: fegaras <fe...@cse.uta.edu>
Committed: Sun Jun 15 08:52:58 2014 +0300
----------------------------------------------------------------------
NOTICE | 2 +-
README | 4 +-
RELEASE_NOTES | 13 +-
bin/mrql.bsp | 6 +-
conf/mrql-env.sh | 15 +-
.../multipleinputs/DelegatingInputFormat.java | 127 ---------------
.../java/multipleinputs/DelegatingMapper.java | 54 -------
.../multipleinputs/DelegatingRecordReader.java | 88 ----------
.../java/multipleinputs/MultipleInputs.java | 141 ----------------
.../java/multipleinputs/TaggedInputSplit.java | 159 -------------------
src/site/xdoc/getting_started.xml | 2 +-
src/site/xdoc/team-list.xml | 2 +-
12 files changed, 24 insertions(+), 589 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-mrql/blob/a03407c4/NOTICE
----------------------------------------------------------------------
diff --git a/NOTICE b/NOTICE
index 30c5cad..aae7c68 100644
--- a/NOTICE
+++ b/NOTICE
@@ -1,5 +1,5 @@
Apache MRQL (incubating)
-Copyright 2013 The Apache Software Foundation
+Copyright 2013-2014 The Apache Software Foundation
This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
http://git-wip-us.apache.org/repos/asf/incubator-mrql/blob/a03407c4/README
----------------------------------------------------------------------
diff --git a/README b/README
index 231dbd2..9a98ca0 100644
--- a/README
+++ b/README
@@ -18,8 +18,8 @@
***************************************************************************
-Apache MRQL 0.9.2
-=================
+Apache MRQL 0.9.2-incubating
+============================
Apache MRQL (pronounced miracle) is a query processing and optimization
system for large-scale, distributed data analysis. MRQL (the MapReduce
http://git-wip-us.apache.org/repos/asf/incubator-mrql/blob/a03407c4/RELEASE_NOTES
----------------------------------------------------------------------
diff --git a/RELEASE_NOTES b/RELEASE_NOTES
index a6addf1..6fef315 100644
--- a/RELEASE_NOTES
+++ b/RELEASE_NOTES
@@ -16,8 +16,8 @@
limitations under the License.
***************************************************************************
-Release Notes - Apache MRQL 0.9.2
----------------------------------
+Release Notes - Apache MRQL 0.9.2-incubating
+--------------------------------------------
New Features
@@ -43,13 +43,15 @@ Improvements
[MRQL-32] Refactoring directory structure for Eclipse
[MRQL-34] Introduce junit for testing
[MRQL-37] Support installation on a Cloudera CDH distribution
+[MRQL-38] Support for Spark 1.0.0
Tasks
-[MRQL-20] Fix maven problems to prepare a new release candidate
+[MRQL-39] Prepare for 0.9.2 release
+
-Release Notes - Apache MRQL 0.9.0
----------------------------------
+Release Notes - Apache MRQL 0.9.0-incubating
+--------------------------------------------
New Features
@@ -81,3 +83,4 @@ Tasks
[MRQL-17] Update the MRQL status page
[MRQL-18] correct NOTICE, DISCLAIMER, xdoc
[MRQL-19] Fix license issues to prepare a new release candidate
+[MRQL-20] Fix maven problems to prepare a new release candidate
http://git-wip-us.apache.org/repos/asf/incubator-mrql/blob/a03407c4/bin/mrql.bsp
----------------------------------------------------------------------
diff --git a/bin/mrql.bsp b/bin/mrql.bsp
index 2f4c8a9..20a65ac 100755
--- a/bin/mrql.bsp
+++ b/bin/mrql.bsp
@@ -49,8 +49,12 @@ if (! [ -a $FULL_JAR ]); then
popd > /dev/null
fi
-if [ "$1" == "-local" ] || [ "$1" == "-dist" ]; then
+if [ "$1" == "-local" ]; then
+ export HAMA_CLASSPATH=$FULL_JAR # needed for Hama 0.6.4 in local mode
+ $HAMA_HOME/bin/hama jar $FULL_JAR org.apache.mrql.Main -bsp $*
+else if [ "$1" == "-dist" ]; then
$HAMA_HOME/bin/hama jar $FULL_JAR org.apache.mrql.Main -bsp $*
else
$JAVA_HOME/bin/java -classpath "$FULL_JAR:$HAMA_JAR:$HADOOP_JARS" org.apache.mrql.Main -bsp $*
fi
+fi
http://git-wip-us.apache.org/repos/asf/incubator-mrql/blob/a03407c4/conf/mrql-env.sh
----------------------------------------------------------------------
diff --git a/conf/mrql-env.sh b/conf/mrql-env.sh
index 56274d6..031255c 100644
--- a/conf/mrql-env.sh
+++ b/conf/mrql-env.sh
@@ -28,9 +28,6 @@
# build MRQL on Hadoop 2.x (yarn):
# mvn -Pyarn -Dyarn.version=2.2.0 -Dhadoop.version=1.2.1 install
#
-# build MRQL on Hadoop 0.20.x:
-# mvn -PMultipleInputs -Dhadoop.version=0.20.2 install
-#
#--------------------------------------------------------------------------------
#
# Set Apache MRQL-specific environment variables here:
@@ -48,8 +45,8 @@ CUP_JAR=${HOME}/.m2/repository/net/sf/squirrel-sql/thirdparty/non-maven/java-cup
JLINE_JAR=${HOME}/.m2/repository/jline/jline/1.0/jline-1.0.jar
-# Required: Hadoop configuration. Supports versions 0.20.x, 1.x, and 2.x (YARN)
-HADOOP_VERSION=2.2.0
+# Required: Hadoop configuration. Supports versions 1.x and 2.x (YARN)
+HADOOP_VERSION=1.2.1
# The Hadoop installation directory
HADOOP_HOME=${HOME}/hadoop-${HADOOP_VERSION}
# The Hadoop job tracker (as defined in mapred-site.xml)
@@ -58,7 +55,7 @@ MAPRED_JOB_TRACKER=localhost:9001
FS_DEFAULT_NAME=hdfs://localhost:9000/
-# Optional: Hama configuration. Supports versions 0.6.2 and 0.6.3 (but not 0.6.4)
+# Optional: Hama configuration. Supports versions 0.6.2, 0.6.3, and 0.6.4
HAMA_VERSION=0.6.3
# The Hadoop installation directory
HAMA_HOME=${HOME}/hama-${HAMA_VERSION}
@@ -69,9 +66,9 @@ HAMA_ZOOKEEPER_QUORUM=localhost
# Optional: Spark configuration. Supports version 1.0.0 only
# (Spark versions 0.8.1, 0.9.0, and 0.9.1 are supported by MRQL 0.9.0)
-# Use either the Spark prebuilts bin-hadoop1 or bin-hadoop2 (Yarn)
+# Use the Spark prebuilts bin-hadoop1 or bin-hadoop2 (Yarn)
# Tested in local, standalone deploy, and Yarn modes
-SPARK_HOME=${HOME}/spark-1.0.0-bin-hadoop2
+SPARK_HOME=${HOME}/spark-1.0.0-bin-hadoop1
# URI of the Spark master node (to run Spark on a YARN cluster, set it to "yarn-client")
SPARK_MASTER=spark://crete:7077
# For a Yarn cluster set it to the number of workers to start on,
@@ -80,7 +77,7 @@ SPARK_WORKER_INSTANCES=1
# Number of cores for the workers
SPARK_WORKER_CORES=2
# Memory per Worker (e.g. 1000M, 2G)
-SPARK_WORKER_MEMORY=2G
+SPARK_WORKER_MEMORY=1G
# Memory for Master (e.g. 1000M, 2G)
SPARK_MASTER_MEMORY=512M
http://git-wip-us.apache.org/repos/asf/incubator-mrql/blob/a03407c4/src/main/java/multipleinputs/DelegatingInputFormat.java
----------------------------------------------------------------------
diff --git a/src/main/java/multipleinputs/DelegatingInputFormat.java b/src/main/java/multipleinputs/DelegatingInputFormat.java
deleted file mode 100644
index 0d9b8f9..0000000
--- a/src/main/java/multipleinputs/DelegatingInputFormat.java
+++ /dev/null
@@ -1,127 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hadoop.mapreduce.lib.input;
-
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
-
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.mapreduce.InputFormat;
-import org.apache.hadoop.mapreduce.InputSplit;
-import org.apache.hadoop.mapreduce.Job;
-import org.apache.hadoop.mapreduce.JobContext;
-import org.apache.hadoop.mapreduce.Mapper;
-import org.apache.hadoop.mapreduce.RecordReader;
-import org.apache.hadoop.mapreduce.TaskAttemptContext;
-import org.apache.hadoop.util.ReflectionUtils;
-
-/**
- * An {@link InputFormat} that delegates behavior of paths to multiple other
- * InputFormats.
- *
- * @see MultipleInputs#addInputPath(Job, Path, Class, Class)
- */
-public class DelegatingInputFormat<K, V> extends InputFormat<K, V> {
-
- @SuppressWarnings("unchecked")
- public List<InputSplit> getSplits(JobContext job)
- throws IOException, InterruptedException {
- Configuration conf = job.getConfiguration();
- Job jobCopy =new Job(conf);
- List<InputSplit> splits = new ArrayList<InputSplit>();
- Map<Path, InputFormat> formatMap =
- MultipleInputs.getInputFormatMap(job);
- Map<Path, Class<? extends Mapper>> mapperMap = MultipleInputs
- .getMapperTypeMap(job);
- Map<Class<? extends InputFormat>, List<Path>> formatPaths
- = new HashMap<Class<? extends InputFormat>, List<Path>>();
-
- // First, build a map of InputFormats to Paths
- for (Entry<Path, InputFormat> entry : formatMap.entrySet()) {
- if (!formatPaths.containsKey(entry.getValue().getClass())) {
- formatPaths.put(entry.getValue().getClass(), new LinkedList<Path>());
- }
-
- formatPaths.get(entry.getValue().getClass()).add(entry.getKey());
- }
-
- for (Entry<Class<? extends InputFormat>, List<Path>> formatEntry :
- formatPaths.entrySet()) {
- Class<? extends InputFormat> formatClass = formatEntry.getKey();
- InputFormat format = (InputFormat) ReflectionUtils.newInstance(
- formatClass, conf);
- List<Path> paths = formatEntry.getValue();
-
- Map<Class<? extends Mapper>, List<Path>> mapperPaths
- = new HashMap<Class<? extends Mapper>, List<Path>>();
-
- // Now, for each set of paths that have a common InputFormat, build
- // a map of Mappers to the paths they're used for
- for (Path path : paths) {
- Class<? extends Mapper> mapperClass = mapperMap.get(path);
- if (!mapperPaths.containsKey(mapperClass)) {
- mapperPaths.put(mapperClass, new LinkedList<Path>());
- }
-
- mapperPaths.get(mapperClass).add(path);
- }
-
- // Now each set of paths that has a common InputFormat and Mapper can
- // be added to the same job, and split together.
- for (Entry<Class<? extends Mapper>, List<Path>> mapEntry :
- mapperPaths.entrySet()) {
- paths = mapEntry.getValue();
- Class<? extends Mapper> mapperClass = mapEntry.getKey();
-
- if (mapperClass == null) {
- try {
- mapperClass = job.getMapperClass();
- } catch (ClassNotFoundException e) {
- throw new IOException("Mapper class is not found", e);
- }
- }
-
- FileInputFormat.setInputPaths(jobCopy, paths.toArray(new Path[paths
- .size()]));
-
- // Get splits for each input path and tag with InputFormat
- // and Mapper types by wrapping in a TaggedInputSplit.
- List<InputSplit> pathSplits = format.getSplits(jobCopy);
- for (InputSplit pathSplit : pathSplits) {
- splits.add(new TaggedInputSplit(pathSplit, conf, format.getClass(),
- mapperClass));
- }
- }
- }
-
- return splits;
- }
-
- @Override
- public RecordReader<K, V> createRecordReader(InputSplit split,
- TaskAttemptContext context) throws IOException, InterruptedException {
- return new DelegatingRecordReader<K, V>(split, context);
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-mrql/blob/a03407c4/src/main/java/multipleinputs/DelegatingMapper.java
----------------------------------------------------------------------
diff --git a/src/main/java/multipleinputs/DelegatingMapper.java b/src/main/java/multipleinputs/DelegatingMapper.java
deleted file mode 100644
index d72d2b5..0000000
--- a/src/main/java/multipleinputs/DelegatingMapper.java
+++ /dev/null
@@ -1,54 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hadoop.mapreduce.lib.input;
-
-import java.io.IOException;
-
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.mapreduce.Mapper;
-import org.apache.hadoop.util.ReflectionUtils;
-
-/**
- * An {@link Mapper} that delegates behavior of paths to multiple other
- * mappers.
- *
- * @see MultipleInputs#addInputPath(Job, Path, Class, Class)
- */
-public class DelegatingMapper<K1, V1, K2, V2> extends Mapper<K1, V1, K2, V2> {
-
- private Mapper<K1, V1, K2, V2> mapper;
-
- @SuppressWarnings("unchecked")
- protected void setup(Context context)
- throws IOException, InterruptedException {
- // Find the Mapper from the TaggedInputSplit.
- TaggedInputSplit inputSplit = (TaggedInputSplit) context.getInputSplit();
- mapper = (Mapper<K1, V1, K2, V2>) ReflectionUtils.newInstance(inputSplit
- .getMapperClass(), context.getConfiguration());
-
- }
-
- @SuppressWarnings("unchecked")
- public void run(Context context)
- throws IOException, InterruptedException {
- setup(context);
- mapper.run(context);
- cleanup(context);
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-mrql/blob/a03407c4/src/main/java/multipleinputs/DelegatingRecordReader.java
----------------------------------------------------------------------
diff --git a/src/main/java/multipleinputs/DelegatingRecordReader.java b/src/main/java/multipleinputs/DelegatingRecordReader.java
deleted file mode 100644
index f0d060e..0000000
--- a/src/main/java/multipleinputs/DelegatingRecordReader.java
+++ /dev/null
@@ -1,88 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.hadoop.mapreduce.lib.input;
-
-import java.io.IOException;
-
-import org.apache.hadoop.mapreduce.InputFormat;
-import org.apache.hadoop.mapreduce.InputSplit;
-import org.apache.hadoop.mapreduce.RecordReader;
-import org.apache.hadoop.mapreduce.TaskAttemptContext;
-import org.apache.hadoop.util.ReflectionUtils;
-
-/**
- * This is a delegating RecordReader, which delegates the functionality to the
- * underlying record reader in {@link TaggedInputSplit}
- */
-public class DelegatingRecordReader<K, V> extends RecordReader<K, V> {
- RecordReader<K, V> originalRR;
-
- /**
- * Constructs the DelegatingRecordReader.
- *
- * @param split TaggegInputSplit object
- * @param context TaskAttemptContext object
- *
- * @throws IOException
- * @throws InterruptedException
- */
- @SuppressWarnings("unchecked")
- public DelegatingRecordReader(InputSplit split, TaskAttemptContext context)
- throws IOException, InterruptedException {
- // Find the InputFormat and then the RecordReader from the
- // TaggedInputSplit.
- TaggedInputSplit taggedInputSplit = (TaggedInputSplit) split;
- InputFormat<K, V> inputFormat = (InputFormat<K, V>) ReflectionUtils
- .newInstance(taggedInputSplit.getInputFormatClass(), context
- .getConfiguration());
- originalRR = inputFormat.createRecordReader(taggedInputSplit
- .getInputSplit(), context);
- }
-
- @Override
- public void close() throws IOException {
- originalRR.close();
- }
-
- @Override
- public K getCurrentKey() throws IOException, InterruptedException {
- return originalRR.getCurrentKey();
- }
-
- @Override
- public V getCurrentValue() throws IOException, InterruptedException {
- return originalRR.getCurrentValue();
- }
-
- @Override
- public float getProgress() throws IOException, InterruptedException {
- return originalRR.getProgress();
- }
-
- @Override
- public void initialize(InputSplit split, TaskAttemptContext context)
- throws IOException, InterruptedException {
- originalRR.initialize(((TaggedInputSplit) split).getInputSplit(), context);
- }
-
- @Override
- public boolean nextKeyValue() throws IOException, InterruptedException {
- return originalRR.nextKeyValue();
- }
-
-}
http://git-wip-us.apache.org/repos/asf/incubator-mrql/blob/a03407c4/src/main/java/multipleinputs/MultipleInputs.java
----------------------------------------------------------------------
diff --git a/src/main/java/multipleinputs/MultipleInputs.java b/src/main/java/multipleinputs/MultipleInputs.java
deleted file mode 100644
index 9140512..0000000
--- a/src/main/java/multipleinputs/MultipleInputs.java
+++ /dev/null
@@ -1,141 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.hadoop.mapreduce.lib.input;
-
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.Map;
-
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.mapreduce.InputFormat;
-import org.apache.hadoop.mapreduce.Job;
-import org.apache.hadoop.mapreduce.JobContext;
-import org.apache.hadoop.mapreduce.Mapper;
-import org.apache.hadoop.util.ReflectionUtils;
-
-/**
- * This class supports MapReduce jobs that have multiple input paths with
- * a different {@link InputFormat} and {@link Mapper} for each path
- */
-public class MultipleInputs {
- /**
- * Add a {@link Path} with a custom {@link InputFormat} to the list of
- * inputs for the map-reduce job.
- *
- * @param job The {@link Job}
- * @param path {@link Path} to be added to the list of inputs for the job
- * @param inputFormatClass {@link InputFormat} class to use for this path
- */
- @SuppressWarnings("unchecked")
- public static void addInputPath(Job job, Path path,
- Class<? extends InputFormat> inputFormatClass) {
- String inputFormatMapping = path.toString() + ";"
- + inputFormatClass.getName();
- Configuration conf = job.getConfiguration();
- String inputFormats = conf.get("mapred.input.dir.formats");
- conf.set("mapred.input.dir.formats",
- inputFormats == null ? inputFormatMapping : inputFormats + ","
- + inputFormatMapping);
-
- job.setInputFormatClass(DelegatingInputFormat.class);
- }
-
- /**
- * Add a {@link Path} with a custom {@link InputFormat} and
- * {@link Mapper} to the list of inputs for the map-reduce job.
- *
- * @param job The {@link Job}
- * @param path {@link Path} to be added to the list of inputs for the job
- * @param inputFormatClass {@link InputFormat} class to use for this path
- * @param mapperClass {@link Mapper} class to use for this path
- */
- @SuppressWarnings("unchecked")
- public static void addInputPath(Job job, Path path,
- Class<? extends InputFormat> inputFormatClass,
- Class<? extends Mapper> mapperClass) {
-
- addInputPath(job, path, inputFormatClass);
- Configuration conf = job.getConfiguration();
- String mapperMapping = path.toString() + ";" + mapperClass.getName();
- String mappers = conf.get("mapred.input.dir.mappers");
- conf.set("mapred.input.dir.mappers", mappers == null ? mapperMapping
- : mappers + "," + mapperMapping);
-
- job.setMapperClass(DelegatingMapper.class);
- }
-
- /**
- * Retrieves a map of {@link Path}s to the {@link InputFormat} class
- * that should be used for them.
- *
- * @param job The {@link JobContext}
- * @see #addInputPath(JobConf, Path, Class)
- * @return A map of paths to inputformats for the job
- */
- @SuppressWarnings("unchecked")
- static Map<Path, InputFormat> getInputFormatMap(JobContext job) {
- Map<Path, InputFormat> m = new HashMap<Path, InputFormat>();
- Configuration conf = job.getConfiguration();
- String[] pathMappings = conf.get("mapred.input.dir.formats").split(",");
- for (String pathMapping : pathMappings) {
- String[] split = pathMapping.split(";");
- InputFormat inputFormat;
- try {
- inputFormat = (InputFormat) ReflectionUtils.newInstance(conf
- .getClassByName(split[1]), conf);
- } catch (ClassNotFoundException e) {
- throw new RuntimeException(e);
- }
- m.put(new Path(split[0]), inputFormat);
- }
- return m;
- }
-
- /**
- * Retrieves a map of {@link Path}s to the {@link Mapper} class that
- * should be used for them.
- *
- * @param job The {@link JobContext}
- * @see #addInputPath(JobConf, Path, Class, Class)
- * @return A map of paths to mappers for the job
- */
- @SuppressWarnings("unchecked")
- static Map<Path, Class<? extends Mapper>>
- getMapperTypeMap(JobContext job) {
- Configuration conf = job.getConfiguration();
- if (conf.get("mapred.input.dir.mappers") == null) {
- return Collections.emptyMap();
- }
- Map<Path, Class<? extends Mapper>> m =
- new HashMap<Path, Class<? extends Mapper>>();
- String[] pathMappings = conf.get("mapred.input.dir.mappers").split(",");
- for (String pathMapping : pathMappings) {
- String[] split = pathMapping.split(";");
- Class<? extends Mapper> mapClass;
- try {
- mapClass =
- (Class<? extends Mapper>) conf.getClassByName(split[1]);
- } catch (ClassNotFoundException e) {
- throw new RuntimeException(e);
- }
- m.put(new Path(split[0]), mapClass);
- }
- return m;
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-mrql/blob/a03407c4/src/main/java/multipleinputs/TaggedInputSplit.java
----------------------------------------------------------------------
diff --git a/src/main/java/multipleinputs/TaggedInputSplit.java b/src/main/java/multipleinputs/TaggedInputSplit.java
deleted file mode 100644
index 68bb789..0000000
--- a/src/main/java/multipleinputs/TaggedInputSplit.java
+++ /dev/null
@@ -1,159 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hadoop.mapreduce.lib.input;
-
-import java.io.DataInput;
-import java.io.DataInputStream;
-import java.io.DataOutput;
-import java.io.DataOutputStream;
-import java.io.IOException;
-
-import org.apache.hadoop.conf.Configurable;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.io.Text;
-import org.apache.hadoop.io.Writable;
-import org.apache.hadoop.io.serializer.Deserializer;
-import org.apache.hadoop.io.serializer.SerializationFactory;
-import org.apache.hadoop.io.serializer.Serializer;
-import org.apache.hadoop.mapreduce.InputFormat;
-import org.apache.hadoop.mapreduce.InputSplit;
-import org.apache.hadoop.mapreduce.Mapper;
-import org.apache.hadoop.util.ReflectionUtils;
-
-/**
- * An {@link InputSplit} that tags another InputSplit with extra data for use
- * by {@link DelegatingInputFormat}s and {@link DelegatingMapper}s.
- */
-class TaggedInputSplit extends InputSplit implements Configurable, Writable {
-
- private Class<? extends InputSplit> inputSplitClass;
-
- private InputSplit inputSplit;
-
- @SuppressWarnings("unchecked")
- private Class<? extends InputFormat> inputFormatClass;
-
- @SuppressWarnings("unchecked")
- private Class<? extends Mapper> mapperClass;
-
- private Configuration conf;
-
- public TaggedInputSplit() {
- // Default constructor.
- }
-
- /**
- * Creates a new TaggedInputSplit.
- *
- * @param inputSplit The InputSplit to be tagged
- * @param conf The configuration to use
- * @param inputFormatClass The InputFormat class to use for this job
- * @param mapperClass The Mapper class to use for this job
- */
- @SuppressWarnings("unchecked")
- public TaggedInputSplit(InputSplit inputSplit, Configuration conf,
- Class<? extends InputFormat> inputFormatClass,
- Class<? extends Mapper> mapperClass) {
- this.inputSplitClass = inputSplit.getClass();
- this.inputSplit = inputSplit;
- this.conf = conf;
- this.inputFormatClass = inputFormatClass;
- this.mapperClass = mapperClass;
- }
-
- /**
- * Retrieves the original InputSplit.
- *
- * @return The InputSplit that was tagged
- */
- public InputSplit getInputSplit() {
- return inputSplit;
- }
-
- /**
- * Retrieves the InputFormat class to use for this split.
- *
- * @return The InputFormat class to use
- */
- @SuppressWarnings("unchecked")
- public Class<? extends InputFormat> getInputFormatClass() {
- return inputFormatClass;
- }
-
- /**
- * Retrieves the Mapper class to use for this split.
- *
- * @return The Mapper class to use
- */
- @SuppressWarnings("unchecked")
- public Class<? extends Mapper> getMapperClass() {
- return mapperClass;
- }
-
- public long getLength() throws IOException, InterruptedException {
- return inputSplit.getLength();
- }
-
- public String[] getLocations() throws IOException, InterruptedException {
- return inputSplit.getLocations();
- }
-
- @SuppressWarnings("unchecked")
- public void readFields(DataInput in) throws IOException {
- inputSplitClass = (Class<? extends InputSplit>) readClass(in);
- inputFormatClass = (Class<? extends InputFormat<?, ?>>) readClass(in);
- mapperClass = (Class<? extends Mapper<?, ?, ?, ?>>) readClass(in);
- inputSplit = (InputSplit) ReflectionUtils
- .newInstance(inputSplitClass, conf);
- SerializationFactory factory = new SerializationFactory(conf);
- Deserializer deserializer = factory.getDeserializer(inputSplitClass);
- deserializer.open((DataInputStream)in);
- inputSplit = (InputSplit)deserializer.deserialize(inputSplit);
- }
-
- private Class<?> readClass(DataInput in) throws IOException {
- String className = Text.readString(in);
- try {
- return conf.getClassByName(className);
- } catch (ClassNotFoundException e) {
- throw new RuntimeException("readObject can't find class", e);
- }
- }
-
- @SuppressWarnings("unchecked")
- public void write(DataOutput out) throws IOException {
- Text.writeString(out, inputSplitClass.getName());
- Text.writeString(out, inputFormatClass.getName());
- Text.writeString(out, mapperClass.getName());
- SerializationFactory factory = new SerializationFactory(conf);
- Serializer serializer =
- factory.getSerializer(inputSplitClass);
- serializer.open((DataOutputStream)out);
- serializer.serialize(inputSplit);
- }
-
- public Configuration getConf() {
- return conf;
- }
-
- public void setConf(Configuration conf) {
- this.conf = conf;
- }
-
-}
http://git-wip-us.apache.org/repos/asf/incubator-mrql/blob/a03407c4/src/site/xdoc/getting_started.xml
----------------------------------------------------------------------
diff --git a/src/site/xdoc/getting_started.xml b/src/site/xdoc/getting_started.xml
index b1ca7d6..33dbdd9 100644
--- a/src/site/xdoc/getting_started.xml
+++ b/src/site/xdoc/getting_started.xml
@@ -28,7 +28,7 @@ xsi:schemaLocation="http://maven.apache.org/XDOC/2.0 http://maven.apache.org/xsd
Apache MRQL can run in three modes: in Map-Reduce mode using Apache Hadoop, in BSP mode (Bulk Synchronous Parallel mode) using Apache Hama, and in Spark mode using Apache Spark.
</p>
<p>
-The MRQL MapReduce mode has been tested on Apache Hadoop MapReduce releases 1.x, and 2.x (Yarn). You can download the latest tarball from <a href="http://hadoop.apache.org/releases.html">Apache Hadoop</a>. The BSP and Spark modes are optional. The BSP mode has been tested on Apache Hama 0.6.2 and 0.6.3. You can download the latest tarball from <a href="http://hama.apache.org/">Apache Hama</a>. The Spark mode has been tested on Apache Spark 1.0.0 in local, standalone deploy, and Yarn modes.
+The MRQL MapReduce mode has been tested on Apache Hadoop MapReduce releases 1.x, and 2.x (Yarn). You can download the latest tarball from <a href="http://hadoop.apache.org/releases.html">Apache Hadoop</a>. The BSP and Spark modes are optional. The BSP mode has been tested on Apache Hama 0.6.2, 0.6.3, and 0.6.4. You can download the latest tarball from <a href="http://hama.apache.org/">Apache Hama</a>. The Spark mode has been tested on Apache Spark 1.0.0 in local, standalone deploy, and Yarn modes.
You can download the latest tarball prebuilt for Hadoop1 or Hadoop2 from <a href="http://spark.apache.org/">Apache Spark</a>.
</p>
<p>
http://git-wip-us.apache.org/repos/asf/incubator-mrql/blob/a03407c4/src/site/xdoc/team-list.xml
----------------------------------------------------------------------
diff --git a/src/site/xdoc/team-list.xml b/src/site/xdoc/team-list.xml
index ef3691a..0811a89 100644
--- a/src/site/xdoc/team-list.xml
+++ b/src/site/xdoc/team-list.xml
@@ -131,7 +131,7 @@ xsi:schemaLocation="http://maven.apache.org/XDOC/2.0 http://maven.apache.org/xsd
<td align="center">committer</td>
</tr>
<tr valign="top">
- <td align="center"></td>
+ <td align="center">moon</td>
<td align="center">Lee Moon Soo</td>
<td align="center"></td>
<td align="center">committer</td>