You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by aw...@apache.org on 2016/09/09 11:08:31 UTC
[01/21] hadoop git commit: HDFS-10844.
test_libhdfs_threaded_hdfs_static and test_libhdfs_zerocopy_hdfs_static are
failing. [Forced Update!]
Repository: hadoop
Updated Branches:
refs/heads/HADOOP-13341 5b7a3df75 -> b417ba32d (forced update)
HDFS-10844. test_libhdfs_threaded_hdfs_static and test_libhdfs_zerocopy_hdfs_static are failing.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/2d1bf53c
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/2d1bf53c
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/2d1bf53c
Branch: refs/heads/HADOOP-13341
Commit: 2d1bf53c7e208ad951ebed7ee3f2e44582dfd151
Parents: 63f5948
Author: Akira Ajisaka <aa...@apache.org>
Authored: Fri Sep 9 00:49:22 2016 +0900
Committer: Akira Ajisaka <aa...@apache.org>
Committed: Fri Sep 9 00:49:22 2016 +0900
----------------------------------------------------------------------
.../src/main/native/libhdfs/hdfs.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2d1bf53c/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c b/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c
index 4618dbb..1dcc768 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c
+++ b/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c
@@ -114,7 +114,7 @@ int hdfsFileGetReadStatistics(hdfsFile file,
jthr = invokeMethod(env, &jVal, INSTANCE, file->file,
"org/apache/hadoop/hdfs/client/HdfsDataInputStream",
"getReadStatistics",
- "()Lorg/apache/hadoop/hdfs/DFSInputStream$ReadStatistics;");
+ "()Lorg/apache/hadoop/hdfs/ReadStatistics;");
if (jthr) {
ret = printExceptionAndFree(env, jthr, PRINT_EXC_ALL,
"hdfsFileGetReadStatistics: getReadStatistics failed");
@@ -127,7 +127,7 @@ int hdfsFileGetReadStatistics(hdfsFile file,
goto done;
}
jthr = invokeMethod(env, &jVal, INSTANCE, readStats,
- "org/apache/hadoop/hdfs/DFSInputStream$ReadStatistics",
+ "org/apache/hadoop/hdfs/ReadStatistics",
"getTotalBytesRead", "()J");
if (jthr) {
ret = printExceptionAndFree(env, jthr, PRINT_EXC_ALL,
@@ -137,7 +137,7 @@ int hdfsFileGetReadStatistics(hdfsFile file,
s->totalBytesRead = jVal.j;
jthr = invokeMethod(env, &jVal, INSTANCE, readStats,
- "org/apache/hadoop/hdfs/DFSInputStream$ReadStatistics",
+ "org/apache/hadoop/hdfs/ReadStatistics",
"getTotalLocalBytesRead", "()J");
if (jthr) {
ret = printExceptionAndFree(env, jthr, PRINT_EXC_ALL,
@@ -147,7 +147,7 @@ int hdfsFileGetReadStatistics(hdfsFile file,
s->totalLocalBytesRead = jVal.j;
jthr = invokeMethod(env, &jVal, INSTANCE, readStats,
- "org/apache/hadoop/hdfs/DFSInputStream$ReadStatistics",
+ "org/apache/hadoop/hdfs/ReadStatistics",
"getTotalShortCircuitBytesRead", "()J");
if (jthr) {
ret = printExceptionAndFree(env, jthr, PRINT_EXC_ALL,
@@ -156,7 +156,7 @@ int hdfsFileGetReadStatistics(hdfsFile file,
}
s->totalShortCircuitBytesRead = jVal.j;
jthr = invokeMethod(env, &jVal, INSTANCE, readStats,
- "org/apache/hadoop/hdfs/DFSInputStream$ReadStatistics",
+ "org/apache/hadoop/hdfs/ReadStatistics",
"getTotalZeroCopyBytesRead", "()J");
if (jthr) {
ret = printExceptionAndFree(env, jthr, PRINT_EXC_ALL,
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[06/21] hadoop git commit: HADOOP-13519. Make Path Serializable.
Contributed by Steve Loughran
Posted by aw...@apache.org.
HADOOP-13519. Make Path Serializable. Contributed by Steve Loughran
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/1192781a
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/1192781a
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/1192781a
Branch: refs/heads/HADOOP-13341
Commit: 1192781a78e7fcfcbf01b15de5d5a1f22d982e0d
Parents: b6d839a
Author: Chris Douglas <cd...@apache.org>
Authored: Thu Sep 8 17:46:12 2016 -0700
Committer: Chris Douglas <cd...@apache.org>
Committed: Thu Sep 8 17:46:12 2016 -0700
----------------------------------------------------------------------
.../main/java/org/apache/hadoop/fs/Path.java | 20 +++++++++++++++++++-
.../java/org/apache/hadoop/fs/TestPath.java | 20 ++++++++++++++++++++
2 files changed, 39 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1192781a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Path.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Path.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Path.java
index f18a675..252b3cc 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Path.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Path.java
@@ -19,6 +19,9 @@
package org.apache.hadoop.fs;
import java.io.IOException;
+import java.io.InvalidObjectException;
+import java.io.ObjectInputValidation;
+import java.io.Serializable;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.regex.Pattern;
@@ -37,7 +40,7 @@ import org.apache.hadoop.conf.Configuration;
@Stringable
@InterfaceAudience.Public
@InterfaceStability.Stable
-public class Path implements Comparable {
+public class Path implements Comparable, Serializable, ObjectInputValidation {
/**
* The directory separator, a slash.
@@ -66,6 +69,8 @@ public class Path implements Comparable {
private static final Pattern HAS_DRIVE_LETTER_SPECIFIER =
Pattern.compile("^/?[a-zA-Z]:");
+ private static final long serialVersionUID = 0xad00f;
+
private URI uri; // a hierarchical uri
/**
@@ -565,4 +570,17 @@ public class Path implements Comparable {
}
return new Path(newUri);
}
+
+ /**
+ * Validate the contents of a deserialized Path, so as
+ * to defend against malicious object streams.
+ * @throws InvalidObjectException if there's no URI
+ */
+ @Override
+ public void validateObject() throws InvalidObjectException {
+ if (uri == null) {
+ throw new InvalidObjectException("No URI in deserialized Path");
+ }
+
+ }
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1192781a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestPath.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestPath.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestPath.java
index dc48a10..5123526 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestPath.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestPath.java
@@ -17,9 +17,14 @@
*/
package org.apache.hadoop.fs;
+import org.junit.Assert;
import org.junit.Test;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.Arrays;
@@ -506,4 +511,19 @@ public class TestPath {
assertFalse(Path.isWindowsAbsolutePath("C:test", false));
assertFalse(Path.isWindowsAbsolutePath("/C:test", true));
}
+
+ @Test(timeout = 30000)
+ public void testSerDeser() throws Throwable {
+ Path source = new Path("hdfs://localhost:4040/scratch");
+ ByteArrayOutputStream baos = new ByteArrayOutputStream(256);
+ try(ObjectOutputStream oos = new ObjectOutputStream(baos)) {
+ oos.writeObject(source);
+ }
+ ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray());
+ try (ObjectInputStream ois = new ObjectInputStream(bais)) {
+ Path deser = (Path) ois.readObject();
+ Assert.assertEquals(source, deser);
+ }
+
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[07/21] hadoop git commit: HDFS-10742. Measure lock time in
FsDatasetImpl. Contributed by Chen Liang
Posted by aw...@apache.org.
HDFS-10742. Measure lock time in FsDatasetImpl. Contributed by Chen Liang
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/011f3b24
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/011f3b24
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/011f3b24
Branch: refs/heads/HADOOP-13341
Commit: 011f3b24d4bfda505a90ab5b5576916a41f869c5
Parents: 1192781
Author: Chris Douglas <cd...@apache.org>
Authored: Thu Sep 8 17:52:02 2016 -0700
Committer: Chris Douglas <cd...@apache.org>
Committed: Thu Sep 8 17:53:40 2016 -0700
----------------------------------------------------------------------
.../apache/hadoop/util/AutoCloseableLock.java | 28 ++-
.../org/apache/hadoop/hdfs/DFSConfigKeys.java | 5 +
.../apache/hadoop/hdfs/InstrumentedLock.java | 185 +++++++++++++++++++
.../datanode/fsdataset/impl/FsDatasetImpl.java | 10 +-
.../src/main/resources/hdfs-default.xml | 8 +
.../hadoop/hdfs/TestInstrumentedLock.java | 166 +++++++++++++++++
6 files changed, 395 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/011f3b24/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/AutoCloseableLock.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/AutoCloseableLock.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/AutoCloseableLock.java
index 2aa8578..d920bc6 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/AutoCloseableLock.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/AutoCloseableLock.java
@@ -17,22 +17,33 @@
*/
package org.apache.hadoop.util;
+import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
+import com.google.common.annotations.VisibleForTesting;
+
/**
* This is a wrap class of a ReentrantLock. Extending AutoCloseable
* interface such that the users can use a try-with-resource syntax.
*/
public class AutoCloseableLock implements AutoCloseable {
- private final ReentrantLock lock;
+ private final Lock lock;
/**
* Creates an instance of {@code AutoCloseableLock}, initializes
- * the underlying {@code ReentrantLock} object.
+ * the underlying lock instance with a new {@code ReentrantLock}.
*/
public AutoCloseableLock() {
- this.lock = new ReentrantLock();
+ this(new ReentrantLock());
+ }
+
+ /**
+ * Wrap provided Lock instance.
+ * @param lock Lock instance to wrap in AutoCloseable API.
+ */
+ public AutoCloseableLock(Lock lock) {
+ this.lock = lock;
}
/**
@@ -86,7 +97,7 @@ public class AutoCloseableLock implements AutoCloseable {
/**
* A wrapper method that makes a call to {@code tryLock()} of
- * the underlying {@code ReentrantLock} object.
+ * the underlying {@code Lock} object.
*
* If the lock is not held by another thread, acquires the lock, set the
* hold count to one and returns {@code true}.
@@ -116,7 +127,12 @@ public class AutoCloseableLock implements AutoCloseable {
* @return {@code true} if any thread holds this lock and
* {@code false} otherwise
*/
- public boolean isLocked() {
- return lock.isLocked();
+ @VisibleForTesting
+ boolean isLocked() {
+ if (lock instanceof ReentrantLock) {
+ return ((ReentrantLock)lock).isLocked();
+ }
+ throw new UnsupportedOperationException();
}
+
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/011f3b24/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
index 7bdfd44..caf6b60 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
@@ -419,6 +419,11 @@ public class DFSConfigKeys extends CommonConfigurationKeys {
public static final String DFS_NAMENODE_READ_LOCK_REPORTING_THRESHOLD_MS_KEY =
"dfs.namenode.read-lock-reporting-threshold-ms";
public static final long DFS_NAMENODE_READ_LOCK_REPORTING_THRESHOLD_MS_DEFAULT = 5000L;
+ // Threshold for how long the lock warnings must be suppressed
+ public static final String DFS_LOCK_SUPPRESS_WARNING_INTERVAL_KEY =
+ "dfs.lock.suppress.warning.interval";
+ public static final long DFS_LOCK_SUPPRESS_WARNING_INTERVAL_DEFAULT =
+ 10000; //ms
public static final String DFS_UPGRADE_DOMAIN_FACTOR = "dfs.namenode.upgrade.domain.factor";
public static final int DFS_UPGRADE_DOMAIN_FACTOR_DEFAULT = DFS_REPLICATION_DEFAULT;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/011f3b24/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/InstrumentedLock.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/InstrumentedLock.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/InstrumentedLock.java
new file mode 100644
index 0000000..6279e95
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/InstrumentedLock.java
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs;
+
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.commons.logging.Log;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.hadoop.util.Timer;
+
+import com.google.common.annotations.VisibleForTesting;
+
+/**
+ * This is a debugging class that can be used by callers to track
+ * whether a specifc lock is being held for too long and periodically
+ * log a warning and stack trace, if so.
+ *
+ * The logged warnings are throttled so that logs are not spammed.
+ *
+ * A new instance of InstrumentedLock can be created for each object
+ * that needs to be instrumented.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+public class InstrumentedLock implements Lock {
+
+ private final Lock lock;
+ private final Log logger;
+ private final String name;
+ private final Timer clock;
+
+ /** Minimum gap between two lock warnings. */
+ private final long minLoggingGap;
+ /** Threshold for detecting long lock held time. */
+ private final long lockWarningThreshold;
+
+ // Tracking counters for lock statistics.
+ private volatile long lockAcquireTimestamp;
+ private final AtomicLong lastLogTimestamp;
+ private final AtomicLong warningsSuppressed = new AtomicLong(0);
+
+ /**
+ * Create a instrumented lock instance which logs a warning message
+ * when lock held time is above given threshold.
+ *
+ * @param name the identifier of the lock object
+ * @param logger this class does not have its own logger, will log to the
+ * given logger instead
+ * @param minLoggingGapMs the minimum time gap between two log messages,
+ * this is to avoid spamming to many logs
+ * @param lockWarningThresholdMs the time threshold to view lock held
+ * time as being "too long"
+ */
+ public InstrumentedLock(String name, Log logger, long minLoggingGapMs,
+ long lockWarningThresholdMs) {
+ this(name, logger, new ReentrantLock(),
+ minLoggingGapMs, lockWarningThresholdMs);
+ }
+
+ public InstrumentedLock(String name, Log logger, Lock lock,
+ long minLoggingGapMs, long lockWarningThresholdMs) {
+ this(name, logger, lock,
+ minLoggingGapMs, lockWarningThresholdMs, new Timer());
+ }
+
+ @VisibleForTesting
+ InstrumentedLock(String name, Log logger, Lock lock,
+ long minLoggingGapMs, long lockWarningThresholdMs, Timer clock) {
+ this.name = name;
+ this.lock = lock;
+ this.clock = clock;
+ this.logger = logger;
+ minLoggingGap = minLoggingGapMs;
+ lockWarningThreshold = lockWarningThresholdMs;
+ lastLogTimestamp = new AtomicLong(
+ clock.monotonicNow() - Math.max(minLoggingGap, lockWarningThreshold));
+ }
+
+ @Override
+ public void lock() {
+ lock.lock();
+ lockAcquireTimestamp = clock.monotonicNow();
+ }
+
+ @Override
+ public void lockInterruptibly() throws InterruptedException {
+ lock.lockInterruptibly();
+ lockAcquireTimestamp = clock.monotonicNow();
+ }
+
+ @Override
+ public boolean tryLock() {
+ if (lock.tryLock()) {
+ lockAcquireTimestamp = clock.monotonicNow();
+ return true;
+ }
+ return false;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) throws InterruptedException {
+ if (lock.tryLock(time, unit)) {
+ lockAcquireTimestamp = clock.monotonicNow();
+ return true;
+ }
+ return false;
+ }
+
+ @Override
+ public void unlock() {
+ long localLockReleaseTime = clock.monotonicNow();
+ long localLockAcquireTime = lockAcquireTimestamp;
+ lock.unlock();
+ check(localLockAcquireTime, localLockReleaseTime);
+ }
+
+ @Override
+ public Condition newCondition() {
+ return lock.newCondition();
+ }
+
+ @VisibleForTesting
+ void logWarning(long lockHeldTime, long suppressed) {
+ logger.warn(String.format("Lock held time above threshold: " +
+ "lock identifier: %s " +
+ "lockHeldTimeMs=%d ms. Suppressed %d lock warnings. " +
+ "The stack trace is: %s" ,
+ name, lockHeldTime, suppressed,
+ StringUtils.getStackTrace(Thread.currentThread())));
+ }
+
+ /**
+ * Log a warning if the lock was held for too long.
+ *
+ * Should be invoked by the caller immediately AFTER releasing the lock.
+ *
+ * @param acquireTime - timestamp just after acquiring the lock.
+ * @param releaseTime - timestamp just before releasing the lock.
+ */
+ private void check(long acquireTime, long releaseTime) {
+ if (!logger.isWarnEnabled()) {
+ return;
+ }
+
+ final long lockHeldTime = releaseTime - acquireTime;
+ if (lockWarningThreshold - lockHeldTime < 0) {
+ long now;
+ long localLastLogTs;
+ do {
+ now = clock.monotonicNow();
+ localLastLogTs = lastLogTimestamp.get();
+ long deltaSinceLastLog = now - localLastLogTs;
+ // check should print log or not
+ if (deltaSinceLastLog - minLoggingGap < 0) {
+ warningsSuppressed.incrementAndGet();
+ return;
+ }
+ } while (!lastLogTimestamp.compareAndSet(localLastLogTs, now));
+ long suppressed = warningsSuppressed.getAndSet(0);
+ logWarning(lockHeldTime, suppressed);
+ }
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/011f3b24/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
index ffd2f8a..e5da0e5 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
@@ -40,6 +40,7 @@ import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Executor;
+import java.util.concurrent.TimeUnit;
import javax.management.NotCompliantMBeanException;
import javax.management.ObjectName;
@@ -60,6 +61,7 @@ import org.apache.hadoop.fs.StorageType;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSUtilClient;
import org.apache.hadoop.hdfs.ExtendedBlockId;
+import org.apache.hadoop.hdfs.InstrumentedLock;
import org.apache.hadoop.util.AutoCloseableLock;
import org.apache.hadoop.hdfs.protocol.Block;
import org.apache.hadoop.hdfs.protocol.BlockListAsLongs;
@@ -278,7 +280,13 @@ class FsDatasetImpl implements FsDatasetSpi<FsVolumeImpl> {
this.dataStorage = storage;
this.conf = conf;
this.smallBufferSize = DFSUtilClient.getSmallBufferSize(conf);
- this.datasetLock = new AutoCloseableLock();
+ this.datasetLock = new AutoCloseableLock(
+ new InstrumentedLock(getClass().getName(), LOG,
+ conf.getTimeDuration(
+ DFSConfigKeys.DFS_LOCK_SUPPRESS_WARNING_INTERVAL_KEY,
+ DFSConfigKeys.DFS_LOCK_SUPPRESS_WARNING_INTERVAL_DEFAULT,
+ TimeUnit.MILLISECONDS),
+ 300));
// The number of volumes required for operation is the total number
// of volumes minus the number of failed volumes we can tolerate.
volFailuresTolerated = datanode.getDnConf().getVolFailuresTolerated();
http://git-wip-us.apache.org/repos/asf/hadoop/blob/011f3b24/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
index 3f78233..3a5de3e 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
@@ -4273,4 +4273,12 @@
a plan.
</description>
</property>
+
+ <property>
+ <name>dfs.lock.suppress.warning.interval</name>
+ <value>10s</value>
+ <description>Instrumentation reporting long critical sections will suppress
+ consecutive warnings within this interval.</description>
+ </property>
+
</configuration>
http://git-wip-us.apache.org/repos/asf/hadoop/blob/011f3b24/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestInstrumentedLock.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestInstrumentedLock.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestInstrumentedLock.java
new file mode 100644
index 0000000..f470688
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestInstrumentedLock.java
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs;
+
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.concurrent.locks.Lock;
+
+import org.apache.hadoop.util.AutoCloseableLock;
+import org.apache.hadoop.util.Timer;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TestName;
+import static org.mockito.Mockito.*;
+import static org.junit.Assert.*;
+
+/**
+ * A test class for InstrumentedLock.
+ */
+public class TestInstrumentedLock {
+
+ static final Log LOG = LogFactory.getLog(TestInstrumentedLock.class);
+
+ @Rule public TestName name = new TestName();
+
+ /**
+ * Test exclusive access of the lock.
+ * @throws Exception
+ */
+ @Test(timeout=10000)
+ public void testMultipleThread() throws Exception {
+ String testname = name.getMethodName();
+ InstrumentedLock lock = new InstrumentedLock(testname, LOG, 0, 300);
+ lock.lock();
+ try {
+ Thread competingThread = new Thread() {
+ @Override
+ public void run() {
+ assertFalse(lock.tryLock());
+ }
+ };
+ competingThread.start();
+ competingThread.join();
+ } finally {
+ lock.unlock();
+ }
+ }
+
+ /**
+ * Test the correctness with try-with-resource syntax.
+ * @throws Exception
+ */
+ @Test(timeout=10000)
+ public void testTryWithResourceSyntax() throws Exception {
+ String testname = name.getMethodName();
+ final AtomicReference<Thread> lockThread = new AtomicReference<>(null);
+ Lock lock = new InstrumentedLock(testname, LOG, 0, 300) {
+ @Override
+ public void lock() {
+ super.lock();
+ lockThread.set(Thread.currentThread());
+ }
+ @Override
+ public void unlock() {
+ super.unlock();
+ lockThread.set(null);
+ }
+ };
+ AutoCloseableLock acl = new AutoCloseableLock(lock);
+ try (AutoCloseable localLock = acl.acquire()) {
+ assertEquals(acl, localLock);
+ Thread competingThread = new Thread() {
+ @Override
+ public void run() {
+ assertNotEquals(Thread.currentThread(), lockThread.get());
+ assertFalse(lock.tryLock());
+ }
+ };
+ competingThread.start();
+ competingThread.join();
+ assertEquals(Thread.currentThread(), lockThread.get());
+ }
+ assertNull(lockThread.get());
+ }
+
+ /**
+ * Test the lock logs warning when lock held time is greater than threshold
+ * and not log warning otherwise.
+ * @throws Exception
+ */
+ @Test(timeout=10000)
+ public void testLockLongHoldingReport() throws Exception {
+ String testname = name.getMethodName();
+ final AtomicLong time = new AtomicLong(0);
+ Timer mclock = new Timer() {
+ @Override
+ public long monotonicNow() {
+ return time.get();
+ }
+ };
+ Lock mlock = mock(Lock.class);
+
+ final AtomicLong wlogged = new AtomicLong(0);
+ final AtomicLong wsuppresed = new AtomicLong(0);
+ InstrumentedLock lock = new InstrumentedLock(
+ testname, LOG, mlock, 2000, 300, mclock) {
+ @Override
+ void logWarning(long lockHeldTime, long suppressed) {
+ wlogged.incrementAndGet();
+ wsuppresed.set(suppressed);
+ }
+ };
+
+ // do not log warning when the lock held time is short
+ lock.lock(); // t = 0
+ time.set(200);
+ lock.unlock(); // t = 200
+ assertEquals(0, wlogged.get());
+ assertEquals(0, wsuppresed.get());
+
+ lock.lock(); // t = 200
+ time.set(700);
+ lock.unlock(); // t = 700
+ assertEquals(1, wlogged.get());
+ assertEquals(0, wsuppresed.get());
+
+ // despite the lock held time is greater than threshold
+ // suppress the log warning due to the logging gap
+ // (not recorded in wsuppressed until next log message)
+ lock.lock(); // t = 700
+ time.set(1100);
+ lock.unlock(); // t = 1100
+ assertEquals(1, wlogged.get());
+ assertEquals(0, wsuppresed.get());
+
+ // log a warning message when the lock held time is greater the threshold
+ // and the logging time gap is satisfied. Also should display suppressed
+ // previous warnings.
+ time.set(2400);
+ lock.lock(); // t = 2400
+ time.set(2800);
+ lock.unlock(); // t = 2800
+ assertEquals(2, wlogged.get());
+ assertEquals(1, wsuppresed.get());
+ }
+
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[14/21] hadoop git commit: HADOOP-13562. Change
hadoop_subcommand_opts to use only uppercase
Posted by aw...@apache.org.
HADOOP-13562. Change hadoop_subcommand_opts to use only uppercase
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/fc8d8c88
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/fc8d8c88
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/fc8d8c88
Branch: refs/heads/HADOOP-13341
Commit: fc8d8c88ab1dfa29536f7546454305fc8acdca26
Parents: ff16b57
Author: Allen Wittenauer <aw...@apache.org>
Authored: Tue Aug 30 08:52:41 2016 -0700
Committer: Allen Wittenauer <aw...@apache.org>
Committed: Fri Sep 9 04:07:43 2016 -0700
----------------------------------------------------------------------
.../src/main/bin/hadoop-functions.sh | 31 +++++++-------------
.../test/scripts/hadoop_subcommand_opts.bats | 18 ++++--------
2 files changed, 16 insertions(+), 33 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/fc8d8c88/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
index dd82347..695fd41 100755
--- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
+++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
@@ -2007,8 +2007,8 @@ function hadoop_subcommand_opts
{
declare program=$1
declare command=$2
- declare var
declare uvar
+ declare depvar
declare uprogram
declare ucommand
@@ -2029,34 +2029,25 @@ function hadoop_subcommand_opts
ucommand=${command^^}
fi
- # HDFS_namenode_OPTS
- # HADOOP_distcp_OPTS
- # MAPRED_distcp_OPTS
- # YARN_sharedcachemanger_OPTS
- # ...
- var="${uprogram}_${command}_OPTS"
+ uvar="${uprogram}_${ucommand}_OPTS"
# Let's handle all of the deprecation cases early
- # HADOOP_NAMENODE_OPTS -> HDFS_namenode_OPTS
- # YARN_RESOURCEMANAGER_OPTS -> YARN_resourcemanager_OPTS
+ # HADOOP_NAMENODE_OPTS -> HDFS_NAMENODE_OPTS
- uvar="${uprogram}_${ucommand}_OPTS"
- if [[ -n ${!uvar} ]]; then
- hadoop_deprecate_envvar "${uvar}" "${var}"
- fi
+ depvar="HADOOP_${ucommand}_OPTS"
- uvar="HADOOP_${ucommand}_OPTS"
- if [[ -n ${!uvar} ]]; then
- hadoop_deprecate_envvar "${uvar}" "${var}"
+ if [[ "${depvar}" != "${uvar}" ]]; then
+ if [[ -n "${!depvar}" ]]; then
+ hadoop_deprecate_envvar "${depvar}" "${uvar}"
+ fi
fi
- if [[ -n ${!var} ]]; then
- hadoop_debug "Appending ${!var} onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${!var}"
+ if [[ -n ${!uvar} ]]; then
+ hadoop_debug "Appending ${!uvar} onto HADOOP_OPTS"
+ HADOOP_OPTS="${HADOOP_OPTS} ${!uvar}"
return 0
fi
}
-
## @description Add custom (program)_(command)_SECURE_EXTRA_OPTS to HADOOP_OPTS.
## @description This *does not* handle the pre-3.x deprecated cases
## @audience public
http://git-wip-us.apache.org/repos/asf/hadoop/blob/fc8d8c88/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_subcommand_opts.bats
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_subcommand_opts.bats b/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_subcommand_opts.bats
index 1fbf343..0010a58 100644
--- a/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_subcommand_opts.bats
+++ b/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_subcommand_opts.bats
@@ -29,7 +29,7 @@ load hadoop-functions_test_helper
@test "hadoop_subcommand_opts (hadoop simple exist)" {
HADOOP_OPTS="x"
- HADOOP_test_OPTS="y"
+ HADOOP_TEST_OPTS="y"
hadoop_subcommand_opts hadoop test
echo "${HADOOP_OPTS}"
[ "${HADOOP_OPTS}" = "x y" ]
@@ -37,7 +37,7 @@ load hadoop-functions_test_helper
@test "hadoop_subcommand_opts (hadoop complex exist)" {
HADOOP_OPTS="x"
- HADOOP_test_OPTS="y z"
+ HADOOP_TEST_OPTS="y z"
hadoop_subcommand_opts hadoop test
echo "${HADOOP_OPTS}"
[ "${HADOOP_OPTS}" = "x y z" ]
@@ -45,7 +45,7 @@ load hadoop-functions_test_helper
@test "hadoop_subcommand_opts (hdfs simple exist)" {
HADOOP_OPTS="x"
- HDFS_test_OPTS="y"
+ HDFS_TEST_OPTS="y"
hadoop_subcommand_opts hdfs test
echo "${HADOOP_OPTS}"
[ "${HADOOP_OPTS}" = "x y" ]
@@ -53,24 +53,16 @@ load hadoop-functions_test_helper
@test "hadoop_subcommand_opts (yarn simple exist)" {
HADOOP_OPTS="x"
- YARN_test_OPTS="y"
+ YARN_TEST_OPTS="y"
hadoop_subcommand_opts yarn test
echo "${HADOOP_OPTS}"
[ "${HADOOP_OPTS}" = "x y" ]
}
-@test "hadoop_subcommand_opts (deprecation case #1)" {
+@test "hadoop_subcommand_opts (deprecation case)" {
HADOOP_OPTS="x"
HADOOP_NAMENODE_OPTS="y"
hadoop_subcommand_opts hdfs namenode
echo "${HADOOP_OPTS}"
[ "${HADOOP_OPTS}" = "x y" ]
}
-
-@test "hadoop_subcommand_opts (deprecation case #2)" {
- HADOOP_OPTS="x"
- YARN_RESOURCEMANAGER_OPTS="y"
- hadoop_subcommand_opts yarn resourcemanager
- echo "${HADOOP_OPTS}"
- [ "${HADOOP_OPTS}" = "x y" ]
-}
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[13/21] hadoop git commit: HADOOP-13564. modify mapred to use
hadoop_subcommand_opts
Posted by aw...@apache.org.
HADOOP-13564. modify mapred to use hadoop_subcommand_opts
Signed-off-by: Allen Wittenauer <aw...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/c7a5b7d1
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/c7a5b7d1
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/c7a5b7d1
Branch: refs/heads/HADOOP-13341
Commit: c7a5b7d152c8db7e6c4a93cf3923a5299176f638
Parents: ab578fb
Author: Allen Wittenauer <aw...@apache.org>
Authored: Tue Aug 30 15:11:24 2016 -0700
Committer: Allen Wittenauer <aw...@apache.org>
Committed: Fri Sep 9 04:07:43 2016 -0700
----------------------------------------------------------------------
hadoop-mapreduce-project/bin/mapred | 11 +++++++----
hadoop-mapreduce-project/bin/mapred-config.sh | 10 ++++++----
hadoop-mapreduce-project/conf/mapred-env.sh | 4 ++--
3 files changed, 15 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c7a5b7d1/hadoop-mapreduce-project/bin/mapred
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/bin/mapred b/hadoop-mapreduce-project/bin/mapred
index c566d5a..fe7c56a 100755
--- a/hadoop-mapreduce-project/bin/mapred
+++ b/hadoop-mapreduce-project/bin/mapred
@@ -69,8 +69,6 @@ function mapredcmd_case
historyserver)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
HADOOP_CLASSNAME=org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
- hadoop_debug "Appending HADOOP_JOB_HISTORYSERVER_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_JOB_HISTORYSERVER_OPTS}"
if [ -n "${HADOOP_JOB_HISTORYSERVER_HEAPSIZE}" ]; then
# shellcheck disable=SC2034
HADOOP_HEAPSIZE_MAX="${HADOOP_JOB_HISTORYSERVER_HEAPSIZE}"
@@ -144,13 +142,18 @@ fi
hadoop_add_client_opts
-if [[ ${HADOOP_SLAVE_MODE} = true ]]; then
- hadoop_common_slave_mode_execute "${HADOOP_MAPRED_HOME}/bin/mapred" "${HADOOP_USER_PARAMS[@]}"
+if [[ ${HADOOP_WORKER_MODE} = true ]]; then
+ hadoop_common_worker_mode_execute "${HADOOP_MAPRED_HOME}/bin/mapred" "${HADOOP_USER_PARAMS[@]}"
exit $?
fi
+hadoop_subcommand_opts "${HADOOP_SHELL_EXECNAME}" "${HADOOP_SUBCMD}"
+
if [[ "${HADOOP_SUBCMD_SECURESERVICE}" = true ]]; then
HADOOP_SECURE_USER="${HADOOP_SUBCMD_SECUREUSER}"
+
+ hadoop_subcommand_secure_opts "${HADOOP_SHELL_EXECNAME}" "${HADOOP_SUBCMD}"
+
hadoop_verify_secure_prereq
hadoop_setup_secure_service
priv_outfile="${HADOOP_LOG_DIR}/privileged-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.out"
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c7a5b7d1/hadoop-mapreduce-project/bin/mapred-config.sh
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/bin/mapred-config.sh b/hadoop-mapreduce-project/bin/mapred-config.sh
old mode 100644
new mode 100755
index a989792..68d3463d
--- a/hadoop-mapreduce-project/bin/mapred-config.sh
+++ b/hadoop-mapreduce-project/bin/mapred-config.sh
@@ -26,7 +26,7 @@ function hadoop_subproject_init
export HADOOP_MAPRED_ENV_PROCESSED=true
fi
fi
-
+
# at some point in time, someone thought it would be a good idea to
# create separate vars for every subproject. *sigh*
# let's perform some overrides and setup some defaults for bw compat
@@ -38,15 +38,17 @@ function hadoop_subproject_init
hadoop_deprecate_envvar HADOOP_MAPRED_LOG_DIR HADOOP_LOG_DIR
hadoop_deprecate_envvar HADOOP_MAPRED_LOGFILE HADOOP_LOGFILE
-
+
hadoop_deprecate_envvar HADOOP_MAPRED_NICENESS HADOOP_NICENESS
-
+
hadoop_deprecate_envvar HADOOP_MAPRED_STOP_TIMEOUT HADOOP_STOP_TIMEOUT
-
+
hadoop_deprecate_envvar HADOOP_MAPRED_PID_DIR HADOOP_PID_DIR
hadoop_deprecate_envvar HADOOP_MAPRED_ROOT_LOGGER HADOOP_ROOT_LOGGER
+ hadoop_deprecate_envvar HADOOP_JOB_HISTORY_OPTS MAPRED_HISTORYSERVER_OPTS
+
HADOOP_MAPRED_HOME="${HADOOP_MAPRED_HOME:-$HADOOP_HOME}"
hadoop_deprecate_envvar HADOOP_MAPRED_IDENT_STRING HADOOP_IDENT_STRING
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c7a5b7d1/hadoop-mapreduce-project/conf/mapred-env.sh
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/conf/mapred-env.sh b/hadoop-mapreduce-project/conf/mapred-env.sh
index bbe4a49..53bc296 100644
--- a/hadoop-mapreduce-project/conf/mapred-env.sh
+++ b/hadoop-mapreduce-project/conf/mapred-env.sh
@@ -31,14 +31,14 @@
# Specify the max heapsize for the JobHistoryServer. If no units are
# given, it will be assumed to be in MB.
# This value will be overridden by an Xmx setting specified in HADOOP_OPTS,
-# and/or HADOOP_JOB_HISTORYSERVER_OPTS.
+# and/or MAPRED_HISTORYSERVER_OPTS.
# Default is the same as HADOOP_HEAPSIZE_MAX.
#export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=
# Specify the JVM options to be used when starting the HistoryServer.
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
-#export HADOOP_JOB_HISTORYSERVER_OPTS=
+#export MAPRED_HISTORYSERVER_OPTS=
# Specify the log4j settings for the JobHistoryServer
# Java property: hadoop.root.logger
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[15/21] hadoop git commit: HADOOP-13355. Handle HADOOP_CLIENT_OPTS in
a function (aw)
Posted by aw...@apache.org.
HADOOP-13355. Handle HADOOP_CLIENT_OPTS in a function (aw)
Signed-off-by: Allen Wittenauer <aw...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/548158f2
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/548158f2
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/548158f2
Branch: refs/heads/HADOOP-13341
Commit: 548158f24fc37da0cb1946597fbf20d44e269397
Parents: 5b7b9fc
Author: Allen Wittenauer <aw...@apache.org>
Authored: Fri Aug 26 14:33:33 2016 -0700
Committer: Allen Wittenauer <aw...@apache.org>
Committed: Fri Sep 9 04:07:43 2016 -0700
----------------------------------------------------------------------
.../hadoop-common/src/main/bin/hadoop | 6 +--
.../src/main/bin/hadoop-functions.sh | 21 ++++++++++
.../test/scripts/hadoop_add_client_opts.bats | 40 ++++++++++++++++++++
.../hadoop-hdfs/src/main/bin/hdfs | 14 +------
hadoop-mapreduce-project/bin/mapred | 12 +-----
.../main/shellprofile.d/hadoop-archive-logs.sh | 2 -
.../src/main/shellprofile.d/hadoop-distcp.sh | 4 --
.../src/main/shellprofile.d/hadoop-extras.sh | 2 -
.../src/main/shellprofile.d/hadoop-rumen.sh | 4 --
.../hadoop-sls/src/main/bin/rumen2sls.sh | 3 +-
hadoop-tools/hadoop-sls/src/main/bin/slsrun.sh | 3 +-
.../src/main/shellprofile.d/hadoop-streaming.sh | 4 --
hadoop-yarn-project/hadoop-yarn/bin/yarn | 34 ++++++-----------
13 files changed, 81 insertions(+), 68 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/548158f2/hadoop-common-project/hadoop-common/src/main/bin/hadoop
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop b/hadoop-common-project/hadoop-common/src/main/bin/hadoop
index b57a4c1..4aae621 100755
--- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop
+++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop
@@ -161,10 +161,6 @@ function hadoopcmd_case
fi
;;
esac
-
- # Always respect HADOOP_OPTS and HADOOP_CLIENT_OPTS
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
}
# This script runs the hadoop core commands.
@@ -205,6 +201,8 @@ fi
hadoop_verify_user "${HADOOP_SUBCMD}"
+hadoop_add_client_opts
+
if [[ ${HADOOP_WORKER_MODE} = true ]]; then
hadoop_common_worker_mode_execute "${HADOOP_COMMON_HOME}/bin/hadoop" "${HADOOP_USER_PARAMS[@]}"
exit $?
http://git-wip-us.apache.org/repos/asf/hadoop/blob/548158f2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
index 6e58dca..9003913 100755
--- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
+++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
@@ -306,6 +306,13 @@ function hadoop_bootstrap
HADOOP_TOOLS_DIR=${HADOOP_TOOLS_DIR:-"share/hadoop/tools"}
HADOOP_TOOLS_LIB_JARS_DIR=${HADOOP_TOOLS_LIB_JARS_DIR:-"${HADOOP_TOOLS_DIR}/lib"}
+ # by default, whatever we are about to run doesn't support
+ # daemonization
+ HADOOP_SUBCMD_SUPPORTDAEMONIZATION=false
+
+ # shellcheck disable=SC2034
+ HADOOP_SUBCMD_SECURESERVICE=false
+
# usage output set to zero
hadoop_reset_usage
@@ -1230,6 +1237,20 @@ function hadoop_translate_cygwin_path
fi
}
+## @description Adds the HADOOP_CLIENT_OPTS variable to
+## @description HADOOP_OPTS if HADOOP_SUBCMD_SUPPORTDAEMONIZATION is false
+## @audience public
+## @stability stable
+## @replaceable yes
+function hadoop_add_client_opts
+{
+ if [[ "${HADOOP_SUBCMD_SUPPORTDAEMONIZATION}" = false
+ || -z "${HADOOP_SUBCMD_SUPPORTDAEMONIZATION}" ]]; then
+ hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
+ HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
+ fi
+}
+
## @description Finish configuring Hadoop specific system properties
## @description prior to executing Java
## @audience private
http://git-wip-us.apache.org/repos/asf/hadoop/blob/548158f2/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_add_client_opts.bats
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_add_client_opts.bats b/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_add_client_opts.bats
new file mode 100644
index 0000000..bc5051d
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_add_client_opts.bats
@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+load hadoop-functions_test_helper
+
+@test "hadoop_subcommand_opts (daemonization false)" {
+ HADOOP_OPTS="1"
+ HADOOP_CLIENT_OPTS="2"
+ HADOOP_SUBCMD_SUPPORTDAEMONIZATION="false"
+ hadoop_add_client_opts
+ [ "${HADOOP_OPTS}" = "1 2" ]
+}
+
+@test "hadoop_subcommand_opts (daemonization true)" {
+ HADOOP_OPTS="1"
+ HADOOP_CLIENT_OPTS="2"
+ HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
+ hadoop_add_client_opts
+ [ "${HADOOP_OPTS}" = "1" ]
+}
+
+@test "hadoop_subcommand_opts (daemonization empty)" {
+ HADOOP_OPTS="1"
+ HADOOP_CLIENT_OPTS="2"
+ unset HADOOP_SUBCMD_SUPPORTDAEMONIZATION
+ hadoop_add_client_opts
+ [ "${HADOOP_OPTS}" = "1 2" ]
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/548158f2/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs b/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
index 7a90f08..af4997d8 100755
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
@@ -118,18 +118,12 @@ function hdfscmd_case
;;
dfs)
HADOOP_CLASSNAME=org.apache.hadoop.fs.FsShell
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
;;
dfsadmin)
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.DFSAdmin
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
;;
diskbalancer)
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.DiskBalancerCLI
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
;;
envvars)
echo "JAVA_HOME='${JAVA_HOME}'"
@@ -144,16 +138,12 @@ function hdfscmd_case
;;
erasurecode)
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.erasurecode.ECCli
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
;;
fetchdt)
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.DelegationTokenFetcher
;;
fsck)
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.DFSck
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
;;
getconf)
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.GetConf
@@ -163,8 +153,6 @@ function hdfscmd_case
;;
haadmin)
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.DFSHAAdmin
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
;;
journalnode)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
@@ -293,6 +281,8 @@ fi
hadoop_verify_user "${HADOOP_SUBCMD}"
+hadoop_add_client_opts
+
if [[ ${HADOOP_WORKER_MODE} = true ]]; then
hadoop_common_worker_mode_execute "${HADOOP_HDFS_HOME}/bin/hdfs" "${HADOOP_USER_PARAMS[@]}"
exit $?
http://git-wip-us.apache.org/repos/asf/hadoop/blob/548158f2/hadoop-mapreduce-project/bin/mapred
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/bin/mapred b/hadoop-mapreduce-project/bin/mapred
index 046d48c..3243d20 100755
--- a/hadoop-mapreduce-project/bin/mapred
+++ b/hadoop-mapreduce-project/bin/mapred
@@ -79,31 +79,21 @@ function mapredcmd_case
;;
hsadmin)
HADOOP_CLASSNAME=org.apache.hadoop.mapreduce.v2.hs.client.HSAdmin
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
;;
job)
HADOOP_CLASSNAME=org.apache.hadoop.mapred.JobClient
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
;;
pipes)
HADOOP_CLASSNAME=org.apache.hadoop.mapred.pipes.Submitter
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
;;
queue)
HADOOP_CLASSNAME=org.apache.hadoop.mapred.JobQueueClient
;;
sampler)
HADOOP_CLASSNAME=org.apache.hadoop.mapred.lib.InputSampler
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
;;
version)
HADOOP_CLASSNAME=org.apache.hadoop.util.VersionInfo
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
;;
*)
HADOOP_CLASSNAME="${subcmd}"
@@ -152,6 +142,8 @@ fi
hadoop_verify_user "${HADOOP_SUBCMD}"
+hadoop_add_client_opts
+
if [[ ${HADOOP_SLAVE_MODE} = true ]]; then
hadoop_common_slave_mode_execute "${HADOOP_MAPRED_HOME}/bin/mapred" "${HADOOP_USER_PARAMS[@]}"
exit $?
http://git-wip-us.apache.org/repos/asf/hadoop/blob/548158f2/hadoop-tools/hadoop-archive-logs/src/main/shellprofile.d/hadoop-archive-logs.sh
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-archive-logs/src/main/shellprofile.d/hadoop-archive-logs.sh b/hadoop-tools/hadoop-archive-logs/src/main/shellprofile.d/hadoop-archive-logs.sh
index ae7b6c6..c889816 100755
--- a/hadoop-tools/hadoop-archive-logs/src/main/shellprofile.d/hadoop-archive-logs.sh
+++ b/hadoop-tools/hadoop-archive-logs/src/main/shellprofile.d/hadoop-archive-logs.sh
@@ -32,8 +32,6 @@ function mapred_subcommand_archive-logs
# shellcheck disable=SC2034
HADOOP_CLASSNAME=org.apache.hadoop.tools.HadoopArchiveLogs
hadoop_add_to_classpath_tools hadoop-archive-logs
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
}
fi
http://git-wip-us.apache.org/repos/asf/hadoop/blob/548158f2/hadoop-tools/hadoop-distcp/src/main/shellprofile.d/hadoop-distcp.sh
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-distcp/src/main/shellprofile.d/hadoop-distcp.sh b/hadoop-tools/hadoop-distcp/src/main/shellprofile.d/hadoop-distcp.sh
index 0178c54..6e93ec1 100755
--- a/hadoop-tools/hadoop-distcp/src/main/shellprofile.d/hadoop-distcp.sh
+++ b/hadoop-tools/hadoop-distcp/src/main/shellprofile.d/hadoop-distcp.sh
@@ -32,8 +32,6 @@ function hadoop_subcommand_distcp
# shellcheck disable=SC2034
HADOOP_CLASSNAME=org.apache.hadoop.tools.DistCp
hadoop_add_to_classpath_tools hadoop-distcp
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
}
fi
@@ -55,8 +53,6 @@ function mapred_subcommand_distcp
# shellcheck disable=SC2034
HADOOP_CLASSNAME=org.apache.hadoop.tools.DistCp
hadoop_add_to_classpath_tools hadoop-distcp
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
}
fi
http://git-wip-us.apache.org/repos/asf/hadoop/blob/548158f2/hadoop-tools/hadoop-extras/src/main/shellprofile.d/hadoop-extras.sh
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-extras/src/main/shellprofile.d/hadoop-extras.sh b/hadoop-tools/hadoop-extras/src/main/shellprofile.d/hadoop-extras.sh
index 829d406..1ce9aee 100755
--- a/hadoop-tools/hadoop-extras/src/main/shellprofile.d/hadoop-extras.sh
+++ b/hadoop-tools/hadoop-extras/src/main/shellprofile.d/hadoop-extras.sh
@@ -32,8 +32,6 @@ function hadoop_subcommand_distch
# shellcheck disable=SC2034
HADOOP_CLASSNAME=org.apache.hadoop.tools.DistCh
hadoop_add_to_classpath_tools hadoop-extras
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
}
fi
http://git-wip-us.apache.org/repos/asf/hadoop/blob/548158f2/hadoop-tools/hadoop-rumen/src/main/shellprofile.d/hadoop-rumen.sh
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-rumen/src/main/shellprofile.d/hadoop-rumen.sh b/hadoop-tools/hadoop-rumen/src/main/shellprofile.d/hadoop-rumen.sh
index d7d4022..77023ff 100755
--- a/hadoop-tools/hadoop-rumen/src/main/shellprofile.d/hadoop-rumen.sh
+++ b/hadoop-tools/hadoop-rumen/src/main/shellprofile.d/hadoop-rumen.sh
@@ -30,8 +30,6 @@ function hadoop_subcommand_rumenfolder
# shellcheck disable=SC2034
HADOOP_CLASSNAME=org.apache.hadoop.tools.rumen.Folder
hadoop_add_to_classpath_tools hadoop-rumen
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
}
fi
@@ -51,8 +49,6 @@ function hadoop_subcommand_rumentrace
# shellcheck disable=SC2034
HADOOP_CLASSNAME=org.apache.hadoop.tools.rumen.TraceBuilder
hadoop_add_to_classpath_tools hadoop-rumen
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
}
fi
http://git-wip-us.apache.org/repos/asf/hadoop/blob/548158f2/hadoop-tools/hadoop-sls/src/main/bin/rumen2sls.sh
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-sls/src/main/bin/rumen2sls.sh b/hadoop-tools/hadoop-sls/src/main/bin/rumen2sls.sh
index 0bd291b..565dfe6 100644
--- a/hadoop-tools/hadoop-sls/src/main/bin/rumen2sls.sh
+++ b/hadoop-tools/hadoop-sls/src/main/bin/rumen2sls.sh
@@ -68,8 +68,7 @@ function run_sls_generator()
hadoop_add_param args -outputJobs "-outputJobs ${outputdir}/${outputprefix}-jobs.json"
hadoop_add_param args -outputNodes "-outputNodes ${outputdir}/${outputprefix}-nodes.json"
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
+ hadoop_add_client_opts
hadoop_finalize
# shellcheck disable=SC2086
http://git-wip-us.apache.org/repos/asf/hadoop/blob/548158f2/hadoop-tools/hadoop-sls/src/main/bin/slsrun.sh
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-sls/src/main/bin/slsrun.sh b/hadoop-tools/hadoop-sls/src/main/bin/slsrun.sh
index 403c4bb..218dee4 100644
--- a/hadoop-tools/hadoop-sls/src/main/bin/slsrun.sh
+++ b/hadoop-tools/hadoop-sls/src/main/bin/slsrun.sh
@@ -96,8 +96,7 @@ function run_simulation() {
hadoop_add_param args -printsimulation "-printsimulation"
fi
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
+ hadoop_add_client_opts
hadoop_finalize
# shellcheck disable=SC2086
http://git-wip-us.apache.org/repos/asf/hadoop/blob/548158f2/hadoop-tools/hadoop-streaming/src/main/shellprofile.d/hadoop-streaming.sh
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-streaming/src/main/shellprofile.d/hadoop-streaming.sh b/hadoop-tools/hadoop-streaming/src/main/shellprofile.d/hadoop-streaming.sh
index cca016d..c3010ff 100755
--- a/hadoop-tools/hadoop-streaming/src/main/shellprofile.d/hadoop-streaming.sh
+++ b/hadoop-tools/hadoop-streaming/src/main/shellprofile.d/hadoop-streaming.sh
@@ -46,10 +46,6 @@ function mapred_subcommand_streaming
done
IFS=${oldifs}
-
- hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
-
}
fi
http://git-wip-us.apache.org/repos/asf/hadoop/blob/548158f2/hadoop-yarn-project/hadoop-yarn/bin/yarn
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/bin/yarn b/hadoop-yarn-project/hadoop-yarn/bin/yarn
index bd91633..66a87b6 100755
--- a/hadoop-yarn-project/hadoop-yarn/bin/yarn
+++ b/hadoop-yarn-project/hadoop-yarn/bin/yarn
@@ -68,8 +68,6 @@ function yarncmd_case
case ${subcmd} in
application|applicationattempt|container)
HADOOP_CLASSNAME=org.apache.hadoop.yarn.client.cli.ApplicationCLI
- hadoop_debug "Append YARN_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${YARN_CLIENT_OPTS}"
set -- "${subcmd}" "$@"
HADOOP_SUBCMD_ARGS=("$@")
;;
@@ -78,13 +76,9 @@ function yarncmd_case
;;
cluster)
HADOOP_CLASSNAME=org.apache.hadoop.yarn.client.cli.ClusterCLI
- hadoop_debug "Append YARN_CLIENT_OPTS onto YARN_OPTS"
- YARN_OPTS="${YARN_OPTS} ${YARN_CLIENT_OPTS}"
;;
daemonlog)
HADOOP_CLASSNAME=org.apache.hadoop.log.LogLevel
- hadoop_debug "Append YARN_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${YARN_CLIENT_OPTS}"
;;
envvars)
echo "JAVA_HOME='${JAVA_HOME}'"
@@ -99,8 +93,6 @@ function yarncmd_case
;;
jar)
HADOOP_CLASSNAME=org.apache.hadoop.util.RunJar
- hadoop_debug "Append YARN_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${YARN_CLIENT_OPTS}"
;;
historyserver)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
@@ -111,13 +103,9 @@ function yarncmd_case
;;
logs)
HADOOP_CLASSNAME=org.apache.hadoop.yarn.client.cli.LogsCLI
- hadoop_debug "Append YARN_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${YARN_CLIENT_OPTS}"
;;
node)
HADOOP_CLASSNAME=org.apache.hadoop.yarn.client.cli.NodeCLI
- hadoop_debug "Append YARN_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${YARN_CLIENT_OPTS}"
;;
nodemanager)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
@@ -142,8 +130,6 @@ function yarncmd_case
;;
queue)
HADOOP_CLASSNAME=org.apache.hadoop.yarn.client.cli.QueueCLI
- hadoop_debug "Append YARN_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${YARN_CLIENT_OPTS}"
;;
resourcemanager)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
@@ -158,13 +144,9 @@ function yarncmd_case
;;
rmadmin)
HADOOP_CLASSNAME='org.apache.hadoop.yarn.client.cli.RMAdminCLI'
- hadoop_debug "Append YARN_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${YARN_CLIENT_OPTS}"
;;
scmadmin)
HADOOP_CLASSNAME='org.apache.hadoop.yarn.client.SCMAdmin'
- hadoop_debug "Append YARN_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${YARN_CLIENT_OPTS}"
;;
sharedcachemanager)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
@@ -191,8 +173,6 @@ function yarncmd_case
;;
version)
HADOOP_CLASSNAME=org.apache.hadoop.util.VersionInfo
- hadoop_debug "Append YARN_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${YARN_CLIENT_OPTS}"
;;
top)
doNotSetCols=0
@@ -222,8 +202,6 @@ function yarncmd_case
fi
fi
HADOOP_CLASSNAME=org.apache.hadoop.yarn.client.cli.TopCLI
- hadoop_debug "Append YARN_CLIENT_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${YARN_CLIENT_OPTS}"
HADOOP_SUBCMD_ARGS=("$@")
;;
*)
@@ -273,6 +251,18 @@ fi
hadoop_verify_user "${HADOOP_SUBCMD}"
+# It's unclear if YARN_CLIENT_OPTS is actually a useful
+# thing to have separate from HADOOP_CLIENT_OPTS. Someone
+# might use it, so let's not deprecate it and just override
+# HADOOP_CLIENT_OPTS instead before we (potentially) add it
+# to the command line
+if [[ -n "${YARN_CLIENT_OPTS}" ]]; then
+ # shellcheck disable=SC2034
+ HADOOP_CLIENT_OPTS=${YARN_CLIENT_OPTS}
+fi
+
+hadoop_add_client_opts
+
if [[ ${HADOOP_WORKER_MODE} = true ]]; then
hadoop_common_worker_mode_execute "${HADOOP_YARN_HOME}/bin/yarn" "${HADOOP_USER_PARAMS[@]}"
exit $?
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[10/21] hadoop git commit: HDFS-9849. DiskBalancer: reduce lock path
in shutdown code. Contributed by Yuanbo Liu.
Posted by aw...@apache.org.
HDFS-9849. DiskBalancer: reduce lock path in shutdown code. Contributed by Yuanbo Liu.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/baab4892
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/baab4892
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/baab4892
Branch: refs/heads/HADOOP-13341
Commit: baab48922a301d639ea84ecf00d8a7616acd950d
Parents: 35c5943
Author: Anu Engineer <ae...@apache.org>
Authored: Thu Sep 8 20:00:42 2016 -0700
Committer: Anu Engineer <ae...@apache.org>
Committed: Thu Sep 8 20:00:42 2016 -0700
----------------------------------------------------------------------
.../hdfs/server/datanode/DiskBalancer.java | 19 +++++++++++++++----
1 file changed, 15 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/baab4892/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DiskBalancer.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DiskBalancer.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DiskBalancer.java
index ec72d97..e9e2e5b 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DiskBalancer.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DiskBalancer.java
@@ -121,17 +121,23 @@ public class DiskBalancer {
*/
public void shutdown() {
lock.lock();
+ boolean needShutdown = false;
try {
this.isDiskBalancerEnabled = false;
this.currentResult = Result.NO_PLAN;
if ((this.future != null) && (!this.future.isDone())) {
this.currentResult = Result.PLAN_CANCELLED;
this.blockMover.setExitFlag();
- shutdownExecutor();
+ scheduler.shutdown();
+ needShutdown = true;
}
} finally {
lock.unlock();
}
+ // no need to hold lock while shutting down executor.
+ if (needShutdown) {
+ shutdownExecutor();
+ }
}
/**
@@ -139,7 +145,6 @@ public class DiskBalancer {
*/
private void shutdownExecutor() {
final int secondsTowait = 10;
- scheduler.shutdown();
try {
if (!scheduler.awaitTermination(secondsTowait, TimeUnit.SECONDS)) {
scheduler.shutdownNow();
@@ -228,6 +233,7 @@ public class DiskBalancer {
*/
public void cancelPlan(String planID) throws DiskBalancerException {
lock.lock();
+ boolean needShutdown = false;
try {
checkDiskBalancerEnabled();
if (this.planID == null ||
@@ -239,13 +245,18 @@ public class DiskBalancer {
DiskBalancerException.Result.NO_SUCH_PLAN);
}
if (!this.future.isDone()) {
- this.blockMover.setExitFlag();
- shutdownExecutor();
this.currentResult = Result.PLAN_CANCELLED;
+ this.blockMover.setExitFlag();
+ scheduler.shutdown();
+ needShutdown = true;
}
} finally {
lock.unlock();
}
+ // no need to hold lock while shutting down executor.
+ if (needShutdown) {
+ shutdownExecutor();
+ }
}
/**
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[08/21] hadoop git commit: HDFS-10831. Add log when
URLConnectionFactory.openConnection failed. Contributed by yunjiong zhao.
Posted by aw...@apache.org.
HDFS-10831. Add log when URLConnectionFactory.openConnection failed. Contributed by yunjiong zhao.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/b07c266d
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/b07c266d
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/b07c266d
Branch: refs/heads/HADOOP-13341
Commit: b07c266dca7f303c793b432a0738d593728cf2b3
Parents: 011f3b2
Author: Andrew Wang <wa...@apache.org>
Authored: Thu Sep 8 18:30:18 2016 -0700
Committer: Andrew Wang <wa...@apache.org>
Committed: Thu Sep 8 18:30:18 2016 -0700
----------------------------------------------------------------------
.../main/java/org/apache/hadoop/hdfs/web/URLConnectionFactory.java | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b07c266d/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/URLConnectionFactory.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/URLConnectionFactory.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/URLConnectionFactory.java
index 975f72e..96095db 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/URLConnectionFactory.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/URLConnectionFactory.java
@@ -183,6 +183,7 @@ public class URLConnectionFactory {
return openConnection(url, false);
} catch (AuthenticationException e) {
// Unreachable
+ LOG.error("Open connection {} failed", url, e);
return null;
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[05/21] hadoop git commit: HDFS-10845. Change defaults in
hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin
Posted by aw...@apache.org.
HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/b6d839a6
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/b6d839a6
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/b6d839a6
Branch: refs/heads/HADOOP-13341
Commit: b6d839a60ceed733bfacb791fc5ed06116720dd0
Parents: cae33118
Author: Chris Douglas <cd...@apache.org>
Authored: Thu Sep 8 13:29:15 2016 -0700
Committer: Chris Douglas <cd...@apache.org>
Committed: Thu Sep 8 13:29:30 2016 -0700
----------------------------------------------------------------------
.../src/main/resources/hdfs-default.xml | 22 ++++++++++----------
1 file changed, 11 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6d839a6/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
index d427f88..3f78233 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
@@ -650,7 +650,7 @@
<property>
<name>dfs.blockreport.initialDelay</name>
- <value>0</value>
+ <value>0s</value>
<description>
Delay for first block report in seconds. Support multiple time unit
suffix(case insensitive), as described in dfs.heartbeat.interval.
@@ -694,7 +694,7 @@
<property>
<name>dfs.datanode.directoryscan.interval</name>
- <value>21600</value>
+ <value>21600s</value>
<description>Interval in seconds for Datanode to scan data directories and
reconcile the difference between blocks in memory and on the disk.
Support multiple time unit suffix(case insensitive), as described
@@ -732,7 +732,7 @@
<property>
<name>dfs.heartbeat.interval</name>
- <value>3</value>
+ <value>3s</value>
<description>
Determines datanode heartbeat interval in seconds.
Can use the following suffix (case insensitive):
@@ -942,7 +942,7 @@
<property>
<name>dfs.namenode.decommission.interval</name>
- <value>30</value>
+ <value>30s</value>
<description>Namenode periodicity in seconds to check if decommission is
complete. Support multiple time unit suffix(case insensitive), as described
in dfs.heartbeat.interval.
@@ -973,7 +973,7 @@
<property>
<name>dfs.namenode.replication.interval</name>
- <value>3</value>
+ <value>3s</value>
<description>The periodicity in seconds with which the namenode computes
replication work for datanodes. Support multiple time unit suffix(case insensitive),
as described in dfs.heartbeat.interval.
@@ -1071,7 +1071,7 @@
<property>
<name>dfs.namenode.checkpoint.period</name>
- <value>3600</value>
+ <value>3600s</value>
<description>
The number of seconds between two periodic checkpoints.
Support multiple time unit suffix(case insensitive), as described
@@ -1090,7 +1090,7 @@
<property>
<name>dfs.namenode.checkpoint.check.period</name>
- <value>60</value>
+ <value>60s</value>
<description>The SecondaryNameNode and CheckpointNode will poll the NameNode
every 'dfs.namenode.checkpoint.check.period' seconds to query the number
of uncheckpointed transactions. Support multiple time unit suffix(case insensitive),
@@ -1433,7 +1433,7 @@
<property>
<name>dfs.client.datanode-restart.timeout</name>
- <value>30</value>
+ <value>30s</value>
<description>
Expert only. The time to wait, in seconds, from reception of an
datanode shutdown notification for quick restart, until declaring
@@ -1502,7 +1502,7 @@
<property>
<name>dfs.ha.log-roll.period</name>
- <value>120</value>
+ <value>120s</value>
<description>
How often, in seconds, the StandbyNode should ask the active to
roll edit logs. Since the StandbyNode only reads from finalized
@@ -1516,7 +1516,7 @@
<property>
<name>dfs.ha.tail-edits.period</name>
- <value>60</value>
+ <value>60s</value>
<description>
How often, in seconds, the StandbyNode should check for new
finalized log segments in the shared edits log.
@@ -2950,7 +2950,7 @@
<property>
<name>dfs.datanode.bp-ready.timeout</name>
- <value>20</value>
+ <value>20s</value>
<description>
The maximum wait time for datanode to be ready before failing the
received request. Setting this to 0 fails requests right away if the
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[11/21] hadoop git commit: HADOOP-13356. Add a function to handle
command_subcommand_OPTS (aw)
Posted by aw...@apache.org.
HADOOP-13356. Add a function to handle command_subcommand_OPTS (aw)
Signed-off-by: Allen Wittenauer <aw...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/5b7b9fc6
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/5b7b9fc6
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/5b7b9fc6
Branch: refs/heads/HADOOP-13341
Commit: 5b7b9fc6a94abfada24659a560507a3d6dbae45e
Parents: baab489
Author: Allen Wittenauer <aw...@apache.org>
Authored: Fri Jul 8 09:25:09 2016 -0700
Committer: Allen Wittenauer <aw...@apache.org>
Committed: Fri Sep 9 04:05:10 2016 -0700
----------------------------------------------------------------------
.../src/main/bin/hadoop-functions.sh | 62 ++++++++++++++++
.../test/scripts/hadoop_subcommand_opts.bats | 76 ++++++++++++++++++++
2 files changed, 138 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/5b7b9fc6/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
index 75554f0..6e58dca 100755
--- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
+++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
@@ -1974,6 +1974,68 @@ function hadoop_verify_user
fi
}
+## @description Add custom (program)_(command)_OPTS to HADOOP_OPTS.
+## @description Also handles the deprecated cases from pre-3.x.
+## @audience public
+## @stability stable
+## @replaceable yes
+## @param program
+## @param subcommand
+## @return will exit on failure conditions
+function hadoop_subcommand_opts
+{
+ declare program=$1
+ declare command=$2
+ declare var
+ declare uvar
+ declare uprogram
+ declare ucommand
+
+ if [[ -z "${program}" || -z "${command}" ]]; then
+ return 1
+ fi
+
+ # bash 4 and up have built-in ways to upper and lower
+ # case the contents of vars. This is faster than
+ # calling tr.
+
+ if [[ -z "${BASH_VERSINFO[0]}" ]] \
+ || [[ "${BASH_VERSINFO[0]}" -lt 4 ]]; then
+ uprogram=$(echo "${program}" | tr '[:lower:]' '[:upper:]')
+ ucommand=$(echo "${command}" | tr '[:lower:]' '[:upper:]')
+ else
+ uprogram=${program^^}
+ ucommand=${command^^}
+ fi
+
+ # HDFS_namenode_OPTS
+ # HADOOP_distcp_OPTS
+ # MAPRED_distcp_OPTS
+ # YARN_sharedcachemanger_OPTS
+ # ...
+ var="${uprogram}_${command}_OPTS"
+
+ # Let's handle all of the deprecation cases early
+ # HADOOP_NAMENODE_OPTS -> HDFS_namenode_OPTS
+ # YARN_RESOURCEMANAGER_OPTS -> YARN_resourcemanager_OPTS
+
+ uvar="${uprogram}_${ucommand}_OPTS"
+ if [[ -n ${!uvar} ]]; then
+ hadoop_deprecate_envvar "${uvar}" "${var}"
+ fi
+
+ uvar="HADOOP_${ucommand}_OPTS"
+ if [[ -n ${!uvar} ]]; then
+ hadoop_deprecate_envvar "${uvar}" "${var}"
+ fi
+
+ if [[ -n ${!var} ]]; then
+ hadoop_debug "Appending ${!var} onto HADOOP_OPTS"
+ HADOOP_OPTS="${HADOOP_OPTS} ${!var}"
+ return 0
+ fi
+}
+
## @description Perform the 'hadoop classpath', etc subcommand with the given
## @description parameters
## @audience private
http://git-wip-us.apache.org/repos/asf/hadoop/blob/5b7b9fc6/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_subcommand_opts.bats
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_subcommand_opts.bats b/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_subcommand_opts.bats
new file mode 100644
index 0000000..1fbf343
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_subcommand_opts.bats
@@ -0,0 +1,76 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+load hadoop-functions_test_helper
+
+@test "hadoop_subcommand_opts (missing param)" {
+ HADOOP_OPTS="x"
+ run hadoop_subcommand_opts testvar
+ [ "${status}" = "1" ]
+}
+
+@test "hadoop_subcommand_opts (simple not exist)" {
+ HADOOP_OPTS="x"
+ hadoop_subcommand_opts hadoop subcommand
+ [ "${HADOOP_OPTS}" = "x" ]
+}
+
+@test "hadoop_subcommand_opts (hadoop simple exist)" {
+ HADOOP_OPTS="x"
+ HADOOP_test_OPTS="y"
+ hadoop_subcommand_opts hadoop test
+ echo "${HADOOP_OPTS}"
+ [ "${HADOOP_OPTS}" = "x y" ]
+}
+
+@test "hadoop_subcommand_opts (hadoop complex exist)" {
+ HADOOP_OPTS="x"
+ HADOOP_test_OPTS="y z"
+ hadoop_subcommand_opts hadoop test
+ echo "${HADOOP_OPTS}"
+ [ "${HADOOP_OPTS}" = "x y z" ]
+}
+
+@test "hadoop_subcommand_opts (hdfs simple exist)" {
+ HADOOP_OPTS="x"
+ HDFS_test_OPTS="y"
+ hadoop_subcommand_opts hdfs test
+ echo "${HADOOP_OPTS}"
+ [ "${HADOOP_OPTS}" = "x y" ]
+}
+
+@test "hadoop_subcommand_opts (yarn simple exist)" {
+ HADOOP_OPTS="x"
+ YARN_test_OPTS="y"
+ hadoop_subcommand_opts yarn test
+ echo "${HADOOP_OPTS}"
+ [ "${HADOOP_OPTS}" = "x y" ]
+}
+
+@test "hadoop_subcommand_opts (deprecation case #1)" {
+ HADOOP_OPTS="x"
+ HADOOP_NAMENODE_OPTS="y"
+ hadoop_subcommand_opts hdfs namenode
+ echo "${HADOOP_OPTS}"
+ [ "${HADOOP_OPTS}" = "x y" ]
+}
+
+@test "hadoop_subcommand_opts (deprecation case #2)" {
+ HADOOP_OPTS="x"
+ YARN_RESOURCEMANAGER_OPTS="y"
+ hadoop_subcommand_opts yarn resourcemanager
+ echo "${HADOOP_OPTS}"
+ [ "${HADOOP_OPTS}" = "x y" ]
+}
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[19/21] hadoop git commit: HADOOP-13563. hadoop_subcommand_opts
should print name not actual content during debug
Posted by aw...@apache.org.
HADOOP-13563. hadoop_subcommand_opts should print name not actual content during debug
Signed-off-by: Allen Wittenauer <aw...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d2942006
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d2942006
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d2942006
Branch: refs/heads/HADOOP-13341
Commit: d294200635162368e37906594fa4446d9fafd1af
Parents: c7a5b7d
Author: Allen Wittenauer <aw...@apache.org>
Authored: Tue Aug 30 15:23:29 2016 -0700
Committer: Allen Wittenauer <aw...@apache.org>
Committed: Fri Sep 9 04:07:43 2016 -0700
----------------------------------------------------------------------
.../hadoop-common/src/main/bin/hadoop-functions.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d2942006/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
index 91546d4..b6e2b59 100755
--- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
+++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
@@ -2057,7 +2057,7 @@ function hadoop_subcommand_opts
fi
if [[ -n ${!uvar} ]]; then
- hadoop_debug "Appending ${!uvar} onto HADOOP_OPTS"
+ hadoop_debug "Appending ${uvar} onto HADOOP_OPTS"
HADOOP_OPTS="${HADOOP_OPTS} ${!uvar}"
return 0
fi
@@ -2102,7 +2102,7 @@ function hadoop_subcommand_secure_opts
uvar="${uprogram}_${ucommand}_SECURE_EXTRA_OPTS"
if [[ -n ${!uvar} ]]; then
- hadoop_debug "Appending ${!uvar} onto HADOOP_OPTS"
+ hadoop_debug "Appending ${uvar} onto HADOOP_OPTS"
HADOOP_OPTS="${HADOOP_OPTS} ${!uvar}"
return 0
fi
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[16/21] hadoop git commit: HADOOP-13361. Modify hadoop_verify_user to
be consistent with hadoop_subcommand_opts (ie more granularity)
Posted by aw...@apache.org.
HADOOP-13361. Modify hadoop_verify_user to be consistent with hadoop_subcommand_opts (ie more granularity)
Signed-off-by: Allen Wittenauer <aw...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ab578fb2
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ab578fb2
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ab578fb2
Branch: refs/heads/HADOOP-13341
Commit: ab578fb2dead6422b368123dca4ebb8b31cfa9f7
Parents: 9bae198
Author: Allen Wittenauer <aw...@apache.org>
Authored: Tue Aug 30 13:55:04 2016 -0700
Committer: Allen Wittenauer <aw...@apache.org>
Committed: Fri Sep 9 04:07:43 2016 -0700
----------------------------------------------------------------------
.../hadoop-common/src/main/bin/hadoop | 4 +-
.../src/main/bin/hadoop-functions.sh | 24 +++++++--
.../hadoop-common/src/main/conf/hadoop-env.sh | 3 +-
.../src/test/scripts/hadoop_verify_user.bats | 53 ++++++++++++++++++++
.../hadoop-hdfs/src/main/bin/hdfs | 4 +-
hadoop-mapreduce-project/bin/mapred | 4 +-
hadoop-yarn-project/hadoop-yarn/bin/yarn | 4 +-
7 files changed, 82 insertions(+), 14 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ab578fb2/hadoop-common-project/hadoop-common/src/main/bin/hadoop
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop b/hadoop-common-project/hadoop-common/src/main/bin/hadoop
index 9b682e6..450543d 100755
--- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop
+++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop
@@ -190,6 +190,8 @@ fi
HADOOP_SUBCMD=$1
shift
+hadoop_verify_user "${HADOOP_SHELL_EXECNAME}" "${HADOOP_SUBCMD}"
+
HADOOP_SUBCMD_ARGS=("$@")
if declare -f hadoop_subcommand_"${HADOOP_SUBCMD}" >/dev/null 2>&1; then
@@ -199,8 +201,6 @@ else
hadoopcmd_case "${HADOOP_SUBCMD}" "${HADOOP_SUBCMD_ARGS[@]}"
fi
-hadoop_verify_user "${HADOOP_SUBCMD}"
-
hadoop_add_client_opts
if [[ ${HADOOP_WORKER_MODE} = true ]]; then
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ab578fb2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
index db868bd..91546d4 100755
--- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
+++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
@@ -1984,12 +1984,26 @@ function hadoop_secure_daemon_handler
## @return will exit on failure conditions
function hadoop_verify_user
{
- local command=$1
- local uservar="HADOOP_${command}_USER"
+ declare program=$1
+ declare command=$2
+ declare uprogram
+ declare ucommand
+ declare uvar
- if [[ -n ${!uservar} ]]; then
- if [[ ${!uservar} != "${USER}" ]]; then
- hadoop_error "ERROR: ${command} can only be executed by ${!uservar}."
+ if [[ -z "${BASH_VERSINFO[0]}" ]] \
+ || [[ "${BASH_VERSINFO[0]}" -lt 4 ]]; then
+ uprogram=$(echo "${program}" | tr '[:lower:]' '[:upper:]')
+ ucommand=$(echo "${command}" | tr '[:lower:]' '[:upper:]')
+ else
+ uprogram=${program^^}
+ ucommand=${command^^}
+ fi
+
+ uvar="${uprogram}_${ucommand}_USER"
+
+ if [[ -n ${!uvar} ]]; then
+ if [[ ${!uvar} != "${USER}" ]]; then
+ hadoop_error "ERROR: ${command} can only be executed by ${!uvar}."
exit 1
fi
fi
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ab578fb2/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh b/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
index 4656f4d..a78f3f6 100644
--- a/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
+++ b/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
@@ -417,6 +417,7 @@ esac
#
# To prevent accidents, shell commands be (superficially) locked
# to only allow certain users to execute certain subcommands.
+# It uses the format of (command)_(subcommand)_USER.
#
# For example, to limit who can execute the namenode command,
-# export HADOOP_namenode_USER=hdfs
+# export HDFS_NAMENODE_USER=hdfs
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ab578fb2/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_verify_user.bats
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_verify_user.bats b/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_verify_user.bats
new file mode 100644
index 0000000..ac9fa9f
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_verify_user.bats
@@ -0,0 +1,53 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+load hadoop-functions_test_helper
+
+@test "hadoop_verify_user (hadoop: no setting)" {
+ run hadoop_verify_user hadoop test
+ [ "${status}" = "0" ]
+}
+
+@test "hadoop_verify_user (yarn: no setting)" {
+ run hadoop_verify_user yarn test
+ [ "${status}" = "0" ]
+}
+
+@test "hadoop_verify_user (hadoop: allow)" {
+ HADOOP_TEST_USER=${USER}
+ run hadoop_verify_user hadoop test
+ [ "${status}" = "0" ]
+}
+
+@test "hadoop_verify_user (yarn: allow)" {
+ YARN_TEST_USER=${USER}
+ run hadoop_verify_user yarn test
+ [ "${status}" = "0" ]
+}
+
+# colon isn't a valid username, so let's use it
+# this should fail regardless of who the user is
+# that is running the test code
+@test "hadoop_verify_user (hadoop: disallow)" {
+ HADOOP_TEST_USER=:
+ run hadoop_verify_user hadoop test
+ [ "${status}" = "1" ]
+}
+
+@test "hadoop_verify_user (yarn: disallow)" {
+ YARN_TEST_USER=:
+ run hadoop_verify_user yarn test
+ [ "${status}" = "1" ]
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ab578fb2/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs b/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
index 566bd7f..6d6088f 100755
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
@@ -246,6 +246,8 @@ fi
HADOOP_SUBCMD=$1
shift
+hadoop_verify_user "${HADOOP_SHELL_EXECNAME}" "${HADOOP_SUBCMD}"
+
HADOOP_SUBCMD_ARGS=("$@")
if declare -f hdfs_subcommand_"${HADOOP_SUBCMD}" >/dev/null 2>&1; then
@@ -255,8 +257,6 @@ else
hdfscmd_case "${HADOOP_SUBCMD}" "${HADOOP_SUBCMD_ARGS[@]}"
fi
-hadoop_verify_user "${HADOOP_SUBCMD}"
-
hadoop_add_client_opts
if [[ ${HADOOP_WORKER_MODE} = true ]]; then
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ab578fb2/hadoop-mapreduce-project/bin/mapred
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/bin/mapred b/hadoop-mapreduce-project/bin/mapred
index 3243d20..c566d5a 100755
--- a/hadoop-mapreduce-project/bin/mapred
+++ b/hadoop-mapreduce-project/bin/mapred
@@ -131,6 +131,8 @@ fi
HADOOP_SUBCMD=$1
shift
+hadoop_verify_user "${HADOOP_SHELL_EXECNAME}" "${HADOOP_SUBCMD}"
+
HADOOP_SUBCMD_ARGS=("$@")
if declare -f mapred_subcommand_"${HADOOP_SUBCMD}" >/dev/null 2>&1; then
@@ -140,8 +142,6 @@ else
mapredcmd_case "${HADOOP_SUBCMD}" "${HADOOP_SUBCMD_ARGS[@]}"
fi
-hadoop_verify_user "${HADOOP_SUBCMD}"
-
hadoop_add_client_opts
if [[ ${HADOOP_SLAVE_MODE} = true ]]; then
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ab578fb2/hadoop-yarn-project/hadoop-yarn/bin/yarn
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/bin/yarn b/hadoop-yarn-project/hadoop-yarn/bin/yarn
index d2a8a50..804fd1a 100755
--- a/hadoop-yarn-project/hadoop-yarn/bin/yarn
+++ b/hadoop-yarn-project/hadoop-yarn/bin/yarn
@@ -228,6 +228,8 @@ fi
HADOOP_SUBCMD=$1
shift
+hadoop_verify_user "${HADOOP_SHELL_EXECNAME}" "${HADOOP_SUBCMD}"
+
HADOOP_SUBCMD_ARGS=("$@")
if declare -f yarn_subcommand_"${HADOOP_SUBCMD}" >/dev/null 2>&1; then
@@ -237,8 +239,6 @@ else
yarncmd_case "${HADOOP_SUBCMD}" "${HADOOP_SUBCMD_ARGS[@]}"
fi
-hadoop_verify_user "${HADOOP_SUBCMD}"
-
# It's unclear if YARN_CLIENT_OPTS is actually a useful
# thing to have separate from HADOOP_CLIENT_OPTS. Someone
# might use it, so let's not deprecate it and just override
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[17/21] hadoop git commit: HADOOP-13357. Modify common to use
hadoop_subcommand_opts
Posted by aw...@apache.org.
HADOOP-13357. Modify common to use hadoop_subcommand_opts
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/dd8e8f31
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/dd8e8f31
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/dd8e8f31
Branch: refs/heads/HADOOP-13341
Commit: dd8e8f3191f8eee3b777392ac3ded9a0a6f34a66
Parents: 359bb90
Author: Allen Wittenauer <aw...@apache.org>
Authored: Tue Aug 30 12:42:32 2016 -0700
Committer: Allen Wittenauer <aw...@apache.org>
Committed: Fri Sep 9 04:07:43 2016 -0700
----------------------------------------------------------------------
hadoop-common-project/hadoop-common/src/main/bin/hadoop | 5 +++++
1 file changed, 5 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/dd8e8f31/hadoop-common-project/hadoop-common/src/main/bin/hadoop
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop b/hadoop-common-project/hadoop-common/src/main/bin/hadoop
index 4aae621..9b682e6 100755
--- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop
+++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop
@@ -208,8 +208,13 @@ if [[ ${HADOOP_WORKER_MODE} = true ]]; then
exit $?
fi
+hadoop_subcommand_opts "${HADOOP_SHELL_EXECNAME}" "${HADOOP_SUBCMD}"
+
if [[ "${HADOOP_SUBCMD_SECURESERVICE}" = true ]]; then
HADOOP_SECURE_USER="${HADOOP_SUBCMD_SECUREUSER}"
+
+ hadoop_subcommand_secure_opts "${HADOOP_SHELL_EXECNAME}" "${HADOOP_SUBCMD}"
+
hadoop_verify_secure_prereq
hadoop_setup_secure_service
priv_outfile="${HADOOP_LOG_DIR}/privileged-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.out"
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[21/21] hadoop git commit: HADOOP-13554. Add an equivalent of
hadoop_subcmd_opts for secure opts (aw)
Posted by aw...@apache.org.
HADOOP-13554. Add an equivalent of hadoop_subcmd_opts for secure opts (aw)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ff16b572
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ff16b572
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ff16b572
Branch: refs/heads/HADOOP-13341
Commit: ff16b5721d4057e7239013936c0cbeecb2c63877
Parents: 548158f
Author: Allen Wittenauer <aw...@apache.org>
Authored: Tue Aug 30 08:16:23 2016 -0700
Committer: Allen Wittenauer <aw...@apache.org>
Committed: Fri Sep 9 04:07:43 2016 -0700
----------------------------------------------------------------------
.../src/main/bin/hadoop-functions.sh | 45 +++++++++++++++++
.../scripts/hadoop_subcommand_secure_opts.bats | 52 ++++++++++++++++++++
2 files changed, 97 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ff16b572/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
index 9003913..dd82347 100755
--- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
+++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
@@ -2057,6 +2057,51 @@ function hadoop_subcommand_opts
fi
}
+## @description Add custom (program)_(command)_SECURE_EXTRA_OPTS to HADOOP_OPTS.
+## @description This *does not* handle the pre-3.x deprecated cases
+## @audience public
+## @stability stable
+## @replaceable yes
+## @param program
+## @param subcommand
+## @return will exit on failure conditions
+function hadoop_subcommand_secure_opts
+{
+ declare program=$1
+ declare command=$2
+ declare uvar
+ declare uprogram
+ declare ucommand
+
+ if [[ -z "${program}" || -z "${command}" ]]; then
+ return 1
+ fi
+
+ # bash 4 and up have built-in ways to upper and lower
+ # case the contents of vars. This is faster than
+ # calling tr.
+
+ if [[ -z "${BASH_VERSINFO[0]}" ]] \
+ || [[ "${BASH_VERSINFO[0]}" -lt 4 ]]; then
+ uprogram=$(echo "${program}" | tr '[:lower:]' '[:upper:]')
+ ucommand=$(echo "${command}" | tr '[:lower:]' '[:upper:]')
+ else
+ uprogram=${program^^}
+ ucommand=${command^^}
+ fi
+
+ # HDFS_DATANODE_SECURE_EXTRA_OPTS
+ # HDFS_NFS3_SECURE_EXTRA_OPTS
+ # ...
+ uvar="${uprogram}_${ucommand}_SECURE_EXTRA_OPTS"
+
+ if [[ -n ${!uvar} ]]; then
+ hadoop_debug "Appending ${!uvar} onto HADOOP_OPTS"
+ HADOOP_OPTS="${HADOOP_OPTS} ${!uvar}"
+ return 0
+ fi
+}
+
## @description Perform the 'hadoop classpath', etc subcommand with the given
## @description parameters
## @audience private
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ff16b572/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_subcommand_secure_opts.bats
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_subcommand_secure_opts.bats b/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_subcommand_secure_opts.bats
new file mode 100644
index 0000000..1b3506c
--- /dev/null
+++ b/hadoop-common-project/hadoop-common/src/test/scripts/hadoop_subcommand_secure_opts.bats
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+load hadoop-functions_test_helper
+
+@test "hadoop_subcommand_secure_opts (missing param)" {
+ HADOOP_OPTS="x"
+ run hadoop_subcommand_secure_opts testvar
+ [ "${status}" = "1" ]
+}
+
+@test "hadoop_subcommand_secure_opts (simple not exist)" {
+ HADOOP_OPTS="x"
+ hadoop_subcommand_secure_opts hadoop subcommand
+ [ "${HADOOP_OPTS}" = "x" ]
+}
+
+@test "hadoop_subcommand_secure_opts (hadoop simple exist)" {
+ HADOOP_OPTS="x"
+ HADOOP_TEST_SECURE_EXTRA_OPTS="y"
+ hadoop_subcommand_secure_opts hadoop test
+ echo "${HADOOP_OPTS}"
+ [ "${HADOOP_OPTS}" = "x y" ]
+}
+
+@test "hadoop_subcommand_secure_opts (hadoop complex exist)" {
+ HADOOP_OPTS="x"
+ HADOOP_TEST_SECURE_EXTRA_OPTS="y z"
+ hadoop_subcommand_secure_opts hadoop test
+ echo "${HADOOP_OPTS}"
+ [ "${HADOOP_OPTS}" = "x y z" ]
+}
+
+@test "hadoop_subcommand_secure_opts (hdfs simple exist)" {
+ HADOOP_OPTS="x"
+ HDFS_TEST_SECURE_EXTRA_OPTS="y"
+ hadoop_subcommand_secure_opts hdfs test
+ echo "${HADOOP_OPTS}"
+ [ "${HADOOP_OPTS}" = "x y" ]
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[04/21] hadoop git commit: Revert "YARN-5567. Fix script exit code
checking in NodeHealthScriptRunner#reportHealthStatus. (Yufei Gu via
rchiang)"
Posted by aw...@apache.org.
Revert "YARN-5567. Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus. (Yufei Gu via rchiang)"
This reverts commit 05ede003868871addc30162e9707c3dc14ed6b7a.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/cae33118
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/cae33118
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/cae33118
Branch: refs/heads/HADOOP-13341
Commit: cae331186da266eea1b0a6fc2c82604907ab0153
Parents: 401db4f
Author: Allen Wittenauer <aw...@apache.org>
Authored: Thu Sep 8 12:40:30 2016 -0700
Committer: Allen Wittenauer <aw...@apache.org>
Committed: Thu Sep 8 12:40:30 2016 -0700
----------------------------------------------------------------------
.../java/org/apache/hadoop/util/NodeHealthScriptRunner.java | 3 +--
.../org/apache/hadoop/util/TestNodeHealthScriptRunner.java | 7 -------
2 files changed, 1 insertion(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/cae33118/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
index c3bef37..fc392c4 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
@@ -106,7 +106,6 @@ public class NodeHealthScriptRunner extends AbstractService {
shexec.execute();
} catch (ExitCodeException e) {
// ignore the exit code of the script
- exceptionStackTrace = StringUtils.stringifyException(e);
status = HealthCheckerExitStatus.FAILED_WITH_EXIT_CODE;
// On Windows, we will not hit the Stream closed IOException
// thrown by stdout buffered reader for timeout event.
@@ -163,7 +162,7 @@ public class NodeHealthScriptRunner extends AbstractService {
setHealthStatus(false, exceptionStackTrace);
break;
case FAILED_WITH_EXIT_CODE:
- setHealthStatus(false, exceptionStackTrace);
+ setHealthStatus(true, "", now);
break;
case FAILED:
setHealthStatus(false, shexec.getOutput());
http://git-wip-us.apache.org/repos/asf/hadoop/blob/cae33118/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
index db61f5a..8fc64d1 100644
--- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
+++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
@@ -91,7 +91,6 @@ public class TestNodeHealthScriptRunner {
public void testNodeHealthScript() throws Exception {
String errorScript = "echo ERROR\n echo \"Tracker not healthy\"";
String normalScript = "echo \"I am all fine\"";
- String failWithExitCodeScript = "echo \"Not healthy\"; exit -1";
String timeOutScript =
Shell.WINDOWS ? "@echo off\nping -n 4 127.0.0.1 >nul\necho \"I am fine\""
: "sleep 4\necho \"I am fine\"";
@@ -125,12 +124,6 @@ public class TestNodeHealthScriptRunner {
nodeHealthScriptRunner.isHealthy());
Assert.assertEquals("", nodeHealthScriptRunner.getHealthReport());
- // Script which fails with exit code.
- writeNodeHealthScriptFile(failWithExitCodeScript, true);
- timerTask.run();
- Assert.assertFalse("Node health status reported healthy",
- nodeHealthScriptRunner.isHealthy());
-
// Timeout script.
writeNodeHealthScriptFile(timeOutScript, true);
timerTask.run();
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[03/21] hadoop git commit: HDFS-8901. Use ByteBuffer in striping
positional read. Contributed by Sammi Chen and Kai Zheng.
Posted by aw...@apache.org.
HDFS-8901. Use ByteBuffer in striping positional read. Contributed by Sammi Chen and Kai Zheng.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/401db4fc
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/401db4fc
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/401db4fc
Branch: refs/heads/HADOOP-13341
Commit: 401db4fc65140979fe7665983e36905e886df971
Parents: 20a20c2
Author: Zhe Zhang <zh...@apache.org>
Authored: Thu Sep 8 11:54:33 2016 -0700
Committer: Zhe Zhang <zh...@apache.org>
Committed: Thu Sep 8 11:54:33 2016 -0700
----------------------------------------------------------------------
.../org/apache/hadoop/util/DataChecksum.java | 2 +-
.../org/apache/hadoop/hdfs/DFSInputStream.java | 68 ++++---
.../hadoop/hdfs/DFSStripedInputStream.java | 24 ++-
.../hadoop/hdfs/util/StripedBlockUtil.java | 177 ++++++++++---------
.../hadoop/hdfs/TestDFSStripedInputStream.java | 121 ++++++++++++-
.../hadoop/hdfs/util/TestStripedBlockUtil.java | 22 +--
6 files changed, 281 insertions(+), 133 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/401db4fc/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java
index 3a53ed9..6982a92 100644
--- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java
+++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java
@@ -304,7 +304,7 @@ public class DataChecksum implements Checksum {
bytesPerChecksum, checksums.array(), crcsOffset, fileName, basePos);
return;
}
- if (NativeCrc32.isAvailable()) {
+ if (NativeCrc32.isAvailable() && data.isDirect()) {
NativeCrc32.verifyChunkedSums(bytesPerChecksum, type.id, checksums, data,
fileName, basePos);
} else {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/401db4fc/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
index 7a10ba4..31fa897 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
@@ -533,7 +533,8 @@ public class DFSInputStream extends FSInputStream
* Open a DataInputStream to a DataNode so that it can be read from.
* We get block ID and the IDs of the destinations at startup, from the namenode.
*/
- private synchronized DatanodeInfo blockSeekTo(long target) throws IOException {
+ private synchronized DatanodeInfo blockSeekTo(long target)
+ throws IOException {
if (target >= getFileLength()) {
throw new IOException("Attempted to read past end of file");
}
@@ -962,14 +963,14 @@ public class DFSInputStream extends FSInputStream
}
protected void fetchBlockByteRange(LocatedBlock block, long start, long end,
- byte[] buf, int offset, CorruptedBlocks corruptedBlocks)
+ ByteBuffer buf, CorruptedBlocks corruptedBlocks)
throws IOException {
block = refreshLocatedBlock(block);
while (true) {
DNAddrPair addressPair = chooseDataNode(block, null);
try {
actualGetFromOneDataNode(addressPair, block, start, end,
- buf, offset, corruptedBlocks);
+ buf, corruptedBlocks);
return;
} catch (IOException e) {
checkInterrupted(e); // check if the read has been interrupted
@@ -988,12 +989,10 @@ public class DFSInputStream extends FSInputStream
return new Callable<ByteBuffer>() {
@Override
public ByteBuffer call() throws Exception {
- byte[] buf = bb.array();
- int offset = bb.position();
try (TraceScope ignored = dfsClient.getTracer().
newScope("hedgedRead" + hedgedReadId, parentSpanId)) {
- actualGetFromOneDataNode(datanode, block, start, end, buf,
- offset, corruptedBlocks);
+ actualGetFromOneDataNode(datanode, block, start, end, bb,
+ corruptedBlocks);
return bb;
}
}
@@ -1007,13 +1006,12 @@ public class DFSInputStream extends FSInputStream
* @param block the located block containing the requested data
* @param startInBlk the startInBlk offset of the block
* @param endInBlk the endInBlk offset of the block
- * @param buf the given byte array into which the data is read
- * @param offset the offset in buf
+ * @param buf the given byte buffer into which the data is read
* @param corruptedBlocks map recording list of datanodes with corrupted
* block replica
*/
void actualGetFromOneDataNode(final DNAddrPair datanode, LocatedBlock block,
- final long startInBlk, final long endInBlk, byte[] buf, int offset,
+ final long startInBlk, final long endInBlk, ByteBuffer buf,
CorruptedBlocks corruptedBlocks)
throws IOException {
DFSClientFaultInjector.get().startFetchFromDatanode();
@@ -1031,7 +1029,22 @@ public class DFSInputStream extends FSInputStream
DFSClientFaultInjector.get().fetchFromDatanodeException();
reader = getBlockReader(block, startInBlk, len, datanode.addr,
datanode.storageType, datanode.info);
- int nread = reader.readAll(buf, offset, len);
+
+ //Behave exactly as the readAll() call
+ ByteBuffer tmp = buf.duplicate();
+ tmp.limit(tmp.position() + len);
+ tmp = tmp.slice();
+ int nread = 0;
+ int ret;
+ while (true) {
+ ret = reader.read(tmp);
+ if (ret <= 0) {
+ break;
+ }
+ nread += ret;
+ }
+ buf.position(buf.position() + nread);
+
IOUtilsClient.updateReadStatistics(readStatistics, nread, reader);
dfsClient.updateFileSystemReadStats(
reader.getNetworkDistance(), nread);
@@ -1098,7 +1111,7 @@ public class DFSInputStream extends FSInputStream
* time. We then wait on which ever read returns first.
*/
private void hedgedFetchBlockByteRange(LocatedBlock block, long start,
- long end, byte[] buf, int offset, CorruptedBlocks corruptedBlocks)
+ long end, ByteBuffer buf, CorruptedBlocks corruptedBlocks)
throws IOException {
final DfsClientConf conf = dfsClient.getConf();
ArrayList<Future<ByteBuffer>> futures = new ArrayList<>();
@@ -1130,8 +1143,8 @@ public class DFSInputStream extends FSInputStream
conf.getHedgedReadThresholdMillis(), TimeUnit.MILLISECONDS);
if (future != null) {
ByteBuffer result = future.get();
- System.arraycopy(result.array(), result.position(), buf, offset,
- len);
+ result.flip();
+ buf.put(result);
return;
}
DFSClient.LOG.debug("Waited {}ms to read from {}; spawning hedged "
@@ -1173,8 +1186,8 @@ public class DFSInputStream extends FSInputStream
// cancel the rest.
cancelAll(futures);
dfsClient.getHedgedReadMetrics().incHedgedReadWins();
- System.arraycopy(result.array(), result.position(), buf, offset,
- len);
+ result.flip();
+ buf.put(result);
return;
} catch (InterruptedException ie) {
// Ignore and retry
@@ -1244,7 +1257,8 @@ public class DFSInputStream extends FSInputStream
* access key from its memory since it's considered expired based on
* the estimated expiration date.
*/
- if (ex instanceof InvalidBlockTokenException || ex instanceof InvalidToken) {
+ if (ex instanceof InvalidBlockTokenException ||
+ ex instanceof InvalidToken) {
DFSClient.LOG.info("Access token was invalid when connecting to "
+ targetAddr + " : " + ex);
return true;
@@ -1272,7 +1286,8 @@ public class DFSInputStream extends FSInputStream
try (TraceScope scope = dfsClient.
newReaderTraceScope("DFSInputStream#byteArrayPread",
src, position, length)) {
- int retLen = pread(position, buffer, offset, length);
+ ByteBuffer bb = ByteBuffer.wrap(buffer, offset, length);
+ int retLen = pread(position, bb);
if (retLen < length) {
dfsClient.addRetLenToReaderScope(scope, retLen);
}
@@ -1280,7 +1295,7 @@ public class DFSInputStream extends FSInputStream
}
}
- private int pread(long position, byte[] buffer, int offset, int length)
+ private int pread(long position, ByteBuffer buffer)
throws IOException {
// sanity checks
dfsClient.checkOpen();
@@ -1292,6 +1307,7 @@ public class DFSInputStream extends FSInputStream
if ((position < 0) || (position >= filelen)) {
return -1;
}
+ int length = buffer.remaining();
int realLen = length;
if ((position + length) > filelen) {
realLen = (int)(filelen - position);
@@ -1304,14 +1320,16 @@ public class DFSInputStream extends FSInputStream
CorruptedBlocks corruptedBlocks = new CorruptedBlocks();
for (LocatedBlock blk : blockRange) {
long targetStart = position - blk.getStartOffset();
- long bytesToRead = Math.min(remaining, blk.getBlockSize() - targetStart);
+ int bytesToRead = (int) Math.min(remaining,
+ blk.getBlockSize() - targetStart);
+ long targetEnd = targetStart + bytesToRead - 1;
try {
if (dfsClient.isHedgedReadsEnabled() && !blk.isStriped()) {
hedgedFetchBlockByteRange(blk, targetStart,
- targetStart + bytesToRead - 1, buffer, offset, corruptedBlocks);
+ targetEnd, buffer, corruptedBlocks);
} else {
- fetchBlockByteRange(blk, targetStart, targetStart + bytesToRead - 1,
- buffer, offset, corruptedBlocks);
+ fetchBlockByteRange(blk, targetStart, targetEnd,
+ buffer, corruptedBlocks);
}
} finally {
// Check and report if any block replicas are corrupted.
@@ -1323,7 +1341,6 @@ public class DFSInputStream extends FSInputStream
remaining -= bytesToRead;
position += bytesToRead;
- offset += bytesToRead;
}
assert remaining == 0 : "Wrong number of bytes read.";
return realLen;
@@ -1457,7 +1474,8 @@ public class DFSInputStream extends FSInputStream
* If another node could not be found, then returns false.
*/
@Override
- public synchronized boolean seekToNewSource(long targetPos) throws IOException {
+ public synchronized boolean seekToNewSource(long targetPos)
+ throws IOException {
if (currentNode == null) {
return seekToBlockSource(targetPos);
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/401db4fc/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
index 9ca8005..ccaf6a7 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
@@ -307,8 +307,8 @@ public class DFSStripedInputStream extends DFSInputStream {
stripeLimit - stripeBufOffset);
LocatedStripedBlock blockGroup = (LocatedStripedBlock) currentLocatedBlock;
- AlignedStripe[] stripes = StripedBlockUtil.divideOneStripe(ecPolicy, cellSize,
- blockGroup, offsetInBlockGroup,
+ AlignedStripe[] stripes = StripedBlockUtil.divideOneStripe(ecPolicy,
+ cellSize, blockGroup, offsetInBlockGroup,
offsetInBlockGroup + stripeRange.length - 1, curStripeBuf);
final LocatedBlock[] blks = StripedBlockUtil.parseStripedBlockGroup(
blockGroup, cellSize, dataBlkNum, parityBlkNum);
@@ -523,13 +523,13 @@ public class DFSStripedInputStream extends DFSInputStream {
*/
@Override
protected void fetchBlockByteRange(LocatedBlock block, long start,
- long end, byte[] buf, int offset, CorruptedBlocks corruptedBlocks)
+ long end, ByteBuffer buf, CorruptedBlocks corruptedBlocks)
throws IOException {
// Refresh the striped block group
LocatedStripedBlock blockGroup = getBlockGroupAt(block.getStartOffset());
AlignedStripe[] stripes = StripedBlockUtil.divideByteRangeIntoStripes(
- ecPolicy, cellSize, blockGroup, start, end, buf, offset);
+ ecPolicy, cellSize, blockGroup, start, end, buf);
CompletionService<Void> readService = new ExecutorCompletionService<>(
dfsClient.getStripedReadsThreadPool());
final LocatedBlock[] blks = StripedBlockUtil.parseStripedBlockGroup(
@@ -542,6 +542,7 @@ public class DFSStripedInputStream extends DFSInputStream {
blks, preaderInfos, corruptedBlocks);
preader.readStripe();
}
+ buf.position(buf.position() + (int)(end - start + 1));
} finally {
for (BlockReaderInfo preaderInfo : preaderInfos) {
closeReader(preaderInfo);
@@ -698,16 +699,15 @@ public class DFSStripedInputStream extends DFSInputStream {
}
private ByteBufferStrategy[] getReadStrategies(StripingChunk chunk) {
- if (chunk.byteBuffer != null) {
- ByteBufferStrategy strategy =
- new ByteBufferStrategy(chunk.byteBuffer, readStatistics, dfsClient);
+ if (chunk.useByteBuffer()) {
+ ByteBufferStrategy strategy = new ByteBufferStrategy(
+ chunk.getByteBuffer(), readStatistics, dfsClient);
return new ByteBufferStrategy[]{strategy};
} else {
ByteBufferStrategy[] strategies =
- new ByteBufferStrategy[chunk.byteArray.getOffsets().length];
+ new ByteBufferStrategy[chunk.getChunkBuffer().getSlices().size()];
for (int i = 0; i < strategies.length; i++) {
- ByteBuffer buffer = ByteBuffer.wrap(chunk.byteArray.buf(),
- chunk.byteArray.getOffsets()[i], chunk.byteArray.getLengths()[i]);
+ ByteBuffer buffer = chunk.getChunkBuffer().getSlice(i);
strategies[i] =
new ByteBufferStrategy(buffer, readStatistics, dfsClient);
}
@@ -814,7 +814,7 @@ public class DFSStripedInputStream extends DFSInputStream {
}
class PositionStripeReader extends StripeReader {
- private byte[][] decodeInputs = null;
+ private ByteBuffer[] decodeInputs = null;
PositionStripeReader(CompletionService<Void> service,
AlignedStripe alignedStripe, LocatedBlock[] targetBlocks,
@@ -836,8 +836,6 @@ public class DFSStripedInputStream extends DFSInputStream {
Preconditions.checkState(index >= dataBlkNum &&
alignedStripe.chunks[index] == null);
alignedStripe.chunks[index] = new StripingChunk(decodeInputs[index]);
- alignedStripe.chunks[index].addByteArraySlice(0,
- (int) alignedStripe.getSpanInBlock());
return true;
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/401db4fc/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java
index c8827d9..4dbbc3d 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java
@@ -73,7 +73,8 @@ import java.util.concurrent.TimeUnit;
@InterfaceAudience.Private
public class StripedBlockUtil {
- public static final Logger LOG = LoggerFactory.getLogger(StripedBlockUtil.class);
+ public static final Logger LOG =
+ LoggerFactory.getLogger(StripedBlockUtil.class);
/**
* Parses a striped block group into individual blocks.
@@ -312,16 +313,17 @@ public class StripedBlockUtil {
* schedule a new fetch request with the decoding input buffer as transfer
* destination.
*/
- public static byte[][] initDecodeInputs(AlignedStripe alignedStripe,
+ public static ByteBuffer[] initDecodeInputs(AlignedStripe alignedStripe,
int dataBlkNum, int parityBlkNum) {
- byte[][] decodeInputs =
- new byte[dataBlkNum + parityBlkNum][(int) alignedStripe.getSpanInBlock()];
+ ByteBuffer[] decodeInputs = new ByteBuffer[dataBlkNum + parityBlkNum];
+ for (int i = 0; i < decodeInputs.length; i++) {
+ decodeInputs[i] = ByteBuffer.allocate(
+ (int) alignedStripe.getSpanInBlock());
+ }
// read the full data aligned stripe
for (int i = 0; i < dataBlkNum; i++) {
if (alignedStripe.chunks[i] == null) {
alignedStripe.chunks[i] = new StripingChunk(decodeInputs[i]);
- alignedStripe.chunks[i].addByteArraySlice(0,
- (int) alignedStripe.getSpanInBlock());
}
}
return decodeInputs;
@@ -334,14 +336,21 @@ public class StripedBlockUtil {
* When all pending requests have returned, this method should be called to
* finalize decode input buffers.
*/
- public static void finalizeDecodeInputs(final byte[][] decodeInputs,
+ public static void finalizeDecodeInputs(final ByteBuffer[] decodeInputs,
AlignedStripe alignedStripe) {
for (int i = 0; i < alignedStripe.chunks.length; i++) {
final StripingChunk chunk = alignedStripe.chunks[i];
if (chunk != null && chunk.state == StripingChunk.FETCHED) {
- chunk.copyTo(decodeInputs[i]);
+ if (chunk.useChunkBuffer()) {
+ chunk.getChunkBuffer().copyTo(decodeInputs[i]);
+ } else {
+ chunk.getByteBuffer().flip();
+ }
} else if (chunk != null && chunk.state == StripingChunk.ALLZERO) {
- Arrays.fill(decodeInputs[i], (byte) 0);
+ //ZERO it. Will be better handled in other following issue.
+ byte[] emptyBytes = new byte[decodeInputs[i].limit()];
+ decodeInputs[i].put(emptyBytes);
+ decodeInputs[i].flip();
} else {
decodeInputs[i] = null;
}
@@ -351,7 +360,7 @@ public class StripedBlockUtil {
/**
* Decode based on the given input buffers and erasure coding policy.
*/
- public static void decodeAndFillBuffer(final byte[][] decodeInputs,
+ public static void decodeAndFillBuffer(final ByteBuffer[] decodeInputs,
AlignedStripe alignedStripe, int dataBlkNum, int parityBlkNum,
RawErasureDecoder decoder) {
// Step 1: prepare indices and output buffers for missing data units
@@ -364,8 +373,11 @@ public class StripedBlockUtil {
}
}
decodeIndices = Arrays.copyOf(decodeIndices, pos);
- byte[][] decodeOutputs =
- new byte[decodeIndices.length][(int) alignedStripe.getSpanInBlock()];
+ ByteBuffer[] decodeOutputs = new ByteBuffer[decodeIndices.length];
+ for (int i = 0; i < decodeOutputs.length; i++) {
+ decodeOutputs[i] = ByteBuffer.allocate(
+ (int) alignedStripe.getSpanInBlock());
+ }
// Step 2: decode into prepared output buffers
decoder.decode(decodeInputs, decodeIndices, decodeOutputs);
@@ -374,8 +386,8 @@ public class StripedBlockUtil {
for (int i = 0; i < decodeIndices.length; i++) {
int missingBlkIdx = decodeIndices[i];
StripingChunk chunk = alignedStripe.chunks[missingBlkIdx];
- if (chunk.state == StripingChunk.MISSING) {
- chunk.copyFrom(decodeOutputs[i]);
+ if (chunk.state == StripingChunk.MISSING && chunk.useChunkBuffer()) {
+ chunk.getChunkBuffer().copyFrom(decodeOutputs[i]);
}
}
}
@@ -402,7 +414,8 @@ public class StripedBlockUtil {
// Step 4: calculate each chunk's position in destination buffer. Since the
// whole read range is within a single stripe, the logic is simpler here.
- int bufOffset = (int) (rangeStartInBlockGroup % ((long) cellSize * dataBlkNum));
+ int bufOffset =
+ (int) (rangeStartInBlockGroup % ((long) cellSize * dataBlkNum));
for (StripingCell cell : cells) {
long cellStart = cell.idxInInternalBlk * cellSize + cell.offset;
long cellEnd = cellStart + cell.size - 1;
@@ -437,15 +450,14 @@ public class StripedBlockUtil {
* @param rangeStartInBlockGroup The byte range's start offset in block group
* @param rangeEndInBlockGroup The byte range's end offset in block group
* @param buf Destination buffer of the read operation for the byte range
- * @param offsetInBuf Start offset into the destination buffer
*
* At most 5 stripes will be generated from each logical range, as
* demonstrated in the header of {@link AlignedStripe}.
*/
- public static AlignedStripe[] divideByteRangeIntoStripes(ErasureCodingPolicy ecPolicy,
+ public static AlignedStripe[] divideByteRangeIntoStripes(
+ ErasureCodingPolicy ecPolicy,
int cellSize, LocatedStripedBlock blockGroup,
- long rangeStartInBlockGroup, long rangeEndInBlockGroup, byte[] buf,
- int offsetInBuf) {
+ long rangeStartInBlockGroup, long rangeEndInBlockGroup, ByteBuffer buf) {
// Step 0: analyze range and calculate basic parameters
final int dataBlkNum = ecPolicy.getNumDataUnits();
@@ -462,7 +474,7 @@ public class StripedBlockUtil {
AlignedStripe[] stripes = mergeRangesForInternalBlocks(ecPolicy, ranges);
// Step 4: calculate each chunk's position in destination buffer
- calcualteChunkPositionsInBuf(cellSize, stripes, cells, buf, offsetInBuf);
+ calcualteChunkPositionsInBuf(cellSize, stripes, cells, buf);
// Step 5: prepare ALLZERO blocks
prepareAllZeroChunks(blockGroup, stripes, cellSize, dataBlkNum);
@@ -476,7 +488,8 @@ public class StripedBlockUtil {
* used by {@link DFSStripedOutputStream} in encoding
*/
@VisibleForTesting
- private static StripingCell[] getStripingCellsOfByteRange(ErasureCodingPolicy ecPolicy,
+ private static StripingCell[] getStripingCellsOfByteRange(
+ ErasureCodingPolicy ecPolicy,
int cellSize, LocatedStripedBlock blockGroup,
long rangeStartInBlockGroup, long rangeEndInBlockGroup) {
Preconditions.checkArgument(
@@ -511,7 +524,8 @@ public class StripedBlockUtil {
* the physical byte range (inclusive) on each stored internal block.
*/
@VisibleForTesting
- private static VerticalRange[] getRangesForInternalBlocks(ErasureCodingPolicy ecPolicy,
+ private static VerticalRange[] getRangesForInternalBlocks(
+ ErasureCodingPolicy ecPolicy,
int cellSize, StripingCell[] cells) {
int dataBlkNum = ecPolicy.getNumDataUnits();
int parityBlkNum = ecPolicy.getNumParityUnits();
@@ -575,8 +589,7 @@ public class StripedBlockUtil {
}
private static void calcualteChunkPositionsInBuf(int cellSize,
- AlignedStripe[] stripes, StripingCell[] cells, byte[] buf,
- int offsetInBuf) {
+ AlignedStripe[] stripes, StripingCell[] cells, ByteBuffer buf) {
/**
* | <--------------- AlignedStripe --------------->|
*
@@ -598,6 +611,7 @@ public class StripedBlockUtil {
for (StripingCell cell : cells) {
long cellStart = cell.idxInInternalBlk * cellSize + cell.offset;
long cellEnd = cellStart + cell.size - 1;
+ StripingChunk chunk;
for (AlignedStripe s : stripes) {
long stripeEnd = s.getOffsetInBlock() + s.getSpanInBlock() - 1;
long overlapStart = Math.max(cellStart, s.getOffsetInBlock());
@@ -606,11 +620,13 @@ public class StripedBlockUtil {
if (overLapLen <= 0) {
continue;
}
- if (s.chunks[cell.idxInStripe] == null) {
- s.chunks[cell.idxInStripe] = new StripingChunk(buf);
+ chunk = s.chunks[cell.idxInStripe];
+ if (chunk == null) {
+ chunk = new StripingChunk();
+ s.chunks[cell.idxInStripe] = chunk;
}
- s.chunks[cell.idxInStripe].addByteArraySlice(
- (int)(offsetInBuf + done + overlapStart - cellStart), overLapLen);
+ chunk.getChunkBuffer().addSlice(buf,
+ (int) (done + overlapStart - cellStart), overLapLen);
}
done += cell.size;
}
@@ -833,88 +849,89 @@ public class StripedBlockUtil {
*/
public int state = REQUESTED;
- public final ChunkByteArray byteArray;
- public final ByteBuffer byteBuffer;
+ private final ChunkByteBuffer chunkBuffer;
+ private final ByteBuffer byteBuffer;
- public StripingChunk(byte[] buf) {
- this.byteArray = new ChunkByteArray(buf);
+ public StripingChunk() {
+ this.chunkBuffer = new ChunkByteBuffer();
byteBuffer = null;
}
public StripingChunk(ByteBuffer buf) {
- this.byteArray = null;
+ this.chunkBuffer = null;
this.byteBuffer = buf;
}
public StripingChunk(int state) {
- this.byteArray = null;
+ this.chunkBuffer = null;
this.byteBuffer = null;
this.state = state;
}
- public void addByteArraySlice(int offset, int length) {
- assert byteArray != null;
- byteArray.offsetsInBuf.add(offset);
- byteArray.lengthsInBuf.add(length);
+ public boolean useByteBuffer(){
+ return byteBuffer != null;
}
- void copyTo(byte[] target) {
- assert byteArray != null;
- byteArray.copyTo(target);
+ public boolean useChunkBuffer() {
+ return chunkBuffer != null;
}
- void copyFrom(byte[] src) {
- assert byteArray != null;
- byteArray.copyFrom(src);
+ public ByteBuffer getByteBuffer() {
+ assert byteBuffer != null;
+ return byteBuffer;
+ }
+
+ public ChunkByteBuffer getChunkBuffer() {
+ assert chunkBuffer != null;
+ return chunkBuffer;
}
}
- public static class ChunkByteArray {
- private final byte[] buf;
- private final List<Integer> offsetsInBuf;
- private final List<Integer> lengthsInBuf;
+ /**
+ * A utility to manage ByteBuffer slices for a reader.
+ */
+ public static class ChunkByteBuffer {
+ private final List<ByteBuffer> slices;
- ChunkByteArray(byte[] buf) {
- this.buf = buf;
- this.offsetsInBuf = new ArrayList<>();
- this.lengthsInBuf = new ArrayList<>();
+ ChunkByteBuffer() {
+ this.slices = new ArrayList<>();
}
- public int[] getOffsets() {
- int[] offsets = new int[offsetsInBuf.size()];
- for (int i = 0; i < offsets.length; i++) {
- offsets[i] = offsetsInBuf.get(i);
- }
- return offsets;
+ public void addSlice(ByteBuffer buffer, int offset, int len) {
+ ByteBuffer tmp = buffer.duplicate();
+ tmp.position(buffer.position() + offset);
+ tmp.limit(buffer.position() + offset + len);
+ slices.add(tmp.slice());
}
- public int[] getLengths() {
- int[] lens = new int[this.lengthsInBuf.size()];
- for (int i = 0; i < lens.length; i++) {
- lens[i] = this.lengthsInBuf.get(i);
- }
- return lens;
+ public ByteBuffer getSlice(int i) {
+ return slices.get(i);
}
- public byte[] buf() {
- return buf;
+ public List<ByteBuffer> getSlices() {
+ return slices;
}
- void copyTo(byte[] target) {
- int posInBuf = 0;
- for (int i = 0; i < offsetsInBuf.size(); i++) {
- System.arraycopy(buf, offsetsInBuf.get(i),
- target, posInBuf, lengthsInBuf.get(i));
- posInBuf += lengthsInBuf.get(i);
+ /**
+ * Note: target will be ready-to-read state after the call.
+ */
+ void copyTo(ByteBuffer target) {
+ for (ByteBuffer slice : slices) {
+ slice.flip();
+ target.put(slice);
}
- }
-
- void copyFrom(byte[] src) {
- int srcPos = 0;
- for (int j = 0; j < offsetsInBuf.size(); j++) {
- System.arraycopy(src, srcPos, buf, offsetsInBuf.get(j),
- lengthsInBuf.get(j));
- srcPos += lengthsInBuf.get(j);
+ target.flip();
+ }
+
+ void copyFrom(ByteBuffer src) {
+ ByteBuffer tmp;
+ int len;
+ for (ByteBuffer slice : slices) {
+ len = slice.remaining();
+ tmp = src.duplicate();
+ tmp.limit(tmp.position() + len);
+ slice.put(tmp);
+ src.position(src.position() + len);
}
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/401db4fc/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java
index 18c2de9..1e27745 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java
@@ -57,7 +57,8 @@ import static org.junit.Assert.assertTrue;
public class TestDFSStripedInputStream {
- public static final Log LOG = LogFactory.getLog(TestDFSStripedInputStream.class);
+ public static final Log LOG =
+ LogFactory.getLog(TestDFSStripedInputStream.class);
private MiniDFSCluster cluster;
private Configuration conf = new Configuration();
@@ -272,12 +273,16 @@ public class TestDFSStripedInputStream {
// |10 |
done += in.read(0, readBuffer, 0, delta);
assertEquals(delta, done);
+ assertArrayEquals(Arrays.copyOf(expected, done),
+ Arrays.copyOf(readBuffer, done));
// both head and trail cells are partial
// |c_0 |c_1 |c_2 |c_3 |c_4 |c_5 |
// |256K - 10|missing|256K|256K|256K - 10|not in range|
done += in.read(delta, readBuffer, delta,
CELLSIZE * (DATA_BLK_NUM - 1) - 2 * delta);
assertEquals(CELLSIZE * (DATA_BLK_NUM - 1) - delta, done);
+ assertArrayEquals(Arrays.copyOf(expected, done),
+ Arrays.copyOf(readBuffer, done));
// read the rest
done += in.read(done, readBuffer, done, readSize - done);
assertEquals(readSize, done);
@@ -291,8 +296,8 @@ public class TestDFSStripedInputStream {
testStatefulRead(true, true);
}
- private void testStatefulRead(boolean useByteBuffer, boolean cellMisalignPacket)
- throws Exception {
+ private void testStatefulRead(boolean useByteBuffer,
+ boolean cellMisalignPacket) throws Exception {
final int numBlocks = 2;
final int fileSize = numBlocks * BLOCK_GROUP_SIZE;
if (cellMisalignPacket) {
@@ -302,7 +307,8 @@ public class TestDFSStripedInputStream {
}
DFSTestUtil.createStripedFile(cluster, filePath, null, numBlocks,
NUM_STRIPE_PER_BLOCK, false);
- LocatedBlocks lbs = fs.getClient().namenode.getBlockLocations(filePath.toString(), 0, fileSize);
+ LocatedBlocks lbs = fs.getClient().namenode.
+ getBlockLocations(filePath.toString(), 0, fileSize);
assert lbs.getLocatedBlocks().size() == numBlocks;
for (LocatedBlock lb : lbs.getLocatedBlocks()) {
@@ -360,4 +366,111 @@ public class TestDFSStripedInputStream {
}
fs.delete(filePath, true);
}
+
+ @Test
+ public void testStatefulReadWithDNFailure() throws Exception {
+ final int numBlocks = 4;
+ final int failedDNIdx = DATA_BLK_NUM - 1;
+ DFSTestUtil.createStripedFile(cluster, filePath, null, numBlocks,
+ NUM_STRIPE_PER_BLOCK, false);
+ LocatedBlocks lbs = fs.getClient().namenode.getBlockLocations(
+ filePath.toString(), 0, BLOCK_GROUP_SIZE);
+
+ assert lbs.get(0) instanceof LocatedStripedBlock;
+ LocatedStripedBlock bg = (LocatedStripedBlock) (lbs.get(0));
+ for (int i = 0; i < DATA_BLK_NUM + PARITY_BLK_NUM; i++) {
+ Block blk = new Block(bg.getBlock().getBlockId() + i,
+ NUM_STRIPE_PER_BLOCK * CELLSIZE,
+ bg.getBlock().getGenerationStamp());
+ blk.setGenerationStamp(bg.getBlock().getGenerationStamp());
+ cluster.injectBlocks(i, Arrays.asList(blk),
+ bg.getBlock().getBlockPoolId());
+ }
+ DFSStripedInputStream in =
+ new DFSStripedInputStream(fs.getClient(), filePath.toString(), false,
+ ecPolicy, null);
+ int readSize = BLOCK_GROUP_SIZE;
+ byte[] readBuffer = new byte[readSize];
+ byte[] expected = new byte[readSize];
+ /** A variation of {@link DFSTestUtil#fillExpectedBuf} for striped blocks */
+ for (int i = 0; i < NUM_STRIPE_PER_BLOCK; i++) {
+ for (int j = 0; j < DATA_BLK_NUM; j++) {
+ for (int k = 0; k < CELLSIZE; k++) {
+ int posInBlk = i * CELLSIZE + k;
+ int posInFile = i * CELLSIZE * DATA_BLK_NUM + j * CELLSIZE + k;
+ expected[posInFile] = SimulatedFSDataset.simulatedByte(
+ new Block(bg.getBlock().getBlockId() + j), posInBlk);
+ }
+ }
+ }
+
+ ErasureCoderOptions coderOptions = new ErasureCoderOptions(
+ DATA_BLK_NUM, PARITY_BLK_NUM);
+ RawErasureDecoder rawDecoder = CodecUtil.createRawDecoder(conf,
+ ecPolicy.getCodecName(), coderOptions);
+
+ // Update the expected content for decoded data
+ int[] missingBlkIdx = new int[PARITY_BLK_NUM];
+ for (int i = 0; i < missingBlkIdx.length; i++) {
+ if (i == 0) {
+ missingBlkIdx[i] = failedDNIdx;
+ } else {
+ missingBlkIdx[i] = DATA_BLK_NUM + i;
+ }
+ }
+ cluster.stopDataNode(failedDNIdx);
+ for (int i = 0; i < NUM_STRIPE_PER_BLOCK; i++) {
+ byte[][] decodeInputs = new byte[DATA_BLK_NUM + PARITY_BLK_NUM][CELLSIZE];
+ byte[][] decodeOutputs = new byte[missingBlkIdx.length][CELLSIZE];
+ for (int j = 0; j < DATA_BLK_NUM; j++) {
+ int posInBuf = i * CELLSIZE * DATA_BLK_NUM + j * CELLSIZE;
+ if (j != failedDNIdx) {
+ System.arraycopy(expected, posInBuf, decodeInputs[j], 0, CELLSIZE);
+ }
+ }
+ for (int j = DATA_BLK_NUM; j < DATA_BLK_NUM + PARITY_BLK_NUM; j++) {
+ for (int k = 0; k < CELLSIZE; k++) {
+ int posInBlk = i * CELLSIZE + k;
+ decodeInputs[j][k] = SimulatedFSDataset.simulatedByte(
+ new Block(bg.getBlock().getBlockId() + j), posInBlk);
+ }
+ }
+ for (int m : missingBlkIdx) {
+ decodeInputs[m] = null;
+ }
+ rawDecoder.decode(decodeInputs, missingBlkIdx, decodeOutputs);
+ int posInBuf = i * CELLSIZE * DATA_BLK_NUM + failedDNIdx * CELLSIZE;
+ System.arraycopy(decodeOutputs[0], 0, expected, posInBuf, CELLSIZE);
+ }
+
+ int delta = 10;
+ int done = 0;
+ // read a small delta, shouldn't trigger decode
+ // |cell_0 |
+ // |10 |
+ done += in.read(readBuffer, 0, delta);
+ assertEquals(delta, done);
+ // both head and trail cells are partial
+ // |c_0 |c_1 |c_2 |c_3 |c_4 |c_5 |
+ // |256K - 10|missing|256K|256K|256K - 10|not in range|
+ while (done < (CELLSIZE * (DATA_BLK_NUM - 1) - 2 * delta)) {
+ int ret = in.read(readBuffer, delta,
+ CELLSIZE * (DATA_BLK_NUM - 1) - 2 * delta);
+ assertTrue(ret > 0);
+ done += ret;
+ }
+ assertEquals(CELLSIZE * (DATA_BLK_NUM - 1) - delta, done);
+ // read the rest
+
+ int restSize;
+ restSize = readSize - done;
+ while (done < restSize) {
+ int ret = in.read(readBuffer, done, restSize);
+ assertTrue(ret > 0);
+ done += ret;
+ }
+
+ assertEquals(readSize, done);
+ assertArrayEquals(expected, readBuffer);
+ }
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/401db4fc/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestStripedBlockUtil.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestStripedBlockUtil.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestStripedBlockUtil.java
index 96fc79c..7d9d7dc 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestStripedBlockUtil.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestStripedBlockUtil.java
@@ -36,6 +36,7 @@ import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.Timeout;
+import java.nio.ByteBuffer;
import java.util.Random;
import static org.junit.Assert.assertEquals;
@@ -242,7 +243,8 @@ public class TestStripedBlockUtil {
*/
@Test
public void testDivideByteRangeIntoStripes() {
- byte[] assembled = new byte[BLK_GROUP_STRIPE_NUM * FULL_STRIPE_SIZE];
+ ByteBuffer assembled =
+ ByteBuffer.allocate(BLK_GROUP_STRIPE_NUM * FULL_STRIPE_SIZE);
for (int bgSize : blockGroupSizes) {
LocatedStripedBlock blockGroup = createDummyLocatedBlock(bgSize);
byte[][] internalBlkBufs = createInternalBlkBuffers(bgSize);
@@ -252,7 +254,7 @@ public class TestStripedBlockUtil {
continue;
}
AlignedStripe[] stripes = divideByteRangeIntoStripes(EC_POLICY,
- CELLSIZE, blockGroup, brStart, brStart + brSize - 1, assembled, 0);
+ CELLSIZE, blockGroup, brStart, brStart + brSize - 1, assembled);
for (AlignedStripe stripe : stripes) {
for (int i = 0; i < DATA_BLK_NUM; i++) {
@@ -261,21 +263,21 @@ public class TestStripedBlockUtil {
continue;
}
int done = 0;
- for (int j = 0; j < chunk.byteArray.getLengths().length; j++) {
- System.arraycopy(internalBlkBufs[i],
- (int) stripe.getOffsetInBlock() + done, assembled,
- chunk.byteArray.getOffsets()[j],
- chunk.byteArray.getLengths()[j]);
- done += chunk.byteArray.getLengths()[j];
+ int len;
+ for (ByteBuffer slice : chunk.getChunkBuffer().getSlices()) {
+ len = slice.remaining();
+ slice.put(internalBlkBufs[i],
+ (int) stripe.getOffsetInBlock() + done, len);
+ done += len;
}
}
}
for (int i = 0; i < brSize; i++) {
- if (hashIntToByte(brStart + i) != assembled[i]) {
+ if (hashIntToByte(brStart + i) != assembled.get(i)) {
System.out.println("Oops");
}
assertEquals("Byte at " + (brStart + i) + " should be the same",
- hashIntToByte(brStart + i), assembled[i]);
+ hashIntToByte(brStart + i), assembled.get(i));
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[02/21] hadoop git commit: HDFS-10847. Complete the document for
FileDistribution processor in OfflineImageViewer. Contributed by Yiqun Lin.
Posted by aw...@apache.org.
HDFS-10847. Complete the document for FileDistribution processor in OfflineImageViewer. Contributed by Yiqun Lin.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/20a20c2f
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/20a20c2f
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/20a20c2f
Branch: refs/heads/HADOOP-13341
Commit: 20a20c2f6e1b3b8aa6a58a824ad3aadc349dc761
Parents: 2d1bf53
Author: Akira Ajisaka <aa...@apache.org>
Authored: Fri Sep 9 01:34:34 2016 +0900
Committer: Akira Ajisaka <aa...@apache.org>
Committed: Fri Sep 9 01:34:34 2016 +0900
----------------------------------------------------------------------
.../hadoop-hdfs/src/site/markdown/HDFSCommands.md | 3 +++
.../hadoop-hdfs/src/site/markdown/HdfsImageViewer.md | 12 +++++++++---
2 files changed, 12 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/20a20c2f/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
index 4075878..83035bb 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
@@ -261,6 +261,9 @@ Usage: `hdfs oiv_legacy [OPTIONS] -i INPUT_FILE -o OUTPUT_FILE`
| COMMAND\_OPTION | Description |
|:---- |:---- |
| `-p`\|`--processor` *processor* | Specify the image processor to apply against the image file. Valid options are Ls (default), XML, Delimited, Indented, and FileDistribution. |
+| `-maxSize` *size* | Specify the range [0, maxSize] of file sizes to be analyzed in bytes (128GB by default). This option is used with FileDistribution processor. |
+| `-step` *size* | Specify the granularity of the distribution in bytes (2MB by default). This option is used with FileDistribution processor. |
+| `-format` | Format the output result in a human-readable fashion rather than a number of bytes. (false by default). This option is used with FileDistribution processor. |
| `-skipBlocks` | Do not enumerate individual blocks within files. This may save processing time and outfile file space on namespaces with very large files. The Ls processor reads the blocks to correctly determine file sizes and ignores this option. |
| `-printToScreen` | Pipe output of processor to console as well as specified file. On extremely large namespaces, this may increase processing time by an order of magnitude. |
| `-delimiter` *arg* | When used in conjunction with the Delimited processor, replaces the default tab delimiter with the string specified by *arg*. |
http://git-wip-us.apache.org/repos/asf/hadoop/blob/20a20c2f/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsImageViewer.md
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsImageViewer.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsImageViewer.md
index f991b44..f55c9fd 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsImageViewer.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsImageViewer.md
@@ -50,10 +50,13 @@ The Offline Image Viewer provides several output processors:
..., s[n-1], maxSize], and the processor calculates how many files
in the system fall into each segment [s[i-1], s[i]). Note that
files larger than maxSize always fall into the very last segment.
- The output file is formatted as a tab separated two column table:
- Size and NumFiles. Where Size represents the start of the segment,
+ By default, the output file is formatted as a tab separated two column
+ table: Size and NumFiles. Where Size represents the start of the segment,
and numFiles is the number of files form the image which size falls
- in this segment.
+ in this segment. By specifying the option -format, the output file will be
+ formatted in a human-readable fashion rather than a number of bytes that
+ showed in Size column. In addition, the Size column will be changed to the
+ Size Range column.
4. Delimited (experimental): Generate a text file with all of the elements
common to both inodes and inodes-under-construction, separated by a
@@ -182,6 +185,9 @@ Due to the internal layout changes introduced by the ProtocolBuffer-based fsimag
| `-i`\|`--inputFile` *input file* | Specify the input fsimage file to process. Required. |
| `-o`\|`--outputFile` *output file* | Specify the output filename, if the specified output processor generates one. If the specified file already exists, it is silently overwritten. Required. |
| `-p`\|`--processor` *processor* | Specify the image processor to apply against the image file. Valid options are Ls (default), XML, Delimited, Indented, and FileDistribution. |
+| `-maxSize` *size* | Specify the range [0, maxSize] of file sizes to be analyzed in bytes (128GB by default). This option is used with FileDistribution processor. |
+| `-step` *size* | Specify the granularity of the distribution in bytes (2MB by default). This option is used with FileDistribution processor. |
+| `-format` | Format the output result in a human-readable fashion rather than a number of bytes. (false by default). This option is used with FileDistribution processor. |
| `-skipBlocks` | Do not enumerate individual blocks within files. This may save processing time and outfile file space on namespaces with very large files. The Ls processor reads the blocks to correctly determine file sizes and ignores this option. |
| `-printToScreen` | Pipe output of processor to console as well as specified file. On extremely large namespaces, this may increase processing time by an order of magnitude. |
| `-delimiter` *arg* | When used in conjunction with the Delimited processor, replaces the default tab delimiter with the string specified by *arg*. |
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[09/21] hadoop git commit: HDFS-10553. DiskBalancer: Rename
Tools/DiskBalancer class to Tools/DiskBalancerCLI. Contributed by Manoj
Govindassamy.
Posted by aw...@apache.org.
HDFS-10553. DiskBalancer: Rename Tools/DiskBalancer class to Tools/DiskBalancerCLI. Contributed by Manoj Govindassamy.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/35c5943b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/35c5943b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/35c5943b
Branch: refs/heads/HADOOP-13341
Commit: 35c5943b8ba394191405555cdfc5e6127053ee97
Parents: b07c266
Author: Anu Engineer <ae...@apache.org>
Authored: Thu Sep 8 19:26:56 2016 -0700
Committer: Anu Engineer <ae...@apache.org>
Committed: Thu Sep 8 19:26:56 2016 -0700
----------------------------------------------------------------------
.../hadoop-hdfs/src/main/bin/hdfs | 2 +-
.../diskbalancer/command/CancelCommand.java | 23 +-
.../server/diskbalancer/command/Command.java | 6 +-
.../diskbalancer/command/ExecuteCommand.java | 15 +-
.../diskbalancer/command/HelpCommand.java | 22 +-
.../diskbalancer/command/PlanCommand.java | 63 +--
.../diskbalancer/command/QueryCommand.java | 19 +-
.../diskbalancer/command/ReportCommand.java | 18 +-
.../apache/hadoop/hdfs/tools/DiskBalancer.java | 482 -------------------
.../hadoop/hdfs/tools/DiskBalancerCLI.java | 482 +++++++++++++++++++
.../command/TestDiskBalancerCommand.java | 16 +-
11 files changed, 576 insertions(+), 572 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/35c5943b/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs b/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
index 5059528..7a90f08 100755
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
@@ -127,7 +127,7 @@ function hdfscmd_case
HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
;;
diskbalancer)
- HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.DiskBalancer
+ HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.DiskBalancerCLI
hadoop_debug "Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS"
HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}"
;;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/35c5943b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/CancelCommand.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/CancelCommand.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/CancelCommand.java
index 8b83e27..007272e 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/CancelCommand.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/CancelCommand.java
@@ -29,7 +29,7 @@ import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol;
import org.apache.hadoop.hdfs.server.diskbalancer.DiskBalancerException;
import org.apache.hadoop.hdfs.server.diskbalancer.planner.NodePlan;
-import org.apache.hadoop.hdfs.tools.DiskBalancer;
+import org.apache.hadoop.hdfs.tools.DiskBalancerCLI;
import java.io.IOException;
@@ -44,9 +44,10 @@ public class CancelCommand extends Command {
*/
public CancelCommand(Configuration conf) {
super(conf);
- addValidCommandParameters(DiskBalancer.CANCEL, "Cancels a running plan.");
- addValidCommandParameters(DiskBalancer.NODE, "Node to run the command " +
- "against in node:port format.");
+ addValidCommandParameters(DiskBalancerCLI.CANCEL,
+ "Cancels a running plan.");
+ addValidCommandParameters(DiskBalancerCLI.NODE,
+ "Node to run the command against in node:port format.");
}
/**
@@ -57,20 +58,20 @@ public class CancelCommand extends Command {
@Override
public void execute(CommandLine cmd) throws Exception {
LOG.info("Executing \"Cancel plan\" command.");
- Preconditions.checkState(cmd.hasOption(DiskBalancer.CANCEL));
- verifyCommandOptions(DiskBalancer.CANCEL, cmd);
+ Preconditions.checkState(cmd.hasOption(DiskBalancerCLI.CANCEL));
+ verifyCommandOptions(DiskBalancerCLI.CANCEL, cmd);
// We can cancel a plan using datanode address and plan ID
// that you can read from a datanode using queryStatus
- if(cmd.hasOption(DiskBalancer.NODE)) {
- String nodeAddress = cmd.getOptionValue(DiskBalancer.NODE);
- String planHash = cmd.getOptionValue(DiskBalancer.CANCEL);
+ if(cmd.hasOption(DiskBalancerCLI.NODE)) {
+ String nodeAddress = cmd.getOptionValue(DiskBalancerCLI.NODE);
+ String planHash = cmd.getOptionValue(DiskBalancerCLI.CANCEL);
cancelPlanUsingHash(nodeAddress, planHash);
} else {
// Or you can cancel a plan using the plan file. If the user
// points us to the plan file, we can compute the hash as well as read
// the address of the datanode from the plan file.
- String planFile = cmd.getOptionValue(DiskBalancer.CANCEL);
+ String planFile = cmd.getOptionValue(DiskBalancerCLI.CANCEL);
Preconditions.checkArgument(planFile != null && !planFile.isEmpty(),
"Invalid plan file specified.");
String planData = null;
@@ -142,6 +143,6 @@ public class CancelCommand extends Command {
HelpFormatter helpFormatter = new HelpFormatter();
helpFormatter.printHelp("hdfs diskbalancer -cancel <planFile> | -cancel " +
"<planID> -node <hostname>",
- header, DiskBalancer.getCancelOptions(), footer);
+ header, DiskBalancerCLI.getCancelOptions(), footer);
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/35c5943b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/Command.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/Command.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/Command.java
index 5acd0ac..7641b44 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/Command.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/Command.java
@@ -43,7 +43,7 @@ import org.apache.hadoop.hdfs.server.diskbalancer.datamodel.DiskBalancerCluster;
import org.apache.hadoop.hdfs.server.diskbalancer.datamodel.DiskBalancerDataNode;
import org.apache.hadoop.hdfs.server.diskbalancer.datamodel.DiskBalancerVolume;
import org.apache.hadoop.hdfs.server.diskbalancer.datamodel.DiskBalancerVolumeSet;
-import org.apache.hadoop.hdfs.tools.DiskBalancer;
+import org.apache.hadoop.hdfs.tools.DiskBalancerCLI;
import org.apache.hadoop.net.NetUtils;
import org.apache.hadoop.security.UserGroupInformation;
import org.codehaus.jackson.map.ObjectMapper;
@@ -418,7 +418,7 @@ public abstract class Command extends Configured {
* @return default top number of nodes.
*/
protected int getDefaultTop() {
- return DiskBalancer.DEFAULT_TOP;
+ return DiskBalancerCLI.DEFAULT_TOP;
}
/**
@@ -437,7 +437,7 @@ public abstract class Command extends Configured {
protected int parseTopNodes(final CommandLine cmd, final StrBuilder result) {
String outputLine = "";
int nodes = 0;
- final String topVal = cmd.getOptionValue(DiskBalancer.TOP);
+ final String topVal = cmd.getOptionValue(DiskBalancerCLI.TOP);
if (StringUtils.isBlank(topVal)) {
outputLine = String.format(
"No top limit specified, using default top value %d.",
http://git-wip-us.apache.org/repos/asf/hadoop/blob/35c5943b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/ExecuteCommand.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/ExecuteCommand.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/ExecuteCommand.java
index f363c34..3a348c9 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/ExecuteCommand.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/ExecuteCommand.java
@@ -29,7 +29,7 @@ import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol;
import org.apache.hadoop.hdfs.server.diskbalancer.DiskBalancerException;
import org.apache.hadoop.hdfs.server.diskbalancer.planner.NodePlan;
-import org.apache.hadoop.hdfs.tools.DiskBalancer;
+import org.apache.hadoop.hdfs.tools.DiskBalancerCLI;
import java.io.IOException;
@@ -46,7 +46,8 @@ public class ExecuteCommand extends Command {
*/
public ExecuteCommand(Configuration conf) {
super(conf);
- addValidCommandParameters(DiskBalancer.EXECUTE, "Executes a given plan.");
+ addValidCommandParameters(DiskBalancerCLI.EXECUTE,
+ "Executes a given plan.");
}
/**
@@ -57,10 +58,10 @@ public class ExecuteCommand extends Command {
@Override
public void execute(CommandLine cmd) throws Exception {
LOG.info("Executing \"execute plan\" command");
- Preconditions.checkState(cmd.hasOption(DiskBalancer.EXECUTE));
- verifyCommandOptions(DiskBalancer.EXECUTE, cmd);
+ Preconditions.checkState(cmd.hasOption(DiskBalancerCLI.EXECUTE));
+ verifyCommandOptions(DiskBalancerCLI.EXECUTE, cmd);
- String planFile = cmd.getOptionValue(DiskBalancer.EXECUTE);
+ String planFile = cmd.getOptionValue(DiskBalancerCLI.EXECUTE);
Preconditions.checkArgument(planFile != null && !planFile.isEmpty(),
"Invalid plan file specified.");
@@ -88,7 +89,7 @@ public class ExecuteCommand extends Command {
String planHash = DigestUtils.shaHex(planData);
try {
// TODO : Support skipping date check.
- dataNode.submitDiskBalancerPlan(planHash, DiskBalancer.PLAN_VERSION,
+ dataNode.submitDiskBalancerPlan(planHash, DiskBalancerCLI.PLAN_VERSION,
planFile, planData, false);
} catch (DiskBalancerException ex) {
LOG.error("Submitting plan on {} failed. Result: {}, Message: {}",
@@ -111,6 +112,6 @@ public class ExecuteCommand extends Command {
HelpFormatter helpFormatter = new HelpFormatter();
helpFormatter.printHelp("hdfs diskbalancer -execute <planfile>",
- header, DiskBalancer.getExecuteOptions(), footer);
+ header, DiskBalancerCLI.getExecuteOptions(), footer);
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/35c5943b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/HelpCommand.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/HelpCommand.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/HelpCommand.java
index 3c2fd0c..c735299 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/HelpCommand.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/HelpCommand.java
@@ -23,7 +23,7 @@ import com.google.common.base.Preconditions;
import org.apache.commons.cli.CommandLine;
import org.apache.commons.cli.HelpFormatter;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hdfs.tools.DiskBalancer;
+import org.apache.hadoop.hdfs.tools.DiskBalancerCLI;
/**
* Help Command prints out detailed help about each command.
@@ -37,7 +37,7 @@ public class HelpCommand extends Command {
*/
public HelpCommand(Configuration conf) {
super(conf);
- addValidCommandParameters(DiskBalancer.HELP, "Help Command");
+ addValidCommandParameters(DiskBalancerCLI.HELP, "Help Command");
}
/**
@@ -53,9 +53,9 @@ public class HelpCommand extends Command {
return;
}
- Preconditions.checkState(cmd.hasOption(DiskBalancer.HELP));
- verifyCommandOptions(DiskBalancer.HELP, cmd);
- String helpCommand = cmd.getOptionValue(DiskBalancer.HELP);
+ Preconditions.checkState(cmd.hasOption(DiskBalancerCLI.HELP));
+ verifyCommandOptions(DiskBalancerCLI.HELP, cmd);
+ String helpCommand = cmd.getOptionValue(DiskBalancerCLI.HELP);
if (helpCommand == null || helpCommand.isEmpty()) {
this.printHelp();
return;
@@ -65,19 +65,19 @@ public class HelpCommand extends Command {
helpCommand = helpCommand.toLowerCase();
Command command = null;
switch (helpCommand) {
- case DiskBalancer.PLAN:
+ case DiskBalancerCLI.PLAN:
command = new PlanCommand(getConf());
break;
- case DiskBalancer.EXECUTE:
+ case DiskBalancerCLI.EXECUTE:
command = new ExecuteCommand(getConf());
break;
- case DiskBalancer.QUERY:
+ case DiskBalancerCLI.QUERY:
command = new QueryCommand(getConf());
break;
- case DiskBalancer.CANCEL:
+ case DiskBalancerCLI.CANCEL:
command = new CancelCommand(getConf());
break;
- case DiskBalancer.REPORT:
+ case DiskBalancerCLI.REPORT:
command = new ReportCommand(getConf(), null);
break;
default:
@@ -102,7 +102,7 @@ public class HelpCommand extends Command {
HelpFormatter helpFormatter = new HelpFormatter();
helpFormatter.printHelp("hdfs diskbalancer [command] [options]",
- header, DiskBalancer.getHelpOptions(), "");
+ header, DiskBalancerCLI.getHelpOptions(), "");
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/35c5943b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/PlanCommand.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/PlanCommand.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/PlanCommand.java
index 72ad2c6..9749409 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/PlanCommand.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/PlanCommand.java
@@ -28,7 +28,7 @@ import org.apache.hadoop.hdfs.server.diskbalancer.datamodel
.DiskBalancerDataNode;
import org.apache.hadoop.hdfs.server.diskbalancer.planner.NodePlan;
import org.apache.hadoop.hdfs.server.diskbalancer.planner.Step;
-import org.apache.hadoop.hdfs.tools.DiskBalancer;
+import org.apache.hadoop.hdfs.tools.DiskBalancerCLI;
import java.nio.charset.StandardCharsets;
import java.util.List;
@@ -53,18 +53,18 @@ public class PlanCommand extends Command {
this.thresholdPercentage = 1;
this.bandwidth = 0;
this.maxError = 0;
- addValidCommandParameters(DiskBalancer.OUTFILE, "Output directory in " +
+ addValidCommandParameters(DiskBalancerCLI.OUTFILE, "Output directory in " +
"HDFS. The generated plan will be written to a file in this " +
"directory.");
- addValidCommandParameters(DiskBalancer.BANDWIDTH, "Maximum Bandwidth to " +
- "be used while copying.");
- addValidCommandParameters(DiskBalancer.THRESHOLD, "Percentage skew that " +
- "we tolerate before diskbalancer starts working.");
- addValidCommandParameters(DiskBalancer.MAXERROR, "Max errors to tolerate " +
- "between 2 disks");
- addValidCommandParameters(DiskBalancer.VERBOSE, "Run plan command in " +
+ addValidCommandParameters(DiskBalancerCLI.BANDWIDTH,
+ "Maximum Bandwidth to be used while copying.");
+ addValidCommandParameters(DiskBalancerCLI.THRESHOLD,
+ "Percentage skew that we tolerate before diskbalancer starts working.");
+ addValidCommandParameters(DiskBalancerCLI.MAXERROR,
+ "Max errors to tolerate between 2 disks");
+ addValidCommandParameters(DiskBalancerCLI.VERBOSE, "Run plan command in " +
"verbose mode.");
- addValidCommandParameters(DiskBalancer.PLAN, "Plan Command");
+ addValidCommandParameters(DiskBalancerCLI.PLAN, "Plan Command");
}
/**
@@ -77,36 +77,37 @@ public class PlanCommand extends Command {
@Override
public void execute(CommandLine cmd) throws Exception {
LOG.debug("Processing Plan Command.");
- Preconditions.checkState(cmd.hasOption(DiskBalancer.PLAN));
- verifyCommandOptions(DiskBalancer.PLAN, cmd);
+ Preconditions.checkState(cmd.hasOption(DiskBalancerCLI.PLAN));
+ verifyCommandOptions(DiskBalancerCLI.PLAN, cmd);
- if (cmd.getOptionValue(DiskBalancer.PLAN) == null) {
+ if (cmd.getOptionValue(DiskBalancerCLI.PLAN) == null) {
throw new IllegalArgumentException("A node name is required to create a" +
" plan.");
}
- if (cmd.hasOption(DiskBalancer.BANDWIDTH)) {
- this.bandwidth = Integer.parseInt(cmd.getOptionValue(DiskBalancer
+ if (cmd.hasOption(DiskBalancerCLI.BANDWIDTH)) {
+ this.bandwidth = Integer.parseInt(cmd.getOptionValue(DiskBalancerCLI
.BANDWIDTH));
}
- if (cmd.hasOption(DiskBalancer.MAXERROR)) {
- this.maxError = Integer.parseInt(cmd.getOptionValue(DiskBalancer
+ if (cmd.hasOption(DiskBalancerCLI.MAXERROR)) {
+ this.maxError = Integer.parseInt(cmd.getOptionValue(DiskBalancerCLI
.MAXERROR));
}
readClusterInfo(cmd);
String output = null;
- if (cmd.hasOption(DiskBalancer.OUTFILE)) {
- output = cmd.getOptionValue(DiskBalancer.OUTFILE);
+ if (cmd.hasOption(DiskBalancerCLI.OUTFILE)) {
+ output = cmd.getOptionValue(DiskBalancerCLI.OUTFILE);
}
setOutputPath(output);
// -plan nodename is the command line argument.
- DiskBalancerDataNode node = getNode(cmd.getOptionValue(DiskBalancer.PLAN));
+ DiskBalancerDataNode node =
+ getNode(cmd.getOptionValue(DiskBalancerCLI.PLAN));
if (node == null) {
throw new IllegalArgumentException("Unable to find the specified node. " +
- cmd.getOptionValue(DiskBalancer.PLAN));
+ cmd.getOptionValue(DiskBalancerCLI.PLAN));
}
this.thresholdPercentage = getThresholdPercentage(cmd);
@@ -124,8 +125,8 @@ public class PlanCommand extends Command {
try (FSDataOutputStream beforeStream = create(String.format(
- DiskBalancer.BEFORE_TEMPLATE,
- cmd.getOptionValue(DiskBalancer.PLAN)))) {
+ DiskBalancerCLI.BEFORE_TEMPLATE,
+ cmd.getOptionValue(DiskBalancerCLI.PLAN)))) {
beforeStream.write(getCluster().toJson()
.getBytes(StandardCharsets.UTF_8));
}
@@ -133,17 +134,17 @@ public class PlanCommand extends Command {
if (plan != null && plan.getVolumeSetPlans().size() > 0) {
LOG.info("Writing plan to : {}", getOutputPath());
try (FSDataOutputStream planStream = create(String.format(
- DiskBalancer.PLAN_TEMPLATE,
- cmd.getOptionValue(DiskBalancer.PLAN)))) {
+ DiskBalancerCLI.PLAN_TEMPLATE,
+ cmd.getOptionValue(DiskBalancerCLI.PLAN)))) {
planStream.write(plan.toJson().getBytes(StandardCharsets.UTF_8));
}
} else {
LOG.info("No plan generated. DiskBalancing not needed for node: {} " +
- "threshold used: {}", cmd.getOptionValue(DiskBalancer.PLAN),
+ "threshold used: {}", cmd.getOptionValue(DiskBalancerCLI.PLAN),
this.thresholdPercentage);
}
- if (cmd.hasOption(DiskBalancer.VERBOSE) && plans.size() > 0) {
+ if (cmd.hasOption(DiskBalancerCLI.VERBOSE) && plans.size() > 0) {
printToScreen(plans);
}
}
@@ -162,8 +163,8 @@ public class PlanCommand extends Command {
" will balance the data.";
HelpFormatter helpFormatter = new HelpFormatter();
- helpFormatter.printHelp("hdfs diskbalancer -plan " +
- "<hostname> [options]", header, DiskBalancer.getPlanOptions(), footer);
+ helpFormatter.printHelp("hdfs diskbalancer -plan <hostname> [options]",
+ header, DiskBalancerCLI.getPlanOptions(), footer);
}
/**
@@ -174,8 +175,8 @@ public class PlanCommand extends Command {
*/
private double getThresholdPercentage(CommandLine cmd) {
Double value = 0.0;
- if (cmd.hasOption(DiskBalancer.THRESHOLD)) {
- value = Double.parseDouble(cmd.getOptionValue(DiskBalancer.THRESHOLD));
+ if (cmd.hasOption(DiskBalancerCLI.THRESHOLD)) {
+ value = Double.parseDouble(cmd.getOptionValue(DiskBalancerCLI.THRESHOLD));
}
if ((value <= 0.0) || (value > 100.0)) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/35c5943b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/QueryCommand.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/QueryCommand.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/QueryCommand.java
index 1557a02..a8adcbd 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/QueryCommand.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/QueryCommand.java
@@ -27,7 +27,7 @@ import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol;
import org.apache.hadoop.hdfs.server.datanode.DiskBalancerWorkStatus;
import org.apache.hadoop.hdfs.server.diskbalancer.DiskBalancerException;
-import org.apache.hadoop.hdfs.tools.DiskBalancer;
+import org.apache.hadoop.hdfs.tools.DiskBalancerCLI;
import org.apache.hadoop.net.NetUtils;
/**
@@ -42,9 +42,10 @@ public class QueryCommand extends Command {
*/
public QueryCommand(Configuration conf) {
super(conf);
- addValidCommandParameters(DiskBalancer.QUERY, "Queries the status of disk" +
- " plan running on a given datanode.");
- addValidCommandParameters(DiskBalancer.VERBOSE, "Prints verbose results.");
+ addValidCommandParameters(DiskBalancerCLI.QUERY,
+ "Queries the status of disk plan running on a given datanode.");
+ addValidCommandParameters(DiskBalancerCLI.VERBOSE,
+ "Prints verbose results.");
}
/**
@@ -55,9 +56,9 @@ public class QueryCommand extends Command {
@Override
public void execute(CommandLine cmd) throws Exception {
LOG.info("Executing \"query plan\" command.");
- Preconditions.checkState(cmd.hasOption(DiskBalancer.QUERY));
- verifyCommandOptions(DiskBalancer.QUERY, cmd);
- String nodeName = cmd.getOptionValue(DiskBalancer.QUERY);
+ Preconditions.checkState(cmd.hasOption(DiskBalancerCLI.QUERY));
+ verifyCommandOptions(DiskBalancerCLI.QUERY, cmd);
+ String nodeName = cmd.getOptionValue(DiskBalancerCLI.QUERY);
Preconditions.checkNotNull(nodeName);
nodeName = nodeName.trim();
String nodeAddress = nodeName;
@@ -79,7 +80,7 @@ public class QueryCommand extends Command {
workStatus.getPlanID(),
workStatus.getResult().toString());
- if (cmd.hasOption(DiskBalancer.VERBOSE)) {
+ if (cmd.hasOption(DiskBalancerCLI.VERBOSE)) {
System.out.printf("%s", workStatus.currentStateString());
}
} catch (DiskBalancerException ex) {
@@ -101,6 +102,6 @@ public class QueryCommand extends Command {
HelpFormatter helpFormatter = new HelpFormatter();
helpFormatter.printHelp("hdfs diskbalancer -query <hostname> [options]",
- header, DiskBalancer.getQueryOptions(), footer);
+ header, DiskBalancerCLI.getQueryOptions(), footer);
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/35c5943b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/ReportCommand.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/ReportCommand.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/ReportCommand.java
index 18dd77e..69b765f 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/ReportCommand.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/ReportCommand.java
@@ -30,7 +30,7 @@ import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hdfs.server.diskbalancer.datamodel.DiskBalancerDataNode;
import org.apache.hadoop.hdfs.server.diskbalancer.datamodel.DiskBalancerVolume;
import org.apache.hadoop.hdfs.server.diskbalancer.datamodel.DiskBalancerVolumeSet;
-import org.apache.hadoop.hdfs.tools.DiskBalancer;
+import org.apache.hadoop.hdfs.tools.DiskBalancerCLI;
import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
@@ -52,15 +52,15 @@ public class ReportCommand extends Command {
super(conf);
this.out = out;
- addValidCommandParameters(DiskBalancer.REPORT,
+ addValidCommandParameters(DiskBalancerCLI.REPORT,
"Report volume information of nodes.");
String desc = String.format(
"Top number of nodes to be processed. Default: %d", getDefaultTop());
- addValidCommandParameters(DiskBalancer.TOP, desc);
+ addValidCommandParameters(DiskBalancerCLI.TOP, desc);
desc = String.format("Print out volume information for a DataNode.");
- addValidCommandParameters(DiskBalancer.NODE, desc);
+ addValidCommandParameters(DiskBalancerCLI.NODE, desc);
}
@Override
@@ -69,8 +69,8 @@ public class ReportCommand extends Command {
String outputLine = "Processing report command";
recordOutput(result, outputLine);
- Preconditions.checkState(cmd.hasOption(DiskBalancer.REPORT));
- verifyCommandOptions(DiskBalancer.REPORT, cmd);
+ Preconditions.checkState(cmd.hasOption(DiskBalancerCLI.REPORT));
+ verifyCommandOptions(DiskBalancerCLI.REPORT, cmd);
readClusterInfo(cmd);
final String nodeFormat =
@@ -81,7 +81,7 @@ public class ReportCommand extends Command {
"[%s: volume-%s] - %.2f used: %d/%d, %.2f free: %d/%d, "
+ "isFailed: %s, isReadOnly: %s, isSkip: %s, isTransient: %s.";
- if (cmd.hasOption(DiskBalancer.NODE)) {
+ if (cmd.hasOption(DiskBalancerCLI.NODE)) {
/*
* Reporting volume information for a specific DataNode
*/
@@ -136,7 +136,7 @@ public class ReportCommand extends Command {
* get value that identifies a DataNode from command line, it could be UUID,
* IP address or host name.
*/
- final String nodeVal = cmd.getOptionValue(DiskBalancer.NODE);
+ final String nodeVal = cmd.getOptionValue(DiskBalancerCLI.NODE);
if (StringUtils.isBlank(nodeVal)) {
outputLine = "The value for '-node' is neither specified or empty.";
@@ -211,6 +211,6 @@ public class ReportCommand extends Command {
HelpFormatter helpFormatter = new HelpFormatter();
helpFormatter.printHelp("hdfs diskbalancer -fs http://namenode.uri " +
"-report [options]",
- header, DiskBalancer.getReportOptions(), footer);
+ header, DiskBalancerCLI.getReportOptions(), footer);
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/35c5943b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DiskBalancer.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DiskBalancer.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DiskBalancer.java
deleted file mode 100644
index 1ed2fdc..0000000
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DiskBalancer.java
+++ /dev/null
@@ -1,482 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with this
- * work for additional information regarding copyright ownership. The ASF
- * licenses this file to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- * <p>
- * http://www.apache.org/licenses/LICENSE-2.0
- * <p>
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
- * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
- * License for the specific language governing permissions and limitations under
- * the License.
- */
-package org.apache.hadoop.hdfs.tools;
-
-import org.apache.commons.cli.BasicParser;
-import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.Option;
-import org.apache.commons.cli.OptionBuilder;
-import org.apache.commons.cli.Options;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.conf.Configured;
-import org.apache.hadoop.hdfs.HdfsConfiguration;
-import org.apache.hadoop.hdfs.server.diskbalancer.command.CancelCommand;
-import org.apache.hadoop.hdfs.server.diskbalancer.command.Command;
-import org.apache.hadoop.hdfs.server.diskbalancer.command.ExecuteCommand;
-import org.apache.hadoop.hdfs.server.diskbalancer.command.HelpCommand;
-import org.apache.hadoop.hdfs.server.diskbalancer.command.PlanCommand;
-import org.apache.hadoop.hdfs.server.diskbalancer.command.QueryCommand;
-import org.apache.hadoop.hdfs.server.diskbalancer.command.ReportCommand;
-import org.apache.hadoop.util.Tool;
-import org.apache.hadoop.util.ToolRunner;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.PrintStream;
-
-/**
- * DiskBalancer is a tool that can be used to ensure that data is spread evenly
- * across volumes of same storage type.
- * <p>
- * For example, if you have 3 disks, with 100 GB , 600 GB and 200 GB on each
- * disk, this tool will ensure that each disk will have 300 GB.
- * <p>
- * This tool can be run while data nodes are fully functional.
- * <p>
- * At very high level diskbalancer computes a set of moves that will make disk
- * utilization equal and then those moves are executed by the datanode.
- */
-public class DiskBalancer extends Configured implements Tool {
- /**
- * Computes a plan for a given set of nodes.
- */
- public static final String PLAN = "plan";
- /**
- * Output file name, for commands like report, plan etc. This is an optional
- * argument, by default diskbalancer will write all its output to
- * /system/reports/diskbalancer of the current cluster it is operating
- * against.
- */
- public static final String OUTFILE = "out";
- /**
- * Help for the program.
- */
- public static final String HELP = "help";
- /**
- * Percentage of data unevenness that we are willing to live with. For example
- * - a value like 10 indicates that we are okay with 10 % +/- from
- * idealStorage Target.
- */
- public static final String THRESHOLD = "thresholdPercentage";
- /**
- * Specifies the maximum disk bandwidth to use per second.
- */
- public static final String BANDWIDTH = "bandwidth";
- /**
- * Specifies the maximum errors to tolerate.
- */
- public static final String MAXERROR = "maxerror";
- /**
- * Executes a given plan file on the target datanode.
- */
- public static final String EXECUTE = "execute";
- /**
- * The report command prints out a disk fragmentation report about the data
- * cluster. By default it prints the DEFAULT_TOP machines names with high
- * nodeDataDensity {DiskBalancerDataNode#getNodeDataDensity} values. This
- * means that these are the nodes that deviates from the ideal data
- * distribution.
- */
- public static final String REPORT = "report";
- /**
- * specify top number of nodes to be processed.
- */
- public static final String TOP = "top";
- /**
- * specify default top number of nodes to be processed.
- */
- public static final int DEFAULT_TOP = 100;
- /**
- * Name or address of the node to execute against.
- */
- public static final String NODE = "node";
- /**
- * Runs the command in verbose mode.
- */
- public static final String VERBOSE = "v";
- public static final int PLAN_VERSION = 1;
- /**
- * Reports the status of disk balancer operation.
- */
- public static final String QUERY = "query";
- /**
- * Cancels a running plan.
- */
- public static final String CANCEL = "cancel";
- /**
- * Template for the Before File. It is node.before.json.
- */
- public static final String BEFORE_TEMPLATE = "%s.before.json";
- /**
- * Template for the plan file. it is node.plan.json.
- */
- public static final String PLAN_TEMPLATE = "%s.plan.json";
- private static final Logger LOG =
- LoggerFactory.getLogger(DiskBalancer.class);
-
- private static final Options PLAN_OPTIONS = new Options();
- private static final Options EXECUTE_OPTIONS = new Options();
- private static final Options QUERY_OPTIONS = new Options();
- private static final Options HELP_OPTIONS = new Options();
- private static final Options CANCEL_OPTIONS = new Options();
- private static final Options REPORT_OPTIONS = new Options();
-
- /**
- * Construct a DiskBalancer.
- *
- * @param conf
- */
- public DiskBalancer(Configuration conf) {
- super(conf);
- }
-
- /**
- * Main for the DiskBalancer Command handling.
- *
- * @param argv - System Args Strings[]
- * @throws Exception
- */
- public static void main(String[] argv) throws Exception {
- DiskBalancer shell = new DiskBalancer(new HdfsConfiguration());
- int res = 0;
- try {
- res = ToolRunner.run(shell, argv);
- } catch (Exception ex) {
- LOG.error(ex.toString());
- res = 1;
- }
- System.exit(res);
- }
-
- /**
- * Execute the command with the given arguments.
- *
- * @param args command specific arguments.
- * @return exit code.
- * @throws Exception
- */
- @Override
- public int run(String[] args) throws Exception {
- return run(args, System.out);
- }
-
- /**
- * Execute the command with the given arguments.
- *
- * @param args command specific arguments.
- * @param out the output stream used for printing
- * @return exit code.
- * @throws Exception
- */
- public int run(String[] args, final PrintStream out) throws Exception {
- Options opts = getOpts();
- CommandLine cmd = parseArgs(args, opts);
- return dispatch(cmd, opts, out);
- }
-
- /**
- * returns the Command Line Options.
- *
- * @return Options
- */
- private Options getOpts() {
- Options opts = new Options();
- addPlanCommands(opts);
- addHelpCommands(opts);
- addExecuteCommands(opts);
- addQueryCommands(opts);
- addCancelCommands(opts);
- addReportCommands(opts);
- return opts;
- }
-
- /**
- * Returns Plan options.
- *
- * @return Options.
- */
- public static Options getPlanOptions() {
- return PLAN_OPTIONS;
- }
-
- /**
- * Returns help options.
- *
- * @return - help options.
- */
- public static Options getHelpOptions() {
- return HELP_OPTIONS;
- }
-
- /**
- * Retuns execute options.
- *
- * @return - execute options.
- */
- public static Options getExecuteOptions() {
- return EXECUTE_OPTIONS;
- }
-
- /**
- * Returns Query Options.
- *
- * @return query Options
- */
- public static Options getQueryOptions() {
- return QUERY_OPTIONS;
- }
-
- /**
- * Returns Cancel Options.
- *
- * @return Options
- */
- public static Options getCancelOptions() {
- return CANCEL_OPTIONS;
- }
-
- /**
- * Returns Report Options.
- *
- * @return Options
- */
- public static Options getReportOptions() {
- return REPORT_OPTIONS;
- }
-
- /**
- * Adds commands for plan command.
- *
- * @return Options.
- */
- private void addPlanCommands(Options opt) {
-
- Option plan = OptionBuilder.withLongOpt(PLAN)
- .withDescription("Hostname, IP address or UUID of datanode " +
- "for which a plan is created.")
- .hasArg()
- .create();
- getPlanOptions().addOption(plan);
- opt.addOption(plan);
-
-
- Option outFile = OptionBuilder.withLongOpt(OUTFILE).hasArg()
- .withDescription(
- "Local path of file to write output to, if not specified "
- + "defaults will be used.")
- .create();
- getPlanOptions().addOption(outFile);
- opt.addOption(outFile);
-
- Option bandwidth = OptionBuilder.withLongOpt(BANDWIDTH).hasArg()
- .withDescription(
- "Maximum disk bandwidth (MB/s) in integer to be consumed by "
- + "diskBalancer. e.g. 10 MB/s.")
- .create();
- getPlanOptions().addOption(bandwidth);
- opt.addOption(bandwidth);
-
- Option threshold = OptionBuilder.withLongOpt(THRESHOLD)
- .hasArg()
- .withDescription("Percentage of data skew that is tolerated before"
- + " disk balancer starts working. For example, if"
- + " total data on a 2 disk node is 100 GB then disk"
- + " balancer calculates the expected value on each disk,"
- + " which is 50 GB. If the tolerance is 10% then data"
- + " on a single disk needs to be more than 60 GB"
- + " (50 GB + 10% tolerance value) for Disk balancer to"
- + " balance the disks.")
- .create();
- getPlanOptions().addOption(threshold);
- opt.addOption(threshold);
-
-
- Option maxError = OptionBuilder.withLongOpt(MAXERROR)
- .hasArg()
- .withDescription("Describes how many errors " +
- "can be tolerated while copying between a pair of disks.")
- .create();
- getPlanOptions().addOption(maxError);
- opt.addOption(maxError);
-
- Option verbose = OptionBuilder.withLongOpt(VERBOSE)
- .withDescription("Print out the summary of the plan on console")
- .create();
- getPlanOptions().addOption(verbose);
- opt.addOption(verbose);
- }
-
- /**
- * Adds Help to the options.
- */
- private void addHelpCommands(Options opt) {
- Option help = OptionBuilder.withLongOpt(HELP)
- .hasOptionalArg()
- .withDescription("valid commands are plan | execute | query | cancel" +
- " | report")
- .create();
- getHelpOptions().addOption(help);
- opt.addOption(help);
- }
-
- /**
- * Adds execute command options.
- *
- * @param opt Options
- */
- private void addExecuteCommands(Options opt) {
- Option execute = OptionBuilder.withLongOpt(EXECUTE)
- .hasArg()
- .withDescription("Takes a plan file and " +
- "submits it for execution by the datanode.")
- .create();
- getExecuteOptions().addOption(execute);
- opt.addOption(execute);
- }
-
- /**
- * Adds query command options.
- *
- * @param opt Options
- */
- private void addQueryCommands(Options opt) {
- Option query = OptionBuilder.withLongOpt(QUERY)
- .hasArg()
- .withDescription("Queries the disk balancer " +
- "status of a given datanode.")
- .create();
- getQueryOptions().addOption(query);
- opt.addOption(query);
-
- // Please note: Adding this only to Query options since -v is already
- // added to global table.
- Option verbose = OptionBuilder.withLongOpt(VERBOSE)
- .withDescription("Prints details of the plan that is being executed " +
- "on the node.")
- .create();
- getQueryOptions().addOption(verbose);
- }
-
- /**
- * Adds cancel command options.
- *
- * @param opt Options
- */
- private void addCancelCommands(Options opt) {
- Option cancel = OptionBuilder.withLongOpt(CANCEL)
- .hasArg()
- .withDescription("Cancels a running plan using a plan file.")
- .create();
- getCancelOptions().addOption(cancel);
- opt.addOption(cancel);
-
- Option node = OptionBuilder.withLongOpt(NODE)
- .hasArg()
- .withDescription("Cancels a running plan using a plan ID and hostName")
- .create();
-
- getCancelOptions().addOption(node);
- opt.addOption(node);
- }
-
- /**
- * Adds report command options.
- *
- * @param opt Options
- */
- private void addReportCommands(Options opt) {
- Option report = OptionBuilder.withLongOpt(REPORT)
- .withDescription("List nodes that will benefit from running " +
- "DiskBalancer.")
- .create();
- getReportOptions().addOption(report);
- opt.addOption(report);
-
- Option top = OptionBuilder.withLongOpt(TOP)
- .hasArg()
- .withDescription("specify the number of nodes to be listed which has" +
- " data imbalance.")
- .create();
- getReportOptions().addOption(top);
- opt.addOption(top);
-
- Option node = OptionBuilder.withLongOpt(NODE)
- .hasArg()
- .withDescription("Datanode address, " +
- "it can be DataNodeID, IP or hostname.")
- .create();
- getReportOptions().addOption(node);
- opt.addOption(node);
- }
-
- /**
- * This function parses all command line arguments and returns the appropriate
- * values.
- *
- * @param argv - Argv from main
- * @return CommandLine
- */
- private CommandLine parseArgs(String[] argv, Options opts)
- throws org.apache.commons.cli.ParseException {
- BasicParser parser = new BasicParser();
- return parser.parse(opts, argv);
- }
-
- /**
- * Dispatches calls to the right command Handler classes.
- *
- * @param cmd - CommandLine
- * @param opts options of command line
- * @param out the output stream used for printing
- */
- private int dispatch(CommandLine cmd, Options opts, final PrintStream out)
- throws Exception {
- Command currentCommand = null;
- if (cmd.hasOption(DiskBalancer.PLAN)) {
- currentCommand = new PlanCommand(getConf());
- }
-
- if (cmd.hasOption(DiskBalancer.EXECUTE)) {
- currentCommand = new ExecuteCommand(getConf());
- }
-
- if (cmd.hasOption(DiskBalancer.QUERY)) {
- currentCommand = new QueryCommand(getConf());
- }
-
- if (cmd.hasOption(DiskBalancer.CANCEL)) {
- currentCommand = new CancelCommand(getConf());
- }
-
- if (cmd.hasOption(DiskBalancer.REPORT)) {
- currentCommand = new ReportCommand(getConf(), out);
- }
-
- if (cmd.hasOption(DiskBalancer.HELP)) {
- currentCommand = new HelpCommand(getConf());
- }
-
- // Invoke main help here.
- if (currentCommand == null) {
- new HelpCommand(getConf()).execute(null);
- return 1;
- }
-
- currentCommand.execute(cmd);
- return 0;
- }
-}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/35c5943b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DiskBalancerCLI.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DiskBalancerCLI.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DiskBalancerCLI.java
new file mode 100644
index 0000000..e961c14
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DiskBalancerCLI.java
@@ -0,0 +1,482 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.hdfs.tools;
+
+import org.apache.commons.cli.BasicParser;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.OptionBuilder;
+import org.apache.commons.cli.Options;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
+import org.apache.hadoop.hdfs.server.diskbalancer.command.CancelCommand;
+import org.apache.hadoop.hdfs.server.diskbalancer.command.Command;
+import org.apache.hadoop.hdfs.server.diskbalancer.command.ExecuteCommand;
+import org.apache.hadoop.hdfs.server.diskbalancer.command.HelpCommand;
+import org.apache.hadoop.hdfs.server.diskbalancer.command.PlanCommand;
+import org.apache.hadoop.hdfs.server.diskbalancer.command.QueryCommand;
+import org.apache.hadoop.hdfs.server.diskbalancer.command.ReportCommand;
+import org.apache.hadoop.util.Tool;
+import org.apache.hadoop.util.ToolRunner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.PrintStream;
+
+/**
+ * DiskBalancer is a tool that can be used to ensure that data is spread evenly
+ * across volumes of same storage type.
+ * <p>
+ * For example, if you have 3 disks, with 100 GB , 600 GB and 200 GB on each
+ * disk, this tool will ensure that each disk will have 300 GB.
+ * <p>
+ * This tool can be run while data nodes are fully functional.
+ * <p>
+ * At very high level diskbalancer computes a set of moves that will make disk
+ * utilization equal and then those moves are executed by the datanode.
+ */
+public class DiskBalancerCLI extends Configured implements Tool {
+ /**
+ * Computes a plan for a given set of nodes.
+ */
+ public static final String PLAN = "plan";
+ /**
+ * Output file name, for commands like report, plan etc. This is an optional
+ * argument, by default diskbalancer will write all its output to
+ * /system/reports/diskbalancer of the current cluster it is operating
+ * against.
+ */
+ public static final String OUTFILE = "out";
+ /**
+ * Help for the program.
+ */
+ public static final String HELP = "help";
+ /**
+ * Percentage of data unevenness that we are willing to live with. For example
+ * - a value like 10 indicates that we are okay with 10 % +/- from
+ * idealStorage Target.
+ */
+ public static final String THRESHOLD = "thresholdPercentage";
+ /**
+ * Specifies the maximum disk bandwidth to use per second.
+ */
+ public static final String BANDWIDTH = "bandwidth";
+ /**
+ * Specifies the maximum errors to tolerate.
+ */
+ public static final String MAXERROR = "maxerror";
+ /**
+ * Executes a given plan file on the target datanode.
+ */
+ public static final String EXECUTE = "execute";
+ /**
+ * The report command prints out a disk fragmentation report about the data
+ * cluster. By default it prints the DEFAULT_TOP machines names with high
+ * nodeDataDensity {DiskBalancerDataNode#getNodeDataDensity} values. This
+ * means that these are the nodes that deviates from the ideal data
+ * distribution.
+ */
+ public static final String REPORT = "report";
+ /**
+ * specify top number of nodes to be processed.
+ */
+ public static final String TOP = "top";
+ /**
+ * specify default top number of nodes to be processed.
+ */
+ public static final int DEFAULT_TOP = 100;
+ /**
+ * Name or address of the node to execute against.
+ */
+ public static final String NODE = "node";
+ /**
+ * Runs the command in verbose mode.
+ */
+ public static final String VERBOSE = "v";
+ public static final int PLAN_VERSION = 1;
+ /**
+ * Reports the status of disk balancer operation.
+ */
+ public static final String QUERY = "query";
+ /**
+ * Cancels a running plan.
+ */
+ public static final String CANCEL = "cancel";
+ /**
+ * Template for the Before File. It is node.before.json.
+ */
+ public static final String BEFORE_TEMPLATE = "%s.before.json";
+ /**
+ * Template for the plan file. it is node.plan.json.
+ */
+ public static final String PLAN_TEMPLATE = "%s.plan.json";
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DiskBalancerCLI.class);
+
+ private static final Options PLAN_OPTIONS = new Options();
+ private static final Options EXECUTE_OPTIONS = new Options();
+ private static final Options QUERY_OPTIONS = new Options();
+ private static final Options HELP_OPTIONS = new Options();
+ private static final Options CANCEL_OPTIONS = new Options();
+ private static final Options REPORT_OPTIONS = new Options();
+
+ /**
+ * Construct a DiskBalancer.
+ *
+ * @param conf
+ */
+ public DiskBalancerCLI(Configuration conf) {
+ super(conf);
+ }
+
+ /**
+ * Main for the DiskBalancer Command handling.
+ *
+ * @param argv - System Args Strings[]
+ * @throws Exception
+ */
+ public static void main(String[] argv) throws Exception {
+ DiskBalancerCLI shell = new DiskBalancerCLI(new HdfsConfiguration());
+ int res = 0;
+ try {
+ res = ToolRunner.run(shell, argv);
+ } catch (Exception ex) {
+ LOG.error(ex.toString());
+ res = 1;
+ }
+ System.exit(res);
+ }
+
+ /**
+ * Execute the command with the given arguments.
+ *
+ * @param args command specific arguments.
+ * @return exit code.
+ * @throws Exception
+ */
+ @Override
+ public int run(String[] args) throws Exception {
+ return run(args, System.out);
+ }
+
+ /**
+ * Execute the command with the given arguments.
+ *
+ * @param args command specific arguments.
+ * @param out the output stream used for printing
+ * @return exit code.
+ * @throws Exception
+ */
+ public int run(String[] args, final PrintStream out) throws Exception {
+ Options opts = getOpts();
+ CommandLine cmd = parseArgs(args, opts);
+ return dispatch(cmd, opts, out);
+ }
+
+ /**
+ * returns the Command Line Options.
+ *
+ * @return Options
+ */
+ private Options getOpts() {
+ Options opts = new Options();
+ addPlanCommands(opts);
+ addHelpCommands(opts);
+ addExecuteCommands(opts);
+ addQueryCommands(opts);
+ addCancelCommands(opts);
+ addReportCommands(opts);
+ return opts;
+ }
+
+ /**
+ * Returns Plan options.
+ *
+ * @return Options.
+ */
+ public static Options getPlanOptions() {
+ return PLAN_OPTIONS;
+ }
+
+ /**
+ * Returns help options.
+ *
+ * @return - help options.
+ */
+ public static Options getHelpOptions() {
+ return HELP_OPTIONS;
+ }
+
+ /**
+ * Retuns execute options.
+ *
+ * @return - execute options.
+ */
+ public static Options getExecuteOptions() {
+ return EXECUTE_OPTIONS;
+ }
+
+ /**
+ * Returns Query Options.
+ *
+ * @return query Options
+ */
+ public static Options getQueryOptions() {
+ return QUERY_OPTIONS;
+ }
+
+ /**
+ * Returns Cancel Options.
+ *
+ * @return Options
+ */
+ public static Options getCancelOptions() {
+ return CANCEL_OPTIONS;
+ }
+
+ /**
+ * Returns Report Options.
+ *
+ * @return Options
+ */
+ public static Options getReportOptions() {
+ return REPORT_OPTIONS;
+ }
+
+ /**
+ * Adds commands for plan command.
+ *
+ * @return Options.
+ */
+ private void addPlanCommands(Options opt) {
+
+ Option plan = OptionBuilder.withLongOpt(PLAN)
+ .withDescription("Hostname, IP address or UUID of datanode " +
+ "for which a plan is created.")
+ .hasArg()
+ .create();
+ getPlanOptions().addOption(plan);
+ opt.addOption(plan);
+
+
+ Option outFile = OptionBuilder.withLongOpt(OUTFILE).hasArg()
+ .withDescription(
+ "Local path of file to write output to, if not specified "
+ + "defaults will be used.")
+ .create();
+ getPlanOptions().addOption(outFile);
+ opt.addOption(outFile);
+
+ Option bandwidth = OptionBuilder.withLongOpt(BANDWIDTH).hasArg()
+ .withDescription(
+ "Maximum disk bandwidth (MB/s) in integer to be consumed by "
+ + "diskBalancer. e.g. 10 MB/s.")
+ .create();
+ getPlanOptions().addOption(bandwidth);
+ opt.addOption(bandwidth);
+
+ Option threshold = OptionBuilder.withLongOpt(THRESHOLD)
+ .hasArg()
+ .withDescription("Percentage of data skew that is tolerated before"
+ + " disk balancer starts working. For example, if"
+ + " total data on a 2 disk node is 100 GB then disk"
+ + " balancer calculates the expected value on each disk,"
+ + " which is 50 GB. If the tolerance is 10% then data"
+ + " on a single disk needs to be more than 60 GB"
+ + " (50 GB + 10% tolerance value) for Disk balancer to"
+ + " balance the disks.")
+ .create();
+ getPlanOptions().addOption(threshold);
+ opt.addOption(threshold);
+
+
+ Option maxError = OptionBuilder.withLongOpt(MAXERROR)
+ .hasArg()
+ .withDescription("Describes how many errors " +
+ "can be tolerated while copying between a pair of disks.")
+ .create();
+ getPlanOptions().addOption(maxError);
+ opt.addOption(maxError);
+
+ Option verbose = OptionBuilder.withLongOpt(VERBOSE)
+ .withDescription("Print out the summary of the plan on console")
+ .create();
+ getPlanOptions().addOption(verbose);
+ opt.addOption(verbose);
+ }
+
+ /**
+ * Adds Help to the options.
+ */
+ private void addHelpCommands(Options opt) {
+ Option help = OptionBuilder.withLongOpt(HELP)
+ .hasOptionalArg()
+ .withDescription("valid commands are plan | execute | query | cancel" +
+ " | report")
+ .create();
+ getHelpOptions().addOption(help);
+ opt.addOption(help);
+ }
+
+ /**
+ * Adds execute command options.
+ *
+ * @param opt Options
+ */
+ private void addExecuteCommands(Options opt) {
+ Option execute = OptionBuilder.withLongOpt(EXECUTE)
+ .hasArg()
+ .withDescription("Takes a plan file and " +
+ "submits it for execution by the datanode.")
+ .create();
+ getExecuteOptions().addOption(execute);
+ opt.addOption(execute);
+ }
+
+ /**
+ * Adds query command options.
+ *
+ * @param opt Options
+ */
+ private void addQueryCommands(Options opt) {
+ Option query = OptionBuilder.withLongOpt(QUERY)
+ .hasArg()
+ .withDescription("Queries the disk balancer " +
+ "status of a given datanode.")
+ .create();
+ getQueryOptions().addOption(query);
+ opt.addOption(query);
+
+ // Please note: Adding this only to Query options since -v is already
+ // added to global table.
+ Option verbose = OptionBuilder.withLongOpt(VERBOSE)
+ .withDescription("Prints details of the plan that is being executed " +
+ "on the node.")
+ .create();
+ getQueryOptions().addOption(verbose);
+ }
+
+ /**
+ * Adds cancel command options.
+ *
+ * @param opt Options
+ */
+ private void addCancelCommands(Options opt) {
+ Option cancel = OptionBuilder.withLongOpt(CANCEL)
+ .hasArg()
+ .withDescription("Cancels a running plan using a plan file.")
+ .create();
+ getCancelOptions().addOption(cancel);
+ opt.addOption(cancel);
+
+ Option node = OptionBuilder.withLongOpt(NODE)
+ .hasArg()
+ .withDescription("Cancels a running plan using a plan ID and hostName")
+ .create();
+
+ getCancelOptions().addOption(node);
+ opt.addOption(node);
+ }
+
+ /**
+ * Adds report command options.
+ *
+ * @param opt Options
+ */
+ private void addReportCommands(Options opt) {
+ Option report = OptionBuilder.withLongOpt(REPORT)
+ .withDescription("List nodes that will benefit from running " +
+ "DiskBalancer.")
+ .create();
+ getReportOptions().addOption(report);
+ opt.addOption(report);
+
+ Option top = OptionBuilder.withLongOpt(TOP)
+ .hasArg()
+ .withDescription("specify the number of nodes to be listed which has" +
+ " data imbalance.")
+ .create();
+ getReportOptions().addOption(top);
+ opt.addOption(top);
+
+ Option node = OptionBuilder.withLongOpt(NODE)
+ .hasArg()
+ .withDescription("Datanode address, " +
+ "it can be DataNodeID, IP or hostname.")
+ .create();
+ getReportOptions().addOption(node);
+ opt.addOption(node);
+ }
+
+ /**
+ * This function parses all command line arguments and returns the appropriate
+ * values.
+ *
+ * @param argv - Argv from main
+ * @return CommandLine
+ */
+ private CommandLine parseArgs(String[] argv, Options opts)
+ throws org.apache.commons.cli.ParseException {
+ BasicParser parser = new BasicParser();
+ return parser.parse(opts, argv);
+ }
+
+ /**
+ * Dispatches calls to the right command Handler classes.
+ *
+ * @param cmd - CommandLine
+ * @param opts options of command line
+ * @param out the output stream used for printing
+ */
+ private int dispatch(CommandLine cmd, Options opts, final PrintStream out)
+ throws Exception {
+ Command currentCommand = null;
+ if (cmd.hasOption(DiskBalancerCLI.PLAN)) {
+ currentCommand = new PlanCommand(getConf());
+ }
+
+ if (cmd.hasOption(DiskBalancerCLI.EXECUTE)) {
+ currentCommand = new ExecuteCommand(getConf());
+ }
+
+ if (cmd.hasOption(DiskBalancerCLI.QUERY)) {
+ currentCommand = new QueryCommand(getConf());
+ }
+
+ if (cmd.hasOption(DiskBalancerCLI.CANCEL)) {
+ currentCommand = new CancelCommand(getConf());
+ }
+
+ if (cmd.hasOption(DiskBalancerCLI.REPORT)) {
+ currentCommand = new ReportCommand(getConf(), out);
+ }
+
+ if (cmd.hasOption(DiskBalancerCLI.HELP)) {
+ currentCommand = new HelpCommand(getConf());
+ }
+
+ // Invoke main help here.
+ if (currentCommand == null) {
+ new HelpCommand(getConf()).execute(null);
+ return 1;
+ }
+
+ currentCommand.execute(cmd);
+ return 0;
+ }
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/35c5943b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/diskbalancer/command/TestDiskBalancerCommand.java
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/diskbalancer/command/TestDiskBalancerCommand.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/diskbalancer/command/TestDiskBalancerCommand.java
index 7d659af..451ca04 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/diskbalancer/command/TestDiskBalancerCommand.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/diskbalancer/command/TestDiskBalancerCommand.java
@@ -41,18 +41,19 @@ import org.apache.hadoop.hdfs.server.diskbalancer.connectors.ClusterConnector;
import org.apache.hadoop.hdfs.server.diskbalancer.connectors.ConnectorFactory;
import org.apache.hadoop.hdfs.server.diskbalancer.datamodel.DiskBalancerCluster;
import org.apache.hadoop.hdfs.server.diskbalancer.datamodel.DiskBalancerDataNode;
+import org.apache.hadoop.hdfs.tools.DiskBalancerCLI;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import com.google.common.collect.Lists;
-import static org.apache.hadoop.hdfs.tools.DiskBalancer.CANCEL;
-import static org.apache.hadoop.hdfs.tools.DiskBalancer.HELP;
-import static org.apache.hadoop.hdfs.tools.DiskBalancer.NODE;
-import static org.apache.hadoop.hdfs.tools.DiskBalancer.PLAN;
-import static org.apache.hadoop.hdfs.tools.DiskBalancer.QUERY;
-import static org.apache.hadoop.hdfs.tools.DiskBalancer.REPORT;
+import static org.apache.hadoop.hdfs.tools.DiskBalancerCLI.CANCEL;
+import static org.apache.hadoop.hdfs.tools.DiskBalancerCLI.HELP;
+import static org.apache.hadoop.hdfs.tools.DiskBalancerCLI.NODE;
+import static org.apache.hadoop.hdfs.tools.DiskBalancerCLI.PLAN;
+import static org.apache.hadoop.hdfs.tools.DiskBalancerCLI.QUERY;
+import static org.apache.hadoop.hdfs.tools.DiskBalancerCLI.REPORT;
import org.junit.Rule;
import org.junit.rules.ExpectedException;
@@ -387,8 +388,7 @@ public class TestDiskBalancerCommand {
private List<String> runCommandInternal(final String cmdLine) throws
Exception {
String[] cmds = StringUtils.split(cmdLine, ' ');
- org.apache.hadoop.hdfs.tools.DiskBalancer db =
- new org.apache.hadoop.hdfs.tools.DiskBalancer(conf);
+ DiskBalancerCLI db = new DiskBalancerCLI(conf);
ByteArrayOutputStream bufOut = new ByteArrayOutputStream();
PrintStream out = new PrintStream(bufOut);
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[20/21] hadoop git commit: HADOOP-13359. Modify YARN to use
hadoop_subcommand_opts
Posted by aw...@apache.org.
HADOOP-13359. Modify YARN to use hadoop_subcommand_opts
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9bae198c
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9bae198c
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9bae198c
Branch: refs/heads/HADOOP-13341
Commit: 9bae198cd14dd6a55eb0f09c725d4049c2a0c5d9
Parents: dd8e8f3
Author: Allen Wittenauer <aw...@apache.org>
Authored: Tue Aug 30 13:03:45 2016 -0700
Committer: Allen Wittenauer <aw...@apache.org>
Committed: Fri Sep 9 04:07:43 2016 -0700
----------------------------------------------------------------------
hadoop-yarn-project/hadoop-yarn/bin/yarn | 17 +++++------------
1 file changed, 5 insertions(+), 12 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9bae198c/hadoop-yarn-project/hadoop-yarn/bin/yarn
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/bin/yarn b/hadoop-yarn-project/hadoop-yarn/bin/yarn
index 66a87b6..d2a8a50 100755
--- a/hadoop-yarn-project/hadoop-yarn/bin/yarn
+++ b/hadoop-yarn-project/hadoop-yarn/bin/yarn
@@ -110,8 +110,6 @@ function yarncmd_case
nodemanager)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
HADOOP_CLASSNAME='org.apache.hadoop.yarn.server.nodemanager.NodeManager'
- hadoop_debug "Append YARN_NODEMANAGER_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${YARN_NODEMANAGER_OPTS}"
# Backwards compatibility
if [[ -n "${YARN_NODEMANAGER_HEAPSIZE}" ]]; then
HADOOP_HEAPSIZE_MAX="${YARN_NODEMANAGER_HEAPSIZE}"
@@ -120,8 +118,6 @@ function yarncmd_case
proxyserver)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
HADOOP_CLASSNAME='org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer'
- hadoop_debug "Append YARN_PROXYSERVER_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${YARN_PROXYSERVER_OPTS}"
# Backwards compatibility
if [[ -n "${YARN_PROXYSERVER_HEAPSIZE}" ]]; then
# shellcheck disable=SC2034
@@ -134,8 +130,6 @@ function yarncmd_case
resourcemanager)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
HADOOP_CLASSNAME='org.apache.hadoop.yarn.server.resourcemanager.ResourceManager'
- HADOOP_OPTS="${HADOOP_OPTS} ${YARN_RESOURCEMANAGER_OPTS}"
- hadoop_debug "Append YARN_RESOURCEMANAGER_OPTS onto HADOOP_OPTS"
# Backwards compatibility
if [[ -n "${YARN_RESOURCEMANAGER_HEAPSIZE}" ]]; then
# shellcheck disable=SC2034
@@ -151,20 +145,14 @@ function yarncmd_case
sharedcachemanager)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
HADOOP_CLASSNAME='org.apache.hadoop.yarn.server.sharedcachemanager.SharedCacheManager'
- hadoop_debug "Append YARN_SHAREDCACHEMANAGER_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${YARN_SHAREDCACHEMANAGER_OPTS}"
;;
timelinereader)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
HADOOP_CLASSNAME='org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer'
- hadoop_debug "Append YARN_TIMELINEREADER_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${YARN_TIMELINEREADER_OPTS}"
;;
timelineserver)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
HADOOP_CLASSNAME='org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer'
- hadoop_debug "Append YARN_TIMELINESERVER_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${YARN_TIMELINESERVER_OPTS}"
# Backwards compatibility
if [[ -n "${YARN_TIMELINESERVER_HEAPSIZE}" ]]; then
# shellcheck disable=SC2034
@@ -268,8 +256,13 @@ if [[ ${HADOOP_WORKER_MODE} = true ]]; then
exit $?
fi
+hadoop_subcommand_opts "${HADOOP_SHELL_EXECNAME}" "${HADOOP_SUBCMD}"
+
if [[ "${HADOOP_SUBCMD_SECURESERVICE}" = true ]]; then
HADOOP_SECURE_USER="${HADOOP_SUBCMD_SECUREUSER}"
+
+ hadoop_subcommand_secure_opts "${HADOOP_SHELL_EXECNAME}" "${HADOOP_SUBCMD}"
+
hadoop_verify_secure_prereq
hadoop_setup_secure_service
priv_outfile="${HADOOP_LOG_DIR}/privileged-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.out"
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[12/21] hadoop git commit: HADOOP-13358. Modify HDFS to use
hadoop_subcommand_opts
Posted by aw...@apache.org.
HADOOP-13358. Modify HDFS to use hadoop_subcommand_opts
Signed-off-by: Allen Wittenauer <aw...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/359bb908
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/359bb908
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/359bb908
Branch: refs/heads/HADOOP-13341
Commit: 359bb90837d367322f87dc66dfd076228abaa53e
Parents: fc8d8c8
Author: Allen Wittenauer <aw...@apache.org>
Authored: Sun Aug 28 10:45:00 2016 -0700
Committer: Allen Wittenauer <aw...@apache.org>
Committed: Fri Sep 9 04:07:43 2016 -0700
----------------------------------------------------------------------
.../src/main/bin/hadoop-functions.sh | 1 +
.../hadoop-common/src/main/conf/hadoop-env.sh | 28 +++++++++----------
.../hadoop-hdfs/src/main/bin/hdfs | 29 ++++----------------
.../hadoop-hdfs/src/main/bin/hdfs-config.sh | 28 ++++++++++++-------
4 files changed, 38 insertions(+), 48 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/359bb908/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
index 695fd41..db868bd 100755
--- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
+++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
@@ -2048,6 +2048,7 @@ function hadoop_subcommand_opts
return 0
fi
}
+
## @description Add custom (program)_(command)_SECURE_EXTRA_OPTS to HADOOP_OPTS.
## @description This *does not* handle the pre-3.x deprecated cases
## @audience public
http://git-wip-us.apache.org/repos/asf/hadoop/blob/359bb908/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh b/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
index f4493f1..4656f4d 100644
--- a/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
+++ b/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
@@ -294,16 +294,16 @@ esac
# and therefore may override any similar flags set in HADOOP_OPTS
#
# a) Set JMX options
-# export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1026"
+# export HDFS_NAMENODE_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1026"
#
# b) Set garbage collection logs
-# export HADOOP_NAMENODE_OPTS="${HADOOP_GC_SETTINGS} -Xloggc:${HADOOP_LOG_DIR}/gc-rm.log-$(date +'%Y%m%d%H%M')"
+# export HDFS_NAMENODE_OPTS="${HADOOP_GC_SETTINGS} -Xloggc:${HADOOP_LOG_DIR}/gc-rm.log-$(date +'%Y%m%d%H%M')"
#
# c) ... or set them directly
-# export HADOOP_NAMENODE_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:${HADOOP_LOG_DIR}/gc-rm.log-$(date +'%Y%m%d%H%M')"
+# export HDFS_NAMENODE_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:${HADOOP_LOG_DIR}/gc-rm.log-$(date +'%Y%m%d%H%M')"
# this is the default:
-# export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=INFO,RFAS"
+# export HDFS_NAMENODE_OPTS="-Dhadoop.security.logger=INFO,RFAS"
###
# SecondaryNameNode specific parameters
@@ -313,7 +313,7 @@ esac
# and therefore may override any similar flags set in HADOOP_OPTS
#
# This is the default:
-# export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=INFO,RFAS"
+# export HDFS_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=INFO,RFAS"
###
# DataNode specific parameters
@@ -323,7 +323,7 @@ esac
# and therefore may override any similar flags set in HADOOP_OPTS
#
# This is the default:
-# export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS"
+# export HDFS_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS"
# On secure datanodes, user to run the datanode as after dropping privileges.
# This **MUST** be uncommented to enable secure HDFS if using privileged ports
@@ -336,7 +336,7 @@ esac
# Supplemental options for secure datanodes
# By default, Hadoop uses jsvc which needs to know to launch a
# server jvm.
-# export HADOOP_DN_SECURE_EXTRA_OPTS="-jvm server"
+# export HDFS_DATANODE_SECURE_EXTRA_OPTS="-jvm server"
# Where datanode log files are stored in the secure data environment.
# This will replace the hadoop.log.dir Java property in secure mode.
@@ -352,18 +352,18 @@ esac
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
-# export HADOOP_NFS3_OPTS=""
+# export HDFS_NFS3_OPTS=""
# Specify the JVM options to be used when starting the Hadoop portmapper.
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
-# export HADOOP_PORTMAP_OPTS="-Xmx512m"
+# export HDFS_PORTMAP_OPTS="-Xmx512m"
# Supplemental options for priviliged gateways
# By default, Hadoop uses jsvc which needs to know to launch a
# server jvm.
-# export HADOOP_NFS3_SECURE_EXTRA_OPTS="-jvm server"
+# export HDFS_NFS3_SECURE_EXTRA_OPTS="-jvm server"
# On privileged gateways, user to run the gateway as after dropping privileges
# This will replace the hadoop.id.str Java property in secure mode.
@@ -376,7 +376,7 @@ esac
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
-# export HADOOP_ZKFC_OPTS=""
+# export HDFS_ZKFC_OPTS=""
###
# QuorumJournalNode specific parameters
@@ -385,7 +385,7 @@ esac
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
-# export HADOOP_JOURNALNODE_OPTS=""
+# export HDFS_JOURNALNODE_OPTS=""
###
# HDFS Balancer specific parameters
@@ -394,7 +394,7 @@ esac
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
-# export HADOOP_BALANCER_OPTS=""
+# export HDFS_BALANCER_OPTS=""
###
# HDFS Mover specific parameters
@@ -403,7 +403,7 @@ esac
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
-# export HADOOP_MOVER_OPTS=""
+# export HDFS_MOVER_OPTS=""
###
# Advanced Users Only!
http://git-wip-us.apache.org/repos/asf/hadoop/blob/359bb908/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs b/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
index af4997d8..566bd7f 100755
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
@@ -79,8 +79,6 @@ function hdfscmd_case
balancer)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.server.balancer.Balancer
- hadoop_debug "Appending HADOOP_BALANCER_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_BALANCER_OPTS}"
;;
cacheadmin)
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.CacheAdmin
@@ -103,13 +101,8 @@ function hdfscmd_case
HADOOP_SECURE_PID_DIR="${HADOOP_SECURE_PID_DIR:-$HADOOP_SECURE_DN_PID_DIR}"
HADOOP_SECURE_LOG_DIR="${HADOOP_SECURE_LOG_DIR:-$HADOOP_SECURE_DN_LOG_DIR}"
- hadoop_debug "Appending HADOOP_DATANODE_OPTS onto HADOOP_OPTS"
- hadoop_debug "Appending HADOOP_DN_SECURE_EXTRA_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_DATANODE_OPTS} ${HADOOP_DN_SECURE_EXTRA_OPTS}"
HADOOP_CLASSNAME="org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter"
else
- hadoop_debug "Appending HADOOP_DATANODE_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_DATANODE_OPTS}"
HADOOP_CLASSNAME='org.apache.hadoop.hdfs.server.datanode.DataNode'
fi
;;
@@ -157,8 +150,6 @@ function hdfscmd_case
journalnode)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
HADOOP_CLASSNAME='org.apache.hadoop.hdfs.qjournal.server.JournalNode'
- hadoop_debug "Appending HADOOP_JOURNALNODE_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_JOURNALNODE_OPTS}"
;;
jmxget)
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.tools.JMXGet
@@ -169,14 +160,10 @@ function hdfscmd_case
mover)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.server.mover.Mover
- hadoop_debug "Appending HADOOP_MOVER_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_MOVER_OPTS}"
;;
namenode)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
HADOOP_CLASSNAME='org.apache.hadoop.hdfs.server.namenode.NameNode'
- hadoop_debug "Appending HADOOP_NAMENODE_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_NAMENODE_OPTS}"
hadoop_add_param HADOOP_OPTS hdfs.audit.logger "-Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER}"
;;
nfs3)
@@ -189,13 +176,8 @@ function hdfscmd_case
HADOOP_SECURE_PID_DIR="${HADOOP_SECURE_PID_DIR:-$HADOOP_SECURE_NFS3_PID_DIR}"
HADOOP_SECURE_LOG_DIR="${HADOOP_SECURE_LOG_DIR:-$HADOOP_SECURE_NFS3_LOG_DIR}"
- hadoop_debug "Appending HADOOP_NFS3_OPTS onto HADOOP_OPTS"
- hadoop_debug "Appending HADOOP_NFS3_SECURE_EXTRA_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_NFS3_OPTS} ${HADOOP_NFS3_SECURE_EXTRA_OPTS}"
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.nfs.nfs3.PrivilegedNfsGatewayStarter
else
- hadoop_debug "Appending HADOOP_NFS3_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_NFS3_OPTS}"
HADOOP_CLASSNAME=org.apache.hadoop.hdfs.nfs.nfs3.Nfs3
fi
;;
@@ -211,14 +193,10 @@ function hdfscmd_case
portmap)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
HADOOP_CLASSNAME=org.apache.hadoop.portmap.Portmap
- hadoop_debug "Appending HADOOP_PORTMAP_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_PORTMAP_OPTS}"
;;
secondarynamenode)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
HADOOP_CLASSNAME='org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode'
- hadoop_debug "Appending HADOOP_SECONDARYNAMENODE_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_SECONDARYNAMENODE_OPTS}"
hadoop_add_param HADOOP_OPTS hdfs.audit.logger "-Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER}"
;;
snapshotDiff)
@@ -233,8 +211,6 @@ function hdfscmd_case
zkfc)
HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
HADOOP_CLASSNAME='org.apache.hadoop.hdfs.tools.DFSZKFailoverController'
- hadoop_debug "Appending HADOOP_ZKFC_OPTS onto HADOOP_OPTS"
- HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_ZKFC_OPTS}"
;;
*)
HADOOP_CLASSNAME="${subcmd}"
@@ -288,8 +264,13 @@ if [[ ${HADOOP_WORKER_MODE} = true ]]; then
exit $?
fi
+hadoop_subcommand_opts "${HADOOP_SHELL_EXECNAME}" "${HADOOP_SUBCMD}"
+
if [[ "${HADOOP_SUBCMD_SECURESERVICE}" = true ]]; then
HADOOP_SECURE_USER="${HADOOP_SUBCMD_SECUREUSER}"
+
+ hadoop_subcommand_secure_opts "${HADOOP_SHELL_EXECNAME}" "${HADOOP_SUBCMD}"
+
hadoop_verify_secure_prereq
hadoop_setup_secure_service
priv_outfile="${HADOOP_LOG_DIR}/privileged-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.out"
http://git-wip-us.apache.org/repos/asf/hadoop/blob/359bb908/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh b/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh
old mode 100644
new mode 100755
index d440210..cba37a4
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh
@@ -26,7 +26,7 @@ function hadoop_subproject_init
export HADOOP_HDFS_ENV_PROCESSED=true
fi
fi
-
+
# at some point in time, someone thought it would be a good idea to
# create separate vars for every subproject. *sigh*
# let's perform some overrides and setup some defaults for bw compat
@@ -42,23 +42,31 @@ function hadoop_subproject_init
hadoop_deprecate_envvar HADOOP_HDFS_NICENESS HADOOP_NICENESS
hadoop_deprecate_envvar HADOOP_HDFS_STOP_TIMEOUT HADOOP_STOP_TIMEOUT
-
+
hadoop_deprecate_envvar HADOOP_HDFS_PID_DIR HADOOP_PID_DIR
hadoop_deprecate_envvar HADOOP_HDFS_ROOT_LOGGER HADOOP_ROOT_LOGGER
hadoop_deprecate_envvar HADOOP_HDFS_IDENT_STRING HADOOP_IDENT_STRING
-
+
+ hadoop_deprecate_envvar HADOOP_DN_SECURE_EXTRA_OPTS HDFS_DATANODE_SECURE_EXTRA_OPTS
+
+ hadoop_deprecate_envvar HADOOP_NFS3_SECURE_EXTRA_OPTS HDFS_NFS3_SECURE_EXTRA_OPTS
+
+
HADOOP_HDFS_HOME="${HADOOP_HDFS_HOME:-$HADOOP_HOME}"
-
+
# turn on the defaults
export HDFS_AUDIT_LOGGER=${HDFS_AUDIT_LOGGER:-INFO,NullAppender}
- export HADOOP_NAMENODE_OPTS=${HADOOP_NAMENODE_OPTS:-"-Dhadoop.security.logger=INFO,RFAS"}
- export HADOOP_SECONDARYNAMENODE_OPTS=${HADOOP_SECONDARYNAMENODE_OPTS:-"-Dhadoop.security.logger=INFO,RFAS"}
- export HADOOP_DATANODE_OPTS=${HADOOP_DATANODE_OPTS:-"-Dhadoop.security.logger=ERROR,RFAS"}
- export HADOOP_DN_SECURE_EXTRA_OPTS=${HADOOP_DN_SECURE_EXTRA_OPTS:-"-jvm server"}
- export HADOOP_NFS3_SECURE_EXTRA_OPTS=${HADOOP_NFS3_SECURE_EXTRA_OPTS:-"-jvm server"}
- export HADOOP_PORTMAP_OPTS=${HADOOP_PORTMAP_OPTS:-"-Xmx512m"}
+ export HDFS_NAMENODE_OPTS=${HDFS_NAMENODE_OPTS:-"-Dhadoop.security.logger=INFO,RFAS"}
+ export HDFS_SECONDARYNAMENODE_OPTS=${HDFS_SECONDARYNAMENODE_OPTS:-"-Dhadoop.security.logger=INFO,RFAS"}
+ export HDFS_DATANODE_OPTS=${HDFS_DATANODE_OPTS:-"-Dhadoop.security.logger=ERROR,RFAS"}
+ export HDFS_PORTMAP_OPTS=${HDFS_PORTMAP_OPTS:-"-Xmx512m"}
+
+ # depending upon what is being used to start Java, these may need to be
+ # set empty. (thus no colon)
+ export HDFS_DATANODE_SECURE_EXTRA_OPTS=${HDFS_DATANODE_SECURE_EXTRA_OPTS-"-jvm server"}
+ export HDFS_NFS3_SECURE_EXTRA_OPTS=${HDFS_NFS3_SECURE_EXTRA_OPTS-"-jvm server"}
}
if [[ -z "${HADOOP_LIBEXEC_DIR}" ]]; then
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[18/21] hadoop git commit: HADOOP-13360. Documentation for
HADOOP_subcommand_OPTS
Posted by aw...@apache.org.
HADOOP-13360. Documentation for HADOOP_subcommand_OPTS
Signed-off-by: Allen Wittenauer <aw...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/b417ba32
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/b417ba32
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/b417ba32
Branch: refs/heads/HADOOP-13341
Commit: b417ba32d4d202bf0bc43ad5b6d17336b3b58ee6
Parents: d294200
Author: Allen Wittenauer <aw...@apache.org>
Authored: Wed Aug 31 07:39:34 2016 -0700
Committer: Allen Wittenauer <aw...@apache.org>
Committed: Fri Sep 9 04:07:43 2016 -0700
----------------------------------------------------------------------
.../src/site/markdown/ClusterSetup.md | 19 ++++-------
.../src/site/markdown/UnixShellGuide.md | 34 +++++++++++++++++---
.../src/site/markdown/HdfsNfsGateway.md | 2 +-
3 files changed, 37 insertions(+), 18 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b417ba32/hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md b/hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
index 0d551b1..f222769 100644
--- a/hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
+++ b/hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
@@ -64,17 +64,17 @@ Administrators can configure individual daemons using the configuration options
| Daemon | Environment Variable |
|:---- |:---- |
-| NameNode | HADOOP\_NAMENODE\_OPTS |
-| DataNode | HADOOP\_DATANODE\_OPTS |
-| Secondary NameNode | HADOOP\_SECONDARYNAMENODE\_OPTS |
+| NameNode | HDFS\_NAMENODE\_OPTS |
+| DataNode | HDFS\_DATANODE\_OPTS |
+| Secondary NameNode | HDFS\_SECONDARYNAMENODE\_OPTS |
| ResourceManager | YARN\_RESOURCEMANAGER\_OPTS |
| NodeManager | YARN\_NODEMANAGER\_OPTS |
| WebAppProxy | YARN\_PROXYSERVER\_OPTS |
-| Map Reduce Job History Server | HADOOP\_JOB\_HISTORYSERVER\_OPTS |
+| Map Reduce Job History Server | MAPRED\_HISTORYSERVER\_OPTS |
-For example, To configure Namenode to use parallelGC, the following statement should be added in hadoop-env.sh :
+For example, To configure Namenode to use parallelGC and a 4GB Java Heap, the following statement should be added in hadoop-env.sh :
- export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC"
+ export HDFS_NAMENODE_OPTS="-XX:+UseParallelGC -Xmx4g"
See `etc/hadoop/hadoop-env.sh` for other examples.
@@ -91,13 +91,6 @@ It is also traditional to configure `HADOOP_HOME` in the system-wide shell envir
HADOOP_HOME=/path/to/hadoop
export HADOOP_HOME
-| Daemon | Environment Variable |
-|:---- |:---- |
-| ResourceManager | YARN\_RESOURCEMANAGER\_HEAPSIZE |
-| NodeManager | YARN\_NODEMANAGER\_HEAPSIZE |
-| WebAppProxy | YARN\_PROXYSERVER\_HEAPSIZE |
-| Map Reduce Job History Server | HADOOP\_JOB\_HISTORYSERVER\_HEAPSIZE |
-
### Configuring the Hadoop Daemons
This section deals with important parameters to be specified in the given configuration files:
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b417ba32/hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md b/hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md
index 940627d..b130f0f 100644
--- a/hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md
+++ b/hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md
@@ -24,7 +24,7 @@ Apache Hadoop has many environment variables that control various aspects of the
### `HADOOP_CLIENT_OPTS`
-This environment variable is used for almost all end-user operations. It can be used to set any Java options as well as any Apache Hadoop options via a system property definition. For example:
+This environment variable is used for all end-user, non-daemon operations. It can be used to set any Java options as well as any Apache Hadoop options via a system property definition. For example:
```bash
HADOOP_CLIENT_OPTS="-Xmx1g -Dhadoop.socks.server=localhost:4000" hadoop fs -ls /tmp
@@ -32,6 +32,18 @@ HADOOP_CLIENT_OPTS="-Xmx1g -Dhadoop.socks.server=localhost:4000" hadoop fs -ls /
will increase the memory and send this command via a SOCKS proxy server.
+### `(command)_(subcommand)_OPTS`
+
+It is also possible to set options on a per subcommand basis. This allows for one to create special options for particular cases. The first part of the pattern is the command being used, but all uppercase. The second part of the command is the subcommand being used. Then finally followed by the string `_OPT`.
+
+For example, to configure `mapred distcp` to use a 2GB heap, one would use:
+
+```bash
+MAPRED_DISTCP_OPTS="-Xmx2g"
+```
+
+These options will appear *after* `HADOOP_CLIENT_OPTS` during execution and will generally take precedence.
+
### `HADOOP_CLASSPATH`
NOTE: Site-wide settings should be configured via a shellprofile entry and permanent user-wide settings should be configured via ${HOME}/.hadooprc using the `hadoop_add_classpath` function. See below for more information.
@@ -56,6 +68,8 @@ For example:
#
HADOOP_CLIENT_OPTS="-Xmx1g"
+MAPRED_DISTCP_OPTS="-Xmx2g"
+HADOOP_DISTCP_OPTS="-Xmx2g"
```
The `.hadoop-env` file can also be used to extend functionality and teach Apache Hadoop new tricks. For example, to run hadoop commands accessing the server referenced in the environment variable `${HADOOP_SERVER}`, the following in the `.hadoop-env` will do just that:
@@ -71,11 +85,23 @@ One word of warning: not all of Unix Shell API routines are available or work c
## Administrator Environment
-There are many environment variables that impact how the system operates. By far, the most important are the series of `_OPTS` variables that control how daemons work. These variables should contain all of the relevant settings for those daemons.
+In addition to the various XML files, there are two key capabilities for administrators to configure Apache Hadoop when using the Unix Shell:
+
+ * Many environment variables that impact how the system operates. This guide will only highlight some key ones. There is generally more information in the various `*-env.sh` files.
+
+ * Supplement or do some platform-specific changes to the existing scripts. Apache Hadoop provides the capabilities to do function overrides so that the existing code base may be changed in place without all of that work. Replacing functions is covered later under the Shell API documentation.
+
+### `(command)_(subcommand)_OPTS`
+
+By far, the most important are the series of `_OPTS` variables that control how daemons work. These variables should contain all of the relevant settings for those daemons.
+
+Similar to the user commands above, all daemons will honor the `(command)_(subcommand)_OPTS` pattern. It is generally recommended that these be set in `hadoop-env.sh` to guarantee that the system will know which settings it should use on restart. Unlike user-facing subcommands, daemons will *NOT* honor `HADOOP_CLIENT_OPTS`.
+
+In addition, daemons that run in an extra security mode also support `(command)_(subcommand)_SECURE_EXTRA_OPTS`. These options are *supplemental* to the generic `*_OPTS` and will appear after, therefore generally taking precedence.
-More, detailed information is contained in `hadoop-env.sh` and the other env.sh files.
+### `(command)_(subcommand)_USER`
-Advanced administrators may wish to supplement or do some platform-specific fixes to the existing scripts. In some systems, this means copying the errant script or creating a custom build with these changes. Apache Hadoop provides the capabilities to do function overrides so that the existing code base may be changed in place without all of that work. Replacing functions is covered later under the Shell API documentation.
+Apache Hadoop provides a way to do a user check per-subcommand. While this method is easily circumvented and should not be considered a security-feature, it does provide a mechanism by which to prevent accidents. For example, setting `HDFS_NAMENODE_USER=hdfs` will make the `hdfs namenode` and `hdfs --daemon start namenode` commands verify that the user running the commands are the hdfs user by checking the `USER` environment variable. This also works for non-daemons. Setting `HADOOP_DISTCP_USER=jane` will verify that `USER` is set to `jane` before being allowed to execute the `hadoop distcp` command.
## Developer and Advanced Administrator Environment
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b417ba32/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md
index 6731189..4742637 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md
@@ -183,7 +183,7 @@ It's strongly recommended for the users to update a few configuration properties
</property>
* JVM and log settings. You can export JVM settings (e.g., heap size and GC log) in
- HADOOP\_NFS3\_OPTS. More NFS related settings can be found in hadoop-env.sh.
+ HDFS\_NFS3\_OPTS. More NFS related settings can be found in hadoop-env.sh.
To get NFS debug trace, you can edit the log4j.property file
to add the following. Note, debug trace, especially for ONCRPC, can be very verbose.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org