You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by jh...@apache.org on 2019/01/08 20:30:10 UTC
[hadoop] branch YARN-8200 updated (aae2333 -> d5a5bb8)
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a change to branch YARN-8200
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
discard aae2333 YARN-6232. Update resource usage and preempted resource calculations to take into account all resource types. Contributed by Varun Vasudev.
new a64c758 YARN-9181. Backport YARN-6232 for generic resource type usage to branch-2
new a1861b1 YARN-9182. Backport YARN-6445 resource profile performance improvements to branch-2
new 074569a YARN-6761. Fix build for YARN-3926 branch. Contributed by Varun Vasudev.
new d5a5bb8 YARN-6786. [YARN-3926] ResourcePBImpl imports cleanup. Contributed by Yeliang Cang.
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (aae2333)
\
N -- N -- N refs/heads/YARN-8200 (d5a5bb8)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../apache/hadoop/yarn/api/records/Resource.java | 117 +++++++++++++++++----
.../yarn/api/records/ResourceInformation.java | 51 +++++----
.../hadoop/yarn/util/UnitsConversionUtil.java | 34 +++---
.../hadoop/yarn/conf/TestResourceInformation.java | 4 +-
.../hadoop/yarn/util/TestUnitsConversionUtil.java | 60 +++++------
.../yarn/api/records/impl/pb/ResourcePBImpl.java | 27 +++--
.../util/resource/DominantResourceCalculator.java | 53 +++++-----
.../hadoop/yarn/util/resource/Resources.java | 18 ++--
.../yarn/util/resource/TestResourceUtils.java | 1 +
.../hadoop/yarn/util/resource/TestResources.java | 12 ++-
.../scheduler/AbstractYarnScheduler.java | 20 +---
11 files changed, 242 insertions(+), 155 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 01/04: YARN-9181. Backport YARN-6232 for generic resource
type usage to branch-2
Posted by jh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a commit to branch YARN-8200
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit a64c7585676174ee6c275ecdef699a3785594cdc
Author: Sunil G <su...@apache.org>
AuthorDate: Mon Mar 6 11:34:20 2017 +0530
YARN-9181. Backport YARN-6232 for generic resource type usage to branch-2
---
.../records/ApplicationResourceUsageReport.java | 58 +++++++-
.../src/main/proto/yarn_protos.proto | 7 +
.../hadoop/yarn/client/cli/ApplicationCLI.java | 35 ++---
.../apache/hadoop/yarn/client/cli/TestYarnCLI.java | 16 ++-
.../pb/ApplicationResourceUsageReportPBImpl.java | 151 ++++++++++++++++++---
.../yarn/api/records/impl/pb/ProtoUtils.java | 34 +++++
.../org/apache/hadoop/yarn/util/StringHelper.java | 36 +++++
.../hadoop/yarn/api/BasePBImplRecordsTest.java | 12 ++
.../apache/hadoop/yarn/api/TestPBImplRecords.java | 4 +
.../ApplicationHistoryManagerOnTimelineStore.java | 18 ++-
.../hadoop/yarn/server/utils/BuilderUtils.java | 16 +--
.../yarn/server/resourcemanager/RMAppManager.java | 8 +-
.../yarn/server/resourcemanager/RMServerUtils.java | 10 +-
.../resourcemanager/recovery/RMStateStore.java | 7 +-
.../records/ApplicationAttemptStateData.java | 89 ++++++++++--
.../impl/pb/ApplicationAttemptStateDataPBImpl.java | 50 +++++++
.../server/resourcemanager/rmapp/RMAppImpl.java | 43 +++---
.../server/resourcemanager/rmapp/RMAppMetrics.java | 41 ++++--
.../rmapp/attempt/AggregateAppResourceUsage.java | 34 ++---
.../rmapp/attempt/RMAppAttemptImpl.java | 32 ++---
.../rmapp/attempt/RMAppAttemptMetrics.java | 106 ++++++++++++---
.../rmcontainer/RMContainerImpl.java | 16 +--
.../scheduler/SchedulerApplicationAttempt.java | 47 ++++---
.../server/resourcemanager/webapp/RMAppBlock.java | 14 +-
.../server/resourcemanager/webapp/dao/AppInfo.java | 28 ++++
.../resourcemanager/webapp/dao/ResourceInfo.java | 36 ++++-
.../resourcemanager/webapp/dao/SchedulerInfo.java | 2 +-
.../yarn_server_resourcemanager_recovery.proto | 2 +
.../server/resourcemanager/TestAppManager.java | 7 +-
.../TestContainerResourceUsage.java | 7 +-
.../applicationsmanager/MockAsm.java | 9 +-
.../metrics/TestSystemMetricsPublisher.java | 15 +-
.../metrics/TestSystemMetricsPublisherForV2.java | 22 ++-
.../recovery/RMStateStoreTestBase.java | 8 +-
.../recovery/TestZKRMStateStore.java | 23 ++--
.../server/resourcemanager/webapp/TestAppPage.java | 8 +-
.../webapp/TestRMWebAppFairScheduler.java | 5 +-
37 files changed, 803 insertions(+), 253 deletions(-)
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationResourceUsageReport.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationResourceUsageReport.java
index 3cf8f3d..f9c8975 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationResourceUsageReport.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationResourceUsageReport.java
@@ -24,6 +24,9 @@ import org.apache.hadoop.classification.InterfaceStability.Stable;
import org.apache.hadoop.classification.InterfaceStability.Unstable;
import org.apache.hadoop.yarn.util.Records;
+import java.util.HashMap;
+import java.util.Map;
+
/**
* Contains various scheduling metrics to be reported by UI and CLI.
*/
@@ -35,9 +38,9 @@ public abstract class ApplicationResourceUsageReport {
@Unstable
public static ApplicationResourceUsageReport newInstance(
int numUsedContainers, int numReservedContainers, Resource usedResources,
- Resource reservedResources, Resource neededResources, long memorySeconds,
- long vcoreSeconds, float queueUsagePerc, float clusterUsagePerc,
- long preemptedMemorySeconds, long preemptedVcoresSeconds) {
+ Resource reservedResources, Resource neededResources,
+ Map<String, Long> resourceSecondsMap, float queueUsagePerc,
+ float clusterUsagePerc, Map<String, Long> preemtedResourceSecondsMap) {
ApplicationResourceUsageReport report =
Records.newRecord(ApplicationResourceUsageReport.class);
report.setNumUsedContainers(numUsedContainers);
@@ -45,12 +48,10 @@ public abstract class ApplicationResourceUsageReport {
report.setUsedResources(usedResources);
report.setReservedResources(reservedResources);
report.setNeededResources(neededResources);
- report.setMemorySeconds(memorySeconds);
- report.setVcoreSeconds(vcoreSeconds);
+ report.setResourceSecondsMap(resourceSecondsMap);
report.setQueueUsagePercentage(queueUsagePerc);
report.setClusterUsagePercentage(clusterUsagePerc);
- report.setPreemptedMemorySeconds(preemptedMemorySeconds);
- report.setPreemptedVcoreSeconds(preemptedVcoresSeconds);
+ report.setPreemptedResourceSecondsMap(preemtedResourceSecondsMap);
return report;
}
@@ -229,4 +230,47 @@ public abstract class ApplicationResourceUsageReport {
@Public
@Unstable
public abstract long getPreemptedVcoreSeconds();
+
+ /**
+ * Get the aggregated number of resources that the application has
+ * allocated times the number of seconds the application has been running.
+ * @return map containing the resource name and aggregated resource-seconds
+ */
+ @Public
+ @Unstable
+ public abstract Map<String, Long> getResourceSecondsMap();
+
+ /**
+ * Set the aggregated number of resources that the application has
+ * allocated times the number of seconds the application has been running.
+ * @param resourceSecondsMap map containing the resource name and aggregated
+ * resource-seconds
+ */
+ @Private
+ @Unstable
+ public abstract void setResourceSecondsMap(
+ Map<String, Long> resourceSecondsMap);
+
+
+ /**
+ * Get the aggregated number of resources preempted that the application has
+ * allocated times the number of seconds the application has been running.
+ * @return map containing the resource name and aggregated preempted
+ * resource-seconds
+ */
+ @Public
+ @Unstable
+ public abstract Map<String, Long> getPreemptedResourceSecondsMap();
+
+ /**
+ * Set the aggregated number of resources preempted that the application has
+ * allocated times the number of seconds the application has been running.
+ * @param preemptedResourceSecondsMap map containing the resource name and
+ * aggregated preempted resource-seconds
+ */
+ @Private
+ @Unstable
+ public abstract void setPreemptedResourceSecondsMap(
+ Map<String, Long> preemptedResourceSecondsMap);
+
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
index a7bf414..4f4c3e0 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
@@ -217,6 +217,11 @@ message LocalResourceProto {
optional bool should_be_uploaded_to_shared_cache = 7;
}
+message StringLongMapProto {
+ required string key = 1;
+ required int64 value = 2;
+}
+
message ApplicationResourceUsageReportProto {
optional int32 num_used_containers = 1;
optional int32 num_reserved_containers = 2;
@@ -229,6 +234,8 @@ message ApplicationResourceUsageReportProto {
optional float cluster_usage_percentage = 9;
optional int64 preempted_memory_seconds = 10;
optional int64 preempted_vcore_seconds = 11;
+ repeated StringLongMapProto application_resource_usage_map = 12;
+ repeated StringLongMapProto application_preempted_resource_usage_map = 13;
}
message ApplicationReportProto {
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
index 5f6b300..2a9b3bc 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
@@ -62,6 +62,8 @@ import org.apache.hadoop.yarn.util.Times;
import com.google.common.annotations.VisibleForTesting;
+import static org.apache.hadoop.yarn.util.StringHelper.getResourceSecondsString;
+
@Private
@Unstable
public class ApplicationCLI extends YarnCLI {
@@ -711,24 +713,9 @@ public class ApplicationCLI extends YarnCLI {
appReportStr.println(appReport.getRpcPort());
appReportStr.print("\tAM Host : ");
appReportStr.println(appReport.getHost());
- appReportStr.print("\tAggregate Resource Allocation : ");
-
ApplicationResourceUsageReport usageReport =
appReport.getApplicationResourceUsageReport();
- if (usageReport != null) {
- //completed app report in the timeline server doesn't have usage report
- appReportStr.print(usageReport.getMemorySeconds() + " MB-seconds, ");
- appReportStr.println(usageReport.getVcoreSeconds() + " vcore-seconds");
- appReportStr.print("\tAggregate Resource Preempted : ");
- appReportStr.print(usageReport.getPreemptedMemorySeconds() +
- " MB-seconds, ");
- appReportStr.println(usageReport.getPreemptedVcoreSeconds() +
- " vcore-seconds");
- } else {
- appReportStr.println("N/A");
- appReportStr.print("\tAggregate Resource Preempted : ");
- appReportStr.println("N/A");
- }
+ printResourceUsage(appReportStr, usageReport);
appReportStr.print("\tLog Aggregation Status : ");
appReportStr.println(appReport.getLogAggregationStatus() == null ? "N/A"
: appReport.getLogAggregationStatus());
@@ -759,6 +746,22 @@ public class ApplicationCLI extends YarnCLI {
return 0;
}
+ private void printResourceUsage(PrintWriter appReportStr,
+ ApplicationResourceUsageReport usageReport) {
+ appReportStr.print("\tAggregate Resource Allocation : ");
+ if (usageReport != null) {
+ appReportStr.println(
+ getResourceSecondsString(usageReport.getResourceSecondsMap()));
+ appReportStr.print("\tAggregate Resource Preempted : ");
+ appReportStr.println(getResourceSecondsString(
+ usageReport.getPreemptedResourceSecondsMap()));
+ } else {
+ appReportStr.println("N/A");
+ appReportStr.print("\tAggregate Resource Preempted : ");
+ appReportStr.println("N/A");
+ }
+ }
+
private String getAllValidApplicationStates() {
StringBuilder sb = new StringBuilder();
sb.append("The valid application state can be" + " one of the following: ");
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java
index 0f0c9bf..eee3af1 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java
@@ -39,8 +39,10 @@ import java.util.ArrayList;
import java.util.Collections;
import java.util.Date;
import java.util.EnumSet;
+import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
+import java.util.Map;
import java.util.Set;
import java.util.regex.Pattern;
@@ -69,6 +71,7 @@ import org.apache.hadoop.yarn.api.records.Priority;
import org.apache.hadoop.yarn.api.records.QueueInfo;
import org.apache.hadoop.yarn.api.records.QueueState;
import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceInformation;
import org.apache.hadoop.yarn.api.records.ResourceUtilization;
import org.apache.hadoop.yarn.api.records.SignalContainerCommand;
import org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState;
@@ -118,9 +121,18 @@ public class TestYarnCLI {
for (int i = 0; i < 2; ++i) {
ApplicationCLI cli = createAndGetAppCLI();
ApplicationId applicationId = ApplicationId.newInstance(1234, 5);
+ Map<String, Long> resourceSecondsMap = new HashMap<>();
+ Map<String, Long> preemptedResoureSecondsMap = new HashMap<>();
+ resourceSecondsMap.put(ResourceInformation.MEMORY_MB.getName(), 123456L);
+ resourceSecondsMap.put(ResourceInformation.VCORES.getName(), 4567L);
+ preemptedResoureSecondsMap
+ .put(ResourceInformation.MEMORY_MB.getName(), 1111L);
+ preemptedResoureSecondsMap
+ .put(ResourceInformation.VCORES.getName(), 2222L);
ApplicationResourceUsageReport usageReport = i == 0 ? null :
- ApplicationResourceUsageReport.newInstance(
- 2, 0, null, null, null, 123456, 4567, 0, 0, 1111, 2222);
+ ApplicationResourceUsageReport
+ .newInstance(2, 0, null, null, null, resourceSecondsMap, 0, 0,
+ preemptedResoureSecondsMap);
ApplicationReport newApplicationReport = ApplicationReport.newInstance(
applicationId, ApplicationAttemptId.newInstance(applicationId, 1),
"user", "queue", "appname", "host", 124, null,
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationResourceUsageReportPBImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationResourceUsageReportPBImpl.java
index 1c85e28..2f94a24 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationResourceUsageReportPBImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationResourceUsageReportPBImpl.java
@@ -22,12 +22,16 @@ import org.apache.hadoop.classification.InterfaceAudience.Private;
import org.apache.hadoop.classification.InterfaceStability.Unstable;
import org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport;
import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceInformation;
import org.apache.hadoop.yarn.proto.YarnProtos.ApplicationResourceUsageReportProto;
import org.apache.hadoop.yarn.proto.YarnProtos.ApplicationResourceUsageReportProtoOrBuilder;
import org.apache.hadoop.yarn.proto.YarnProtos.ResourceProto;
import com.google.protobuf.TextFormat;
+import java.util.HashMap;
+import java.util.Map;
+
@Private
@Unstable
public class ApplicationResourceUsageReportPBImpl
@@ -41,6 +45,9 @@ extends ApplicationResourceUsageReport {
Resource reservedResources;
Resource neededResources;
+ private Map<String, Long> resourceSecondsMap;
+ private Map<String, Long> preemptedResourceSecondsMap;
+
public ApplicationResourceUsageReportPBImpl() {
builder = ApplicationResourceUsageReportProto.newBuilder();
}
@@ -49,6 +56,8 @@ extends ApplicationResourceUsageReport {
ApplicationResourceUsageReportProto proto) {
this.proto = proto;
viaProto = true;
+ getResourceSecondsMap();
+ getPreemptedResourceSecondsMap();
}
public synchronized ApplicationResourceUsageReportProto getProto() {
@@ -89,6 +98,23 @@ extends ApplicationResourceUsageReport {
if (this.neededResources != null) {
builder.setNeededResources(convertToProtoFormat(this.neededResources));
}
+ builder.clearApplicationResourceUsageMap();
+ builder.clearApplicationPreemptedResourceUsageMap();
+
+ if (preemptedResourceSecondsMap != null && !preemptedResourceSecondsMap
+ .isEmpty()) {
+ builder.addAllApplicationPreemptedResourceUsageMap(ProtoUtils
+ .convertMapToStringLongMapProtoList(preemptedResourceSecondsMap));
+ }
+ if (resourceSecondsMap != null && !resourceSecondsMap.isEmpty()) {
+ builder.addAllApplicationResourceUsageMap(
+ ProtoUtils.convertMapToStringLongMapProtoList(resourceSecondsMap));
+ }
+
+ builder.setMemorySeconds(this.getMemorySeconds());
+ builder.setVcoreSeconds(this.getVcoreSeconds());
+ builder.setPreemptedMemorySeconds(this.getPreemptedMemorySeconds());
+ builder.setPreemptedVcoreSeconds(this.getPreemptedVcoreSeconds());
}
private void mergeLocalToProto() {
@@ -196,54 +222,64 @@ extends ApplicationResourceUsageReport {
@Override
public synchronized void setMemorySeconds(long memory_seconds) {
- maybeInitBuilder();
- builder.setMemorySeconds(memory_seconds);
+ getResourceSecondsMap()
+ .put(ResourceInformation.MEMORY_MB.getName(), memory_seconds);
}
-
+
@Override
public synchronized long getMemorySeconds() {
- ApplicationResourceUsageReportProtoOrBuilder p = viaProto ? proto : builder;
- return p.getMemorySeconds();
+ Map<String, Long> tmp = getResourceSecondsMap();
+ if (tmp.containsKey(ResourceInformation.MEMORY_MB.getName())) {
+ return tmp.get(ResourceInformation.MEMORY_MB.getName());
+ }
+ return 0;
}
@Override
public synchronized void setVcoreSeconds(long vcore_seconds) {
- maybeInitBuilder();
- builder.setVcoreSeconds(vcore_seconds);
+ getResourceSecondsMap()
+ .put(ResourceInformation.VCORES.getName(), vcore_seconds);
}
@Override
public synchronized long getVcoreSeconds() {
- ApplicationResourceUsageReportProtoOrBuilder p = viaProto ? proto : builder;
- return (p.getVcoreSeconds());
+ Map<String, Long> tmp = getResourceSecondsMap();
+ if (tmp.containsKey(ResourceInformation.VCORES.getName())) {
+ return tmp.get(ResourceInformation.VCORES.getName());
+ }
+ return 0;
}
@Override
public synchronized void setPreemptedMemorySeconds(
long preemptedMemorySeconds) {
- maybeInitBuilder();
- builder.setPreemptedMemorySeconds(preemptedMemorySeconds);
+ getPreemptedResourceSecondsMap()
+ .put(ResourceInformation.MEMORY_MB.getName(), preemptedMemorySeconds);
}
@Override
public synchronized long getPreemptedMemorySeconds() {
- ApplicationResourceUsageReportProtoOrBuilder p =
- viaProto ? proto : builder;
- return p.getPreemptedMemorySeconds();
+ Map<String, Long> tmp = getPreemptedResourceSecondsMap();
+ if (tmp.containsKey(ResourceInformation.MEMORY_MB.getName())) {
+ return tmp.get(ResourceInformation.MEMORY_MB.getName());
+ }
+ return 0;
}
@Override
public synchronized void setPreemptedVcoreSeconds(
long vcoreSeconds) {
- maybeInitBuilder();
- builder.setPreemptedVcoreSeconds(vcoreSeconds);
+ getPreemptedResourceSecondsMap()
+ .put(ResourceInformation.VCORES.getName(), vcoreSeconds);
}
@Override
public synchronized long getPreemptedVcoreSeconds() {
- ApplicationResourceUsageReportProtoOrBuilder p =
- viaProto ? proto : builder;
- return (p.getPreemptedVcoreSeconds());
+ Map<String, Long> tmp = getPreemptedResourceSecondsMap();
+ if (tmp.containsKey(ResourceInformation.VCORES.getName())) {
+ return tmp.get(ResourceInformation.VCORES.getName());
+ }
+ return 0;
}
private ResourcePBImpl convertFromProtoFormat(ResourceProto p) {
@@ -277,4 +313,81 @@ extends ApplicationResourceUsageReport {
maybeInitBuilder();
builder.setClusterUsagePercentage((clusterUsagePerc));
}
+
+ @Override
+ public synchronized void setResourceSecondsMap(
+ Map<String, Long> resourceSecondsMap) {
+ this.resourceSecondsMap = resourceSecondsMap;
+ if (resourceSecondsMap == null) {
+ return;
+ }
+ if (!resourceSecondsMap
+ .containsKey(ResourceInformation.MEMORY_MB.getName())) {
+ this.setMemorySeconds(0L);
+ }
+ if (!resourceSecondsMap.containsKey(ResourceInformation.VCORES.getName())) {
+ this.setVcoreSeconds(0L);
+ }
+ }
+
+ @Override
+ public synchronized Map<String, Long> getResourceSecondsMap() {
+ if (this.resourceSecondsMap != null) {
+ return this.resourceSecondsMap;
+ }
+ ApplicationResourceUsageReportProtoOrBuilder p = viaProto ? proto : builder;
+ this.resourceSecondsMap = ProtoUtils
+ .convertStringLongMapProtoListToMap(
+ p.getApplicationResourceUsageMapList());
+ if (!this.resourceSecondsMap
+ .containsKey(ResourceInformation.MEMORY_MB.getName())) {
+ this.setMemorySeconds(p.getMemorySeconds());
+ }
+ if (!this.resourceSecondsMap
+ .containsKey(ResourceInformation.VCORES.getName())) {
+ this.setVcoreSeconds(p.getVcoreSeconds());
+ }
+ this.setMemorySeconds(p.getMemorySeconds());
+ this.setVcoreSeconds(p.getVcoreSeconds());
+ return this.resourceSecondsMap;
+ }
+
+ @Override
+ public synchronized void setPreemptedResourceSecondsMap(
+ Map<String, Long> preemptedResourceSecondsMap) {
+ this.preemptedResourceSecondsMap = preemptedResourceSecondsMap;
+ if (preemptedResourceSecondsMap == null) {
+ return;
+ }
+ if (!preemptedResourceSecondsMap
+ .containsKey(ResourceInformation.MEMORY_MB.getName())) {
+ this.setPreemptedMemorySeconds(0L);
+ }
+ if (!preemptedResourceSecondsMap
+ .containsKey(ResourceInformation.VCORES.getName())) {
+ this.setPreemptedVcoreSeconds(0L);
+ }
+ }
+
+ @Override
+ public synchronized Map<String, Long> getPreemptedResourceSecondsMap() {
+ if (this.preemptedResourceSecondsMap != null) {
+ return this.preemptedResourceSecondsMap;
+ }
+ ApplicationResourceUsageReportProtoOrBuilder p = viaProto ? proto : builder;
+ this.preemptedResourceSecondsMap = ProtoUtils
+ .convertStringLongMapProtoListToMap(
+ p.getApplicationPreemptedResourceUsageMapList());
+ if (!this.preemptedResourceSecondsMap
+ .containsKey(ResourceInformation.MEMORY_MB.getName())) {
+ this.setPreemptedMemorySeconds(p.getPreemptedMemorySeconds());
+ }
+ if (!this.preemptedResourceSecondsMap
+ .containsKey(ResourceInformation.VCORES.getName())) {
+ this.setPreemptedVcoreSeconds(p.getPreemptedVcoreSeconds());
+ }
+ this.setPreemptedMemorySeconds(p.getPreemptedMemorySeconds());
+ this.setPreemptedVcoreSeconds(p.getPreemptedVcoreSeconds());
+ return this.preemptedResourceSecondsMap;
+ }
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
index 662db5e..e8b0594 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
@@ -19,6 +19,10 @@
package org.apache.hadoop.yarn.api.records.impl.pb;
import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
import org.apache.hadoop.classification.InterfaceAudience.Private;
import org.apache.hadoop.classification.InterfaceStability.Unstable;
@@ -46,6 +50,7 @@ import org.apache.hadoop.yarn.api.records.QueueACL;
import org.apache.hadoop.yarn.api.records.QueueState;
import org.apache.hadoop.yarn.api.records.ReservationRequestInterpreter;
import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceInformation;
import org.apache.hadoop.yarn.api.records.UpdateContainerError;
import org.apache.hadoop.yarn.api.records.UpdateContainerRequest;
import org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState;
@@ -462,6 +467,35 @@ public class ProtoUtils {
public static ResourceTypes convertFromProtoFormat(ResourceTypesProto e) {
return ResourceTypes.valueOf(e.name());
}
+
+ public static Map<String, Long> convertStringLongMapProtoListToMap(
+ List<YarnProtos.StringLongMapProto> pList) {
+ Resource tmp = Resource.newInstance(0, 0);
+ Map<String, Long> ret = new HashMap<>();
+ for (Map.Entry<String, ResourceInformation> entry : tmp.getResources()
+ .entrySet()) {
+ ret.put(entry.getKey(), 0L);
+ }
+ if (pList != null) {
+ for (YarnProtos.StringLongMapProto p : pList) {
+ ret.put(p.getKey(), p.getValue());
+ }
+ }
+ return ret;
+ }
+
+ public static List<YarnProtos.StringLongMapProto> convertMapToStringLongMapProtoList(
+ Map<String, Long> map) {
+ List<YarnProtos.StringLongMapProto> ret = new ArrayList<>();
+ for (Map.Entry<String, Long> entry : map.entrySet()) {
+ YarnProtos.StringLongMapProto.Builder tmp =
+ YarnProtos.StringLongMapProto.newBuilder();
+ tmp.setKey(entry.getKey());
+ tmp.setValue(entry.getValue());
+ ret.add(tmp.build());
+ }
+ return ret;
+ }
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/StringHelper.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/StringHelper.java
index 0240fbc..01e8951 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/StringHelper.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/StringHelper.java
@@ -20,9 +20,15 @@ package org.apache.hadoop.yarn.util;
import com.google.common.base.Joiner;
import com.google.common.base.Splitter;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
import java.util.regex.Pattern;
import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.yarn.api.records.ResourceInformation;
+import org.apache.hadoop.yarn.util.resource.ResourceUtils;
/**
* Common string manipulation helpers
@@ -174,4 +180,34 @@ public final class StringHelper {
}
sb.append(part);
}
+
+ public static String getResourceSecondsString(Map<String, Long> targetMap) {
+ List<String> strings = new ArrayList<>(targetMap.size());
+ //completed app report in the timeline server doesn't have usage report
+ Long memorySeconds = 0L;
+ Long vcoreSeconds = 0L;
+ if (targetMap.containsKey(ResourceInformation.MEMORY_MB.getName())) {
+ memorySeconds = targetMap.get(ResourceInformation.MEMORY_MB.getName());
+ }
+ if (targetMap.containsKey(ResourceInformation.VCORES.getName())) {
+ vcoreSeconds = targetMap.get(ResourceInformation.VCORES.getName());
+ }
+ strings.add(memorySeconds + " MB-seconds");
+ strings.add(vcoreSeconds + " vcore-seconds");
+ Map<String, ResourceInformation> tmp = ResourceUtils.getResourceTypes();
+ if (targetMap.size() > 2) {
+ for (Map.Entry<String, Long> entry : targetMap.entrySet()) {
+ if (!entry.getKey().equals(ResourceInformation.MEMORY_MB.getName())
+ && !entry.getKey().equals(ResourceInformation.VCORES.getName())) {
+ String units = "";
+ if (tmp.containsKey(entry.getKey())) {
+ units = tmp.get(entry.getKey()).getUnits();
+ }
+ strings.add(entry.getValue() + " " + entry.getKey() + "-" + units
+ + "seconds");
+ }
+ }
+ }
+ return Joiner.on(", ").join(strings);
+ }
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/BasePBImplRecordsTest.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/BasePBImplRecordsTest.java
index 82170b3..8694651 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/BasePBImplRecordsTest.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/BasePBImplRecordsTest.java
@@ -37,6 +37,9 @@ public class BasePBImplRecordsTest {
@SuppressWarnings("checkstyle:visibilitymodifier")
protected static HashMap<Type, Object> typeValueCache =
new HashMap<Type, Object>();
+ @SuppressWarnings("checkstyle:visibilitymodifier")
+ protected static HashMap<Type, List<String>> excludedPropertiesMap =
+ new HashMap<>();
private static Random rand = new Random();
private static byte [] bytes = new byte[] {'1', '2', '3', '4'};
@@ -167,6 +170,10 @@ public class BasePBImplRecordsTest {
private <R> Map<String, GetSetPair> getGetSetPairs(Class<R> recordClass)
throws Exception {
Map<String, GetSetPair> ret = new HashMap<String, GetSetPair>();
+ List<String> excluded = null;
+ if (excludedPropertiesMap.containsKey(recordClass.getClass())) {
+ excluded = excludedPropertiesMap.get(recordClass.getClass());
+ }
Method [] methods = recordClass.getDeclaredMethods();
// get all get methods
for (int i = 0; i < methods.length; i++) {
@@ -224,6 +231,11 @@ public class BasePBImplRecordsTest {
(gsp.setMethod == null)) {
LOG.info(String.format("Exclude potential property: %s\n", gsp.propertyName));
itr.remove();
+ } else if ((excluded != null && excluded.contains(gsp.propertyName))) {
+ LOG.info(String.format(
+ "Excluding potential property(present in exclusion list): %s\n",
+ gsp.propertyName));
+ itr.remove();
} else {
LOG.info(String.format("New property: %s type: %s", gsp.toString(), gsp.type));
gsp.testValue = genTypeValue(gsp.type);
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java
index 1aed065..f71cd3f 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java
@@ -336,6 +336,8 @@ import org.junit.Ignore;
import org.junit.Test;
import com.google.common.collect.ImmutableSet;
+import java.io.IOException;
+import java.util.Arrays;
/**
* Test class for YARN API protocol records.
@@ -738,6 +740,8 @@ public class TestPBImplRecords extends BasePBImplRecordsTest {
@Test
public void testApplicationResourceUsageReportPBImpl() throws Exception {
+ excludedPropertiesMap.put(ApplicationResourceUsageReportPBImpl.class.getClass(),
+ Arrays.asList("PreemptedResourceSecondsMap", "ResourceSecondsMap"));
validatePBImplRecord(ApplicationResourceUsageReportPBImpl.class,
ApplicationResourceUsageReportProto.class);
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
index b4c91f9..a35f038 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
@@ -46,6 +46,7 @@ import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
import org.apache.hadoop.yarn.api.records.NodeId;
import org.apache.hadoop.yarn.api.records.Priority;
import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceInformation;
import org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState;
import org.apache.hadoop.yarn.api.records.YarnApplicationState;
import org.apache.hadoop.yarn.api.records.timeline.TimelineEntities;
@@ -338,9 +339,20 @@ public class ApplicationHistoryManagerOnTimelineStore extends AbstractService
ApplicationMetricsConstants.APP_MEM_PREEMPT_METRICS);
long preemptedVcoreSeconds = parseLong(entityInfo,
ApplicationMetricsConstants.APP_CPU_PREEMPT_METRICS);
- appResources = ApplicationResourceUsageReport.newInstance(0, 0, null,
- null, null, memorySeconds, vcoreSeconds, 0, 0,
- preemptedMemorySeconds, preemptedVcoreSeconds);
+ Map<String, Long> resourceSecondsMap = new HashMap<>();
+ Map<String, Long> preemptedResoureSecondsMap = new HashMap<>();
+ resourceSecondsMap
+ .put(ResourceInformation.MEMORY_MB.getName(), memorySeconds);
+ resourceSecondsMap
+ .put(ResourceInformation.VCORES.getName(), vcoreSeconds);
+ preemptedResoureSecondsMap.put(ResourceInformation.MEMORY_MB.getName(),
+ preemptedMemorySeconds);
+ preemptedResoureSecondsMap
+ .put(ResourceInformation.VCORES.getName(), preemptedVcoreSeconds);
+
+ appResources = ApplicationResourceUsageReport
+ .newInstance(0, 0, null, null, null, resourceSecondsMap, 0, 0,
+ preemptedResoureSecondsMap);
}
if (entityInfo.containsKey(ApplicationMetricsConstants.APP_TAGS_INFO)) {
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java
index cd146a1..66d95bd 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java
@@ -65,8 +65,6 @@ import org.apache.hadoop.yarn.api.records.ResourceUtilization;
import org.apache.hadoop.yarn.api.records.Token;
import org.apache.hadoop.yarn.api.records.URL;
import org.apache.hadoop.yarn.api.records.YarnApplicationState;
-import org.apache.hadoop.yarn.conf.YarnConfiguration;
-import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.factories.RecordFactory;
import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider;
import org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager;
@@ -435,12 +433,12 @@ public class BuilderUtils {
queue, priority, amContainer, isUnmanagedAM, cancelTokensWhenComplete,
maxAppAttempts, resource, null);
}
-
+
public static ApplicationResourceUsageReport newApplicationResourceUsageReport(
int numUsedContainers, int numReservedContainers, Resource usedResources,
- Resource reservedResources, Resource neededResources, long memorySeconds,
- long vcoreSeconds, long preemptedMemorySeconds,
- long preemptedVcoreSeconds) {
+ Resource reservedResources, Resource neededResources,
+ Map<String, Long> resourceSecondsMap,
+ Map<String, Long> preemptedResourceSecondsMap) {
ApplicationResourceUsageReport report =
recordFactory.newRecordInstance(ApplicationResourceUsageReport.class);
report.setNumUsedContainers(numUsedContainers);
@@ -448,10 +446,8 @@ public class BuilderUtils {
report.setUsedResources(usedResources);
report.setReservedResources(reservedResources);
report.setNeededResources(neededResources);
- report.setMemorySeconds(memorySeconds);
- report.setVcoreSeconds(vcoreSeconds);
- report.setPreemptedMemorySeconds(preemptedMemorySeconds);
- report.setPreemptedVcoreSeconds(preemptedVcoreSeconds);
+ report.setResourceSecondsMap(resourceSecondsMap);
+ report.setPreemptedResourceSecondsMap(preemptedResourceSecondsMap);
return report;
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
index 3b47db2..806d431 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
@@ -69,6 +69,7 @@ import org.apache.hadoop.yarn.util.Times;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.util.concurrent.Futures;
import com.google.common.util.concurrent.SettableFuture;
+import org.apache.hadoop.yarn.util.StringHelper;
/**
* This class manages the list of applications for the resource manager.
@@ -188,7 +189,12 @@ public class RMAppManager implements EventHandler<RMAppManagerEvent>,
.add("preemptedAMContainers", metrics.getNumAMContainersPreempted())
.add("preemptedNonAMContainers", metrics.getNumNonAMContainersPreempted())
.add("preemptedResources", metrics.getResourcePreempted())
- .add("applicationType", app.getApplicationType());
+ .add("applicationType", app.getApplicationType())
+ .add("resourceSeconds", StringHelper
+ .getResourceSecondsString(metrics.getResourceSecondsMap()))
+ .add("preemptedResourceSeconds", StringHelper
+ .getResourceSecondsString(
+ metrics.getPreemptedResourceSecondsMap()));
return summary;
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
index 35b0c98..f802b8e 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
@@ -478,7 +478,7 @@ public class RMServerUtils {
DUMMY_APPLICATION_RESOURCE_USAGE_REPORT =
BuilderUtils.newApplicationResourceUsageReport(-1, -1,
Resources.createResource(-1, -1), Resources.createResource(-1, -1),
- Resources.createResource(-1, -1), 0, 0, 0, 0);
+ Resources.createResource(-1, -1), new HashMap<String, Long>(), new HashMap<String, Long>());
/**
@@ -622,4 +622,12 @@ public class RMServerUtils {
return labelsToNodes.get(label);
}
}
+
+ public static Long getOrDefault(Map<String, Long> map, String key,
+ Long defaultValue) {
+ if (map.containsKey(key)) {
+ return map.get(key);
+ }
+ return defaultValue;
+ }
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
index e8ed0b7..f0ab324 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
@@ -853,11 +853,8 @@ public abstract class RMStateStore extends AbstractService {
appAttempt.getAppAttemptId(),
appAttempt.getMasterContainer(),
credentials, appAttempt.getStartTime(),
- resUsage.getMemorySeconds(),
- resUsage.getVcoreSeconds(),
- attempMetrics.getPreemptedMemory(),
- attempMetrics.getPreemptedVcore()
- );
+ resUsage.getResourceUsageSecondsMap(),
+ attempMetrics.getPreemptedResourceSecondsMap());
getRMStateStoreEventHandler().handle(
new RMStateStoreAppAttemptEvent(attemptState));
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/records/ApplicationAttemptStateData.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/records/ApplicationAttemptStateData.java
index 67aaf94..2de071a 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/records/ApplicationAttemptStateData.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/records/ApplicationAttemptStateData.java
@@ -25,23 +25,28 @@ import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
import org.apache.hadoop.yarn.api.records.Container;
import org.apache.hadoop.yarn.api.records.ContainerExitStatus;
import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
+import org.apache.hadoop.yarn.api.records.ResourceInformation;
import org.apache.hadoop.yarn.proto.YarnServerResourceManagerRecoveryProtos.ApplicationAttemptStateDataProto;
+import org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils;
import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptState;
import org.apache.hadoop.yarn.util.Records;
+import java.util.Map;
+
/*
* Contains the state data that needs to be persisted for an ApplicationAttempt
*/
@Public
@Unstable
public abstract class ApplicationAttemptStateData {
+
public static ApplicationAttemptStateData newInstance(
ApplicationAttemptId attemptId, Container container,
Credentials attemptTokens, long startTime, RMAppAttemptState finalState,
String finalTrackingUrl, String diagnostics,
FinalApplicationStatus amUnregisteredFinalStatus, int exitStatus,
- long finishTime, long memorySeconds, long vcoreSeconds,
- long preemptedMemorySeconds, long preemptedVcoreSeconds) {
+ long finishTime, Map<String, Long> resourceSecondsMap,
+ Map<String, Long> preemptedResourceSecondsMap) {
ApplicationAttemptStateData attemptStateData =
Records.newRecord(ApplicationAttemptStateData.class);
attemptStateData.setAttemptId(attemptId);
@@ -54,23 +59,33 @@ public abstract class ApplicationAttemptStateData {
attemptStateData.setFinalApplicationStatus(amUnregisteredFinalStatus);
attemptStateData.setAMContainerExitStatus(exitStatus);
attemptStateData.setFinishTime(finishTime);
- attemptStateData.setMemorySeconds(memorySeconds);
- attemptStateData.setVcoreSeconds(vcoreSeconds);
- attemptStateData.setPreemptedMemorySeconds(preemptedMemorySeconds);
- attemptStateData.setPreemptedVcoreSeconds(preemptedVcoreSeconds);
+ attemptStateData.setMemorySeconds(RMServerUtils
+ .getOrDefault(resourceSecondsMap,
+ ResourceInformation.MEMORY_MB.getName(), 0L));
+ attemptStateData.setVcoreSeconds(RMServerUtils
+ .getOrDefault(resourceSecondsMap, ResourceInformation.VCORES.getName(),
+ 0L));
+ attemptStateData.setPreemptedMemorySeconds(RMServerUtils
+ .getOrDefault(preemptedResourceSecondsMap,
+ ResourceInformation.MEMORY_MB.getName(), 0L));
+ attemptStateData.setPreemptedVcoreSeconds(RMServerUtils
+ .getOrDefault(preemptedResourceSecondsMap,
+ ResourceInformation.VCORES.getName(), 0L));
+ attemptStateData.setResourceSecondsMap(resourceSecondsMap);
+ attemptStateData
+ .setPreemptedResourceSecondsMap(preemptedResourceSecondsMap);
return attemptStateData;
}
public static ApplicationAttemptStateData newInstance(
ApplicationAttemptId attemptId, Container masterContainer,
- Credentials attemptTokens, long startTime, long memorySeconds,
- long vcoreSeconds, long preemptedMemorySeconds,
- long preemptedVcoreSeconds) {
- return newInstance(attemptId, masterContainer, attemptTokens,
- startTime, null, "N/A", "", null, ContainerExitStatus.INVALID, 0,
- memorySeconds, vcoreSeconds,
- preemptedMemorySeconds, preemptedVcoreSeconds);
- }
+ Credentials attemptTokens, long startTime,
+ Map<String, Long> resourceSeondsMap,
+ Map<String, Long> preemptedResourceSecondsMap) {
+ return newInstance(attemptId, masterContainer, attemptTokens, startTime,
+ null, "N/A", "", null, ContainerExitStatus.INVALID, 0,
+ resourceSeondsMap, preemptedResourceSecondsMap);
+ }
public abstract ApplicationAttemptStateDataProto getProto();
@@ -215,4 +230,50 @@ public abstract class ApplicationAttemptStateData {
@Public
@Unstable
public abstract void setPreemptedVcoreSeconds(long vcoreSeconds);
+
+ /**
+ * Get the aggregated number of resources preempted that the application has
+ * allocated times the number of seconds the application has been running.
+ *
+ * @return map containing the resource name and aggregated preempted
+ * resource-seconds
+ */
+ @Public
+ @Unstable
+ public abstract Map<String, Long> getResourceSecondsMap();
+
+ /**
+ * Set the aggregated number of resources that the application has
+ * allocated times the number of seconds the application has been running.
+ *
+ * @param resourceSecondsMap map containing the resource name and aggregated
+ * resource-seconds
+ */
+ @Public
+ @Unstable
+ public abstract void setResourceSecondsMap(
+ Map<String, Long> resourceSecondsMap);
+
+ /**
+ * Get the aggregated number of resources preempted that the application has
+ * allocated times the number of seconds the application has been running.
+ *
+ * @return map containing the resource name and aggregated preempted
+ * resource-seconds
+ */
+ @Public
+ @Unstable
+ public abstract Map<String, Long> getPreemptedResourceSecondsMap();
+
+ /**
+ * Set the aggregated number of resources preempted that the application has
+ * allocated times the number of seconds the application has been running.
+ *
+ * @param preemptedResourceSecondsMap map containing the resource name and
+ * aggregated preempted resource-seconds
+ */
+ @Public
+ @Unstable
+ public abstract void setPreemptedResourceSecondsMap(
+ Map<String, Long> preemptedResourceSecondsMap);
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/records/impl/pb/ApplicationAttemptStateDataPBImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/records/impl/pb/ApplicationAttemptStateDataPBImpl.java
index e89726f..ed71ea2 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/records/impl/pb/ApplicationAttemptStateDataPBImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/records/impl/pb/ApplicationAttemptStateDataPBImpl.java
@@ -20,6 +20,7 @@ package org.apache.hadoop.yarn.server.resourcemanager.recovery.records.impl.pb;
import java.io.IOException;
import java.nio.ByteBuffer;
+import java.util.Map;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
@@ -55,6 +56,9 @@ public class ApplicationAttemptStateDataPBImpl extends
private Container masterContainer = null;
private ByteBuffer appAttemptTokens = null;
+ private Map<String, Long> resourceSecondsMap;
+ private Map<String, Long> preemptedResourceSecondsMap;
+
public ApplicationAttemptStateDataPBImpl() {
builder = ApplicationAttemptStateDataProto.newBuilder();
}
@@ -404,4 +408,50 @@ public class ApplicationAttemptStateDataPBImpl extends
IOUtils.closeStream(dibb);
}
}
+
+ @Override
+ public Map<String, Long> getResourceSecondsMap() {
+ if (this.resourceSecondsMap != null) {
+ return this.resourceSecondsMap;
+ }
+ ApplicationAttemptStateDataProtoOrBuilder p = viaProto ? proto : builder;
+ this.resourceSecondsMap = ProtoUtils.convertStringLongMapProtoListToMap(
+ p.getApplicationResourceUsageMapList());
+ return this.resourceSecondsMap;
+ }
+
+ @Override
+ public void setResourceSecondsMap(Map<String, Long> resourceSecondsMap) {
+ maybeInitBuilder();
+ builder.clearApplicationResourceUsageMap();
+ this.resourceSecondsMap = resourceSecondsMap;
+ if (resourceSecondsMap != null) {
+ builder.addAllApplicationResourceUsageMap(
+ ProtoUtils.convertMapToStringLongMapProtoList(resourceSecondsMap));
+ }
+ }
+
+ @Override
+ public Map<String, Long> getPreemptedResourceSecondsMap() {
+ if (this.preemptedResourceSecondsMap != null) {
+ return this.preemptedResourceSecondsMap;
+ }
+ ApplicationAttemptStateDataProtoOrBuilder p = viaProto ? proto : builder;
+ this.preemptedResourceSecondsMap = ProtoUtils
+ .convertStringLongMapProtoListToMap(
+ p.getApplicationResourceUsageMapList());
+ return this.preemptedResourceSecondsMap;
+ }
+
+ @Override
+ public void setPreemptedResourceSecondsMap(
+ Map<String, Long> preemptedResourceSecondsMap) {
+ maybeInitBuilder();
+ builder.clearPreemptedResourceUsageMap();
+ this.preemptedResourceSecondsMap = preemptedResourceSecondsMap;
+ if (preemptedResourceSecondsMap != null) {
+ builder.addAllPreemptedResourceUsageMap(ProtoUtils
+ .convertMapToStringLongMapProtoList(preemptedResourceSecondsMap));
+ }
+ }
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
index af5ef5e..e152782 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
@@ -749,14 +749,10 @@ public class RMAppImpl implements RMApp, Recoverable {
}
RMAppMetrics rmAppMetrics = getRMAppMetrics();
- appUsageReport.setMemorySeconds(rmAppMetrics.getMemorySeconds());
- appUsageReport.setVcoreSeconds(rmAppMetrics.getVcoreSeconds());
- appUsageReport.
- setPreemptedMemorySeconds(rmAppMetrics.
- getPreemptedMemorySeconds());
- appUsageReport.
- setPreemptedVcoreSeconds(rmAppMetrics.
- getPreemptedVcoreSeconds());
+ appUsageReport
+ .setResourceSecondsMap(rmAppMetrics.getResourceSecondsMap());
+ appUsageReport.setPreemptedResourceSecondsMap(
+ rmAppMetrics.getPreemptedResourceSecondsMap());
}
if (currentApplicationAttemptId == null) {
@@ -1617,10 +1613,9 @@ public class RMAppImpl implements RMApp, Recoverable {
Resource resourcePreempted = Resource.newInstance(0, 0);
int numAMContainerPreempted = 0;
int numNonAMContainerPreempted = 0;
- long memorySeconds = 0;
- long vcoreSeconds = 0;
- long preemptedMemorySeconds = 0;
- long preemptedVcoreSeconds = 0;
+ Map<String, Long> resourceSecondsMap = new HashMap<>();
+ Map<String, Long> preemptedSecondsMap = new HashMap<>();
+
this.readLock.lock();
try {
for (RMAppAttempt attempt : attempts.values()) {
@@ -1636,20 +1631,28 @@ public class RMAppImpl implements RMApp, Recoverable {
// for both running and finished containers.
AggregateAppResourceUsage resUsage =
attempt.getRMAppAttemptMetrics().getAggregateAppResourceUsage();
- memorySeconds += resUsage.getMemorySeconds();
- vcoreSeconds += resUsage.getVcoreSeconds();
- preemptedMemorySeconds += attemptMetrics.getPreemptedMemory();
- preemptedVcoreSeconds += attemptMetrics.getPreemptedVcore();
+ for (Map.Entry<String, Long> entry : resUsage
+ .getResourceUsageSecondsMap().entrySet()) {
+ long value = RMServerUtils
+ .getOrDefault(resourceSecondsMap, entry.getKey(), 0L);
+ value += entry.getValue();
+ resourceSecondsMap.put(entry.getKey(), value);
+ }
+ for (Map.Entry<String, Long> entry : attemptMetrics
+ .getPreemptedResourceSecondsMap().entrySet()) {
+ long value = RMServerUtils
+ .getOrDefault(preemptedSecondsMap, entry.getKey(), 0L);
+ value += entry.getValue();
+ preemptedSecondsMap.put(entry.getKey(), value);
+ }
}
}
} finally {
this.readLock.unlock();
}
- return new RMAppMetrics(resourcePreempted,
- numNonAMContainerPreempted, numAMContainerPreempted,
- memorySeconds, vcoreSeconds,
- preemptedMemorySeconds, preemptedVcoreSeconds);
+ return new RMAppMetrics(resourcePreempted, numNonAMContainerPreempted,
+ numAMContainerPreempted, resourceSecondsMap, preemptedSecondsMap);
}
@Private
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppMetrics.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppMetrics.java
index fa068ea..2bb7fd1 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppMetrics.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppMetrics.java
@@ -19,27 +19,27 @@
package org.apache.hadoop.yarn.server.resourcemanager.rmapp;
import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceInformation;
+import org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils;
+
+import java.util.Map;
public class RMAppMetrics {
final Resource resourcePreempted;
final int numNonAMContainersPreempted;
final int numAMContainersPreempted;
- final long memorySeconds;
- final long vcoreSeconds;
- private final long preemptedMemorySeconds;
- private final long preemptedVcoreSeconds;
+ private final Map<String, Long> resourceSecondsMap;
+ private final Map<String, Long> preemptedResourceSecondsMap;
public RMAppMetrics(Resource resourcePreempted,
int numNonAMContainersPreempted, int numAMContainersPreempted,
- long memorySeconds, long vcoreSeconds, long preemptedMemorySeconds,
- long preemptedVcoreSeconds) {
+ Map<String, Long> resourceSecondsMap,
+ Map<String, Long> preemptedResourceSecondsMap) {
this.resourcePreempted = resourcePreempted;
this.numNonAMContainersPreempted = numNonAMContainersPreempted;
this.numAMContainersPreempted = numAMContainersPreempted;
- this.memorySeconds = memorySeconds;
- this.vcoreSeconds = vcoreSeconds;
- this.preemptedMemorySeconds = preemptedMemorySeconds;
- this.preemptedVcoreSeconds = preemptedVcoreSeconds;
+ this.resourceSecondsMap = resourceSecondsMap;
+ this.preemptedResourceSecondsMap = preemptedResourceSecondsMap;
}
public Resource getResourcePreempted() {
@@ -55,19 +55,32 @@ public class RMAppMetrics {
}
public long getMemorySeconds() {
- return memorySeconds;
+ return RMServerUtils.getOrDefault(resourceSecondsMap,
+ ResourceInformation.MEMORY_MB.getName(), 0L);
}
public long getVcoreSeconds() {
- return vcoreSeconds;
+ return RMServerUtils
+ .getOrDefault(resourceSecondsMap, ResourceInformation.VCORES.getName(),
+ 0L);
}
public long getPreemptedMemorySeconds() {
- return preemptedMemorySeconds;
+ return RMServerUtils.getOrDefault(preemptedResourceSecondsMap,
+ ResourceInformation.MEMORY_MB.getName(), 0L);
}
public long getPreemptedVcoreSeconds() {
- return preemptedVcoreSeconds;
+ return RMServerUtils.getOrDefault(preemptedResourceSecondsMap,
+ ResourceInformation.VCORES.getName(), 0L);
+ }
+
+ public Map<String, Long> getResourceSecondsMap() {
+ return resourceSecondsMap;
+ }
+
+ public Map<String, Long> getPreemptedResourceSecondsMap() {
+ return preemptedResourceSecondsMap;
}
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/AggregateAppResourceUsage.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/AggregateAppResourceUsage.java
index f0c2b34..b858712 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/AggregateAppResourceUsage.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/AggregateAppResourceUsage.java
@@ -19,42 +19,38 @@
package org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt;
import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.yarn.api.records.ResourceInformation;
+import org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils;
+
+import java.util.HashMap;
+import java.util.Map;
@Private
public class AggregateAppResourceUsage {
- long memorySeconds;
- long vcoreSeconds;
+ private Map<String, Long> resourceSecondsMap = new HashMap<>();
- public AggregateAppResourceUsage(long memorySeconds, long vcoreSeconds) {
- this.memorySeconds = memorySeconds;
- this.vcoreSeconds = vcoreSeconds;
+ public AggregateAppResourceUsage(Map<String, Long> resourceSecondsMap) {
+ this.resourceSecondsMap.putAll(resourceSecondsMap);
}
/**
* @return the memorySeconds
*/
public long getMemorySeconds() {
- return memorySeconds;
- }
-
- /**
- * @param memorySeconds the memorySeconds to set
- */
- public void setMemorySeconds(long memorySeconds) {
- this.memorySeconds = memorySeconds;
+ return RMServerUtils.getOrDefault(resourceSecondsMap,
+ ResourceInformation.MEMORY_MB.getName(), 0L);
}
/**
* @return the vcoreSeconds
*/
public long getVcoreSeconds() {
- return vcoreSeconds;
+ return RMServerUtils
+ .getOrDefault(resourceSecondsMap, ResourceInformation.VCORES.getName(),
+ 0L);
}
- /**
- * @param vcoreSeconds the vcoreSeconds to set
- */
- public void setVcoreSeconds(long vcoreSeconds) {
- this.vcoreSeconds = vcoreSeconds;
+ public Map<String, Long> getResourceUsageSecondsMap() {
+ return resourceSecondsMap;
}
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
index 07f949d..b9dbefc 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
@@ -939,12 +939,9 @@ public class RMAppAttemptImpl implements RMAppAttempt, Recoverable {
}
AggregateAppResourceUsage resUsage =
this.attemptMetrics.getAggregateAppResourceUsage();
- report.setMemorySeconds(resUsage.getMemorySeconds());
- report.setVcoreSeconds(resUsage.getVcoreSeconds());
- report.setPreemptedMemorySeconds(
- this.attemptMetrics.getPreemptedMemory());
- report.setPreemptedVcoreSeconds(
- this.attemptMetrics.getPreemptedVcore());
+ report.setResourceSecondsMap(resUsage.getResourceUsageSecondsMap());
+ report.setPreemptedResourceSecondsMap(
+ this.attemptMetrics.getPreemptedResourceSecondsMap());
return report;
} finally {
this.readLock.unlock();
@@ -981,11 +978,10 @@ public class RMAppAttemptImpl implements RMAppAttempt, Recoverable {
this.finalStatus = attemptState.getFinalApplicationStatus();
this.startTime = attemptState.getStartTime();
this.finishTime = attemptState.getFinishTime();
- this.attemptMetrics.updateAggregateAppResourceUsage(
- attemptState.getMemorySeconds(), attemptState.getVcoreSeconds());
+ this.attemptMetrics
+ .updateAggregateAppResourceUsage(attemptState.getResourceSecondsMap());
this.attemptMetrics.updateAggregatePreemptedAppResourceUsage(
- attemptState.getPreemptedMemorySeconds(),
- attemptState.getPreemptedVcoreSeconds());
+ attemptState.getPreemptedResourceSecondsMap());
}
public void transferStateFromAttempt(RMAppAttempt attempt) {
@@ -1361,16 +1357,12 @@ public class RMAppAttemptImpl implements RMAppAttempt, Recoverable {
RMStateStore rmStore = rmContext.getStateStore();
setFinishTime(System.currentTimeMillis());
- ApplicationAttemptStateData attemptState =
- ApplicationAttemptStateData.newInstance(
- applicationAttemptId, getMasterContainer(),
- rmStore.getCredentialsFromAppAttempt(this),
- startTime, stateToBeStored, finalTrackingUrl, diags.toString(),
- finalStatus, exitStatus,
- getFinishTime(), resUsage.getMemorySeconds(),
- resUsage.getVcoreSeconds(),
- this.attemptMetrics.getPreemptedMemory(),
- this.attemptMetrics.getPreemptedVcore());
+ ApplicationAttemptStateData attemptState = ApplicationAttemptStateData
+ .newInstance(applicationAttemptId, getMasterContainer(),
+ rmStore.getCredentialsFromAppAttempt(this), startTime,
+ stateToBeStored, finalTrackingUrl, diags.toString(), finalStatus, exitStatus,
+ getFinishTime(), resUsage.getResourceUsageSecondsMap(),
+ this.attemptMetrics.getPreemptedResourceSecondsMap());
LOG.info("Updating application attempt " + applicationAttemptId
+ " with final state: " + targetedFinalState + ", and exit status: "
+ exitStatus);
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptMetrics.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptMetrics.java
index 0655609..8ef6d99 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptMetrics.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptMetrics.java
@@ -18,6 +18,8 @@
package org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt;
+import java.util.HashMap;
+import java.util.Map;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;
@@ -25,11 +27,13 @@ import java.util.concurrent.locks.ReentrantReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock.ReadLock;
import java.util.concurrent.locks.ReentrantReadWriteLock.WriteLock;
+import org.apache.commons.lang.time.DateUtils;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
import org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport;
import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceInformation;
import org.apache.hadoop.yarn.server.resourcemanager.RMContext;
import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp;
import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
@@ -49,10 +53,8 @@ public class RMAppAttemptMetrics {
private ReadLock readLock;
private WriteLock writeLock;
- private AtomicLong finishedMemorySeconds = new AtomicLong(0);
- private AtomicLong finishedVcoreSeconds = new AtomicLong(0);
- private AtomicLong preemptedMemorySeconds = new AtomicLong(0);
- private AtomicLong preemptedVcoreSeconds = new AtomicLong(0);
+ private Map<String, AtomicLong> resourceUsageMap = new HashMap<>();
+ private Map<String, AtomicLong> preemptedResourceMap = new HashMap<>();
private RMContext rmContext;
private int[][] localityStatistics =
@@ -102,11 +104,16 @@ public class RMAppAttemptMetrics {
}
public long getPreemptedMemory() {
- return preemptedMemorySeconds.get();
+ return preemptedResourceMap.get(ResourceInformation.MEMORY_MB.getName())
+ .get();
}
public long getPreemptedVcore() {
- return preemptedVcoreSeconds.get();
+ return preemptedResourceMap.get(ResourceInformation.VCORES.getName()).get();
+ }
+
+ public Map<String, Long> getPreemptedResourceSecondsMap() {
+ return convertAtomicLongMaptoLongMap(preemptedResourceMap);
}
public int getNumNonAMContainersPreempted() {
@@ -122,35 +129,90 @@ public class RMAppAttemptMetrics {
}
public AggregateAppResourceUsage getAggregateAppResourceUsage() {
- long memorySeconds = finishedMemorySeconds.get();
- long vcoreSeconds = finishedVcoreSeconds.get();
+ Map<String, Long> resourcesUsed =
+ convertAtomicLongMaptoLongMap(resourceUsageMap);
// Only add in the running containers if this is the active attempt.
RMApp rmApp = rmContext.getRMApps().get(attemptId.getApplicationId());
- if (null != rmApp) {
- RMAppAttempt currentAttempt = rmApp.getCurrentAppAttempt();
+ if (rmApp != null) {
+ RMAppAttempt currentAttempt = rmContext.getRMApps().get(attemptId.getApplicationId()).getCurrentAppAttempt();
if (currentAttempt.getAppAttemptId().equals(attemptId)) {
- ApplicationResourceUsageReport appResUsageReport = rmContext
- .getScheduler().getAppResourceUsageReport(attemptId);
+ ApplicationResourceUsageReport appResUsageReport =
+ rmContext.getScheduler().getAppResourceUsageReport(attemptId);
if (appResUsageReport != null) {
- memorySeconds += appResUsageReport.getMemorySeconds();
- vcoreSeconds += appResUsageReport.getVcoreSeconds();
+ Map<String, Long> tmp = appResUsageReport.getResourceSecondsMap();
+ for (Map.Entry<String, Long> entry : tmp.entrySet()) {
+ if (resourcesUsed.containsKey(entry.getKey())) {
+ Long value = resourcesUsed.get(entry.getKey());
+ value += entry.getValue();
+ resourcesUsed.put(entry.getKey(), value);
+ } else{
+ resourcesUsed.put(entry.getKey(), entry.getValue());
+ }
+ }
}
}
}
- return new AggregateAppResourceUsage(memorySeconds, vcoreSeconds);
+ return new AggregateAppResourceUsage(resourcesUsed);
+ }
+
+ public void updateAggregateAppResourceUsage(Resource allocated,
+ long deltaUsedMillis) {
+ updateUsageMap(allocated, deltaUsedMillis, resourceUsageMap);
+ }
+
+ public void updateAggregatePreemptedAppResourceUsage(Resource allocated,
+ long deltaUsedMillis) {
+ updateUsageMap(allocated, deltaUsedMillis, preemptedResourceMap);
}
- public void updateAggregateAppResourceUsage(long finishedMemorySeconds,
- long finishedVcoreSeconds) {
- this.finishedMemorySeconds.addAndGet(finishedMemorySeconds);
- this.finishedVcoreSeconds.addAndGet(finishedVcoreSeconds);
+ public void updateAggregateAppResourceUsage(
+ Map<String, Long> resourceSecondsMap) {
+ updateUsageMap(resourceSecondsMap, resourceUsageMap);
}
public void updateAggregatePreemptedAppResourceUsage(
- long preemptedMemorySeconds, long preemptedVcoreSeconds) {
- this.preemptedMemorySeconds.addAndGet(preemptedMemorySeconds);
- this.preemptedVcoreSeconds.addAndGet(preemptedVcoreSeconds);
+ Map<String, Long> preemptedResourceSecondsMap) {
+ updateUsageMap(preemptedResourceSecondsMap, preemptedResourceMap);
+ }
+
+ private void updateUsageMap(Resource allocated, long deltaUsedMillis,
+ Map<String, AtomicLong> targetMap) {
+ for (Map.Entry<String, ResourceInformation> entry : allocated.getResources()
+ .entrySet()) {
+ AtomicLong resourceUsed;
+ if (!targetMap.containsKey(entry.getKey())) {
+ resourceUsed = new AtomicLong(0);
+ targetMap.put(entry.getKey(), resourceUsed);
+
+ }
+ resourceUsed = targetMap.get(entry.getKey());
+ resourceUsed.addAndGet((entry.getValue().getValue() * deltaUsedMillis)
+ / DateUtils.MILLIS_PER_SECOND);
+ }
+ }
+
+ private void updateUsageMap(Map<String, Long> sourceMap,
+ Map<String, AtomicLong> targetMap) {
+ for (Map.Entry<String, Long> entry : sourceMap.entrySet()) {
+ AtomicLong resourceUsed;
+ if (!targetMap.containsKey(entry.getKey())) {
+ resourceUsed = new AtomicLong(0);
+ targetMap.put(entry.getKey(), resourceUsed);
+
+ }
+ resourceUsed = targetMap.get(entry.getKey());
+ resourceUsed.set(entry.getValue());
+ }
+ }
+
+ private Map<String, Long> convertAtomicLongMaptoLongMap(
+ Map<String, AtomicLong> source) {
+ Map<String, Long> ret = new HashMap<>();
+ for (Map.Entry<String, AtomicLong> entry : source.entrySet()) {
+ ret.put(entry.getKey(), entry.getValue().get());
+ }
+ return ret;
}
public void incNumAllocatedContainers(NodeType containerType,
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
index b185495..76933f1 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
@@ -25,7 +25,6 @@ import java.util.concurrent.locks.ReentrantReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock.ReadLock;
import java.util.concurrent.locks.ReentrantReadWriteLock.WriteLock;
-import org.apache.commons.lang.time.DateUtils;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
@@ -716,20 +715,15 @@ public class RMContainerImpl implements RMContainer {
if (rmAttempt != null) {
long usedMillis = container.finishTime - container.creationTime;
- long memorySeconds = resource.getMemorySize()
- * usedMillis / DateUtils.MILLIS_PER_SECOND;
- long vcoreSeconds = resource.getVirtualCores()
- * usedMillis / DateUtils.MILLIS_PER_SECOND;
rmAttempt.getRMAppAttemptMetrics()
- .updateAggregateAppResourceUsage(memorySeconds,vcoreSeconds);
+ .updateAggregateAppResourceUsage(resource, usedMillis);
// If this is a preempted container, update preemption metrics
if (ContainerExitStatus.PREEMPTED == container.finishedStatus
- .getExitStatus()) {
- rmAttempt.getRMAppAttemptMetrics().updatePreemptionInfo(resource,
- container);
+ .getExitStatus()) {
rmAttempt.getRMAppAttemptMetrics()
- .updateAggregatePreemptedAppResourceUsage(memorySeconds,
- vcoreSeconds);
+ .updatePreemptionInfo(resource, container);
+ rmAttempt.getRMAppAttemptMetrics()
+ .updateAggregatePreemptedAppResourceUsage(resource, usedMillis);
}
}
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
index 05dc834..1785dfb 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
@@ -55,11 +55,13 @@ import org.apache.hadoop.yarn.api.records.NMToken;
import org.apache.hadoop.yarn.api.records.NodeId;
import org.apache.hadoop.yarn.api.records.Priority;
import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceInformation;
import org.apache.hadoop.yarn.api.records.ResourceRequest;
import org.apache.hadoop.yarn.api.records.UpdateContainerError;
import org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager;
import org.apache.hadoop.yarn.server.api.ContainerType;
import org.apache.hadoop.yarn.server.resourcemanager.RMContext;
+import org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils;
import org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager;
import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp;
import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.AggregateAppResourceUsage;
@@ -107,9 +109,7 @@ public class SchedulerApplicationAttempt implements SchedulableEntity {
private static final long MEM_AGGREGATE_ALLOCATION_CACHE_MSECS = 3000;
protected long lastMemoryAggregateAllocationUpdateTime = 0;
- private long lastMemorySeconds = 0;
- private long lastVcoreSeconds = 0;
-
+ private Map<String, Long> lastResourceSecondsMap = new HashMap<>();
protected final AppSchedulingInfo appSchedulingInfo;
protected ApplicationAttemptId attemptId;
protected Map<ContainerId, RMContainer> liveContainers =
@@ -1002,22 +1002,24 @@ public class SchedulerApplicationAttempt implements SchedulableEntity {
// recently.
if ((currentTimeMillis - lastMemoryAggregateAllocationUpdateTime)
> MEM_AGGREGATE_ALLOCATION_CACHE_MSECS) {
- long memorySeconds = 0;
- long vcoreSeconds = 0;
+ Map<String, Long> resourceSecondsMap = new HashMap<>();
for (RMContainer rmContainer : this.liveContainers.values()) {
long usedMillis = currentTimeMillis - rmContainer.getCreationTime();
Resource resource = rmContainer.getContainer().getResource();
- memorySeconds += resource.getMemorySize() * usedMillis /
- DateUtils.MILLIS_PER_SECOND;
- vcoreSeconds += resource.getVirtualCores() * usedMillis
- / DateUtils.MILLIS_PER_SECOND;
+ for (Map.Entry<String, ResourceInformation> entry : resource
+ .getResources().entrySet()) {
+ long value = RMServerUtils
+ .getOrDefault(resourceSecondsMap, entry.getKey(), 0L);
+ value += entry.getValue().getValue() * usedMillis
+ / DateUtils.MILLIS_PER_SECOND;
+ resourceSecondsMap.put(entry.getKey(), value);
+ }
}
lastMemoryAggregateAllocationUpdateTime = currentTimeMillis;
- lastMemorySeconds = memorySeconds;
- lastVcoreSeconds = vcoreSeconds;
+ lastResourceSecondsMap = resourceSecondsMap;
}
- return new AggregateAppResourceUsage(lastMemorySeconds, lastVcoreSeconds);
+ return new AggregateAppResourceUsage(lastResourceSecondsMap);
}
public ApplicationResourceUsageReport getResourceUsageReport() {
@@ -1032,6 +1034,11 @@ public class SchedulerApplicationAttempt implements SchedulableEntity {
Resource cluster = rmContext.getScheduler().getClusterResource();
ResourceCalculator calc =
rmContext.getScheduler().getResourceCalculator();
+ Map<String, Long> preemptedResourceSecondsMaps = new HashMap<>();
+ preemptedResourceSecondsMaps
+ .put(ResourceInformation.MEMORY_MB.getName(), 0L);
+ preemptedResourceSecondsMaps
+ .put(ResourceInformation.VCORES.getName(), 0L);
float queueUsagePerc = 0.0f;
float clusterUsagePerc = 0.0f;
if (!calc.isInvalidDivisor(cluster)) {
@@ -1041,15 +1048,15 @@ public class SchedulerApplicationAttempt implements SchedulableEntity {
queueUsagePerc = calc.divide(cluster, usedResourceClone,
Resources.multiply(cluster, queueCapacityPerc)) * 100;
}
- clusterUsagePerc = calc.divide(cluster, usedResourceClone, cluster)
- * 100;
+ clusterUsagePerc =
+ calc.divide(cluster, usedResourceClone, cluster) * 100;
}
- return ApplicationResourceUsageReport.newInstance(liveContainers.size(),
- reservedContainers.size(), usedResourceClone, reservedResourceClone,
- Resources.add(usedResourceClone, reservedResourceClone),
- runningResourceUsage.getMemorySeconds(),
- runningResourceUsage.getVcoreSeconds(), queueUsagePerc,
- clusterUsagePerc, 0, 0);
+ return ApplicationResourceUsageReport
+ .newInstance(liveContainers.size(), reservedContainers.size(),
+ usedResourceClone, reservedResourceClone,
+ Resources.add(usedResourceClone, reservedResourceClone),
+ runningResourceUsage.getResourceUsageSecondsMap(), queueUsagePerc,
+ clusterUsagePerc, preemptedResourceSecondsMaps);
} finally {
writeLock.unlock();
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppBlock.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppBlock.java
index 3f774e5..bbc35fc 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppBlock.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppBlock.java
@@ -44,6 +44,7 @@ import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttempt;
import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics;
import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.AppAttemptInfo;
import org.apache.hadoop.yarn.server.webapp.AppBlock;
+import org.apache.hadoop.yarn.util.StringHelper;
import org.apache.hadoop.yarn.util.resource.Resources;
import org.apache.hadoop.yarn.webapp.hamlet.Hamlet;
import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.DIV;
@@ -106,15 +107,12 @@ public class RMAppBlock extends AppBlock{
attemptResourcePreempted)
._("Number of Non-AM Containers Preempted from Current Attempt:",
attemptNumNonAMContainerPreempted)
- ._("Aggregate Resource Allocation:",
- String.format("%d MB-seconds, %d vcore-seconds",
- appMetrics == null ? "N/A" : appMetrics.getMemorySeconds(),
- appMetrics == null ? "N/A" : appMetrics.getVcoreSeconds()))
+ ._("Aggregate Resource Allocation:", appMetrics == null ? "N/A" :
+ StringHelper
+ .getResourceSecondsString(appMetrics.getResourceSecondsMap()))
._("Aggregate Preempted Resource Allocation:",
- String.format("%d MB-seconds, %d vcore-seconds",
- appMetrics == null ? "N/A" : appMetrics.getPreemptedMemorySeconds(),
- appMetrics == null ? "N/A" :
- appMetrics.getPreemptedVcoreSeconds()));
+ appMetrics == null ? "N/A" : StringHelper.getResourceSecondsString(
+ appMetrics.getPreemptedResourceSecondsMap()));
pdiv._();
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java
index 7fed3df..bff8612 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java
@@ -101,6 +101,7 @@ public class AppInfo {
private long vcoreSeconds;
protected float queueUsagePercentage;
protected float clusterUsagePercentage;
+ protected Map<String, Long> resourceSecondsMap;
// preemption info fields
private long preemptedResourceMB;
@@ -109,6 +110,7 @@ public class AppInfo {
private int numAMContainerPreempted;
private long preemptedMemorySeconds;
private long preemptedVcoreSeconds;
+ protected Map<String, Long> preemptedResourceSecondsMap;
// list of resource requests
@XmlElement(name = "resourceRequests")
@@ -234,8 +236,10 @@ public class AppInfo {
appMetrics.getResourcePreempted().getVirtualCores();
memorySeconds = appMetrics.getMemorySeconds();
vcoreSeconds = appMetrics.getVcoreSeconds();
+ resourceSecondsMap = appMetrics.getResourceSecondsMap();
preemptedMemorySeconds = appMetrics.getPreemptedMemorySeconds();
preemptedVcoreSeconds = appMetrics.getPreemptedVcoreSeconds();
+ preemptedResourceSecondsMap = appMetrics.getPreemptedResourceSecondsMap();
ApplicationSubmissionContext appSubmissionContext =
app.getApplicationSubmissionContext();
unmanagedApplication = appSubmissionContext.getUnmanagedAM();
@@ -454,6 +458,22 @@ public class AppInfo {
return this.reservedVCores;
}
+ public long getPreemptedMB() {
+ return preemptedResourceMB;
+ }
+
+ public long getPreemptedVCores() {
+ return preemptedResourceVCores;
+ }
+
+ public int getNumNonAMContainersPreempted() {
+ return numNonAMContainerPreempted;
+ }
+
+ public int getNumAMContainersPreempted() {
+ return numAMContainerPreempted;
+ }
+
public long getMemorySeconds() {
return memorySeconds;
}
@@ -462,6 +482,10 @@ public class AppInfo {
return vcoreSeconds;
}
+ public Map<String, Long> getResourceSecondsMap() {
+ return resourceSecondsMap;
+ }
+
public long getPreemptedMemorySeconds() {
return preemptedMemorySeconds;
}
@@ -470,6 +494,10 @@ public class AppInfo {
return preemptedVcoreSeconds;
}
+ public Map<String, Long> getPreemptedResourceSecondsMap() {
+ return preemptedResourceSecondsMap;
+ }
+
public List<ResourceRequestInfo> getResourceRequests() {
return this.resourceRequests;
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ResourceInfo.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ResourceInfo.java
index 5083943..e13980a 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ResourceInfo.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ResourceInfo.java
@@ -20,46 +20,68 @@ package org.apache.hadoop.yarn.server.resourcemanager.webapp.dao;
import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;
import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.util.resource.Resources;
@XmlRootElement
-@XmlAccessorType(XmlAccessType.FIELD)
+@XmlAccessorType(XmlAccessType.NONE)
public class ResourceInfo {
+
+ @XmlElement
long memory;
+ @XmlElement
int vCores;
-
+
+ private Resource resources;
+
public ResourceInfo() {
}
public ResourceInfo(Resource res) {
memory = res.getMemorySize();
vCores = res.getVirtualCores();
+ resources = Resources.clone(res);
}
public long getMemorySize() {
- return memory;
+ if (resources == null) {
+ resources = Resource.newInstance(memory, vCores);
+ }
+ return resources.getMemorySize();
}
public int getvCores() {
- return vCores;
+ if (resources == null) {
+ resources = Resource.newInstance(memory, vCores);
+ }
+ return resources.getVirtualCores();
}
-
+
@Override
public String toString() {
- return "<memory:" + memory + ", vCores:" + vCores + ">";
+ return resources.toString();
}
public void setMemory(int memory) {
+ if (resources == null) {
+ resources = Resource.newInstance(memory, vCores);
+ }
this.memory = memory;
+ resources.setMemorySize(memory);
}
public void setvCores(int vCores) {
+ if (resources == null) {
+ resources = Resource.newInstance(memory, vCores);
+ }
this.vCores = vCores;
+ resources.setVirtualCores(vCores);
}
public Resource getResource() {
- return Resource.newInstance(memory, vCores);
+ return Resource.newInstance(resources);
}
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
index cf93edd..887b854 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
@@ -73,7 +73,7 @@ public class SchedulerInfo {
}
public String getSchedulerResourceTypes() {
- return this.schedulingResourceTypes.toString();
+ return minAllocResource.getResource().getResources().keySet().toString();
}
public int getMaxClusterLevelAppPriority() {
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/proto/yarn_server_resourcemanager_recovery.proto b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/proto/yarn_server_resourcemanager_recovery.proto
index 247cd21..39a56a8 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/proto/yarn_server_resourcemanager_recovery.proto
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/proto/yarn_server_resourcemanager_recovery.proto
@@ -87,6 +87,8 @@ message ApplicationAttemptStateDataProto {
optional int64 finish_time = 12;
optional int64 preempted_memory_seconds = 13;
optional int64 preempted_vcore_seconds = 14;
+ repeated StringLongMapProto application_resource_usage_map = 15;
+ repeated StringLongMapProto preempted_resource_usage_map = 16;
}
message EpochProto {
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
index f93659f..2a9f74b 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
@@ -35,6 +35,7 @@ import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import java.util.concurrent.ConcurrentMap;
import org.apache.commons.logging.Log;
@@ -56,6 +57,7 @@ import org.apache.hadoop.yarn.api.records.ExecutionType;
import org.apache.hadoop.yarn.api.records.ExecutionTypeRequest;
import org.apache.hadoop.yarn.api.records.Priority;
import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceInformation;
import org.apache.hadoop.yarn.api.records.ResourceRequest;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.event.AsyncDispatcher;
@@ -827,9 +829,12 @@ public class TestAppManager{
when(app.getState()).thenReturn(RMAppState.RUNNING);
when(app.getApplicationType()).thenReturn("MAPREDUCE");
when(app.getSubmitTime()).thenReturn(1000L);
+ Map<String, Long> resourceSecondsMap = new HashMap<>();
+ resourceSecondsMap.put(ResourceInformation.MEMORY_MB.getName(), 16384L);
+ resourceSecondsMap.put(ResourceInformation.VCORES.getName(), 64L);
RMAppMetrics metrics =
new RMAppMetrics(Resource.newInstance(1234, 56),
- 10, 1, 16384, 64, 0, 0);
+ 10, 1, resourceSecondsMap, new HashMap<String, Long>());
when(app.getRMAppMetrics()).thenReturn(metrics);
RMAppManager.ApplicationSummary.SummaryBuilder summary =
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestContainerResourceUsage.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestContainerResourceUsage.java
index 11fe056..3508ab4 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestContainerResourceUsage.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestContainerResourceUsage.java
@@ -23,6 +23,7 @@ import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import org.apache.commons.lang.time.DateUtils;
import org.apache.hadoop.security.UserGroupInformation;
@@ -32,6 +33,7 @@ import org.apache.hadoop.yarn.api.records.Container;
import org.apache.hadoop.yarn.api.records.ContainerId;
import org.apache.hadoop.yarn.api.records.ContainerState;
import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceInformation;
import org.apache.hadoop.yarn.api.records.ResourceRequest;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.server.resourcemanager.recovery.MemoryRMStateStore;
@@ -424,6 +426,9 @@ public class TestContainerResourceUsage {
* usedMillis / DateUtils.MILLIS_PER_SECOND;
long vcoreSeconds = resource.getVirtualCores()
* usedMillis / DateUtils.MILLIS_PER_SECOND;
- return new AggregateAppResourceUsage(memorySeconds, vcoreSeconds);
+ Map<String, Long> map = new HashMap<>();
+ map.put(ResourceInformation.MEMORY_MB.getName(), memorySeconds);
+ map.put(ResourceInformation.VCORES.getName(), vcoreSeconds);
+ return new AggregateAppResourceUsage(map);
}
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java
index f826631..cdb3ba1 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager;
import java.util.Collection;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -189,7 +190,8 @@ public abstract class MockAsm extends MockApps {
@Override
public RMAppMetrics getRMAppMetrics() {
- return new RMAppMetrics(Resource.newInstance(0, 0), 0, 0, 0, 0, 0, 0);
+ return new RMAppMetrics(Resource.newInstance(0, 0), 0, 0, new HashMap<String, Long>(),
+ new HashMap<String, Long>());
}
@Override
@@ -337,8 +339,9 @@ public abstract class MockAsm extends MockApps {
public ApplicationReport createAndGetApplicationReport(
String clientUserName, boolean allowAccess) {
ApplicationResourceUsageReport usageReport =
- ApplicationResourceUsageReport.newInstance(0, 0, null, null, null,
- 0, 0, 0, 0, 0, 0);
+ ApplicationResourceUsageReport
+ .newInstance(0, 0, null, null, null, new HashMap<String, Long>(), 0, 0,
+ new HashMap<String, Long>());
ApplicationReport report = ApplicationReport.newInstance(
getApplicationId(), appAttemptId, getUser(), getQueue(),
getName(), null, 0, null, null, getDiagnostics().toString(),
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
index 5321916..36e35d4 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
@@ -24,6 +24,7 @@ import static org.mockito.Mockito.when;
import java.util.Collection;
import java.util.Collections;
import java.util.EnumSet;
+import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
@@ -40,6 +41,7 @@ import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
import org.apache.hadoop.yarn.api.records.NodeId;
import org.apache.hadoop.yarn.api.records.Priority;
import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceInformation;
import org.apache.hadoop.yarn.api.records.ResourceRequest;
import org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState;
import org.apache.hadoop.yarn.api.records.YarnApplicationState;
@@ -505,9 +507,16 @@ public class TestSystemMetricsPublisher {
when(app.getCurrentAppAttempt()).thenReturn(appAttempt);
when(app.getFinalApplicationStatus()).thenReturn(
FinalApplicationStatus.UNDEFINED);
- when(app.getRMAppMetrics()).thenReturn(
- new RMAppMetrics(null, 0, 0, Integer.MAX_VALUE, Long.MAX_VALUE,
- Integer.MAX_VALUE, Long.MAX_VALUE));
+ Map<String, Long> resourceMap = new HashMap<>();
+ resourceMap
+ .put(ResourceInformation.MEMORY_MB.getName(), (long) Integer.MAX_VALUE);
+ resourceMap.put(ResourceInformation.VCORES.getName(), Long.MAX_VALUE);
+ Map<String, Long> preemptedMap = new HashMap<>();
+ preemptedMap
+ .put(ResourceInformation.MEMORY_MB.getName(), (long) Integer.MAX_VALUE);
+ preemptedMap.put(ResourceInformation.VCORES.getName(), Long.MAX_VALUE);
+ when(app.getRMAppMetrics())
+ .thenReturn(new RMAppMetrics(null, 0, 0, resourceMap, preemptedMap));
Set<String> appTags = new HashSet<String>();
appTags.add("test");
appTags.add("tags");
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisherForV2.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisherForV2.java
index ad71f6e..3aeafc9 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisherForV2.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisherForV2.java
@@ -29,6 +29,8 @@ import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
@@ -46,6 +48,7 @@ import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
import org.apache.hadoop.yarn.api.records.NodeId;
import org.apache.hadoop.yarn.api.records.Priority;
import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceInformation;
import org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity;
import org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntityType;
import org.apache.hadoop.yarn.api.records.timelineservice.TimelineEvent;
@@ -358,15 +361,20 @@ public class TestSystemMetricsPublisherForV2 {
when(app.getDiagnostics()).thenReturn(
new StringBuilder("test diagnostics info"));
RMAppAttempt appAttempt = mock(RMAppAttempt.class);
- when(appAttempt.getAppAttemptId()).thenReturn(
- ApplicationAttemptId.newInstance(appId, 1));
+ when(appAttempt.getAppAttemptId())
+ .thenReturn(ApplicationAttemptId.newInstance(appId, 1));
when(app.getCurrentAppAttempt()).thenReturn(appAttempt);
- when(app.getFinalApplicationStatus()).thenReturn(
- FinalApplicationStatus.UNDEFINED);
+ when(app.getFinalApplicationStatus())
+ .thenReturn(FinalApplicationStatus.UNDEFINED);
+ Map<String, Long> resourceSecondsMap = new HashMap<>();
+ resourceSecondsMap
+ .put(ResourceInformation.MEMORY_MB.getName(), (long) Integer.MAX_VALUE);
+ resourceSecondsMap
+ .put(ResourceInformation.VCORES.getName(), Long.MAX_VALUE);
when(app.getRMAppMetrics()).thenReturn(
- new RMAppMetrics(Resource.newInstance(0, 0), 0, 0, Integer.MAX_VALUE,
- Long.MAX_VALUE, Long.MAX_VALUE, Long.MAX_VALUE));
- when(app.getApplicationTags()).thenReturn(Collections.<String> emptySet());
+ new RMAppMetrics(Resource.newInstance(0, 0), 0, 0, resourceSecondsMap,
+ new HashMap<String, Long>()));
+ when(app.getApplicationTags()).thenReturn(Collections.<String>emptySet());
ApplicationSubmissionContext appSubmissionContext =
mock(ApplicationSubmissionContext.class);
when(appSubmissionContext.getPriority())
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java
index fdac89c..a1339f6 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java
@@ -194,7 +194,7 @@ public class RMStateStoreTestBase {
when(mockAttempt.getRMAppAttemptMetrics())
.thenReturn(mockRmAppAttemptMetrics);
when(mockRmAppAttemptMetrics.getAggregateAppResourceUsage())
- .thenReturn(new AggregateAppResourceUsage(0, 0));
+ .thenReturn(new AggregateAppResourceUsage(new HashMap<String, Long>()));
dispatcher.attemptId = attemptId;
store.storeNewApplicationAttempt(mockAttempt);
waitNotify(dispatcher);
@@ -292,7 +292,7 @@ public class RMStateStoreTestBase {
when(mockRemovedAttempt.getRMAppAttemptMetrics())
.thenReturn(mockRmAppAttemptMetrics);
when(mockRmAppAttemptMetrics.getAggregateAppResourceUsage())
- .thenReturn(new AggregateAppResourceUsage(0,0));
+ .thenReturn(new AggregateAppResourceUsage(new HashMap<String, Long>()));
attempts.put(attemptIdRemoved, mockRemovedAttempt);
store.removeApplication(mockRemovedApp);
@@ -369,7 +369,7 @@ public class RMStateStoreTestBase {
oldAttemptState.getStartTime(), RMAppAttemptState.FINISHED,
"myTrackingUrl", "attemptDiagnostics",
FinalApplicationStatus.SUCCEEDED, 100,
- oldAttemptState.getFinishTime(), 0, 0, 0, 0);
+ oldAttemptState.getFinishTime(), new HashMap<String, Long>(), new HashMap<String, Long>());
store.updateApplicationAttemptState(newAttemptState);
// test updating the state of an app/attempt whose initial state was not
@@ -393,7 +393,7 @@ public class RMStateStoreTestBase {
oldAttemptState.getStartTime(), RMAppAttemptState.FINISHED,
"myTrackingUrl", "attemptDiagnostics",
FinalApplicationStatus.SUCCEEDED, 111,
- oldAttemptState.getFinishTime(), 0, 0, 0, 0);
+ oldAttemptState.getFinishTime(), new HashMap<String, Long>(), new HashMap<String, Long>());
store.updateApplicationAttemptState(dummyAttempt);
// let things settle down
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
index fbacac0..5d8f1e1 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
@@ -34,12 +34,7 @@ import org.apache.hadoop.security.token.Token;
import org.apache.hadoop.security.token.delegation.DelegationKey;
import org.apache.hadoop.service.Service;
import org.apache.hadoop.test.GenericTestUtils;
-import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
-import org.apache.hadoop.yarn.api.records.ApplicationId;
-import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
-import org.apache.hadoop.yarn.api.records.Container;
-import org.apache.hadoop.yarn.api.records.ContainerId;
-import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
+import org.apache.hadoop.yarn.api.records.*;
import org.apache.hadoop.yarn.api.records.impl.pb.ApplicationSubmissionContextPBImpl;
import org.apache.hadoop.yarn.api.records.impl.pb.ContainerPBImpl;
import org.apache.hadoop.yarn.conf.HAUtil;
@@ -554,7 +549,7 @@ public class TestZKRMStateStore extends RMStateStoreTestBase {
when(mockAttempt.getRMAppAttemptMetrics())
.thenReturn(mockRmAppAttemptMetrics);
when(mockRmAppAttemptMetrics.getAggregateAppResourceUsage())
- .thenReturn(new AggregateAppResourceUsage(0,0));
+ .thenReturn(new AggregateAppResourceUsage(new HashMap<String, Long>()));
store.storeNewApplicationAttempt(mockAttempt);
assertEquals("RMStateStore should have been in fenced state",
true, store.isFencedState());
@@ -566,7 +561,7 @@ public class TestZKRMStateStore extends RMStateStoreTestBase {
store.getCredentialsFromAppAttempt(mockAttempt),
startTime, RMAppAttemptState.FINISHED, "testUrl",
"test", FinalApplicationStatus.SUCCEEDED, 100,
- finishTime, 0, 0, 0, 0);
+ finishTime, new HashMap<String, Long>(), new HashMap<String, Long>());
store.updateApplicationAttemptState(newAttemptState);
assertEquals("RMStateStore should have been in fenced state",
true, store.isFencedState());
@@ -803,10 +798,20 @@ public class TestZKRMStateStore extends RMStateStoreTestBase {
private static ApplicationAttemptStateData createFinishedAttempt(
ApplicationAttemptId attemptId, Container container, long startTime,
int amExitStatus) {
+ Map<String, Long> resourceSecondsMap = new HashMap<>();
+ Map<String, Long> preemptedResoureSecondsMap = new HashMap<>();
+ resourceSecondsMap
+ .put(ResourceInformation.MEMORY_MB.getName(), 0L);
+ resourceSecondsMap
+ .put(ResourceInformation.VCORES.getName(), 0L);
+ preemptedResoureSecondsMap.put(ResourceInformation.MEMORY_MB.getName(),
+ 0L);
+ preemptedResoureSecondsMap
+ .put(ResourceInformation.VCORES.getName(), 0L);
return ApplicationAttemptStateData.newInstance(attemptId,
container, null, startTime, RMAppAttemptState.FINISHED,
"myTrackingUrl", "attemptDiagnostics", FinalApplicationStatus.SUCCEEDED,
- amExitStatus, 0, 0, 0, 0, 0);
+ amExitStatus, 0, resourceSecondsMap, preemptedResoureSecondsMap);
}
private ApplicationAttemptId storeAttempt(RMStateStore store,
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestAppPage.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestAppPage.java
index d9ed073..e67e1cb 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestAppPage.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestAppPage.java
@@ -22,6 +22,7 @@ import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
import java.io.IOException;
+import java.util.HashMap;
import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
@@ -62,9 +63,10 @@ public class TestAppPage {
when(app.getStartTime()).thenReturn(0L);
when(app.getFinishTime()).thenReturn(0L);
when(app.createApplicationState()).thenReturn(YarnApplicationState.FAILED);
-
- RMAppMetrics appMetrics = new RMAppMetrics(
- Resource.newInstance(0, 0), 0, 0, 0, 0, 0, 0);
+
+ RMAppMetrics appMetrics =
+ new RMAppMetrics(Resource.newInstance(0, 0), 0, 0, new HashMap<String, Long>(),
+ new HashMap<String, Long>());
when(app.getRMAppMetrics()).thenReturn(appMetrics);
// initialize RM Context, and create RMApp, without creating RMAppAttempt
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebAppFairScheduler.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebAppFairScheduler.java
index 8c00b39..6bd78c1 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebAppFairScheduler.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebAppFairScheduler.java
@@ -51,6 +51,7 @@ import org.junit.Test;
import java.io.IOException;
import java.util.Arrays;
+import java.util.HashMap;
import java.util.List;
import java.util.concurrent.ConcurrentMap;
@@ -136,8 +137,8 @@ public class TestRMWebAppFairScheduler {
MockRMApp app = new MockRMApp(i, i, state) {
@Override
public RMAppMetrics getRMAppMetrics() {
- return new RMAppMetrics(Resource.newInstance(0, 0),
- 0, 0, 0, 0, 0, 0);
+ return new RMAppMetrics(Resource.newInstance(0, 0), 0, 0,
+ new HashMap<String, Long>(), new HashMap<String, Long>());
}
@Override
public YarnApplicationState createApplicationState() {
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 02/04: YARN-9182. Backport YARN-6445 resource profile
performance improvements to branch-2
Posted by jh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a commit to branch YARN-8200
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit a1861b102b46ef5b03493220a6bd2b383a3af418
Author: Sunil G <su...@apache.org>
AuthorDate: Tue Apr 25 11:53:11 2017 +0530
YARN-9182. Backport YARN-6445 resource profile performance improvements to branch-2
---
.../apache/hadoop/yarn/api/records/Resource.java | 19 +++++--
.../yarn/api/records/ResourceInformation.java | 51 ++++++++++--------
.../hadoop/yarn/util/UnitsConversionUtil.java | 34 ++++++------
.../hadoop/yarn/conf/TestResourceInformation.java | 4 +-
.../hadoop/yarn/util/TestUnitsConversionUtil.java | 60 +++++++++++-----------
.../yarn/api/records/impl/pb/ResourcePBImpl.java | 23 +++++----
.../util/resource/DominantResourceCalculator.java | 54 +++++++++----------
.../hadoop/yarn/util/resource/Resources.java | 18 +++----
.../yarn/util/resource/TestResourceUtils.java | 1 +
.../hadoop/yarn/util/resource/TestResources.java | 12 ++++-
.../scheduler/AbstractYarnScheduler.java | 20 +-------
11 files changed, 157 insertions(+), 139 deletions(-)
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Resource.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Resource.java
index 10755b4..c51357c 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Resource.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Resource.java
@@ -107,12 +107,23 @@ public abstract class Resource implements Comparable<Resource> {
@InterfaceStability.Unstable
public static Resource newInstance(Resource resource) {
Resource ret = Resource.newInstance(0, 0);
- for (Map.Entry<String, ResourceInformation> entry : resource.getResources()
+ Resource.copy(resource, ret);
+ return ret;
+ }
+
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public static void copy(Resource source, Resource dest) {
+ for (Map.Entry<String, ResourceInformation> entry : source.getResources()
.entrySet()) {
- ret.setResourceInformation(entry.getKey(),
- ResourceInformation.newInstance(entry.getValue()));
+ try {
+ ResourceInformation.copy(entry.getValue(),
+ dest.getResourceInformation(entry.getKey()));
+ } catch (YarnException ye) {
+ dest.setResourceInformation(entry.getKey(),
+ ResourceInformation.newInstance(entry.getValue()));
+ }
}
- return ret;
}
/**
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceInformation.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceInformation.java
index 7d74efc..7c0f055 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceInformation.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceInformation.java
@@ -30,9 +30,9 @@ public class ResourceInformation implements Comparable<ResourceInformation> {
private String name;
private String units;
private ResourceTypes resourceType;
- private Long value;
- private Long minimumAllocation;
- private Long maximumAllocation;
+ private long value;
+ private long minimumAllocation;
+ private long maximumAllocation;
private static final String MEMORY_URI = "memory-mb";
private static final String VCORES_URI = "vcores";
@@ -106,7 +106,7 @@ public class ResourceInformation implements Comparable<ResourceInformation> {
*
* @return the resource value
*/
- public Long getValue() {
+ public long getValue() {
return value;
}
@@ -115,7 +115,7 @@ public class ResourceInformation implements Comparable<ResourceInformation> {
*
* @param rValue the resource value
*/
- public void setValue(Long rValue) {
+ public void setValue(long rValue) {
this.value = rValue;
}
@@ -124,7 +124,7 @@ public class ResourceInformation implements Comparable<ResourceInformation> {
*
* @return the minimum allocation for the resource
*/
- public Long getMinimumAllocation() {
+ public long getMinimumAllocation() {
return minimumAllocation;
}
@@ -133,7 +133,7 @@ public class ResourceInformation implements Comparable<ResourceInformation> {
*
* @param minimumAllocation the minimum allocation for the resource
*/
- public void setMinimumAllocation(Long minimumAllocation) {
+ public void setMinimumAllocation(long minimumAllocation) {
this.minimumAllocation = minimumAllocation;
}
@@ -142,7 +142,7 @@ public class ResourceInformation implements Comparable<ResourceInformation> {
*
* @return the maximum allocation for the resource
*/
- public Long getMaximumAllocation() {
+ public long getMaximumAllocation() {
return maximumAllocation;
}
@@ -151,7 +151,7 @@ public class ResourceInformation implements Comparable<ResourceInformation> {
*
* @param maximumAllocation the maximum allocation for the resource
*/
- public void setMaximumAllocation(Long maximumAllocation) {
+ public void setMaximumAllocation(long maximumAllocation) {
this.maximumAllocation = maximumAllocation;
}
@@ -163,18 +163,13 @@ public class ResourceInformation implements Comparable<ResourceInformation> {
*/
public static ResourceInformation newInstance(ResourceInformation other) {
ResourceInformation ret = new ResourceInformation();
- ret.setName(other.getName());
- ret.setResourceType(other.getResourceType());
- ret.setUnits(other.getUnits());
- ret.setValue(other.getValue());
- ret.setMinimumAllocation(other.getMinimumAllocation());
- ret.setMaximumAllocation(other.getMaximumAllocation());
+ copy(other, ret);
return ret;
}
public static ResourceInformation newInstance(String name, String units,
- Long value, ResourceTypes type, Long minimumAllocation,
- Long maximumAllocation) {
+ long value, ResourceTypes type, long minimumAllocation,
+ long maximumAllocation) {
ResourceInformation ret = new ResourceInformation();
ret.setName(name);
ret.setResourceType(type);
@@ -186,7 +181,7 @@ public class ResourceInformation implements Comparable<ResourceInformation> {
}
public static ResourceInformation newInstance(String name, String units,
- Long value) {
+ long value) {
return ResourceInformation
.newInstance(name, units, value, ResourceTypes.COUNTABLE, 0L,
Long.MAX_VALUE);
@@ -198,7 +193,7 @@ public class ResourceInformation implements Comparable<ResourceInformation> {
Long.MAX_VALUE);
}
- public static ResourceInformation newInstance(String name, Long value) {
+ public static ResourceInformation newInstance(String name, long value) {
return ResourceInformation
.newInstance(name, "", value, ResourceTypes.COUNTABLE, 0L,
Long.MAX_VALUE);
@@ -208,6 +203,22 @@ public class ResourceInformation implements Comparable<ResourceInformation> {
return ResourceInformation.newInstance(name, "");
}
+ /**
+ * Copies the content of the source ResourceInformation object to the
+ * destination object, overwriting all properties of the destination object.
+ * @param src Source ResourceInformation object
+ * @param dst Destination ResourceInformation object
+ */
+
+ public static void copy(ResourceInformation src, ResourceInformation dst) {
+ dst.setName(src.getName());
+ dst.setResourceType(src.getResourceType());
+ dst.setUnits(src.getUnits());
+ dst.setValue(src.getValue());
+ dst.setMinimumAllocation(src.getMinimumAllocation());
+ dst.setMaximumAllocation(src.getMaximumAllocation());
+ }
+
@Override
public String toString() {
return "name: " + this.name + ", units: " + this.units + ", type: "
@@ -244,7 +255,7 @@ public class ResourceInformation implements Comparable<ResourceInformation> {
939769357 + name.hashCode(); // prime * result = 939769357 initially
result = prime * result + resourceType.hashCode();
result = prime * result + units.hashCode();
- result = prime * result + value.hashCode();
+ result = prime * result + Long.valueOf(value).hashCode();
return result;
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/UnitsConversionUtil.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/UnitsConversionUtil.java
index 47bb3df..c7663de 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/UnitsConversionUtil.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/UnitsConversionUtil.java
@@ -128,23 +128,24 @@ public class UnitsConversionUtil {
* @param fromValue the value you wish to convert
* @return the value in toUnit
*/
- public static Long convert(String fromUnit, String toUnit, Long fromValue) {
- if (toUnit == null || fromUnit == null || fromValue == null) {
+ public static long convert(String fromUnit, String toUnit, long fromValue) {
+ if (toUnit == null || fromUnit == null) {
throw new IllegalArgumentException("One or more arguments are null");
}
- String overflowMsg =
- "Converting " + fromValue + " from '" + fromUnit + "' to '" + toUnit
- + "' will result in an overflow of Long";
+
if (fromUnit.equals(toUnit)) {
return fromValue;
}
Converter fc = getConverter(fromUnit);
Converter tc = getConverter(toUnit);
- Long numerator = fc.numerator * tc.denominator;
- Long denominator = fc.denominator * tc.numerator;
- Long numeratorMultiplierLimit = Long.MAX_VALUE / numerator;
+ long numerator = fc.numerator * tc.denominator;
+ long denominator = fc.denominator * tc.numerator;
+ long numeratorMultiplierLimit = Long.MAX_VALUE / numerator;
if (numerator < denominator) {
if (numeratorMultiplierLimit < fromValue) {
+ String overflowMsg =
+ "Converting " + fromValue + " from '" + fromUnit + "' to '" + toUnit
+ + "' will result in an overflow of Long";
throw new IllegalArgumentException(overflowMsg);
}
return (fromValue * numerator) / denominator;
@@ -152,8 +153,11 @@ public class UnitsConversionUtil {
if (numeratorMultiplierLimit > fromValue) {
return (numerator * fromValue) / denominator;
}
- Long tmp = numerator / denominator;
+ long tmp = numerator / denominator;
if ((Long.MAX_VALUE / tmp) < fromValue) {
+ String overflowMsg =
+ "Converting " + fromValue + " from '" + fromUnit + "' to '" + toUnit
+ + "' will result in an overflow of Long";
throw new IllegalArgumentException(overflowMsg);
}
return fromValue * tmp;
@@ -170,8 +174,8 @@ public class UnitsConversionUtil {
* @return +1, 0 or -1 depending on whether the relationship is greater than,
* equal to or lesser than
*/
- public static int compare(String unitA, Long valueA, String unitB,
- Long valueB) {
+ public static int compare(String unitA, long valueA, String unitB,
+ long valueB) {
if (unitA == null || unitB == null || !KNOWN_UNITS.contains(unitA)
|| !KNOWN_UNITS.contains(unitB)) {
throw new IllegalArgumentException("Units cannot be null");
@@ -185,19 +189,19 @@ public class UnitsConversionUtil {
Converter unitAC = getConverter(unitA);
Converter unitBC = getConverter(unitB);
if (unitA.equals(unitB)) {
- return valueA.compareTo(valueB);
+ return Long.valueOf(valueA).compareTo(valueB);
}
int unitAPos = SORTED_UNITS.indexOf(unitA);
int unitBPos = SORTED_UNITS.indexOf(unitB);
try {
- Long tmpA = valueA;
- Long tmpB = valueB;
+ long tmpA = valueA;
+ long tmpB = valueB;
if (unitAPos < unitBPos) {
tmpB = convert(unitB, unitA, valueB);
} else {
tmpA = convert(unitA, unitB, valueA);
}
- return tmpA.compareTo(tmpB);
+ return Long.valueOf(tmpA).compareTo(tmpB);
} catch (IllegalArgumentException ie) {
BigInteger tmpA = BigInteger.valueOf(valueA);
BigInteger tmpB = BigInteger.valueOf(valueB);
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestResourceInformation.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestResourceInformation.java
index 28f69c9..2efb990 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestResourceInformation.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestResourceInformation.java
@@ -50,7 +50,7 @@ public class TestResourceInformation {
@Test
public void testValue() {
String name = "yarn.io/test";
- Long value = 1l;
+ long value = 1L;
ResourceInformation ri = ResourceInformation.newInstance(name, value);
Assert.assertEquals("Resource name incorrect", name, ri.getName());
Assert.assertEquals("Resource value incorrect", value, ri.getValue());
@@ -59,7 +59,7 @@ public class TestResourceInformation {
@Test
public void testResourceInformation() {
String name = "yarn.io/test";
- Long value = 1l;
+ long value = 1L;
String units = "m";
ResourceInformation ri =
ResourceInformation.newInstance(name, units, value);
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/util/TestUnitsConversionUtil.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/util/TestUnitsConversionUtil.java
index 0f999e8..0b512a5 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/util/TestUnitsConversionUtil.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/util/TestUnitsConversionUtil.java
@@ -27,62 +27,62 @@ public class TestUnitsConversionUtil {
public void testUnitsConversion() {
int value = 5;
String fromUnit = "";
- Long test = Long.valueOf(value);
+ long test = value;
Assert.assertEquals("pico test failed",
- Long.valueOf(value * 1000l * 1000l * 1000l * 1000l),
+ value * 1000L * 1000L * 1000L * 1000L,
UnitsConversionUtil.convert(fromUnit, "p", test));
Assert.assertEquals("nano test failed",
- Long.valueOf(value * 1000l * 1000l * 1000l),
+ value * 1000L * 1000L * 1000L,
UnitsConversionUtil.convert(fromUnit, "n", test));
Assert
- .assertEquals("micro test failed", Long.valueOf(value * 1000l * 1000l),
+ .assertEquals("micro test failed", value * 1000L * 1000L,
UnitsConversionUtil.convert(fromUnit, "u", test));
- Assert.assertEquals("milli test failed", Long.valueOf(value * 1000l),
+ Assert.assertEquals("milli test failed", value * 1000L,
UnitsConversionUtil.convert(fromUnit, "m", test));
- test = Long.valueOf(value * 1000l * 1000l * 1000l * 1000l * 1000l);
+ test = value * 1000L * 1000L * 1000L * 1000L * 1000L;
fromUnit = "";
- Assert.assertEquals("kilo test failed", Long.valueOf(test / 1000l),
+ Assert.assertEquals("kilo test failed", test / 1000L,
UnitsConversionUtil.convert(fromUnit, "k", test));
Assert
- .assertEquals("mega test failed", Long.valueOf(test / (1000l * 1000l)),
+ .assertEquals("mega test failed", test / (1000L * 1000L),
UnitsConversionUtil.convert(fromUnit, "M", test));
Assert.assertEquals("giga test failed",
- Long.valueOf(test / (1000l * 1000l * 1000l)),
+ test / (1000L * 1000L * 1000L),
UnitsConversionUtil.convert(fromUnit, "G", test));
Assert.assertEquals("tera test failed",
- Long.valueOf(test / (1000l * 1000l * 1000l * 1000l)),
+ test / (1000L * 1000L * 1000L * 1000L),
UnitsConversionUtil.convert(fromUnit, "T", test));
Assert.assertEquals("peta test failed",
- Long.valueOf(test / (1000l * 1000l * 1000l * 1000l * 1000l)),
+ test / (1000L * 1000L * 1000L * 1000L * 1000L),
UnitsConversionUtil.convert(fromUnit, "P", test));
- Assert.assertEquals("nano to pico test failed", Long.valueOf(value * 1000l),
- UnitsConversionUtil.convert("n", "p", Long.valueOf(value)));
+ Assert.assertEquals("nano to pico test failed", value * 1000L,
+ UnitsConversionUtil.convert("n", "p", value));
- Assert.assertEquals("mega to giga test failed", Long.valueOf(value),
- UnitsConversionUtil.convert("M", "G", Long.valueOf(value * 1000l)));
+ Assert.assertEquals("mega to giga test failed", value,
+ UnitsConversionUtil.convert("M", "G", value * 1000L));
- Assert.assertEquals("Mi to Gi test failed", Long.valueOf(value),
- UnitsConversionUtil.convert("Mi", "Gi", Long.valueOf(value * 1024l)));
+ Assert.assertEquals("Mi to Gi test failed", value,
+ UnitsConversionUtil.convert("Mi", "Gi", value * 1024L));
- Assert.assertEquals("Mi to Ki test failed", Long.valueOf(value * 1024),
- UnitsConversionUtil.convert("Mi", "Ki", Long.valueOf(value)));
+ Assert.assertEquals("Mi to Ki test failed", value * 1024,
+ UnitsConversionUtil.convert("Mi", "Ki", value));
- Assert.assertEquals("Ki to base units test failed", Long.valueOf(5 * 1024),
- UnitsConversionUtil.convert("Ki", "", Long.valueOf(5)));
+ Assert.assertEquals("Ki to base units test failed", 5 * 1024,
+ UnitsConversionUtil.convert("Ki", "", 5));
- Assert.assertEquals("Mi to k test failed", Long.valueOf(1073741),
- UnitsConversionUtil.convert("Mi", "k", Long.valueOf(1024)));
+ Assert.assertEquals("Mi to k test failed", 1073741,
+ UnitsConversionUtil.convert("Mi", "k", 1024));
- Assert.assertEquals("M to Mi test failed", Long.valueOf(953),
- UnitsConversionUtil.convert("M", "Mi", Long.valueOf(1000)));
+ Assert.assertEquals("M to Mi test failed", 953,
+ UnitsConversionUtil.convert("M", "Mi", 1000));
}
@Test
public void testOverflow() {
- Long test = Long.valueOf(5 * 1000l * 1000l * 1000l * 1000l * 1000l);
+ long test = 5 * 1000L * 1000L * 1000L * 1000L * 1000L;
try {
UnitsConversionUtil.convert("P", "p", test);
Assert.fail("this operation should result in an overflow");
@@ -100,9 +100,9 @@ public class TestUnitsConversionUtil {
@Test
public void testCompare() {
String unitA = "P";
- Long valueA = Long.valueOf(1);
+ long valueA = 1;
String unitB = "p";
- Long valueB = Long.valueOf(2);
+ long valueB = 2;
Assert.assertEquals(1,
UnitsConversionUtil.compare(unitA, valueA, unitB, valueB));
Assert.assertEquals(-1,
@@ -120,7 +120,7 @@ public class TestUnitsConversionUtil {
Assert.assertEquals(-1,
UnitsConversionUtil.compare(unitB, valueB, unitA, valueA));
Assert.assertEquals(0,
- UnitsConversionUtil.compare(unitA, valueA, unitB, 1000l));
+ UnitsConversionUtil.compare(unitA, valueA, unitB, 1000L));
unitA = "p";
unitB = "n";
@@ -129,7 +129,7 @@ public class TestUnitsConversionUtil {
Assert.assertEquals(1,
UnitsConversionUtil.compare(unitB, valueB, unitA, valueA));
Assert.assertEquals(0,
- UnitsConversionUtil.compare(unitA, 1000l, unitB, valueA));
+ UnitsConversionUtil.compare(unitA, 1000L, unitB, valueA));
}
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java
index 48d3178..bba5f7e 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.yarn.api.records.impl.pb;
+import org.apache.commons.collections.map.UnmodifiableMap;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.classification.InterfaceAudience.Private;
@@ -46,6 +47,7 @@ public class ResourcePBImpl extends Resource {
boolean viaProto = false;
private Map<String, ResourceInformation> resources;
+ private Map<String, ResourceInformation> readOnlyResources;
// call via ProtoUtils.convertToProtoFormat(Resource)
@@ -68,6 +70,7 @@ public class ResourcePBImpl extends Resource {
public ResourcePBImpl(ResourceProto proto) {
this.proto = proto;
viaProto = true;
+ this.readOnlyResources = null;
this.resources = null;
initResources();
}
@@ -111,10 +114,9 @@ public class ResourcePBImpl extends Resource {
@Override
public void setMemorySize(long memory) {
- setResourceInformation(ResourceInformation.MEMORY_MB.getName(),
- ResourceInformation.newInstance(ResourceInformation.MEMORY_MB.getName(),
- ResourceInformation.MEMORY_MB.getUnits(), memory));
-
+ maybeInitBuilder();
+ getResourceInformation(ResourceInformation.MEMORY_MB.getName())
+ .setValue(memory);
}
@Override
@@ -127,9 +129,9 @@ public class ResourcePBImpl extends Resource {
@Override
public void setVirtualCores(int vCores) {
- setResourceInformation(ResourceInformation.VCORES.getName(),
- ResourceInformation.newInstance(ResourceInformation.VCORES.getName(),
- ResourceInformation.VCORES.getUnits(), (long) vCores));
+ maybeInitBuilder();
+ getResourceInformation(ResourceInformation.VCORES.getName())
+ .setValue(vCores);
}
private void initResources() {
@@ -143,7 +145,7 @@ public class ResourcePBImpl extends Resource {
entry.hasType() ? ProtoUtils.convertFromProtoFormat(entry.getType()) :
ResourceTypes.COUNTABLE;
String units = entry.hasUnits() ? entry.getUnits() : "";
- Long value = entry.hasValue() ? entry.getValue() : 0L;
+ long value = entry.hasValue() ? entry.getValue() : 0L;
ResourceInformation ri = ResourceInformation
.newInstance(entry.getKey(), units, value, type, 0L, Long.MAX_VALUE);
if (resources.containsKey(ri.getName())) {
@@ -171,7 +173,7 @@ public class ResourcePBImpl extends Resource {
}
initResources();
if (resources.containsKey(resource)) {
- resources.put(resource, resourceInformation);
+ ResourceInformation.copy(resourceInformation, resources.get(resource));
}
}
@@ -193,7 +195,7 @@ public class ResourcePBImpl extends Resource {
@Override
public Map<String, ResourceInformation> getResources() {
initResources();
- return Collections.unmodifiableMap(this.resources);
+ return readOnlyResources;
}
@Override
@@ -226,6 +228,7 @@ public class ResourcePBImpl extends Resource {
resources.put(entry.getKey(),
ResourceInformation.newInstance(entry.getValue()));
}
+ readOnlyResources = Collections.unmodifiableMap(resources);
}
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.java
index 7db1da4..76bc364 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.java
@@ -154,7 +154,7 @@ public class DominantResourceCalculator extends ResourceCalculator {
clusterResource.getResourceInformation(rName);
ResourceInformation resourceInformation =
resource.getResourceInformation(rName);
- Long resourceValue = UnitsConversionUtil
+ long resourceValue = UnitsConversionUtil
.convert(resourceInformation.getUnits(),
clusterResourceResourceInformation.getUnits(),
resourceInformation.getValue());
@@ -180,11 +180,11 @@ public class DominantResourceCalculator extends ResourceCalculator {
available.getResourceInformation(resource);
ResourceInformation requiredResource =
required.getResourceInformation(resource);
- Long requiredResourceValue = UnitsConversionUtil
+ long requiredResourceValue = UnitsConversionUtil
.convert(requiredResource.getUnits(), availableResource.getUnits(),
requiredResource.getValue());
if (requiredResourceValue != 0) {
- Long tmp = availableResource.getValue() / requiredResourceValue;
+ long tmp = availableResource.getValue() / requiredResourceValue;
min = min < tmp ? min : tmp;
}
} catch (YarnException ye) {
@@ -228,7 +228,7 @@ public class DominantResourceCalculator extends ResourceCalculator {
a.getResourceInformation(resource);
ResourceInformation bResourceInformation =
b.getResourceInformation(resource);
- Long bResourceValue = UnitsConversionUtil
+ long bResourceValue = UnitsConversionUtil
.convert(bResourceInformation.getUnits(),
aResourceInformation.getUnits(),
bResourceInformation.getValue());
@@ -249,14 +249,13 @@ public class DominantResourceCalculator extends ResourceCalculator {
}
public Resource divideAndCeil(Resource numerator, long denominator) {
- Resource ret = Resources.createResource(0, 0);
+ Resource ret = Resource.newInstance(numerator);
for (String resource : resourceNames) {
try {
- ResourceInformation resourceInformation = ResourceInformation
- .newInstance(numerator.getResourceInformation(resource));
+ ResourceInformation resourceInformation =
+ ret.getResourceInformation(resource);
resourceInformation.setValue(
divideAndCeil(resourceInformation.getValue(), denominator));
- ret.setResourceInformation(resource, resourceInformation);
} catch (YarnException ye) {
throw new IllegalArgumentException(
"Error getting resource information for " + resource, ye);
@@ -276,7 +275,7 @@ public class DominantResourceCalculator extends ResourceCalculator {
@Override
public Resource normalize(Resource r, Resource minimumResource,
Resource maximumResource, Resource stepFactor) {
- Resource ret = Resources.createResource(0, 0);
+ Resource ret = Resource.newInstance(r);
for (String resource : resourceNames) {
try {
ResourceInformation rResourceInformation =
@@ -287,28 +286,26 @@ public class DominantResourceCalculator extends ResourceCalculator {
maximumResource.getResourceInformation(resource);
ResourceInformation stepFactorResourceInformation =
stepFactor.getResourceInformation(resource);
- ResourceInformation tmp =
- ResourceInformation.newInstance(rResourceInformation);
+ ResourceInformation tmp = ret.getResourceInformation(resource);
- Long rValue = rResourceInformation.getValue();
- Long minimumValue = UnitsConversionUtil
+ long rValue = rResourceInformation.getValue();
+ long minimumValue = UnitsConversionUtil
.convert(minimumResourceInformation.getUnits(),
rResourceInformation.getUnits(),
minimumResourceInformation.getValue());
- Long maximumValue = UnitsConversionUtil
+ long maximumValue = UnitsConversionUtil
.convert(maximumResourceInformation.getUnits(),
rResourceInformation.getUnits(),
maximumResourceInformation.getValue());
- Long stepFactorValue = UnitsConversionUtil
+ long stepFactorValue = UnitsConversionUtil
.convert(stepFactorResourceInformation.getUnits(),
rResourceInformation.getUnits(),
stepFactorResourceInformation.getValue());
- Long value = Math.max(rValue, minimumValue);
+ long value = Math.max(rValue, minimumValue);
if (stepFactorValue != 0) {
value = roundUp(value, stepFactorValue);
}
tmp.setValue(Math.min(value, maximumValue));
- ret.setResourceInformation(resource, tmp);
} catch (YarnException ye) {
throw new IllegalArgumentException(
"Error getting resource information for " + resource, ye);
@@ -328,28 +325,27 @@ public class DominantResourceCalculator extends ResourceCalculator {
}
private Resource rounding(Resource r, Resource stepFactor, boolean roundUp) {
- Resource ret = Resources.createResource(0, 0);
+ Resource ret = Resource.newInstance(r);
for (String resource : resourceNames) {
try {
ResourceInformation rResourceInformation =
r.getResourceInformation(resource);
ResourceInformation stepFactorResourceInformation =
stepFactor.getResourceInformation(resource);
- ResourceInformation tmp =
- ResourceInformation.newInstance(rResourceInformation);
- Long rValue = rResourceInformation.getValue();
- Long stepFactorValue = UnitsConversionUtil
+ long rValue = rResourceInformation.getValue();
+ long stepFactorValue = UnitsConversionUtil
.convert(stepFactorResourceInformation.getUnits(),
rResourceInformation.getUnits(),
stepFactorResourceInformation.getValue());
- Long value = rValue;
+ long value = rValue;
if (stepFactorValue != 0) {
value = roundUp ? roundUp(rValue, stepFactorValue) :
roundDown(rValue, stepFactorValue);
}
- tmp.setValue(value);
- ret.setResourceInformation(resource, tmp);
+ ResourceInformation
+ .copy(rResourceInformation, ret.getResourceInformation(resource));
+ ret.getResourceInformation(resource).setValue(value);
} catch (YarnException ye) {
throw new IllegalArgumentException(
"Error getting resource information for " + resource, ye);
@@ -372,15 +368,14 @@ public class DominantResourceCalculator extends ResourceCalculator {
private Resource multiplyAndNormalize(Resource r, double by,
Resource stepFactor, boolean roundUp) {
- Resource ret = Resources.createResource(0, 0);
+ Resource ret = Resource.newInstance(r);
for (String resource : resourceNames) {
try {
ResourceInformation rResourceInformation =
r.getResourceInformation(resource);
ResourceInformation stepFactorResourceInformation =
stepFactor.getResourceInformation(resource);
- ResourceInformation tmp =
- ResourceInformation.newInstance(rResourceInformation);
+ ResourceInformation tmp = ret.getResourceInformation(resource);
Long rValue = rResourceInformation.getValue();
Long stepFactorValue = UnitsConversionUtil
@@ -397,7 +392,6 @@ public class DominantResourceCalculator extends ResourceCalculator {
roundUp ? (long) Math.ceil(rValue * by) : (long) (rValue * by);
}
tmp.setValue(value);
- ret.setResourceInformation(resource, tmp);
} catch (YarnException ye) {
throw new IllegalArgumentException(
"Error getting resource information for " + resource, ye);
@@ -414,7 +408,7 @@ public class DominantResourceCalculator extends ResourceCalculator {
smaller.getResourceInformation(resource);
ResourceInformation bResourceInformation =
bigger.getResourceInformation(resource);
- Long sResourceValue = UnitsConversionUtil
+ long sResourceValue = UnitsConversionUtil
.convert(sResourceInformation.getUnits(),
bResourceInformation.getUnits(),
sResourceInformation.getValue());
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java
index b0ec907..d143e93 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java
@@ -42,7 +42,7 @@ public class Resources {
static class FixedValueResource extends Resource {
private Map<String, ResourceInformation> resources;
- private Long resourceValue;
+ private long resourceValue;
private String name;
/**
@@ -50,7 +50,7 @@ public class Resources {
* @param rName the name of the resource
* @param value the fixed value to be returned for all resource types
*/
- FixedValueResource(String rName, Long value) {
+ FixedValueResource(String rName, long value) {
this.resourceValue = value;
this.name = rName;
resources = initResourceMap();
@@ -60,7 +60,7 @@ public class Resources {
if(this.resourceValue > Integer.MAX_VALUE) {
return Integer.MAX_VALUE;
}
- return this.resourceValue.intValue();
+ return Long.valueOf(this.resourceValue).intValue();
}
@Override
@@ -203,7 +203,7 @@ public class Resources {
try {
ResourceInformation rhsValue = rhs.getResourceInformation(name);
ResourceInformation lhsValue = entry.getValue();
- Long convertedRhs = UnitsConversionUtil
+ long convertedRhs = UnitsConversionUtil
.convert(rhsValue.getUnits(), lhsValue.getUnits(),
rhsValue.getValue());
lhs.setResourceValue(name, lhsValue.getValue() + convertedRhs);
@@ -225,7 +225,7 @@ public class Resources {
try {
ResourceInformation rhsValue = rhs.getResourceInformation(name);
ResourceInformation lhsValue = entry.getValue();
- Long convertedRhs = UnitsConversionUtil
+ long convertedRhs = UnitsConversionUtil
.convert(rhsValue.getUnits(), lhsValue.getUnits(),
rhsValue.getValue());
lhs.setResourceValue(name, lhsValue.getValue() - convertedRhs);
@@ -288,7 +288,7 @@ public class Resources {
try {
ResourceInformation rhsValue = rhs.getResourceInformation(name);
ResourceInformation lhsValue = entry.getValue();
- Long convertedRhs = (long) (UnitsConversionUtil
+ long convertedRhs = (long) (UnitsConversionUtil
.convert(rhsValue.getUnits(), lhsValue.getUnits(),
rhsValue.getValue()) * by);
lhs.setResourceValue(name, lhsValue.getValue() + convertedRhs);
@@ -422,7 +422,7 @@ public class Resources {
try {
ResourceInformation rhsValue = bigger.getResourceInformation(name);
ResourceInformation lhsValue = entry.getValue();
- Long convertedRhs = UnitsConversionUtil
+ long convertedRhs = UnitsConversionUtil
.convert(rhsValue.getUnits(), lhsValue.getUnits(),
rhsValue.getValue());
if(lhsValue.getValue() > convertedRhs) {
@@ -448,7 +448,7 @@ public class Resources {
try {
ResourceInformation rhsValue = rhs.getResourceInformation(name);
ResourceInformation lhsValue = entry.getValue();
- Long convertedRhs = UnitsConversionUtil
+ long convertedRhs = UnitsConversionUtil
.convert(rhsValue.getUnits(), lhsValue.getUnits(),
rhsValue.getValue());
ResourceInformation outInfo =
@@ -469,7 +469,7 @@ public class Resources {
try {
ResourceInformation rhsValue = rhs.getResourceInformation(name);
ResourceInformation lhsValue = entry.getValue();
- Long convertedRhs = UnitsConversionUtil
+ long convertedRhs = UnitsConversionUtil
.convert(rhsValue.getUnits(), lhsValue.getUnits(),
rhsValue.getValue());
ResourceInformation outInfo =
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/resource/TestResourceUtils.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/resource/TestResourceUtils.java
index 6f17659..38554b6 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/resource/TestResourceUtils.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/resource/TestResourceUtils.java
@@ -283,6 +283,7 @@ public class TestResourceUtils {
Map<String, ResourceInformation> actual =
ResourceUtils.getNodeResourceInformation(conf);
Assert.assertEquals(entry.getValue().getResources(), actual);
+ dest.delete();
}
}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/resource/TestResources.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/resource/TestResources.java
index d81dcdb..1555e55 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/resource/TestResources.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/resource/TestResources.java
@@ -42,6 +42,16 @@ import static org.junit.Assert.assertTrue;
public class TestResources {
+ static class ExtendedResources extends Resources {
+ public static Resource unbounded() {
+ return new FixedValueResource("UNBOUNDED", Long.MAX_VALUE);
+ }
+
+ public static Resource none() {
+ return new FixedValueResource("NONE", 0L);
+ }
+ }
+
private static final String EXTRA_RESOURCE_TYPE = "resource2";
private String resourceTypesFile;
@@ -87,7 +97,7 @@ public class TestResources {
@Test(timeout = 10000)
public void testCompareToWithUnboundedResource() {
unsetExtraResourceType();
- Resource unboundedClone = Resources.clone(Resources.unbounded());
+ Resource unboundedClone = Resources.clone(ExtendedResources.unbounded());
assertTrue(unboundedClone
.compareTo(createResource(Long.MAX_VALUE, Integer.MAX_VALUE)) == 0);
assertTrue(unboundedClone.compareTo(createResource(Long.MAX_VALUE, 0)) > 0);
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java
index 7b544ef..c44c068 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java
@@ -1340,15 +1340,7 @@ public abstract class AbstractYarnScheduler
* @return a Resource object with the minimum allocation for the scheduler
*/
public Resource getMinimumAllocation() {
- boolean profilesEnabled = getConfig()
- .getBoolean(YarnConfiguration.RM_RESOURCE_PROFILES_ENABLED,
- YarnConfiguration.DEFAULT_RM_RESOURCE_PROFILES_ENABLED);
- Resource ret;
- if (!profilesEnabled) {
- ret = ResourceUtils.getResourceTypesMinimumAllocation();
- } else {
- ret = rmContext.getResourceProfilesManager().getMinimumProfile();
- }
+ Resource ret = ResourceUtils.getResourceTypesMinimumAllocation();
LOG.info("Minimum allocation = " + ret);
return ret;
}
@@ -1362,15 +1354,7 @@ public abstract class AbstractYarnScheduler
*/
public Resource getMaximumAllocation() {
- boolean profilesEnabled = getConfig()
- .getBoolean(YarnConfiguration.RM_RESOURCE_PROFILES_ENABLED,
- YarnConfiguration.DEFAULT_RM_RESOURCE_PROFILES_ENABLED);
- Resource ret;
- if (!profilesEnabled) {
- ret = ResourceUtils.getResourceTypesMaximumAllocation();
- } else {
- ret = rmContext.getResourceProfilesManager().getMaximumProfile();
- }
+ Resource ret = ResourceUtils.getResourceTypesMaximumAllocation();
LOG.info("Maximum allocation = " + ret);
return ret;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 03/04: YARN-6761. Fix build for YARN-3926 branch.
Contributed by Varun Vasudev.
Posted by jh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a commit to branch YARN-8200
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 074569a71a1ad2e578bb3bf20151cedf6dda4a52
Author: Sunil G <su...@apache.org>
AuthorDate: Mon Jul 10 09:21:26 2017 +0530
YARN-6761. Fix build for YARN-3926 branch. Contributed by Varun Vasudev.
(cherry picked from commit ca2808d8bbe49766029e5dba88ff863f6c993287)
---
.../apache/hadoop/yarn/api/records/Resource.java | 112 ++++++++++++++++-----
.../util/resource/DominantResourceCalculator.java | 1 +
2 files changed, 90 insertions(+), 23 deletions(-)
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Resource.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Resource.java
index c51357c..fb31745 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Resource.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Resource.java
@@ -29,6 +29,8 @@ import org.apache.hadoop.yarn.exceptions.ResourceNotFoundException;
import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.util.Records;
+import java.util.Collections;
+import java.util.HashMap;
import java.util.Map;
/**
@@ -58,12 +60,17 @@ import java.util.Map;
@Stable
public abstract class Resource implements Comparable<Resource> {
+ private static Resource tmpResource = Records.newRecord(Resource.class);
+
private static class SimpleResource extends Resource {
private long memory;
private long vcores;
+ private Map<String, ResourceInformation> resourceInformationMap;
+
SimpleResource(long memory, long vcores) {
this.memory = memory;
this.vcores = vcores;
+
}
@Override
public int getMemory() {
@@ -89,17 +96,44 @@ public abstract class Resource implements Comparable<Resource> {
public void setVirtualCores(int vcores) {
this.vcores = vcores;
}
+ @Override
+ public Map<String, ResourceInformation> getResources() {
+ if (resourceInformationMap == null) {
+ resourceInformationMap = new HashMap<>();
+ resourceInformationMap.put(ResourceInformation.MEMORY_MB.getName(),
+ ResourceInformation.newInstance(ResourceInformation.MEMORY_MB));
+ resourceInformationMap.put(ResourceInformation.VCORES.getName(),
+ ResourceInformation.newInstance(ResourceInformation.VCORES));
+ }
+ resourceInformationMap.get(ResourceInformation.MEMORY_MB.getName())
+ .setValue(this.memory);
+ resourceInformationMap.get(ResourceInformation.VCORES.getName())
+ .setValue(this.vcores);
+ return Collections.unmodifiableMap(resourceInformationMap);
+ }
}
@Public
@Stable
public static Resource newInstance(int memory, int vCores) {
+ if (tmpResource.getResources().size() > 2) {
+ Resource ret = Records.newRecord(Resource.class);
+ ret.setMemorySize(memory);
+ ret.setVirtualCores(vCores);
+ return ret;
+ }
return new SimpleResource(memory, vCores);
}
@Public
@Stable
public static Resource newInstance(long memory, int vCores) {
+ if (tmpResource.getResources().size() > 2) {
+ Resource ret = Records.newRecord(Resource.class);
+ ret.setMemorySize(memory);
+ ret.setVirtualCores(vCores);
+ return ret;
+ }
return new SimpleResource(memory, vCores);
}
@@ -116,13 +150,7 @@ public abstract class Resource implements Comparable<Resource> {
public static void copy(Resource source, Resource dest) {
for (Map.Entry<String, ResourceInformation> entry : source.getResources()
.entrySet()) {
- try {
- ResourceInformation.copy(entry.getValue(),
- dest.getResourceInformation(entry.getKey()));
- } catch (YarnException ye) {
- dest.setResourceInformation(entry.getKey(),
- ResourceInformation.newInstance(entry.getValue()));
- }
+ dest.setResourceInformation(entry.getKey(), entry.getValue());
}
}
@@ -234,8 +262,15 @@ public abstract class Resource implements Comparable<Resource> {
*/
@Public
@Evolving
- public abstract ResourceInformation getResourceInformation(String resource)
- throws YarnException;
+ public ResourceInformation getResourceInformation(String resource)
+ throws YarnException {
+ if (getResources().containsKey(resource)) {
+ return getResources().get(resource);
+ }
+ throw new YarnException(
+ "Unknown resource '" + resource + "'. Known resources are "
+ + getResources().keySet());
+ }
/**
* Get the value for a specified resource. No information about the units is
@@ -247,7 +282,14 @@ public abstract class Resource implements Comparable<Resource> {
*/
@Public
@Evolving
- public abstract Long getResourceValue(String resource) throws YarnException;
+ public Long getResourceValue(String resource) throws YarnException {
+ if (getResources().containsKey(resource)) {
+ return getResources().get(resource).getValue();
+ }
+ throw new YarnException(
+ "Unknown resource '" + resource + "'. Known resources are "
+ + getResources().keySet());
+ }
/**
* Set the ResourceInformation object for a particular resource.
@@ -258,8 +300,25 @@ public abstract class Resource implements Comparable<Resource> {
*/
@Public
@Evolving
- public abstract void setResourceInformation(String resource,
- ResourceInformation resourceInformation) throws ResourceNotFoundException;
+ public void setResourceInformation(String resource,
+ ResourceInformation resourceInformation) throws ResourceNotFoundException {
+ if (resource.equals(ResourceInformation.MEMORY_MB.getName())) {
+ this.setMemorySize(resourceInformation.getValue());
+ return;
+ }
+ if (resource.equals(ResourceInformation.VCORES.getName())) {
+ this.setVirtualCores((int) resourceInformation.getValue());
+ return;
+ }
+ if (getResources().containsKey(resource)) {
+ ResourceInformation
+ .copy(resourceInformation, getResources().get(resource));
+ return;
+ }
+ throw new ResourceNotFoundException(
+ "Unknown resource '" + resource + "'. Known resources are "
+ + getResources().keySet());
+ }
/**
* Set the value of a resource in the ResourceInformation object. The unit of
@@ -271,8 +330,24 @@ public abstract class Resource implements Comparable<Resource> {
*/
@Public
@Evolving
- public abstract void setResourceValue(String resource, Long value)
- throws ResourceNotFoundException;
+ public void setResourceValue(String resource, Long value)
+ throws ResourceNotFoundException {
+ if (resource.equals(ResourceInformation.MEMORY_MB.getName())) {
+ this.setMemorySize(value);
+ return;
+ }
+ if (resource.equals(ResourceInformation.VCORES.getName())) {
+ this.setVirtualCores(value.intValue());
+ return;
+ }
+ if (getResources().containsKey(resource)) {
+ getResources().get(resource).setValue(value);
+ return;
+ }
+ throw new ResourceNotFoundException(
+ "Unknown resource '" + resource + "'. Known resources are "
+ + getResources().keySet());
+ }
@Override
public int hashCode() {
@@ -312,15 +387,6 @@ public abstract class Resource implements Comparable<Resource> {
}
@Override
- public int compareTo(Resource other) {
- long diff = this.getMemorySize() - other.getMemorySize();
- if (diff == 0) {
- diff = this.getVirtualCores() - other.getVirtualCores();
- }
- return diff == 0 ? 0 : (diff > 0 ? 1 : -1);
- }
-
- @Override
public String toString() {
StringBuilder sb = new StringBuilder();
sb.append("<memory:").append(getMemorySize()).append(", vCores:")
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.java
index 76bc364..79bb03d 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.java
@@ -306,6 +306,7 @@ public class DominantResourceCalculator extends ResourceCalculator {
value = roundUp(value, stepFactorValue);
}
tmp.setValue(Math.min(value, maximumValue));
+ ret.setResourceInformation(resource, tmp);
} catch (YarnException ye) {
throw new IllegalArgumentException(
"Error getting resource information for " + resource, ye);
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 04/04: YARN-6786. [YARN-3926] ResourcePBImpl imports
cleanup. Contributed by Yeliang Cang.
Posted by jh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a commit to branch YARN-8200
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit d5a5bb8cf9c99c934e661f51d2c5139a8fc0e083
Author: Sunil G <su...@apache.org>
AuthorDate: Thu Jul 13 16:30:59 2017 +0530
YARN-6786. [YARN-3926] ResourcePBImpl imports cleanup. Contributed by Yeliang Cang.
(cherry picked from commit 4e5632d28eb8f6b1b8830c2aa89ec6321d477977)
---
.../org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java
index bba5f7e..6e51efa 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java
@@ -18,7 +18,6 @@
package org.apache.hadoop.yarn.api.records.impl.pb;
-import org.apache.commons.collections.map.UnmodifiableMap;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.classification.InterfaceAudience.Private;
@@ -34,7 +33,10 @@ import org.apache.hadoop.yarn.proto.YarnProtos.ResourceInformationProto;
import org.apache.hadoop.yarn.util.resource.ResourceUtils;
import org.apache.hadoop.yarn.util.UnitsConversionUtil;
-import java.util.*;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Collections;
+
@Private
@Unstable
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org