You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "gerashegalov (via GitHub)" <gi...@apache.org> on 2026/10/01 04:27:46 UTC
[PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
gerashegalov opened a new pull request, #8770:
URL: https://github.com/apache/hadoop/pull/8770
### Description of PR
Fixes [YARN-11995](https://issues.apache.org/jira/browse/YARN-11995).
Container history currently drops custom allocations such as `yarn.io/gpu`: the timeline publishers store only memory and vcores, and the history readers reconstruct a two-resource report. An allocation of two GPUs therefore appears as zero in history.
Persist custom allocation values and units in the additive `YARN_CONTAINER_ALLOCATED_RESOURCES` info field. Update both ResourceManager timeline publishers, the NodeManager v2 publisher, and both history report converters. Shared helpers snapshot allocations before asynchronous publication, accept integer/long JSON values, and convert stored units to the reader's configured units.
Existing memory/vcore fields and legacy entities remain compatible. Unknown resource types are skipped with a warning while their raw timeline data remains available. Document the resource-type configuration required for history servers and standalone clients. Allocations omitted by older publishers cannot be recovered retroactively.
### How was this patch tested?
- 107 tests passed on JDK 17, with zero failures, errors, or skips, across `TestTimelineServiceHelper`, `TestSystemMetricsPublisher`, `TestSystemMetricsPublisherForV2`, `TestNMTimelinePublisher`, `TestApplicationHistoryManagerOnTimelineStore`, `TestAHSWebServices`, and `TestAHSv2ClientImpl`. Maven ran with `-Dmaven.test.failure.ignore=false`.
- Coverage includes JSON/store round trips, single-container and list REST responses, GPU and other custom resources, zero and large values, unit conversion, legacy entities, unknown resource types, and source-mutation isolation.
- Additional standalone probes passed for the generic storage codec, creation/finish entity merging, container-report protobuf serialization, and fresh-JVM loading of `resource-types.xml`.
- `git diff --check` passed. Checkstyle reported 45 existing diagnostics in the checked files, none on changed lines.
- A live HBase-backed ATSv2 cluster test was not run.
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the endpoint
declared according to the connector-specific documentation? *Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required.* Not applicable: no object storage changes.
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? Not applicable: no new dependencies.
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? Not applicable: no licensing changes.
### AI Tooling
Contains content generated by OpenAI Codex.
If an AI tool was used:
- [x] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#issuecomment-5928314082
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 7m 5s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 3s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 24s | | trunk passed |
| +1 :green_heart: | compile | 3m 31s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | compile | 3m 44s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 6s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 3m 55s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +0 :ok: | spotbugs | 0m 36s | | branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 1m 5s | [/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/1/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 17m 13s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 17s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 28s | | the patch passed |
| +1 :green_heart: | compile | 3m 20s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javac | 3m 20s | | the patch passed |
| +1 :green_heart: | compile | 3m 25s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 3m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 23s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 2s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 37s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 3m 41s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +0 :ok: | spotbugs | 0m 24s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs |
| +1 :green_heart: | shadedclient | 15m 47s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 4m 30s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 2m 54s | | hadoop-yarn-server-applicationhistoryservice in the patch passed. |
| +1 :green_heart: | unit | 91m 16s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| -1 :x: | unit | 23m 56s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt) | hadoop-yarn-server-nodemanager in the patch passed. |
| +1 :green_heart: | unit | 25m 37s | | hadoop-yarn-client in the patch passed. |
| +1 :green_heart: | unit | 0m 25s | | hadoop-yarn-site in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not generate ASF License warnings. |
| | | 279m 7s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8770 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux f68e7bd69b43 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d715d9f9ef407cbef1188d9a4a80618605954697 |
| Default Java | Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/1/testReport/ |
| Max. process+thread count | 968 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/1/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
Posted by "joseluisll (via GitHub)" <gi...@apache.org>.
joseluisll commented on code in PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#discussion_r4204242885
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java:
##########
@@ -60,6 +73,73 @@ public class TimelineUtils {
YarnJacksonJaxbJsonProvider.configObjectMapper(mapper);
}
+ @Private
+ public static Map<String, Map<String, Object>> getCustomResourceInfo(
+ Resource resource) {
+ Map<String, Map<String, Object>> resources = new HashMap<>();
+ for (ResourceInformation information : resource.getResources()) {
+ String name = information.getName();
+ if (!ResourceInformation.MEMORY_URI.equals(name)
+ && !ResourceInformation.VCORES_URI.equals(name)
+ && information.getValue() != 0) {
+ Map<String, Object> allocation = new HashMap<>();
+ allocation.put("value", information.getValue());
Review Comment:
nit, which I should have caught on the first pass: `"value"` and `"units"` are spelled out separately by the writer (`getCustomResourceInfo`) and the reader (`getContainerResource`). They're part of the stored format now, so could they be constants next to `ALLOCATED_RESOURCES_INFO` in `ContainerMetricsConstants`? That way writer and reader can't drift apart.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java:
##########
@@ -60,6 +73,73 @@ public class TimelineUtils {
YarnJacksonJaxbJsonProvider.configObjectMapper(mapper);
}
+ @Private
+ public static Map<String, Map<String, Object>> getCustomResourceInfo(
+ Resource resource) {
+ Map<String, Map<String, Object>> resources = new HashMap<>();
+ for (ResourceInformation information : resource.getResources()) {
+ String name = information.getName();
+ if (!ResourceInformation.MEMORY_URI.equals(name)
+ && !ResourceInformation.VCORES_URI.equals(name)
+ && information.getValue() != 0) {
+ Map<String, Object> allocation = new HashMap<>();
+ allocation.put("value", information.getValue());
+ allocation.put("units", information.getUnits());
+ resources.put(name, allocation);
+ }
+ }
+ return resources;
+ }
+
+ @Private
+ public static Resource getContainerResource(Map<String, Object> entityInfo) {
+ if (entityInfo == null) {
+ return Resource.newInstance(0, 0);
+ }
+ long memory = ((Number) entityInfo.getOrDefault(
+ ContainerMetricsConstants.ALLOCATED_MEMORY_INFO, 0L)).longValue();
+ int vcores = ((Number) entityInfo.getOrDefault(
+ ContainerMetricsConstants.ALLOCATED_VCORE_INFO, 0)).intValue();
+ Resource resource = Resource.newInstance(memory, vcores);
+ Object allocationInfo = entityInfo.get(
+ ContainerMetricsConstants.ALLOCATED_RESOURCES_INFO);
+ if (allocationInfo instanceof Map) {
+ Map<?, ?> allocations = (Map<?, ?>) allocationInfo;
+ for (Map.Entry<?, ?> entry : allocations.entrySet()) {
+ try {
+ String name = (String) entry.getKey();
+ if (ResourceInformation.MEMORY_URI.equals(name)
+ || ResourceInformation.VCORES_URI.equals(name)) {
+ continue;
+ }
+ if (!ResourceUtils.getResourceTypes().containsKey(name)) {
+ LOG.debug("Skipping unknown resource type {} in container history", name);
+ continue;
+ }
+ Map<?, ?> allocation = (Map<?, ?>) entry.getValue();
+ Number storedValue = (Number) allocation.get("value");
+ if (storedValue instanceof Float || storedValue instanceof Double) {
+ throw new IllegalArgumentException("Floating-point resource value");
+ }
+ long value = storedValue instanceof Integer || storedValue instanceof Long
+ ? storedValue.longValue()
+ : new BigDecimal(storedValue.toString()).longValueExact();
+ String units = (String) allocation.get("units");
+ String defaultUnits = resource.getResourceInformation(name).getUnits();
+ resource.setResourceValue(name,
+ UnitsConversionUtil.convert(units, defaultUnits, value));
+ } catch (ClassCastException | NullPointerException
Review Comment:
optional: Following up on my earlier suggestion: listing the four exceptions covers today's code paths. A single `catch (RuntimeException e)` would also cover anything not listed, such as a `ResourceNotFoundException` from `getResourceInformation`, which is what a per-entry skip guard is for. Feel free to leave as is.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java:
##########
@@ -60,6 +73,73 @@ public class TimelineUtils {
YarnJacksonJaxbJsonProvider.configObjectMapper(mapper);
}
+ @Private
+ public static Map<String, Map<String, Object>> getCustomResourceInfo(
Review Comment:
nit, also from the first pass: A short Javadoc on these two public helpers would help. On `getCustomResourceInfo` it could show the map shape (`name -> {value, units}`, leaving out memory, vcores and zero values). On `getContainerResource` it could say that unknown or malformed entries are skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#issuecomment-5977615408
PRs #8776 and #8779 have been merged. I’ve triggered a new Jenkins precommit run against the latest trunk to check whether the CI failures are resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#issuecomment-6007559561
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 48s | | trunk passed |
| +1 :green_heart: | compile | 3m 32s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | compile | 3m 34s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 3s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 4m 6s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +0 :ok: | spotbugs | 0m 32s | | branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 16m 43s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 16s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 36s | | the patch passed |
| +1 :green_heart: | compile | 3m 13s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javac | 3m 13s | | the patch passed |
| +1 :green_heart: | compile | 3m 31s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 3m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 24s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 44 unchanged - 0 fixed = 45 total (was 44) |
| +1 :green_heart: | mvnsite | 4m 2s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 41s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 3m 41s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +0 :ok: | spotbugs | 0m 27s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs |
| +1 :green_heart: | shadedclient | 16m 4s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 4m 35s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 17s | | hadoop-yarn-server-applicationhistoryservice in the patch passed. |
| +1 :green_heart: | unit | 91m 36s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 23m 21s | | hadoop-yarn-server-nodemanager in the patch passed. |
| +1 :green_heart: | unit | 25m 11s | | hadoop-yarn-client in the patch passed. |
| +1 :green_heart: | unit | 0m 32s | | hadoop-yarn-site in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not generate ASF License warnings. |
| | | 272m 56s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8770 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 5dd722c77906 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 48b26fe74000f8fb277b194094e999b81c2f1090 |
| Default Java | Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/3/testReport/ |
| Max. process+thread count | 906 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/3/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#issuecomment-5986486744
@gerashegalov The latest Jenkins/Yetus run is green. The two remaining red checks appear to be stale statuses from the previous build.
Could you push an empty commit to retrigger CI on a new commit SHA? This should prevent the stale statuses from carrying over.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
Posted by "gerashegalov (via GitHub)" <gi...@apache.org>.
gerashegalov commented on PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#issuecomment-5933293759
**CI update for [Jenkins run #1](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/1/):** The fork Build passes, and the changed `TestNMTimelinePublisher` tests pass (6/6). Yetus reports two NodeManager failures that appear unrelated to this patch:
- **SpotBugs:** Trunk has one existing `NP_NULL_ON_SOME_PATH_EXCEPTION` warning in `ResourceLocalizationService$PublicLocalizer.run()`; this patch adds no new warnings. [YARN-11993](https://issues.apache.org/jira/browse/YARN-11993) / [PR #8755](https://github.com/apache/hadoop/pull/8755) addresses it.
- **Unit test:** `TestLogAggregationService.testLocalFileDeletionAfterUpload` fails because the application log directory still exists when asserted. The test waits for individual files, but not for the independently scheduled directory deletion. This failed on all three attempts in this run. [HADOOP-19979](https://issues.apache.org/jira/browse/HADOOP-19979) / [PR #8717](https://github.com/apache/hadoop/pull/8717) adds the missing wait.
Neither failing path is changed by this PR. Once those fixes land, a fresh CI run should establish whether any failures remain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#issuecomment-5979594241
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 7m 6s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 48s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 22s | | trunk passed |
| +1 :green_heart: | compile | 3m 36s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | compile | 3m 29s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 19s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 8s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 4m 8s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +0 :ok: | spotbugs | 0m 35s | | branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 16m 56s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 18s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 28s | | the patch passed |
| +1 :green_heart: | compile | 3m 16s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javac | 3m 16s | | the patch passed |
| +1 :green_heart: | compile | 3m 32s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 3m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 58s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 41s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 3m 37s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +0 :ok: | spotbugs | 0m 26s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs |
| +1 :green_heart: | shadedclient | 15m 53s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 4m 35s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 19s | | hadoop-yarn-server-applicationhistoryservice in the patch passed. |
| +1 :green_heart: | unit | 90m 44s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 22m 38s | | hadoop-yarn-server-nodemanager in the patch passed. |
| +1 :green_heart: | unit | 25m 4s | | hadoop-yarn-client in the patch passed. |
| +1 :green_heart: | unit | 0m 23s | | hadoop-yarn-site in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not generate ASF License warnings. |
| | | 275m 17s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8770 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 6d622352cea3 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d715d9f9ef407cbef1188d9a4a80618605954697 |
| Default Java | Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/2/testReport/ |
| Max. process+thread count | 967 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/2/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#issuecomment-6092849157
Merged. Thanks, @gerashegalov, for the contribution, and @joseluisll for reviewing!
@gerashegalov, should we also backport this change to `branch-3.4`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 merged PR #8770:
URL: https://github.com/apache/hadoop/pull/8770
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#issuecomment-6030594907
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 24s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 1s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 3s | | trunk passed |
| +1 :green_heart: | compile | 3m 34s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | compile | 3m 40s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 10s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 4m 5s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +0 :ok: | spotbugs | 0m 36s | | branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 17m 12s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 31s | | the patch passed |
| +1 :green_heart: | compile | 3m 11s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javac | 3m 11s | | the patch passed |
| +1 :green_heart: | compile | 3m 29s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 3m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 23s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 31s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 3m 40s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +0 :ok: | spotbugs | 0m 26s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs |
| +1 :green_heart: | shadedclient | 16m 3s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 4m 35s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 18s | | hadoop-yarn-server-applicationhistoryservice in the patch passed. |
| +1 :green_heart: | unit | 92m 7s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 23m 20s | | hadoop-yarn-server-nodemanager in the patch passed. |
| +1 :green_heart: | unit | 25m 5s | | hadoop-yarn-client in the patch passed. |
| +1 :green_heart: | unit | 0m 31s | | hadoop-yarn-site in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not generate ASF License warnings. |
| | | 273m 12s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8770 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 4f438f19eeea 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4de2db24f6f5bdc3dd0714a7b84367e2b2725848 |
| Default Java | Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/4/testReport/ |
| Max. process+thread count | 944 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/4/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
Posted by "gerashegalov (via GitHub)" <gi...@apache.org>.
gerashegalov commented on PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#issuecomment-6002921123
Thanks for the green rerun. Rather than adding an empty commit, I pushed substantive review-feedback changes in 48b26fe. This gives the PR a new SHA and should retrigger CI without carrying over the two stale statuses from run #1. The updated description records the local 29/29 targeted test result and notes that YARN-12007 tracks the pre-existing unit-conversion issue separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#issuecomment-5967778638
> **CI update for [Jenkins run #1](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/1/):** The fork Build passes, and the changed `TestNMTimelinePublisher` tests pass (6/6). Yetus reports two NodeManager failures that appear unrelated to this patch:
>
> * **SpotBugs:** Trunk has one existing `NP_NULL_ON_SOME_PATH_EXCEPTION` warning in `ResourceLocalizationService$PublicLocalizer.run()`; this patch adds no new warnings. [YARN-11993](https://issues.apache.org/jira/browse/YARN-11993) / [PR #8755](https://github.com/apache/hadoop/pull/8755) addresses it.
> * **Unit test:** `TestLogAggregationService.testLocalFileDeletionAfterUpload` fails because the application log directory still exists when asserted. The test waits for individual files, but not for the independently scheduled directory deletion. This failed on all three attempts in this run. [HADOOP-19979](https://issues.apache.org/jira/browse/HADOOP-19979) / [PR #8717](https://github.com/apache/hadoop/pull/8717) adds the missing wait.
>
> Neither failing path is changed by this PR. Once those fixes land, a fresh CI run should establish whether any failures remain.
@gerashegalov Thanks for inviting me to review this PR! I'll take another look. I suggest merging #8776 and #8779 first, as they address unrelated flaky YARN tests that can cause CI failures. This should help stabilize the CI baseline and make it easier to assess this PR's results.
Thanks for your patience!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
Posted by "gerashegalov (via GitHub)" <gi...@apache.org>.
gerashegalov commented on PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#issuecomment-6094702294
Thank you @slfan1989 . I think it would be great to backport to branch-3.4, which hopefully speeds up its delivery to customers if the effort is reasonable. I can work on it next
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
Posted by "gerashegalov (via GitHub)" <gi...@apache.org>.
gerashegalov commented on PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#issuecomment-6066282240
Thank you for reviewing @joseluisll @slfan1989
@slfan1989 would you mind merging the PR if there are no more comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
Posted by "joseluisll (via GitHub)" <gi...@apache.org>.
joseluisll commented on code in PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#discussion_r4181416993
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java:
##########
@@ -60,6 +72,57 @@ public class TimelineUtils {
YarnJacksonJaxbJsonProvider.configObjectMapper(mapper);
}
+ @Private
+ public static Map<String, Map<String, Object>> getCustomResourceInfo(
+ Resource resource) {
+ Map<String, Map<String, Object>> resources = new HashMap<>();
+ for (ResourceInformation information : resource.getResources()) {
+ String name = information.getName();
+ if (!ResourceInformation.MEMORY_URI.equals(name)
+ && !ResourceInformation.VCORES_URI.equals(name)) {
+ Map<String, Object> allocation = new HashMap<>();
+ allocation.put("value", information.getValue());
+ allocation.put("units", information.getUnits());
+ resources.put(name, allocation);
+ }
+ }
+ return resources;
+ }
+
+ @Private
+ public static Resource getContainerResource(Map<String, Object> entityInfo) {
+ if (entityInfo == null) {
+ return Resource.newInstance(0, 0);
+ }
+ long memory = ((Number) entityInfo.getOrDefault(
+ ContainerMetricsConstants.ALLOCATED_MEMORY_INFO, 0L)).longValue();
+ int vcores = ((Number) entityInfo.getOrDefault(
+ ContainerMetricsConstants.ALLOCATED_VCORE_INFO, 0)).intValue();
+ Resource resource = Resource.newInstance(memory, vcores);
+ Map<?, ?> allocations = (Map<?, ?>) entityInfo.get(
+ ContainerMetricsConstants.ALLOCATED_RESOURCES_INFO);
+ if (allocations != null) {
+ for (Map.Entry<?, ?> entry : allocations.entrySet()) {
+ String name = (String) entry.getKey();
+ if (ResourceInformation.MEMORY_URI.equals(name)
+ || ResourceInformation.VCORES_URI.equals(name)) {
+ continue;
+ }
+ if (!ResourceUtils.getResourceTypes().containsKey(name)) {
+ LOG.warn("Skipping unknown resource type {} in container history", name);
+ continue;
+ }
+ Map<?, ?> allocation = (Map<?, ?>) entry.getValue();
Review Comment:
**Make per-entry parsing tolerant of bad data**
This loop has no error handling, so one malformed entry fails the whole conversion:
- a missing `value` gives an NPE;
- a value or units of an unexpected type gives a `ClassCastException`;
- an unknown unit, or a unit conversion that overflows, gives an `IllegalArgumentException` from `UnitsConversionUtil.convert`.
Unknown resource types already get "warn and skip". I think bad entries deserve the same treatment instead of failing the request. Otherwise one bad entry breaks the container report, and on the list endpoints it breaks every container of the attempt. Something like:
```java
try {
Map<?, ?> allocation = (Map<?, ?>) entry.getValue();
long value = ((Number) allocation.get("value")).longValue();
String units = (String) allocation.get("units");
String defaultUnits = resource.getResourceInformation(name).getUnits();
resource.setResourceValue(name,
UnitsConversionUtil.convert(units, defaultUnits, value));
} catch (RuntimeException e) {
LOG.debug("Skipping malformed allocation for resource type {}", name, e);
}
```
A small test with a missing `value` and with an unknown unit would cover it.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java:
##########
@@ -60,6 +72,57 @@ public class TimelineUtils {
YarnJacksonJaxbJsonProvider.configObjectMapper(mapper);
}
+ @Private
+ public static Map<String, Map<String, Object>> getCustomResourceInfo(
+ Resource resource) {
+ Map<String, Map<String, Object>> resources = new HashMap<>();
+ for (ResourceInformation information : resource.getResources()) {
+ String name = information.getName();
+ if (!ResourceInformation.MEMORY_URI.equals(name)
+ && !ResourceInformation.VCORES_URI.equals(name)) {
+ Map<String, Object> allocation = new HashMap<>();
+ allocation.put("value", information.getValue());
+ allocation.put("units", information.getUnits());
+ resources.put(name, allocation);
+ }
+ }
+ return resources;
+ }
+
+ @Private
+ public static Resource getContainerResource(Map<String, Object> entityInfo) {
+ if (entityInfo == null) {
+ return Resource.newInstance(0, 0);
+ }
+ long memory = ((Number) entityInfo.getOrDefault(
+ ContainerMetricsConstants.ALLOCATED_MEMORY_INFO, 0L)).longValue();
+ int vcores = ((Number) entityInfo.getOrDefault(
+ ContainerMetricsConstants.ALLOCATED_VCORE_INFO, 0)).intValue();
+ Resource resource = Resource.newInstance(memory, vcores);
+ Map<?, ?> allocations = (Map<?, ?>) entityInfo.get(
+ ContainerMetricsConstants.ALLOCATED_RESOURCES_INFO);
+ if (allocations != null) {
+ for (Map.Entry<?, ?> entry : allocations.entrySet()) {
+ String name = (String) entry.getKey();
+ if (ResourceInformation.MEMORY_URI.equals(name)
+ || ResourceInformation.VCORES_URI.equals(name)) {
+ continue;
+ }
+ if (!ResourceUtils.getResourceTypes().containsKey(name)) {
+ LOG.warn("Skipping unknown resource type {} in container history", name);
Review Comment:
**Avoid log spam for unknown resource types**
This warning runs for every container on every read. For example, take an AHS without `resource-types.xml` that serves a container list for a large app. It would write one warning per container per request. Could we log this at DEBUG, or warn only once per resource type (e.g. track the names already reported in a `ConcurrentHashMap.newKeySet()`)?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java:
##########
@@ -60,6 +72,57 @@ public class TimelineUtils {
YarnJacksonJaxbJsonProvider.configObjectMapper(mapper);
}
+ @Private
+ public static Map<String, Map<String, Object>> getCustomResourceInfo(
Review Comment:
**Skip the field when there are no custom resources**
This always returns a map, and the publishers always `put` it. So on clusters without custom resources, every container entity gets an empty `YARN_CONTAINER_ALLOCATED_RESOURCES: {}`. That's an extra column per container in ATSv2/HBase and extra bytes in ATSv1. Readers already handle the field being absent, so the publishers could skip the `put` when the map is empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
Posted by "gerashegalov (via GitHub)" <gi...@apache.org>.
gerashegalov commented on code in PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#discussion_r4188802157
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java:
##########
@@ -60,6 +72,57 @@ public class TimelineUtils {
YarnJacksonJaxbJsonProvider.configObjectMapper(mapper);
}
+ @Private
+ public static Map<String, Map<String, Object>> getCustomResourceInfo(
+ Resource resource) {
+ Map<String, Map<String, Object>> resources = new HashMap<>();
+ for (ResourceInformation information : resource.getResources()) {
+ String name = information.getName();
+ if (!ResourceInformation.MEMORY_URI.equals(name)
+ && !ResourceInformation.VCORES_URI.equals(name)) {
+ Map<String, Object> allocation = new HashMap<>();
+ allocation.put("value", information.getValue());
+ allocation.put("units", information.getUnits());
+ resources.put(name, allocation);
+ }
+ }
+ return resources;
+ }
+
+ @Private
+ public static Resource getContainerResource(Map<String, Object> entityInfo) {
+ if (entityInfo == null) {
+ return Resource.newInstance(0, 0);
+ }
+ long memory = ((Number) entityInfo.getOrDefault(
+ ContainerMetricsConstants.ALLOCATED_MEMORY_INFO, 0L)).longValue();
+ int vcores = ((Number) entityInfo.getOrDefault(
+ ContainerMetricsConstants.ALLOCATED_VCORE_INFO, 0)).intValue();
+ Resource resource = Resource.newInstance(memory, vcores);
+ Map<?, ?> allocations = (Map<?, ?>) entityInfo.get(
+ ContainerMetricsConstants.ALLOCATED_RESOURCES_INFO);
+ if (allocations != null) {
+ for (Map.Entry<?, ?> entry : allocations.entrySet()) {
+ String name = (String) entry.getKey();
+ if (ResourceInformation.MEMORY_URI.equals(name)
+ || ResourceInformation.VCORES_URI.equals(name)) {
+ continue;
+ }
+ if (!ResourceUtils.getResourceTypes().containsKey(name)) {
+ LOG.warn("Skipping unknown resource type {} in container history", name);
+ continue;
+ }
+ Map<?, ?> allocation = (Map<?, ?>) entry.getValue();
Review Comment:
Thanks for the concrete cases. Addressed in 48b26fe: malformed custom entries are skipped independently so later valid allocations survive. The tests cover missing values, invalid shapes, unknown units, conversion overflow, and invalid numeric values. The separate pre-existing UnitsConversionUtil issue is tracked as YARN-12007.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java:
##########
@@ -60,6 +72,57 @@ public class TimelineUtils {
YarnJacksonJaxbJsonProvider.configObjectMapper(mapper);
}
+ @Private
+ public static Map<String, Map<String, Object>> getCustomResourceInfo(
+ Resource resource) {
+ Map<String, Map<String, Object>> resources = new HashMap<>();
+ for (ResourceInformation information : resource.getResources()) {
+ String name = information.getName();
+ if (!ResourceInformation.MEMORY_URI.equals(name)
+ && !ResourceInformation.VCORES_URI.equals(name)) {
+ Map<String, Object> allocation = new HashMap<>();
+ allocation.put("value", information.getValue());
+ allocation.put("units", information.getUnits());
+ resources.put(name, allocation);
+ }
+ }
+ return resources;
+ }
+
+ @Private
+ public static Resource getContainerResource(Map<String, Object> entityInfo) {
+ if (entityInfo == null) {
+ return Resource.newInstance(0, 0);
+ }
+ long memory = ((Number) entityInfo.getOrDefault(
+ ContainerMetricsConstants.ALLOCATED_MEMORY_INFO, 0L)).longValue();
+ int vcores = ((Number) entityInfo.getOrDefault(
+ ContainerMetricsConstants.ALLOCATED_VCORE_INFO, 0)).intValue();
+ Resource resource = Resource.newInstance(memory, vcores);
+ Map<?, ?> allocations = (Map<?, ?>) entityInfo.get(
+ ContainerMetricsConstants.ALLOCATED_RESOURCES_INFO);
+ if (allocations != null) {
+ for (Map.Entry<?, ?> entry : allocations.entrySet()) {
+ String name = (String) entry.getKey();
+ if (ResourceInformation.MEMORY_URI.equals(name)
+ || ResourceInformation.VCORES_URI.equals(name)) {
+ continue;
+ }
+ if (!ResourceUtils.getResourceTypes().containsKey(name)) {
+ LOG.warn("Skipping unknown resource type {} in container history", name);
Review Comment:
Addressed in 48b26fe: unknown resource types now log at DEBUG rather than WARN, and both timeline docs match.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java:
##########
@@ -60,6 +72,57 @@ public class TimelineUtils {
YarnJacksonJaxbJsonProvider.configObjectMapper(mapper);
}
+ @Private
+ public static Map<String, Map<String, Object>> getCustomResourceInfo(
Review Comment:
Addressed in 48b26fe: all three publishers omit the field for an empty custom-resource map. The helper also omits zero-valued custom types, so even a configured but unused GPU type adds no field. RM v1, RM v2, and NM tests assert the field is absent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#issuecomment-6039330621
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 23s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 23s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 50s | | trunk passed |
| +1 :green_heart: | compile | 3m 35s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | compile | 3m 34s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 12s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 4m 4s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +0 :ok: | spotbugs | 0m 35s | | branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 17m 1s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 18s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 31s | | the patch passed |
| +1 :green_heart: | compile | 3m 21s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javac | 3m 21s | | the patch passed |
| +1 :green_heart: | compile | 3m 53s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 3m 53s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 13s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 2m 54s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +0 :ok: | spotbugs | 0m 21s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs |
| +1 :green_heart: | shadedclient | 17m 7s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 4m 31s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 6s | | hadoop-yarn-server-applicationhistoryservice in the patch passed. |
| +1 :green_heart: | unit | 91m 42s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 22m 58s | | hadoop-yarn-server-nodemanager in the patch passed. |
| +1 :green_heart: | unit | 24m 34s | | hadoop-yarn-client in the patch passed. |
| +1 :green_heart: | unit | 0m 23s | | hadoop-yarn-site in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not generate ASF License warnings. |
| | | 270m 49s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8770 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 170a84b33855 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d60acf73214d49a74e8caf19525f6351d80e56b4 |
| Default Java | Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/5/testReport/ |
| Max. process+thread count | 953 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8770/5/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11995. Preserve custom resource allocations in timeline container history [hadoop]
Posted by "gerashegalov (via GitHub)" <gi...@apache.org>.
gerashegalov commented on code in PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#discussion_r4215813362
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java:
##########
@@ -60,6 +73,73 @@ public class TimelineUtils {
YarnJacksonJaxbJsonProvider.configObjectMapper(mapper);
}
+ @Private
+ public static Map<String, Map<String, Object>> getCustomResourceInfo(
+ Resource resource) {
+ Map<String, Map<String, Object>> resources = new HashMap<>();
+ for (ResourceInformation information : resource.getResources()) {
+ String name = information.getName();
+ if (!ResourceInformation.MEMORY_URI.equals(name)
+ && !ResourceInformation.VCORES_URI.equals(name)
+ && information.getValue() != 0) {
+ Map<String, Object> allocation = new HashMap<>();
+ allocation.put("value", information.getValue());
Review Comment:
Thanks—addressed in `d60acf7`. `value` and `units` now have shared constants in `ContainerMetricsConstants`, used by both the writer and reader. The stored field names are unchanged.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java:
##########
@@ -60,6 +73,73 @@ public class TimelineUtils {
YarnJacksonJaxbJsonProvider.configObjectMapper(mapper);
}
+ @Private
+ public static Map<String, Map<String, Object>> getCustomResourceInfo(
Review Comment:
Added Javadocs in `d60acf7` describing the stored map shape, omitted allocations, and how the reader handles unknown or malformed entries.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java:
##########
@@ -60,6 +73,73 @@ public class TimelineUtils {
YarnJacksonJaxbJsonProvider.configObjectMapper(mapper);
}
+ @Private
+ public static Map<String, Map<String, Object>> getCustomResourceInfo(
+ Resource resource) {
+ Map<String, Map<String, Object>> resources = new HashMap<>();
+ for (ResourceInformation information : resource.getResources()) {
+ String name = information.getName();
+ if (!ResourceInformation.MEMORY_URI.equals(name)
+ && !ResourceInformation.VCORES_URI.equals(name)
+ && information.getValue() != 0) {
+ Map<String, Object> allocation = new HashMap<>();
+ allocation.put("value", information.getValue());
+ allocation.put("units", information.getUnits());
+ resources.put(name, allocation);
+ }
+ }
+ return resources;
+ }
+
+ @Private
+ public static Resource getContainerResource(Map<String, Object> entityInfo) {
+ if (entityInfo == null) {
+ return Resource.newInstance(0, 0);
+ }
+ long memory = ((Number) entityInfo.getOrDefault(
+ ContainerMetricsConstants.ALLOCATED_MEMORY_INFO, 0L)).longValue();
+ int vcores = ((Number) entityInfo.getOrDefault(
+ ContainerMetricsConstants.ALLOCATED_VCORE_INFO, 0)).intValue();
+ Resource resource = Resource.newInstance(memory, vcores);
+ Object allocationInfo = entityInfo.get(
+ ContainerMetricsConstants.ALLOCATED_RESOURCES_INFO);
+ if (allocationInfo instanceof Map) {
+ Map<?, ?> allocations = (Map<?, ?>) allocationInfo;
+ for (Map.Entry<?, ?> entry : allocations.entrySet()) {
+ try {
+ String name = (String) entry.getKey();
+ if (ResourceInformation.MEMORY_URI.equals(name)
+ || ResourceInformation.VCORES_URI.equals(name)) {
+ continue;
+ }
+ if (!ResourceUtils.getResourceTypes().containsKey(name)) {
+ LOG.debug("Skipping unknown resource type {} in container history", name);
+ continue;
+ }
+ Map<?, ?> allocation = (Map<?, ?>) entry.getValue();
+ Number storedValue = (Number) allocation.get("value");
+ if (storedValue instanceof Float || storedValue instanceof Double) {
+ throw new IllegalArgumentException("Floating-point resource value");
+ }
+ long value = storedValue instanceof Integer || storedValue instanceof Long
+ ? storedValue.longValue()
+ : new BigDecimal(storedValue.toString()).longValueExact();
+ String units = (String) allocation.get("units");
+ String defaultUnits = resource.getResourceInformation(name).getUnits();
+ resource.setResourceValue(name,
+ UnitsConversionUtil.convert(units, defaultUnits, value));
+ } catch (ClassCastException | NullPointerException
Review Comment:
Thanks for the optional suggestion. I’d keep the specific catches here: after the configured-type check, a `ResourceNotFoundException` would indicate inconsistent resource metadata rather than a malformed stored entry, so silently skipping it could hide a separate problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org