You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@knox.apache.org by "bonampak (via GitHub)" <gi...@apache.org> on 2026/09/11 07:19:48 UTC
[PR] KNOX-3444: Optimize CM discovery memory usage by only caching role configurations that ServiceModelGenerators actually use [knox]
bonampak opened a new pull request, #1392:
URL: https://github.com/apache/knox/pull/1392
[KNOX-3444](https://issues.apache.org/jira/browse/KNOX-3444) - Exclude role types that are not referenced by ServiceModelGenerators
## What changes were proposed in this pull request?
During CM discovery only cache role configurations that ServiceModelGenerators actually use.
## How was this patch tested?
Updated unit tests and did manual test on a cluster.
Verified that the ClusterConfiguration cache persisted after discovery is the same as before the changes were applied and the topologies have the same generated service URLs.
Acquired a heap dump using jmap:
```bash
jmap -dump:live,format=b,file="/tmp/knox-heap-jmap-${knox_pid}".hprof "${knox_pid}";
```
Used Eclipse Memory Analyzer Tool and Calcite SQL plugin to analyze the heap usage of the ApiRoleConfig objects cached in `ClouderaManagerServiceDiscoveryRepository`
```sql
SELECT toString(r.this['roleType']) roleType, COUNT(r.this) as cnt,
AVG(retainedSize(r.this)) as avg_retained_size, SUM(retainedSize(r.this)) sum_retained_size
FROM "com.cloudera.api.swagger.model.ApiRoleConfig" r
GROUP BY toString(r.this['roleType'])
ORDER BY sum_retained_size desc, roleType
```
The unused roles (e.g. NODEMANAGER, REGIONSERVER, GATEWAY, DATANODE, KNOX_GATEWAY, KAFKA_BROKER, OZONE_DATANODE, etc.) are not cached. For each unused role, the memory savings are 60K - 150K by role instance not cached (with an average of roughly 100K).
## Integration Tests
N/A. Integration tests would require a live CM dependency. Setting up a real (or faithfully simulated) Cloudera Manager server with representative services/roles is heavy infrastructure that the project's test suite does not provide, and a mock-based "integration" test would exercise the same code paths the unit tests already cover.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@knox.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KNOX-3444: Optimize CM discovery memory usage by only caching role configurations that ServiceModelGenerators actually use [knox]
Posted by "bonampak (via GitHub)" <gi...@apache.org>.
bonampak merged PR #1392:
URL: https://github.com/apache/knox/pull/1392
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@knox.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KNOX-3444: Optimize CM discovery memory usage by only caching role configurations that ServiceModelGenerators actually use [knox]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1392:
URL: https://github.com/apache/knox/pull/1392#issuecomment-5631055368
## Test Results
4 files 4 suites 23s ⏱️
84 tests 84 ✅ 0 💤 0 ❌
102 runs 102 ✅ 0 💤 0 ❌
Results for commit e40f7901.
[test-results]:data:application/gzip;base64,H4sIAOeuo2oC/13MSw6CMBSF4a2Qjh30lnIRN2Po4yaNQE0fI+PeLQSwOvz+k5wXIzfZyG6NvDQsZpdOmBzG5PxSKNrisqR1u8oD95i1/isP9yyFn4FGN/0EG4IPewl5WR+Bix3HY12+j5urw831n/bz7FIBs5JTP3Ag2Yt2QNmB5aisUaioRQCDBLozyN4fCxdsogIBAAA=
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@knox.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org