You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hugegraph.apache.org by "byteayan (via GitHub)" <gi...@apache.org> on 2026/10/15 04:54:25 UTC
[PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
byteayan opened a new pull request, #3355:
URL: https://github.com/apache/hugegraph/pull/3355
## Purpose of the PR
- close #3254
Both shipped `application-pd.yml` files set `rocksdb.total_memory_size: 32000000000`, and the `pd` profile is always included, so the `Runtime.maxMemory()` fallback in `AppConfig.init()` never ran. `RaftRocksdbOptions` splits the value into the `WriteBufferManager` cache (66%) and the block cache. Both are native `LRUCache`s, so neither `-Xmx` nor the container limit bounds them, and a separate 1 GiB raft log cache sits on top. In the run in the issue all three Stores were OOM-killed at a 4 GiB limit once each held about 1 GB of RocksDB data.
Docker users had no clean way to change it. `docker-entrypoint.sh` builds `SPRING_APPLICATION_JSON` from five `HG_STORE_*` keys, and none of them is a `rocksdb.*` key. Bind-mounting a config file or passing `-Drocksdb.total_memory_size=<bytes>` in `JAVA_OPTS` both work, but neither is documented.
As the issue points out, dropping the line from the yml alone would hand RocksDB a budget equal to the heap, which is a second heap-sized allocation on top of the first.
## Main Changes
- New `RocksdbMemoryBudget` in `hg-store-core/.../options`. An unset, blank or 0 value means auto: half of (physical memory − max heap − `MaxDirectMemorySize` − 1.5 GiB reserve), clamped to 256 MiB to 32 GB. The reserve covers the raft log cache plus stacks, metaspace and other native memory. Physical memory comes from `OperatingSystemMXBean.getTotalMemorySize()`, which on JDK 17 reports the cgroup v1/v2 limit inside a container. Older JDK 17 updates may not see a limit set only on a parent cgroup; those get host memory, as before this change. An unset `MaxDirectMemorySize` is counted as the heap size, since that is what the JVM allows. If physical memory can't be read, the previous heap-size default applies.
- The issue's setup (4 GiB limit, `-Xmx1g`, `-XX:MaxDirectMemorySize=512m`) gets 512 MiB, and an 8 GiB limit gets 2.5 GiB. I checked the 4 GiB case with `resolve()` inside `eclipse-temurin:17-jre-jammy` under a 4 GiB Docker limit: 536870912 bytes. With `-Xmx2g` and no direct-memory cap it lands on the 256 MiB floor. The 32 GB cap means no host gets a bigger budget than before.
- A positive value is used unchanged, with one exception: values under 1 MiB are refused. The write buffer manager gets 2/3 of the budget, so `1` would give a 0-byte manager, which RocksDB treats as no memtable limit at all. `4GB` and other non-numeric values now fail at startup with "must be a number of bytes", where before they failed with a bare `NumberFormatException`. Nothing shipped (configs, compose files, Helm, CI) sets a value under 1 MiB.
- Startup logs the chosen value and its inputs. It warns when the 256 MiB floor applies (heap plus direct memory already fill the limit), when an explicit value is under 256 MiB, and when an explicit value exceeds the memory left after the heap, direct memory and the reserve. That last case is what an upgraded install with an old mounted `application-pd.yml` will hit.
- `AppConfig.init()` uses it. While there I fixed the raft disruptor sizing for budgets under 1 GB: `log(0)` rounded to a buffer size of 0. It now gives 32. The formula moved to `RocksdbMemoryBudget.disruptorBufferSize()` so it has a test. The shipped `application.yml` pins `disruptorBufferSize: 1024`, so this only matters when that key is absent.
- Both `application-pd.yml` files ship `total_memory_size: 0`, which also removes the #3254 TODO from #3260. The test resources keep their explicit values.
- `docker-entrypoint.sh` accepts `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE`. Values must be digits only; anything else exits with code 2. It is added to `SPRING_APPLICATION_JSON` only when set, so an unset variable leaves the config file or auto-sizing in charge. When set it outranks both the config file and `-Drocksdb.total_memory_size` (checked against Spring Boot 2.5's property order).
- `helm/hugegraph/values-cluster.yaml`: the store comment said container limits don't configure the RocksDB budget. It now says how the budget is derived (2.5 GiB for that preset) and how to pin it with `store.extraEnv`.
In a small container the write buffer manager now flushes memtables much earlier than before, which means more small L0 files and lower write throughput in exchange for staying inside the limit. Both caps are soft, so RocksDB can still go over briefly under heavy writes. I have not measured either effect. Setting the value explicitly restores the old behaviour.
## Verifying these changes
- [x] Need tests and can be verified as follows:
- New `RocksdbMemoryBudgetTest` (8 tests, added to `CoreSuiteTest`, which the `store-core-test` profile runs). It covers:
- explicit values kept unchanged, and values under 1 MiB refused;
- unset, blank, `0` and `00` going to auto;
- refusal of `4GB`, `-1`, `1.5e9` and a value past `Long.MAX_VALUE`;
- the 4 GiB and 8 GiB examples above, both clamp bounds, and the unknown-physical fallback;
- the disruptor size for 512 MiB, 1 GB and 32 GB.
- New `test-store-docker-entrypoint.sh` (9 cases), added to the store job in `pd-store-ci.yml`. It checks that the `rocksdb` key is absent when the variable is unset or empty, that bytes and `0` pass through, that `app.data-path` is still set, and that `4GB`, `-1`, `1.5e9`, `1 2` and `1"}` are refused. It clears the variable from the caller's environment first, so "unset" really is unset.
- `mvn test -pl hugegraph-store/hg-store-test -P store-core-test` on JDK 17: 113 tests, 0 failures. `helm unittest helm/hugegraph`: 294 tests pass. `mvn editorconfig:check` and `git diff --check` are clean.
- Not tested: a full Store started in a memory-limited container, and the load from the issue.
- CI selection gap, left as is: a PR that changes only `test-store-docker-entrypoint.sh` doesn't select the store job, because the script isn't in `CONTRACT_SUITES` in `.github/scripts/ci-policy.py`. Changing the entrypoint itself does select it. The PD entrypoint test has the same gap, so I'd rather fix both in one place.
## Does this PR potentially affect the following parts?
- [ ] Dependencies ([add/update license](https://hugegraph.apache.org/docs/contribution-guidelines/contribute/#321-check-licenses) info & [regenerate_known_dependencies.sh](../install-dist/scripts/dependency/regenerate_known_dependencies.sh))
- [x] Modify configurations
- [ ] The public API
- [ ] Other affects (typed here)
- [ ] Nope
## Documentation Status
- [x] `Doc - TODO`: required documentation is pending; complete it before merging.
- [ ] `Doc - Done`: documentation is included here or linked below.
- [ ] `Doc - No Need`: no user-visible documentation is affected.
Documentation files in this PR or paired hugegraph-doc PR:
- `hugegraph-store/docs/deployment-guide.md`: the auto-sizing rule and its inputs, the 1 MiB minimum, the new env var and its precedence in the compose and Kubernetes examples, and the RocksDB block (`total_memory_size: 0`; the nonexistent `block_cache_size` key replaced with `write_buffer_ratio`).
- `hugegraph-store/README.md`: the env var in the `docker run` section.
- Paired apache/hugegraph-doc PR, link to follow: `content/{en,cn}/docs/quickstart/hugegraph/hugegraph-hstore.md` (the default, the auto-sizing rule and the env var) and `content/{en,cn}/docs/guides/hugegraph-docker-cluster.md` (the new `HG_STORE_*` row). The hstore page's "checked against Server master commit" note should move to this PR's merge commit once both land.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4383329491
##########
hugegraph-store/docs/deployment-guide.md:
##########
@@ -356,14 +356,76 @@ rocksdb:
level0_file_num_compaction_trigger: 4
max_background_jobs: 8 # Background compaction/flush threads
- # Block Cache
- block_cache_size: 16000000000 # Block cache (16GB)
+ # Budget split: this share goes to memtables, the rest to the block cache
+ write_buffer_ratio: 0.66
# SST File Size
target_file_size_base: 134217728 # Target SST size (128MB)
max_bytes_for_level_base: 1073741824 # L1 size (1GB)
```
+`total_memory_size` bounds the memtables (`write_buffer_ratio` of it) and the block cache
+(the rest). Both live outside the Java heap, so `-Xmx` does not limit them. A fixed 1 GiB
+cache for the Raft log comes on top of this budget; the Raft log memtables of all partitions
+share a 256 MiB write buffer charged to that cache, so they don't add to it.
+
+With `total_memory_size: auto` (the shipped value; a missing, blank or `0` value means the same)
+the Store sizes the budget at startup as half of what is left after the heap, the maximum direct
+memory and a 1.5 GiB reserve (the 1 GiB Raft log cache plus 512 MiB for other native memory),
+clamped to 256 MiB–32 GB. Inside a container the starting point is the cgroup memory limit, including a
+limit set only on a parent cgroup when the Store can see that cgroup; inside a private cgroup
+namespace, or when only a subtree of the cgroup2 hierarchy is mounted, a limit above it is not
+visible, so pin the value there if only the pod or slice is limited. The Store uses the lowest
+limit it can see and logs a warning when it can't see the whole hierarchy; it doesn't guess at
+limits it can't read. A Store with a 4 GiB limit, `-Xmx1g` and `-XX:MaxDirectMemorySize=512m`
+gets 512 MiB. Without
+`-XX:MaxDirectMemorySize` the JVM allows as much direct memory as heap, and the budget counts
+it that way. `bin/start-hugegraph-store.sh` sets the flag to the heap it picks; when `-j` sets
+`-Xmx`, `-XX:MaxHeapSize` or `-XX:MaxDirectMemorySize` itself, the launcher leaves the flag out and the JVM's
+default (that heap) or the given cap applies. Either way the budget counts a limit the JVM
+enforces. In small containers set a
+cap below the heap, or the budget falls to the 256 MiB floor.
+
+On a host without a container limit the starting point is physical memory. The default
+`bin/start-hugegraph-store.sh` caps the heap at 2 GiB and direct memory at the same 2 GiB, so a
+16 GiB host gets about 5.25 GiB. When you raise the heap, pass a direct-memory cap with it,
+for example `-j "-Xmx36g -XX:MaxDirectMemorySize=2g"`; otherwise direct memory counts as a
+second heap. With a 2 GiB cap a 16 GiB host with `-Xmx8g` gets 2.25 GiB and a 64 GiB host with
+`-Xmx36g` gets 12.25 GiB. Before this default every host got a fixed `32000000000`, so pin the
+value after upgrading if a large node relied on it.
+
+At startup the Store logs the budget and what it was computed from, for example:
+
+```text
+RocksDB memory budget: 512 MiB (536870912 bytes), mode=auto, physical-limit=4 GiB, source=cgroup-v2, heap=1 GiB, direct=512 MiB, reserve=1.5 GiB, cgroup-view=complete
+```
+
+It warns when the floor applies, or when an explicit value is under 256 MiB or exceeds the
+memory left after the heap, direct memory and the reserve. Set a number of bytes, at least
+1048576 (1 MiB), to pin the budget; a smaller value or a unit suffix such as `4GB` is rejected. Before this default existed the
+shipped value was `32000000000`, which can exceed a container limit and get the Store
+OOM-killed. Both caps are soft: RocksDB can briefly go over them under heavy writes, and a
+small budget with many partitions means frequent memtable flushes.
+
+**Upgrading an existing deployment.** Auto-sizing applies only when the loaded configuration
+has `total_memory_size: auto` (or `0`, or no value). A Store that mounts its own `conf/application-pd.yml`,
+or a tarball install that kept its `conf/` from an earlier release, still has `32000000000` and
+keeps that fixed budget; the Store logs a warning naming the old value at startup. Set the key
+to `auto`, or set `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` in Docker.
+
+PD can also hold the old value. When PD has no shared copy yet, the first Store to start uploads
+its local `application-pd.yml` to PD (KV key `S:FS`). At startup a Store loads that copy over its
+local file only when it has no local file or the file is no newer than PD's `S:Timestamp`. The
+upload sets no timestamp and a missing one counts as 0, so changing `S:FS` alone changes nothing.
+The simplest upgrade is to set `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE=auto` (or
+`-Drocksdb.total_memory_size=auto`) on every Store and restart it; the Store keeps this value
+above both its local file and PD's copy. To change the shared copy instead, set
+`rocksdb.total_memory_size` to `auto` in `S:FS`, then put the current time in
Review Comment:
Important: publishing `S:Timestamp` restarts every running Store, and a Store still on an earlier release reloads `auto` from `S:FS` and fails in `AppConfig.init()` at `Long.parseLong("auto")`, so a rolling upgrade or rollback leaves it unable to start. Recommend `0` (or removing the key) in the shared `S:FS` copy, which both releases accept, or state that every Store must run this release before `S:FS` gets `auto`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#issuecomment-6422165850
Hi @byteayan (after I check it again, the PR need changes):
Please simplify PR #3355 without weakening the original OOM fix.
Focus on these changes:
1. **Keep the automatic RocksDB memory budget**, including heap, direct memory, container/cgroup limit, native reserve, and the shared Raft log `WriteBufferManager`. These protections are necessary.
2. **Reduce the cgroup implementation weight**:
- Move cgroup v1/v2 detection into a small independent helper such as `CgroupMemoryLimit`.
- Keep `RocksdbMemoryBudget` responsible only for parsing the setting and calculating the budget.
- Treat hidden parent limits as an observability boundary: use the lowest visible limit and warn when the hierarchy is incomplete. Do not add more heuristics to guess invisible limits.
3. **Improve the configuration semantics**:
- Prefer `rocksdb.total_memory_size: auto` in the shipped configuration.
- Keep missing, blank, and legacy `0` as backward-compatible aliases for `auto`.
- Keep positive integers as explicit byte values.
- Allow `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE=auto`.
- Do not add unit parsing such as `4GB` in this PR.
4. **Simplify PD/Spring precedence handling**:
- When a JSON or `-Drocksdb.total_memory_size` override exists, remove only this key from the PD property source before adding it.
- Let Spring preserve its native precedence instead of creating another synthetic property source.
5. **Separate unrelated calculations**:
- Move `disruptorBufferSize()` and `writeBufferSize()` out of `RocksdbMemoryBudget`, since they are Store tuning rules rather than cgroup or process-memory detection.
6. **Make constants explicit and reusable**:
- Separate the 1 GiB Raft log cache reserve from the 512 MiB other-native reserve.
- Avoid duplicating the Raft cache size between the budget calculation and `RaftRocksdbOptions`.
7. **Improve startup diagnostics**:
Log the selected budget and its inputs in readable units, for example:
```text
RocksDB memory budget: 512 MiB
mode=auto
physical-limit=4 GiB
source=cgroup-v2
heap=1 GiB
direct=512 MiB
reserve=1.5 GiB
cgroup-view=complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4399521318
##########
hugegraph-store/hg-store-dist/src/assembly/static/bin/start-hugegraph-store.sh:
##########
@@ -169,7 +169,18 @@ if [ "$JAVA_OPTIONS" = "" ]; then
>> ${OUTPUT}
exit 1
fi
- JAVA_OPTIONS="-Xms${MIN_MEM}m -Xmx${XMX}m -XX:MetaspaceSize=256M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${LOGS} ${USER_OPTION} "
+ # Cap direct memory at the heap size picked here, the JVM's own unset default, so the
+ # RocksDB auto-sizing (RocksdbMemoryBudget) counts a limit the JVM actually enforces.
+ # Leave it out when -j sets the heap or the cap itself: a fixed cap from the computed
+ # heap would no longer match a -j -Xmx, and the JVM then defaults it to that heap.
+ DIRECT_MEMORY_OPTION="-XX:MaxDirectMemorySize=${XMX}m"
Review Comment:
Fixed in 527cfdd6c, without eval. The launcher now strips every single and double quote from `JAVA_TOOL_OPTIONS` and `JDK_JAVA_OPTIONS`, as the JVM does anywhere in a token, and turns tabs and newlines into spaces. It then matches ` -XX:MaxDirectMemorySize=` at a token start with a single `case`, so nothing is word-split or globbed. A false match, such as that text inside a quoted `-D` value, only leaves the cap to the JVM, whose default is the same heap the launcher picked. New launcher cases cover a single- and a double-quoted token in each variable, an inner quote (`-XX:"MaxDirectMemorySize"=64m`) and a tab separator; a control `-Dt=-XX:MaxDirectMemorySize=64m` still gets the default cap. All 27 cases pass on bash 3.2 and 5.3, and the five quoted cases fail against the previous launcher. On JDK 17, all three quoted forms give 64 MiB without the launcher flag and 1 GiB with it, matching your OpenJDK 21 result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4343573565
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,458 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
+ static final long MIN_AUTO = 256 * MIB;
+ // RaftRocksdbOptions gives the write buffer manager 2/3 of the budget, and a 0-byte
+ // write buffer manager means no memtable limit at all
+ static final long MIN_EXPLICIT = MIB;
+ // The value shipped before auto-sizing; a large host never gets more than it had
+ static final long MAX_AUTO = 32_000_000_000L;
+
+ private RocksdbMemoryBudget() {
+ }
+
+ /**
+ * Returns the configured budget in bytes, or an automatic one when the value is
+ * missing, blank or 0.
+ */
+ public static long resolve(String configured) {
+ long value = isAuto(configured) ? 0L : parse(configured);
+ long heap = Runtime.getRuntime().maxMemory();
+ CgroupLimit cgroup = cgroupLimit(Paths.get("/proc/self/cgroup"),
+ Paths.get("/proc/self/mountinfo"),
+ Paths.get("/sys/fs/cgroup"));
+ long physical = effectivePhysical(jvmMemory(), cgroup.bytes);
+ if (value > 0) {
+ if (value < MIN_EXPLICIT) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be 0 (auto) or at least %s bytes, " +
+ "got %s", MIN_EXPLICIT, value));
+ }
+ if (value < MIN_AUTO) {
+ log.warn("rocksdb.total_memory_size={} is below the {} bytes auto-sizing " +
+ "never goes under", value, MIN_AUTO);
+ }
+ if (value == MAX_AUTO) {
+ // Most likely an application-pd.yml kept from before auto-sizing
+ log.warn("rocksdb.total_memory_size={} is the fixed value shipped before " +
+ "auto-sizing; set it to 0 or remove it to size the budget from " +
+ "the available memory", value);
+ }
+ long direct = maxDirectMemory(heap);
+ long left = physical - heap - direct - RESERVED;
+ if (physical > 0 && value > left) {
+ log.warn("rocksdb.total_memory_size={} exceeds the {} bytes left after the " +
+ "heap, direct memory and the {} bytes reserve (physical={}, heap={}, " +
+ "direct={}); the process may be OOM-killed",
+ value, left, RESERVED, physical, heap, direct);
+ }
+ return value;
+ }
+ long direct = maxDirectMemory(heap);
+ long budget = auto(physical, heap, direct);
+ log.info("rocksdb.total_memory_size is unset or 0, auto-sizing to {} bytes " +
+ "(physical={}, heap={}, direct={})", budget, physical, heap, direct);
+ if (cgroup.incomplete) {
+ log.warn("The cgroup mounts in /proc/self/mountinfo don't show this process's " +
+ "whole cgroup hierarchy (an inherited mount in a cgroup namespace, or " +
+ "only cgroup2 subtree mounts), so a tighter limit on the process's " +
+ "cgroup or above the mounted subtree may be missed; set " +
+ "rocksdb.total_memory_size or HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE " +
+ "explicitly");
+ }
+ if (physical > 0 && physical - heap - direct - RESERVED < 2 * MIN_AUTO) {
+ log.warn("Heap, direct memory and the {} bytes reserve leave too little of the " +
+ "{} bytes physical memory, rocksdb gets the {} bytes floor; lower -Xmx " +
+ "or -XX:MaxDirectMemorySize", RESERVED, physical, MIN_AUTO);
+ }
+ return budget;
+ }
+
+ static boolean isAuto(String configured) {
+ return configured == null || configured.trim().isEmpty();
+ }
+
+ static long parse(String configured) {
+ try {
+ long value = Long.parseLong(configured.trim());
+ if (value < 0) {
+ throw new NumberFormatException("negative");
+ }
+ return value;
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be a number of bytes, got '%s'",
+ configured), e);
+ }
+ }
+
+ /**
+ * Half of what is left after the heap, direct memory and the reserve, clamped to
+ * [MIN_AUTO, MAX_AUTO]. Falls back to the heap size, the previous default, when
+ * physical memory is unknown.
+ */
+ static long auto(long physical, long heap, long direct) {
+ if (physical <= 0) {
+ return Math.min(Math.max(heap, MIN_AUTO), MAX_AUTO);
+ }
+ long available = physical - heap - direct - RESERVED;
+ return Math.min(Math.max(available / 2, MIN_AUTO), MAX_AUTO);
+ }
+
+ /**
+ * The raft disruptor buffer size AppConfig derives from the budget when
+ * raft.disruptorBufferSize is 0: the budget in GB, at least 1, rounded to a power
+ * of two, times 32.
+ */
+ public static int disruptorBufferSize(long totalMemory) {
+ // A budget under 1 GB would otherwise give log(0) and a zero-sized buffer
+ int size = (int) Math.max(1, totalMemory / 1000 / 1000 / 1000);
+ return (int) Math.pow(2, Math.round(Math.log(size) / Math.log(2))) * 32;
+ }
+
+ /**
+ * The write_buffer_size AppConfig derives when the key is unset or 0: a thousandth
+ * of the budget, but never below the 1 MiB rocksdb.write_buffer_size accepts
+ * (RocksDBOptions.MEMTABLE_SIZE), which an auto-sized budget under 1 GiB would
+ * otherwise fall under.
+ */
+ public static long writeBufferSize(long totalMemory) {
+ return Math.max(totalMemory / 1000, MIB);
+ }
+
+ private static long jvmMemory() {
+ try {
+ // Reports the cgroup v1/v2 memory limit inside a container, but older JDK 17
+ // updates may not see a limit set only on a parent cgroup
+ OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(
+ OperatingSystemMXBean.class);
+ return os == null ? -1L : os.getTotalMemorySize();
+ } catch (RuntimeException | LinkageError e) {
+ log.warn("Failed to read physical memory size", e);
+ return -1L;
+ }
+ }
+
+ /**
+ * A cgroup memory limit in bytes (-1 for none), and whether the mounts may hide a
+ * stricter one: a cgroup mount existed but none contained this process's cgroup, or
+ * every matching cgroup2 mount was a subtree, so bytes only covers what is visible
+ * from those mounts' roots down.
+ */
+ static final class CgroupLimit {
+
+ final long bytes;
+ final boolean incomplete;
+
+ CgroupLimit(long bytes, boolean incomplete) {
+ this.bytes = bytes;
+ this.incomplete = incomplete;
+ }
+ }
+
+ /**
+ * The smaller of what the JVM reports and the cgroup limit, ignoring unknown (<= 0)
+ * values.
+ */
+ static long effectivePhysical(long jvm, long cgroup) {
+ if (cgroup <= 0) {
+ return jvm;
+ }
+ return jvm <= 0 ? cgroup : Math.min(jvm, cgroup);
+ }
+
+ /**
+ * The lowest memory limit on this process's cgroup or any visible ancestor, or -1
+ * when there is none or it can't be read. The membership path from /proc/self/cgroup
+ * is rebased onto every mount of the controller in /proc/self/mountinfo whose root
+ * contains it, so a cgroupfs mounted from a subtree (root /kubepods at /sys/fs/cgroup)
+ * is walked from the right directory, and the lowest limit over all those mounts is
+ * taken: a leaf-only or unrelated mount listed first can't hide a fuller view listed
+ * later. Without a matching mount the default root is used. cgroup v2
+ * walks memory.max from the process's cgroup up to the mount; cgroup v1 reads
+ * hierarchical_memory_limit, which already accounts for ancestors. A limit above the
+ * mount root, as inside a private cgroup namespace, stays invisible.
+ */
+ static long cgroupMemoryLimit(Path procSelfCgroup, Path mountinfo, Path defaultRoot) {
+ return cgroupLimit(procSelfCgroup, mountinfo, defaultRoot).bytes;
+ }
+
+ static CgroupLimit cgroupLimit(Path procSelfCgroup, Path mountinfo, Path defaultRoot) {
+ List<String[]> v2Mounts = cgroupMounts(mountinfo, true);
+ List<String[]> v1Mounts = cgroupMounts(mountinfo, false);
+ List<String> lines;
+ try {
+ lines = Files.readAllLines(procSelfCgroup, StandardCharsets.UTF_8);
+ } catch (IOException | RuntimeException e) {
+ return new CgroupLimit(-1L, false);
+ }
+ long limit = -1L;
+ boolean incomplete = false;
+ for (String line : lines) {
+ String[] parts = line.split(":", 3);
+ if (parts.length < 3) {
+ continue;
+ }
+ boolean v2 = parts[0].equals("0") && parts[1].isEmpty();
+ if (!v2 && !("," + parts[1] + ",").contains(",memory,")) {
+ continue;
+ }
+ List<String[]> mounts = v2 ? v2Mounts : v1Mounts;
+ boolean matched = false;
+ boolean fullView = false;
+ long lineLimit = -1L;
+ for (String[] mount : mounts) {
+ if (!contains(mount[0], parts[2])) {
+ // A mount of another subtree says nothing about this cgroup
+ continue;
+ }
+ matched = true;
+ Path point = Paths.get(mount[1]);
+ String relative = rebase(parts[2], mount[0]);
+ // A full-hierarchy mount only covers the ancestors if this process's own
+ // directory is actually reachable through it
+ fullView |= mount[0].equals("/") && visible(point, relative);
+ lineLimit = lowest(lineLimit, v2 ? cgroupV2Limit(point, relative) :
+ cgroupV1Limit(point, relative));
+ }
+ limit = lowest(limit, lineLimit);
+ if (v2 && matched && parts[2].equals("/") && lineLimit <= 0) {
Review Comment:
⚠️ Important: A finite root `memory.max` does not prove that this is the host hierarchy root. In a private cgroup v2 namespace, `/proc/self/cgroup` and the mount root can both appear as `/`, while a stricter ancestor remains hidden; this branch sets `incomplete` only when the visible root limit is absent or `max`. For example, a visible 8 GiB and hidden parent 4 GiB lets auto-sizing use the wrong 8 GiB ceiling without its explicit-budget warning, so the Store can still exceed its effective cgroup budget. Please treat the namespace-root view as potentially incomplete even with a finite visible limit, and update the regression that currently asserts `incomplete == false` for this case.
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,458 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
Review Comment:
⚠️ Important: The 512 MiB other-allocation reserve does not cover the supported `raft.useRocksDBSegmentLogStorage=false` path. `PartitionEngine` then creates a `RocksDBLogStorage` per local group; `RaftRocksdbOptions` assigns 8 MiB write buffers with up to three per column family and leaves `setWriteBufferManager(...)` disabled. With 64 busy groups, the default log column family alone can hold up to 1.5 GiB of memtables outside this fixed reserve, so a 4 GiB Store can still exceed its limit after the budget is added. Please cap log memtables with a shared `WriteBufferManager` or include active-group count in the auto budget.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#issuecomment-6231014660
@imbajin thanks, closed that edge too in 643db967c. The launcher now splits `-j` the way it reaches the JVM (unquoted, so on spaces, tabs and newlines) and matches each option against `-Xmx*`, `-XX:MaxHeapSize=*` and `-XX:MaxDirectMemorySize=*`, rather than looking for a literal " -Xmx". `test-start-hugegraph-store.sh --gc-only` gains a tab-separated `-Xmx4g` case and a `-XX:MaxHeapSize=4g` case; all 17 pass, and against 119f819a7 exactly those two fail with the computed 1024m cap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#issuecomment-6231325434
@imbajin thanks for tracing that. Master is merged in (a5942ff00), and 9e6dc506b makes the start-server.sh writes idempotent. A `set_property` helper deletes every existing line for the key, dots escaped, before appending it once. It now handles `usePD`, `schema.sync_deletion` and both HBase timeouts. The new `test-start-server.sh` runs the real script twice on one fixture, with init-store and the launcher stubbed, for hstore and hbase, and checks each key appears exactly once. All 7 checks pass, and against master's start-server.sh 5 of them fail on the duplicated keys. The test runs in the Server rocksdb job's startup-contracts step and is mapped in `CONTRACT_SUITES`, and the selector tests cover the mapping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4309570116
##########
.github/scripts/ci-policy.py:
##########
@@ -101,6 +101,8 @@
TRAVIS + "test-check-jacoco-report.sh": {"struct"},
TRAVIS + "test-codecov-upload-config.sh": {"struct"},
TRAVIS + "test-riscv64-java-runtime.sh": {"server", "riscv64"},
+ TRAVIS + "test-store-docker-entrypoint.sh": {"store"},
Review Comment:
⚠️ Important: This PR adds direct-memory cases to `test-start-hugegraph-store.sh`, executed by the `store` job at `pd-store-ci.yml:444`, but `CONTRACT_SUITES` has no entry for that script. If a future change touches only this script, `ci-policy.py::select()` takes the generic `hugegraph-dist` route, which selects `server`, `docker` and macOS checks (plus distributed dependents) but not `store`; these cases would not rerun. Please map the script to `{'store'}` and add it to `test_contract_scripts_reach_their_actual_inner_jobs_and_steps`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4309860399
##########
.github/scripts/ci-policy.py:
##########
@@ -101,6 +101,8 @@
TRAVIS + "test-check-jacoco-report.sh": {"struct"},
TRAVIS + "test-codecov-upload-config.sh": {"struct"},
TRAVIS + "test-riscv64-java-runtime.sh": {"server", "riscv64"},
+ TRAVIS + "test-store-docker-entrypoint.sh": {"store"},
Review Comment:
Agreed, fixed in f4cd98c31. `test-start-hugegraph-store.sh` maps to `{'store'}` in `CONTRACT_SUITES` and is a case in `test_contract_scripts_reach_their_actual_inner_jobs_and_steps`. That step runs under `if: steps.store-preflight.outputs.can_run == 'true'` (fuser/curl/java), so the test now allows exactly that gate on a store contract step and asserts the `store-preflight` step itself has no `if:`. Every other gate still fails the test. With the mapping removed the new case fails; all 38 CI-policy tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4307144928
##########
hugegraph-store/hg-store-dist/docker/docker-entrypoint.sh:
##########
@@ -54,6 +54,22 @@ require_env "HG_STORE_RAFT_ADDRESS"
: "${HG_STORE_GRPC_PORT:=8500}"
: "${HG_STORE_REST_PORT:=8520}"
: "${HG_STORE_DATA_PATH:=/hugegraph-store/storage}"
+: "${HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE:=}"
+
+# Bytes only: the Store parses the value as a long and refuses to start otherwise
+if [[ -n "${HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE}" &&
+ ! "${HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE}" =~ ^[0-9]+$ ]]; then
+ echo "ERROR: HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE must be a number of bytes," \
+ "got '${HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE}'" >&2
+ exit 2
+fi
+
+# Only emitted when set: SPRING_APPLICATION_JSON outranks conf/application-pd.yml
+ROCKSDB_JSON=""
+if [[ -n "${HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE}" ]]; then
+ ROCKSDB_JSON=",
+ \"rocksdb\": { \"total_memory_size\": \"${HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE}\" }"
Review Comment:
⚠️ Important: This value enters through SPRING_APPLICATION_JSON, but StoreNodeApplication installs PdConfigureListener, which can add a PD-managed application-pd.yml at the front of the property sources before RocksdbConfig binds. If that file contains rocksdb.total_memory_size, it shadows HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE, so the documented per-Store override can be ignored and an old large budget can remain active. Please preserve this environment value as highest priority after PD configuration is loaded and cover that precedence case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4295706995
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.lang.management.ManagementFactory;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
+ static final long MIN_AUTO = 256 * MIB;
+ // Direct memory counted when -XX:MaxDirectMemorySize is unset. The JVM then allows
+ // as much as the heap, but the Store's gRPC/Netty buffers do not grow with the heap,
+ // and counting a 36 GB heap twice would leave rocksdb the floor on a large host
+ static final long UNSET_DIRECT = 1024 * MIB;
+ // RaftRocksdbOptions gives the write buffer manager 2/3 of the budget, and a 0-byte
+ // write buffer manager means no memtable limit at all
+ static final long MIN_EXPLICIT = MIB;
+ // The value shipped before auto-sizing; a large host never gets more than it had
+ static final long MAX_AUTO = 32_000_000_000L;
+
+ private RocksdbMemoryBudget() {
+ }
+
+ /**
+ * Returns the configured budget in bytes, or an automatic one when the value is
+ * missing, blank or 0.
+ */
+ public static long resolve(String configured) {
+ long value = isAuto(configured) ? 0L : parse(configured);
+ long heap = Runtime.getRuntime().maxMemory();
+ long physical = physicalMemory();
+ if (value > 0) {
+ if (value < MIN_EXPLICIT) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be 0 (auto) or at least %s bytes, " +
+ "got %s", MIN_EXPLICIT, value));
+ }
+ if (value < MIN_AUTO) {
+ log.warn("rocksdb.total_memory_size={} is below the {} bytes auto-sizing " +
+ "never goes under", value, MIN_AUTO);
+ }
+ long direct = maxDirectMemory(heap);
+ long left = physical - heap - direct - RESERVED;
+ if (physical > 0 && value > left) {
+ log.warn("rocksdb.total_memory_size={} exceeds the {} bytes left after the " +
+ "heap, direct memory and the {} bytes reserve (physical={}, heap={}, " +
+ "direct={}); the process may be OOM-killed",
+ value, left, RESERVED, physical, heap, direct);
+ }
+ return value;
+ }
+ long direct = maxDirectMemory(heap);
+ long budget = auto(physical, heap, direct);
+ log.info("rocksdb.total_memory_size is unset or 0, auto-sizing to {} bytes " +
+ "(physical={}, heap={}, direct={})", budget, physical, heap, direct);
+ if (physical > 0 && physical - heap - direct - RESERVED < 2 * MIN_AUTO) {
+ log.warn("Heap, direct memory and the {} bytes reserve leave too little of the " +
+ "{} bytes physical memory, rocksdb gets the {} bytes floor; lower -Xmx " +
+ "or -XX:MaxDirectMemorySize", RESERVED, physical, MIN_AUTO);
+ }
+ return budget;
+ }
+
+ static boolean isAuto(String configured) {
+ return configured == null || configured.trim().isEmpty();
+ }
+
+ static long parse(String configured) {
+ try {
+ long value = Long.parseLong(configured.trim());
+ if (value < 0) {
+ throw new NumberFormatException("negative");
+ }
+ return value;
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be a number of bytes, got '%s'",
+ configured), e);
+ }
+ }
+
+ /**
+ * Half of what is left after the heap, direct memory and the reserve, clamped to
+ * [MIN_AUTO, MAX_AUTO]. Falls back to the heap size, the previous default, when
+ * physical memory is unknown.
+ */
+ static long auto(long physical, long heap, long direct) {
+ if (physical <= 0) {
+ return Math.min(Math.max(heap, MIN_AUTO), MAX_AUTO);
+ }
+ long available = physical - heap - direct - RESERVED;
+ return Math.min(Math.max(available / 2, MIN_AUTO), MAX_AUTO);
+ }
+
+ /**
+ * The raft disruptor buffer size AppConfig derives from the budget when
+ * raft.disruptorBufferSize is 0: the budget in GB, at least 1, rounded to a power
+ * of two, times 32.
+ */
+ public static int disruptorBufferSize(long totalMemory) {
+ // A budget under 1 GB would otherwise give log(0) and a zero-sized buffer
+ int size = (int) Math.max(1, totalMemory / 1000 / 1000 / 1000);
+ return (int) Math.pow(2, Math.round(Math.log(size) / Math.log(2))) * 32;
+ }
+
+ /**
+ * The write_buffer_size AppConfig derives when the key is unset or 0: a thousandth
+ * of the budget, but never below the 1 MiB rocksdb.write_buffer_size accepts
+ * (RocksDBOptions.MEMTABLE_SIZE), which an auto-sized budget under 1 GiB would
+ * otherwise fall under.
+ */
+ public static long writeBufferSize(long totalMemory) {
+ return Math.max(totalMemory / 1000, MIB);
+ }
+
+ private static long physicalMemory() {
+ try {
+ // Reports the cgroup v1/v2 memory limit inside a container; older JDK 17
+ // updates may not see a limit set only on a parent cgroup
+ OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(
+ OperatingSystemMXBean.class);
+ return os == null ? -1L : os.getTotalMemorySize();
Review Comment:
Fixed in 03a4d4367 by detecting it rather than requiring a JDK update. `RocksdbMemoryBudget` now reads the cgroup itself: on v2 it walks `memory.max` from the process's cgroup (from `/proc/self/cgroup`) up to the mount root, on v1 it reads `hierarchical_memory_limit`, which already includes ancestors. It uses the lower of that and what the JVM reports. New tests build fake v1 and v2 trees, including your case: a 64 GiB host reported by the JVM, `-Xmx1g`, a 512 MiB direct cap and a 4 GiB limit only on the pod now gives 512 MiB. One limit remains: inside a private cgroup namespace the mount root is the container's own cgroup, so a limit above it can't be seen from inside. The deployment guide says so and tells users to pin the value in that setup.
##########
hugegraph-store/docs/deployment-guide.md:
##########
@@ -726,8 +751,15 @@ environment:
HG_STORE_REST_PORT: "8520" # maps to server.port
HG_STORE_RAFT_ADDRESS: store0:8510 # maps to raft.address
HG_STORE_DATA_PATH: /hugegraph-store/storage # maps to app.data-path
+ # optional; maps to rocksdb.total_memory_size (bytes); unset = config file or auto-size
+ # HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE: "2147483648"
```
+`HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` must be a plain number of bytes; the entrypoint refuses
+anything else. When set it outranks `rocksdb.total_memory_size` in a mounted
+`conf/application-pd.yml` and a `-Drocksdb.total_memory_size` in `JAVA_OPTS`. Leave it unset to let the Store size the budget from the container
Review Comment:
Added in 03a4d4367: the guide has an "Upgrading an existing deployment" step (set `total_memory_size` to 0 or remove it, or set `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE`), and this paragraph now says that leaving the variable unset only auto-sizes when the loaded config is also 0 or unset. The Store also logs a WARN when it sees exactly `32000000000`, the old shipped value, telling the operator to set it to 0 or remove it.
##########
.github/workflows/pd-store-ci.yml:
##########
@@ -406,6 +406,10 @@ jobs:
run: |
mvn editorconfig:check -pl hugegraph-store/hg-store-test -am -ntp
+ - name: Run Store docker entrypoint rocksdb memory tests
Review Comment:
Fixed in 3fb0da01f: `test-store-docker-entrypoint.sh` maps to `{"store"}` in `CONTRACT_SUITES`, and `test_contract_scripts_reach_their_actual_inner_jobs_and_steps` now covers it, checking the `store` job's condition and that the step has no `if:` of its own. With the mapping removed, that test fails for this script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4399272963
##########
hugegraph-store/hg-store-dist/src/assembly/static/bin/start-hugegraph-store.sh:
##########
@@ -169,7 +169,18 @@ if [ "$JAVA_OPTIONS" = "" ]; then
>> ${OUTPUT}
exit 1
fi
- JAVA_OPTIONS="-Xms${MIN_MEM}m -Xmx${XMX}m -XX:MetaspaceSize=256M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${LOGS} ${USER_OPTION} "
+ # Cap direct memory at the heap size picked here, the JVM's own unset default, so the
+ # RocksDB auto-sizing (RocksdbMemoryBudget) counts a limit the JVM actually enforces.
+ # Leave it out when -j sets the heap or the cap itself: a fixed cap from the computed
+ # heap would no longer match a -j -Xmx, and the JVM then defaults it to that heap.
+ DIRECT_MEMORY_OPTION="-XX:MaxDirectMemorySize=${XMX}m"
Review Comment:
The unquoted cases are fixed, but this remains incomplete for JVM-valid quoted option tokens. At [lines 186–188](https://github.com/apache/hugegraph/blob/7fdd54ab79e6ba8e205caed512d768cf3ec03562/hugegraph-store/hg-store-dist/src/assembly/static/bin/start-hugegraph-store.sh#L186-L188), shell word splitting retains quote characters, so a whole quoted -XX:MaxDirectMemorySize=64m token in either JAVA_TOOL_OPTIONS or JDK_JAVA_OPTIONS does not match the guard.
An isolated exact-source launcher fixture with calc_xmx=1024 still emits -XX:MaxDirectMemorySize=1024m. On dots OpenJDK 21.0.12.1, bounded -Xms16m -Xmx32m -XX:+PrintFlagsFinal -version checks confirmed 67108864 without that generated flag, but 1073741824 with it. Both single- and double-quoted tokens reproduce in both variables (four cases), while both unquoted controls preserve 64 MiB. This verifies a raised effective limit, not actual direct-memory allocation or OOM; no Store process or stress test was run, and Java17 runtime behavior was not independently tested.
Please account for JVM option quoting without eval and add these quoted-token regressions. The existing 20 launcher fixture cases and 11 entrypoint fixture cases passed. This is a narrow remaining case, not a request to redesign the memory budget mechanism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4295803469
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.lang.management.ManagementFactory;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
+ static final long MIN_AUTO = 256 * MIB;
+ // Direct memory counted when -XX:MaxDirectMemorySize is unset. The JVM then allows
+ // as much as the heap, but the Store's gRPC/Netty buffers do not grow with the heap,
+ // and counting a 36 GB heap twice would leave rocksdb the floor on a large host
+ static final long UNSET_DIRECT = 1024 * MIB;
+ // RaftRocksdbOptions gives the write buffer manager 2/3 of the budget, and a 0-byte
+ // write buffer manager means no memtable limit at all
+ static final long MIN_EXPLICIT = MIB;
+ // The value shipped before auto-sizing; a large host never gets more than it had
+ static final long MAX_AUTO = 32_000_000_000L;
+
+ private RocksdbMemoryBudget() {
+ }
+
+ /**
+ * Returns the configured budget in bytes, or an automatic one when the value is
+ * missing, blank or 0.
+ */
+ public static long resolve(String configured) {
+ long value = isAuto(configured) ? 0L : parse(configured);
+ long heap = Runtime.getRuntime().maxMemory();
+ long physical = physicalMemory();
+ if (value > 0) {
+ if (value < MIN_EXPLICIT) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be 0 (auto) or at least %s bytes, " +
+ "got %s", MIN_EXPLICIT, value));
+ }
+ if (value < MIN_AUTO) {
+ log.warn("rocksdb.total_memory_size={} is below the {} bytes auto-sizing " +
+ "never goes under", value, MIN_AUTO);
+ }
+ long direct = maxDirectMemory(heap);
+ long left = physical - heap - direct - RESERVED;
+ if (physical > 0 && value > left) {
+ log.warn("rocksdb.total_memory_size={} exceeds the {} bytes left after the " +
+ "heap, direct memory and the {} bytes reserve (physical={}, heap={}, " +
+ "direct={}); the process may be OOM-killed",
+ value, left, RESERVED, physical, heap, direct);
+ }
+ return value;
+ }
+ long direct = maxDirectMemory(heap);
+ long budget = auto(physical, heap, direct);
+ log.info("rocksdb.total_memory_size is unset or 0, auto-sizing to {} bytes " +
+ "(physical={}, heap={}, direct={})", budget, physical, heap, direct);
+ if (physical > 0 && physical - heap - direct - RESERVED < 2 * MIN_AUTO) {
+ log.warn("Heap, direct memory and the {} bytes reserve leave too little of the " +
+ "{} bytes physical memory, rocksdb gets the {} bytes floor; lower -Xmx " +
+ "or -XX:MaxDirectMemorySize", RESERVED, physical, MIN_AUTO);
+ }
+ return budget;
+ }
+
+ static boolean isAuto(String configured) {
+ return configured == null || configured.trim().isEmpty();
+ }
+
+ static long parse(String configured) {
+ try {
+ long value = Long.parseLong(configured.trim());
+ if (value < 0) {
+ throw new NumberFormatException("negative");
+ }
+ return value;
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be a number of bytes, got '%s'",
+ configured), e);
+ }
+ }
+
+ /**
+ * Half of what is left after the heap, direct memory and the reserve, clamped to
+ * [MIN_AUTO, MAX_AUTO]. Falls back to the heap size, the previous default, when
+ * physical memory is unknown.
+ */
+ static long auto(long physical, long heap, long direct) {
+ if (physical <= 0) {
+ return Math.min(Math.max(heap, MIN_AUTO), MAX_AUTO);
+ }
+ long available = physical - heap - direct - RESERVED;
+ return Math.min(Math.max(available / 2, MIN_AUTO), MAX_AUTO);
+ }
+
+ /**
+ * The raft disruptor buffer size AppConfig derives from the budget when
+ * raft.disruptorBufferSize is 0: the budget in GB, at least 1, rounded to a power
+ * of two, times 32.
+ */
+ public static int disruptorBufferSize(long totalMemory) {
+ // A budget under 1 GB would otherwise give log(0) and a zero-sized buffer
+ int size = (int) Math.max(1, totalMemory / 1000 / 1000 / 1000);
+ return (int) Math.pow(2, Math.round(Math.log(size) / Math.log(2))) * 32;
+ }
+
+ /**
+ * The write_buffer_size AppConfig derives when the key is unset or 0: a thousandth
+ * of the budget, but never below the 1 MiB rocksdb.write_buffer_size accepts
+ * (RocksDBOptions.MEMTABLE_SIZE), which an auto-sized budget under 1 GiB would
+ * otherwise fall under.
+ */
+ public static long writeBufferSize(long totalMemory) {
+ return Math.max(totalMemory / 1000, MIB);
+ }
+
+ private static long physicalMemory() {
+ try {
+ // Reports the cgroup v1/v2 memory limit inside a container; older JDK 17
+ // updates may not see a limit set only on a parent cgroup
+ OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(
+ OperatingSystemMXBean.class);
+ return os == null ? -1L : os.getTotalMemorySize();
Review Comment:
A remaining mount-root translation case is independently reproducible at this revision: https://github.com/apache/hugegraph/blob/03a4d436754274d144d76101bf43c496bfb2a0d8/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java#L218-L226 . With membership 0::/kubepods/pod1/ctr and a cgroup filesystem whose mountinfo root is /kubepods at /sys/fs/cgroup, the readable effective limit is /sys/fs/cgroup/pod1/memory.max. The resolver instead probes /sys/fs/cgroup/kubepods/pod1/ctr and falls back to the unlimited mount root, skipping that visible ancestor. This differs from an ancestor hidden above the mount root. A bounded Java 21 synthetic-file harness using the current resolver (only Lombok logging replaced with a no-op stub) produced 32,000,000,000 bytes with a simulated 64 GiB host-memory report, 1 GiB heap and 512 MiB direct cap; correct rebasing finds the 4 GiB limit and yields 512 MiB. Please resolve the controller mount and translate
the membership path relative to its mountinfo root before walking ancestors, with an intermediate-subtree regression. This is conditional on the subtree layout and a JVM that misses the effective limit, not every container. No real mounts, JDK17 Store startup or load test was performed. The current CI-policy 20 tests and entrypoint-contract 9 cases passed independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#issuecomment-6230879191
@imbajin thanks, good catch on the heap-only `-j`. Fixed in 119f819a7: the launcher now adds `-XX:MaxDirectMemorySize=<computed heap>` only when `-j` sets neither `-Xmx` nor `-XX:MaxDirectMemorySize`. With a heap-only `-j` the JVM defaults the cap to that heap, and with an explicit cap the cap applies; the budget counts whichever the JVM reports. `test-start-hugegraph-store.sh` (already run in the store CI job) gains four cases: no `-j` gives the computed 1024m cap, `-j -Xmx512m` and `-j -Xmx4g` leave the cap out, and `-j -XX:MaxDirectMemorySize=256m` ends with 256m. Against the previous launcher both heap-only cases fail with the stale 1024m cap. The deployment guide now describes this path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4295872216
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.lang.management.ManagementFactory;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
+ static final long MIN_AUTO = 256 * MIB;
+ // Direct memory counted when -XX:MaxDirectMemorySize is unset. The JVM then allows
+ // as much as the heap, but the Store's gRPC/Netty buffers do not grow with the heap,
+ // and counting a 36 GB heap twice would leave rocksdb the floor on a large host
+ static final long UNSET_DIRECT = 1024 * MIB;
+ // RaftRocksdbOptions gives the write buffer manager 2/3 of the budget, and a 0-byte
+ // write buffer manager means no memtable limit at all
+ static final long MIN_EXPLICIT = MIB;
+ // The value shipped before auto-sizing; a large host never gets more than it had
+ static final long MAX_AUTO = 32_000_000_000L;
+
+ private RocksdbMemoryBudget() {
+ }
+
+ /**
+ * Returns the configured budget in bytes, or an automatic one when the value is
+ * missing, blank or 0.
+ */
+ public static long resolve(String configured) {
+ long value = isAuto(configured) ? 0L : parse(configured);
+ long heap = Runtime.getRuntime().maxMemory();
+ long physical = physicalMemory();
+ if (value > 0) {
+ if (value < MIN_EXPLICIT) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be 0 (auto) or at least %s bytes, " +
+ "got %s", MIN_EXPLICIT, value));
+ }
+ if (value < MIN_AUTO) {
+ log.warn("rocksdb.total_memory_size={} is below the {} bytes auto-sizing " +
+ "never goes under", value, MIN_AUTO);
+ }
+ long direct = maxDirectMemory(heap);
+ long left = physical - heap - direct - RESERVED;
+ if (physical > 0 && value > left) {
+ log.warn("rocksdb.total_memory_size={} exceeds the {} bytes left after the " +
+ "heap, direct memory and the {} bytes reserve (physical={}, heap={}, " +
+ "direct={}); the process may be OOM-killed",
+ value, left, RESERVED, physical, heap, direct);
+ }
+ return value;
+ }
+ long direct = maxDirectMemory(heap);
+ long budget = auto(physical, heap, direct);
+ log.info("rocksdb.total_memory_size is unset or 0, auto-sizing to {} bytes " +
+ "(physical={}, heap={}, direct={})", budget, physical, heap, direct);
+ if (physical > 0 && physical - heap - direct - RESERVED < 2 * MIN_AUTO) {
+ log.warn("Heap, direct memory and the {} bytes reserve leave too little of the " +
+ "{} bytes physical memory, rocksdb gets the {} bytes floor; lower -Xmx " +
+ "or -XX:MaxDirectMemorySize", RESERVED, physical, MIN_AUTO);
+ }
+ return budget;
+ }
+
+ static boolean isAuto(String configured) {
+ return configured == null || configured.trim().isEmpty();
+ }
+
+ static long parse(String configured) {
+ try {
+ long value = Long.parseLong(configured.trim());
+ if (value < 0) {
+ throw new NumberFormatException("negative");
+ }
+ return value;
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be a number of bytes, got '%s'",
+ configured), e);
+ }
+ }
+
+ /**
+ * Half of what is left after the heap, direct memory and the reserve, clamped to
+ * [MIN_AUTO, MAX_AUTO]. Falls back to the heap size, the previous default, when
+ * physical memory is unknown.
+ */
+ static long auto(long physical, long heap, long direct) {
+ if (physical <= 0) {
+ return Math.min(Math.max(heap, MIN_AUTO), MAX_AUTO);
+ }
+ long available = physical - heap - direct - RESERVED;
+ return Math.min(Math.max(available / 2, MIN_AUTO), MAX_AUTO);
+ }
+
+ /**
+ * The raft disruptor buffer size AppConfig derives from the budget when
+ * raft.disruptorBufferSize is 0: the budget in GB, at least 1, rounded to a power
+ * of two, times 32.
+ */
+ public static int disruptorBufferSize(long totalMemory) {
+ // A budget under 1 GB would otherwise give log(0) and a zero-sized buffer
+ int size = (int) Math.max(1, totalMemory / 1000 / 1000 / 1000);
+ return (int) Math.pow(2, Math.round(Math.log(size) / Math.log(2))) * 32;
+ }
+
+ /**
+ * The write_buffer_size AppConfig derives when the key is unset or 0: a thousandth
+ * of the budget, but never below the 1 MiB rocksdb.write_buffer_size accepts
+ * (RocksDBOptions.MEMTABLE_SIZE), which an auto-sized budget under 1 GiB would
+ * otherwise fall under.
+ */
+ public static long writeBufferSize(long totalMemory) {
+ return Math.max(totalMemory / 1000, MIB);
+ }
+
+ private static long physicalMemory() {
+ try {
+ // Reports the cgroup v1/v2 memory limit inside a container; older JDK 17
+ // updates may not see a limit set only on a parent cgroup
+ OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(
+ OperatingSystemMXBean.class);
+ return os == null ? -1L : os.getTotalMemorySize();
Review Comment:
Reproduced and fixed in 33e53ee18. The resolver now reads the cgroup2 mount, or the v1 mount carrying `memory`, from `/proc/self/mountinfo` and strips its root from the membership path before walking ancestors, so `0::/kubepods/pod1/ctr` under a mount rooted at `/kubepods` walks `<mount>/pod1/ctr` and finds the pod's 4 GiB. With your 64 GiB host, `-Xmx1g` and 512 MiB direct cap that gives 512 MiB. The new `testCgroupV2SubtreeMountIsRebased` covers that layout and checks that without mountinfo the old lookup misses it; there are also a v1 subtree case with an escaped (`\040`) mount point and unit cases for the rebasing. Without a mountinfo entry it falls back to the default `/sys/fs/cgroup` layout as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4286093351
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.lang.management.ManagementFactory;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
+ static final long MIN_AUTO = 256 * MIB;
+ // RaftRocksdbOptions gives the write buffer manager 2/3 of the budget, and a 0-byte
+ // write buffer manager means no memtable limit at all
+ static final long MIN_EXPLICIT = MIB;
+ // The value shipped before auto-sizing; a large host never gets more than it had
+ static final long MAX_AUTO = 32_000_000_000L;
+
+ private RocksdbMemoryBudget() {
+ }
+
+ /**
+ * Returns the configured budget in bytes, or an automatic one when the value is
+ * missing, blank or 0.
+ */
+ public static long resolve(String configured) {
+ long value = isAuto(configured) ? 0L : parse(configured);
+ long heap = Runtime.getRuntime().maxMemory();
+ long physical = physicalMemory();
+ if (value > 0) {
+ if (value < MIN_EXPLICIT) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be 0 (auto) or at least %s bytes, " +
+ "got %s", MIN_EXPLICIT, value));
+ }
+ if (value < MIN_AUTO) {
+ log.warn("rocksdb.total_memory_size={} is below the {} bytes auto-sizing " +
+ "never goes under", value, MIN_AUTO);
+ }
+ long direct = maxDirectMemory(heap);
+ long left = physical - heap - direct - RESERVED;
+ if (physical > 0 && value > left) {
+ log.warn("rocksdb.total_memory_size={} exceeds the {} bytes left after the " +
+ "heap, direct memory and the {} bytes reserve (physical={}, heap={}, " +
+ "direct={}); the process may be OOM-killed",
+ value, left, RESERVED, physical, heap, direct);
+ }
+ return value;
+ }
+ long direct = maxDirectMemory(heap);
+ long budget = auto(physical, heap, direct);
+ log.info("rocksdb.total_memory_size is unset or 0, auto-sizing to {} bytes " +
+ "(physical={}, heap={}, direct={})", budget, physical, heap, direct);
+ if (physical > 0 && physical - heap - direct - RESERVED < 2 * MIN_AUTO) {
+ log.warn("Heap, direct memory and the {} bytes reserve leave too little of the " +
+ "{} bytes physical memory, rocksdb gets the {} bytes floor; lower -Xmx " +
+ "or -XX:MaxDirectMemorySize", RESERVED, physical, MIN_AUTO);
+ }
+ return budget;
+ }
+
+ static boolean isAuto(String configured) {
+ return configured == null || configured.trim().isEmpty();
+ }
+
+ static long parse(String configured) {
+ try {
+ long value = Long.parseLong(configured.trim());
+ if (value < 0) {
+ throw new NumberFormatException("negative");
+ }
+ return value;
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be a number of bytes, got '%s'",
+ configured), e);
+ }
+ }
+
+ /**
+ * Half of what is left after the heap, direct memory and the reserve, clamped to
+ * [MIN_AUTO, MAX_AUTO]. Falls back to the heap size, the previous default, when
+ * physical memory is unknown.
+ */
+ static long auto(long physical, long heap, long direct) {
+ if (physical <= 0) {
+ return Math.min(Math.max(heap, MIN_AUTO), MAX_AUTO);
+ }
+ long available = physical - heap - direct - RESERVED;
+ return Math.min(Math.max(available / 2, MIN_AUTO), MAX_AUTO);
+ }
+
+ /**
+ * The raft disruptor buffer size AppConfig derives from the budget when
+ * raft.disruptorBufferSize is 0: the budget in GB, at least 1, rounded to a power
+ * of two, times 32.
+ */
+ public static int disruptorBufferSize(long totalMemory) {
+ // A budget under 1 GB would otherwise give log(0) and a zero-sized buffer
+ int size = (int) Math.max(1, totalMemory / 1000 / 1000 / 1000);
+ return (int) Math.pow(2, Math.round(Math.log(size) / Math.log(2))) * 32;
+ }
+
+ /**
+ * The write_buffer_size AppConfig derives when the key is unset or 0: a thousandth
+ * of the budget, but never below the 1 MiB rocksdb.write_buffer_size accepts
+ * (RocksDBOptions.MEMTABLE_SIZE), which an auto-sized budget under 1 GiB would
+ * otherwise fall under.
+ */
+ public static long writeBufferSize(long totalMemory) {
+ return Math.max(totalMemory / 1000, MIB);
+ }
+
+ private static long physicalMemory() {
+ try {
+ // Reports the cgroup v1/v2 memory limit inside a container; older JDK 17
+ // updates may not see a limit set only on a parent cgroup
+ OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(
+ OperatingSystemMXBean.class);
+ return os == null ? -1L : os.getTotalMemorySize();
+ } catch (RuntimeException | LinkageError e) {
+ log.warn("Failed to read physical memory size", e);
+ return -1L;
+ }
+ }
+
+ private static long maxDirectMemory(long heap) {
+ try {
+ HotSpotDiagnosticMXBean hotspot = ManagementFactory.getPlatformMXBean(
+ HotSpotDiagnosticMXBean.class);
+ long direct = Long.parseLong(
+ hotspot.getVMOption("MaxDirectMemorySize").getValue());
+ // 0 means unset, and the JVM then allows as much direct memory as heap
+ return direct > 0 ? direct : heap;
Review Comment:
Important: Auto-sizing counts an unset `-XX:MaxDirectMemorySize` as a second full heap, so any Store whose heap is near half of physical memory gets the 256 MiB floor. `start-hugegraph-store.sh` never sets that flag, and its own comment says to raise the heap to 36GB for production. A tarball install that upgrades and takes the new `total_memory_size: 0` goes from 32000000000 bytes to 268435456 without changing anything, and the only signal is a WARN line.
Evidence: I compiled this file at e691489e (a stdout logger in place of `@Slf4j`, nothing else changed) and called `resolve(null)` on a 16 GiB host, OpenJDK 17.0.20:
- `-Xmx2g`: 5637144576
- `-Xmx8g`: 268435456, with the floor warning
- `-Xmx8g -XX:MaxDirectMemorySize=512m`: 3221225472
The same `auto()` arithmetic gives the floor for 64 GiB with `-Xmx32g` or `-Xmx36g`, and for 32 GiB with `-Xmx16g`. At the floor the write buffer manager gets about 169 MiB for every partition's memtables and the block cache about 87 MiB, against about 21 GB and 11 GB with the previous shipped value. The PR description and the deployment guide only describe the small-container case.
Please make the default launch path produce a usable budget, for example by having `start-hugegraph-store.sh` pass a bounded `-XX:MaxDirectMemorySize` when none is set (the Helm cluster preset already uses 512m), or by not counting an unset cap as a full heap here. Either way, the deployment guide should say what a bare-metal Store with a large heap and no direct-memory cap gets after upgrading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4290951780
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.lang.management.ManagementFactory;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
+ static final long MIN_AUTO = 256 * MIB;
+ // RaftRocksdbOptions gives the write buffer manager 2/3 of the budget, and a 0-byte
+ // write buffer manager means no memtable limit at all
+ static final long MIN_EXPLICIT = MIB;
+ // The value shipped before auto-sizing; a large host never gets more than it had
+ static final long MAX_AUTO = 32_000_000_000L;
+
+ private RocksdbMemoryBudget() {
+ }
+
+ /**
+ * Returns the configured budget in bytes, or an automatic one when the value is
+ * missing, blank or 0.
+ */
+ public static long resolve(String configured) {
+ long value = isAuto(configured) ? 0L : parse(configured);
+ long heap = Runtime.getRuntime().maxMemory();
+ long physical = physicalMemory();
+ if (value > 0) {
+ if (value < MIN_EXPLICIT) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be 0 (auto) or at least %s bytes, " +
+ "got %s", MIN_EXPLICIT, value));
+ }
+ if (value < MIN_AUTO) {
+ log.warn("rocksdb.total_memory_size={} is below the {} bytes auto-sizing " +
+ "never goes under", value, MIN_AUTO);
+ }
+ long direct = maxDirectMemory(heap);
+ long left = physical - heap - direct - RESERVED;
+ if (physical > 0 && value > left) {
+ log.warn("rocksdb.total_memory_size={} exceeds the {} bytes left after the " +
+ "heap, direct memory and the {} bytes reserve (physical={}, heap={}, " +
+ "direct={}); the process may be OOM-killed",
+ value, left, RESERVED, physical, heap, direct);
+ }
+ return value;
+ }
+ long direct = maxDirectMemory(heap);
+ long budget = auto(physical, heap, direct);
+ log.info("rocksdb.total_memory_size is unset or 0, auto-sizing to {} bytes " +
+ "(physical={}, heap={}, direct={})", budget, physical, heap, direct);
+ if (physical > 0 && physical - heap - direct - RESERVED < 2 * MIN_AUTO) {
+ log.warn("Heap, direct memory and the {} bytes reserve leave too little of the " +
+ "{} bytes physical memory, rocksdb gets the {} bytes floor; lower -Xmx " +
+ "or -XX:MaxDirectMemorySize", RESERVED, physical, MIN_AUTO);
+ }
+ return budget;
+ }
+
+ static boolean isAuto(String configured) {
+ return configured == null || configured.trim().isEmpty();
+ }
+
+ static long parse(String configured) {
+ try {
+ long value = Long.parseLong(configured.trim());
+ if (value < 0) {
+ throw new NumberFormatException("negative");
+ }
+ return value;
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be a number of bytes, got '%s'",
+ configured), e);
+ }
+ }
+
+ /**
+ * Half of what is left after the heap, direct memory and the reserve, clamped to
+ * [MIN_AUTO, MAX_AUTO]. Falls back to the heap size, the previous default, when
+ * physical memory is unknown.
+ */
+ static long auto(long physical, long heap, long direct) {
+ if (physical <= 0) {
+ return Math.min(Math.max(heap, MIN_AUTO), MAX_AUTO);
+ }
+ long available = physical - heap - direct - RESERVED;
+ return Math.min(Math.max(available / 2, MIN_AUTO), MAX_AUTO);
+ }
+
+ /**
+ * The raft disruptor buffer size AppConfig derives from the budget when
+ * raft.disruptorBufferSize is 0: the budget in GB, at least 1, rounded to a power
+ * of two, times 32.
+ */
+ public static int disruptorBufferSize(long totalMemory) {
+ // A budget under 1 GB would otherwise give log(0) and a zero-sized buffer
+ int size = (int) Math.max(1, totalMemory / 1000 / 1000 / 1000);
+ return (int) Math.pow(2, Math.round(Math.log(size) / Math.log(2))) * 32;
+ }
+
+ /**
+ * The write_buffer_size AppConfig derives when the key is unset or 0: a thousandth
+ * of the budget, but never below the 1 MiB rocksdb.write_buffer_size accepts
+ * (RocksDBOptions.MEMTABLE_SIZE), which an auto-sized budget under 1 GiB would
+ * otherwise fall under.
+ */
+ public static long writeBufferSize(long totalMemory) {
+ return Math.max(totalMemory / 1000, MIB);
+ }
+
+ private static long physicalMemory() {
+ try {
+ // Reports the cgroup v1/v2 memory limit inside a container; older JDK 17
+ // updates may not see a limit set only on a parent cgroup
+ OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(
+ OperatingSystemMXBean.class);
+ return os == null ? -1L : os.getTotalMemorySize();
+ } catch (RuntimeException | LinkageError e) {
+ log.warn("Failed to read physical memory size", e);
+ return -1L;
+ }
+ }
+
+ private static long maxDirectMemory(long heap) {
+ try {
+ HotSpotDiagnosticMXBean hotspot = ManagementFactory.getPlatformMXBean(
+ HotSpotDiagnosticMXBean.class);
+ long direct = Long.parseLong(
+ hotspot.getVMOption("MaxDirectMemorySize").getValue());
+ // 0 means unset, and the JVM then allows as much direct memory as heap
+ return direct > 0 ? direct : heap;
Review Comment:
Thanks for running it, that regression is real. Fixed in 84e8e27f3 on the budget side rather than in the launcher: an unset `-XX:MaxDirectMemorySize` now counts as the heap capped at 1 GiB (`directMemory()`), since the Store's gRPC/Netty direct buffers don't scale with the heap. A launcher default would also change what every Store may allocate at runtime. An explicit cap is still used as given, so the container case is unchanged. With the same `auto()` arithmetic: 16 GiB with `-Xmx8g` gets 2.75 GiB, 64 GiB with `-Xmx36g` gets 12.75 GiB, and the default launcher (2 GiB heap) on 16 GiB gets 5.75 GiB; tests cover all three. The deployment guide now spells out the bare-metal numbers and says to pin the value after upgrading if a large node relied on the old 32 GB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4279275783
##########
hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/AppConfig.java:
##########
@@ -104,15 +105,11 @@ public void init() {
}
Map<String, String> rocksdb = rocksdbConfig.rocksdb;
- if (!rocksdb.containsKey("total_memory_size")
- || "0".equals(rocksdb.get("total_memory_size"))) {
- rocksdb.put("total_memory_size", Long.toString(rt.maxMemory()));
- }
- long totalMemory = Long.parseLong(rocksdbConfig.rocksdb.get("total_memory_size"));
+ long totalMemory = RocksdbMemoryBudget.resolve(rocksdb.get(RocksdbMemoryBudget.KEY));
+ rocksdb.put(RocksdbMemoryBudget.KEY, Long.toString(totalMemory));
if (raft.getDisruptorBufferSize() == 0) {
Review Comment:
Good catch, thanks. Fixed in 1927fbee8: `RocksdbMemoryBudget.writeBufferSize()` now clamps the derived value to 1 MiB, and `AppConfig` uses it. The new test takes your exact case (4 GiB, `-Xmx1g`, 512 MiB direct, so a 512 MiB budget), runs the result through the same `OptionSpace`/`HugeConfig` validation `initRocksdb()` does, and checks that the old 536870-byte value is rejected there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#issuecomment-6220896982
@imbajin thanks, fixed in 535c2d01b. The resolver now reads every cgroup2 mount, or every v1 mount carrying `memory`, from mountinfo instead of the first one. It skips mounts whose root doesn't contain the process's cgroup, walks each remaining one from its rebased path, and keeps the lowest limit. A leaf-only or unrelated mount listed first can no longer hide a fuller view listed later. New tests run three mount orders for v2 (unrelated, leaf-only and full views) and two for v1 (with a cpu-only mount that's ignored). Each finds the 4 GiB pod limit and, with a 64 GiB host, 1 GiB heap and 512 MiB direct cap, gives 512 MiB. Store core tests: 126 passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4398705156
##########
hugegraph-store/hg-store-dist/src/assembly/static/bin/start-hugegraph-store.sh:
##########
@@ -169,7 +169,18 @@ if [ "$JAVA_OPTIONS" = "" ]; then
>> ${OUTPUT}
exit 1
fi
- JAVA_OPTIONS="-Xms${MIN_MEM}m -Xmx${XMX}m -XX:MetaspaceSize=256M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${LOGS} ${USER_OPTION} "
+ # Cap direct memory at the heap size picked here, the JVM's own unset default, so the
+ # RocksDB auto-sizing (RocksdbMemoryBudget) counts a limit the JVM actually enforces.
+ # Leave it out when -j sets the heap or the cap itself: a fixed cap from the computed
+ # heap would no longer match a -j -Xmx, and the JVM then defaults it to that heap.
+ DIRECT_MEMORY_OPTION="-XX:MaxDirectMemorySize=${XMX}m"
Review Comment:
⚠️ Important: This default also overrides a limit supplied in `JAVA_TOOL_OPTIONS` or `JDK_JAVA_OPTIONS`, because JVM environment options are processed before command-line flags. For example, `JAVA_TOOL_OPTIONS=-XX:MaxDirectMemorySize=512m` with the generated `-XX:MaxDirectMemorySize=2g` results in a 2 GiB cap (confirmed with `-XX:+PrintFlagsFinal`). An existing deployment using that environment cap can therefore allocate more direct memory after upgrading and exceed its container limit. Please preserve an explicit environment-provided cap and add a launcher regression case for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4392073656
##########
hugegraph-store/docs/deployment-guide.md:
##########
@@ -356,14 +356,82 @@ rocksdb:
level0_file_num_compaction_trigger: 4
max_background_jobs: 8 # Background compaction/flush threads
- # Block Cache
- block_cache_size: 16000000000 # Block cache (16GB)
+ # Budget split: this share goes to memtables, the rest to the block cache
+ write_buffer_ratio: 0.66
# SST File Size
target_file_size_base: 134217728 # Target SST size (128MB)
max_bytes_for_level_base: 1073741824 # L1 size (1GB)
```
+`total_memory_size` bounds the memtables (`write_buffer_ratio` of it) and the block cache
+(the rest). Both live outside the Java heap, so `-Xmx` does not limit them. A fixed 1 GiB
+cache for the Raft log comes on top of this budget; the Raft log memtables of all partitions
+share a 256 MiB write buffer charged to that cache, so they don't add to it.
+
+With `total_memory_size: auto` (the shipped value; a missing, blank or `0` value means the same)
+the Store sizes the budget at startup as half of what is left after the heap, the maximum direct
+memory and a 1.5 GiB reserve (the 1 GiB Raft log cache plus 512 MiB for other native memory),
+clamped to 256 MiB–32 GB. Inside a container the starting point is the cgroup memory limit, including a
+limit set only on a parent cgroup when the Store can see that cgroup; inside a private cgroup
+namespace, or when only a subtree of the cgroup2 hierarchy is mounted, a limit above it is not
+visible, so pin the value there if only the pod or slice is limited. The Store uses the lowest
+limit it can see and logs a warning when it can't see the whole hierarchy; it doesn't guess at
+limits it can't read. A Store with a 4 GiB limit, `-Xmx1g` and `-XX:MaxDirectMemorySize=512m`
+gets 512 MiB. Without
+`-XX:MaxDirectMemorySize` the JVM allows as much direct memory as heap, and the budget counts
+it that way. `bin/start-hugegraph-store.sh` sets the flag to the heap it picks; when `-j` sets
+`-Xmx`, `-XX:MaxHeapSize` or `-XX:MaxDirectMemorySize` itself, the launcher leaves the flag out and the JVM's
+default (that heap) or the given cap applies. Either way the budget counts a limit the JVM
+enforces. In small containers set a
+cap below the heap, or the budget falls to the 256 MiB floor.
+
+On a host without a container limit the starting point is physical memory. The default
+`bin/start-hugegraph-store.sh` caps the heap at 2 GiB and direct memory at the same 2 GiB, so a
+16 GiB host gets about 5.25 GiB. When you raise the heap, pass a direct-memory cap with it,
+for example `-j "-Xmx36g -XX:MaxDirectMemorySize=2g"`; otherwise direct memory counts as a
+second heap. With a 2 GiB cap a 16 GiB host with `-Xmx8g` gets 2.25 GiB and a 64 GiB host with
+`-Xmx36g` gets 12.25 GiB. Before this default every host got a fixed `32000000000`, so pin the
+value after upgrading if a large node relied on it.
+
+At startup the Store logs the budget and what it was computed from, for example:
+
+```text
+RocksDB memory budget: 512 MiB (536870912 bytes), mode=auto, physical-limit=4 GiB, source=cgroup-v2, heap=1 GiB, direct=512 MiB, reserve=1.5 GiB, cgroup-view=complete
+```
+
+It warns when the floor applies, or when an explicit value is under 256 MiB or exceeds the
+memory left after the heap, direct memory and the reserve. Set a number of bytes, at least
+1048576 (1 MiB), to pin the budget; a smaller value or a unit suffix such as `4GB` is rejected. Before this default existed the
+shipped value was `32000000000`, which can exceed a container limit and get the Store
+OOM-killed. Both caps are soft: RocksDB can briefly go over them under heavy writes, and a
+small budget with many partitions means frequent memtable flushes.
+
+**Upgrading an existing deployment.** Auto-sizing applies only when the loaded configuration
+has `total_memory_size: auto` (or `0`, or no value). A Store that mounts its own `conf/application-pd.yml`,
+or a tarball install that kept its `conf/` from an earlier release, still has `32000000000` and
+keeps that fixed budget; the Store logs a warning naming the old value at startup. Set the key
+to `auto`, or set `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` in Docker.
+
+PD can also hold the old value. When PD has no shared copy yet, the first Store to start uploads
+its local `application-pd.yml` to PD (KV key `S:FS`). At startup a Store loads that copy over its
+local file only when it has no local file or the file is no newer than PD's `S:Timestamp`. The
+upload sets no timestamp and a missing one counts as 0, so changing `S:FS` alone changes nothing.
+The simplest upgrade is to set `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE=auto` (or
+`-Drocksdb.total_memory_size=auto`) on every Store and restart it; the Store keeps this value
+above both its local file and PD's copy. To change the shared copy instead, set
+`rocksdb.total_memory_size` to `0` or remove it in `S:FS`, then put the current time in
+milliseconds into `S:Timestamp` through PD's KV service (gRPC `KvService.put`). Use `auto` there
+only once every Store runs this release: an earlier release can't parse `auto` and fails to start,
+while it reads `0` as its old heap-sized default. This also applies to a copy that a Store of this
+release uploaded when PD had none, since the shipped file says `auto`. Every running Store watches
+`S:Timestamp`: it reloads PD's copy, writes it over its local file and restarts itself with
+`bin/restart-hugegraph-store.sh` within 60 seconds. A stopped Store loads the copy at its next
+start. That self-restart passes no arguments, so `-j` and `-g` from the original start are
+dropped, including the `JAVA_OPTS` the Docker entrypoint passes as `-j`, and the launcher's
Review Comment:
Confirmed: with `-d false` the launcher execs Java as the entrypoint's last process, so the stop step ends the container before the start step runs. The guide now says a Docker Store exits with 143 and stays down without a restart policy, and points Docker users to `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` instead of `S:Timestamp`. It also notes that all Stores restart within the same minute. Fixed in 3cbb58fe9; hugegraph-doc#535 has the same change in eab98044.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4345184001
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,458 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
+ static final long MIN_AUTO = 256 * MIB;
+ // RaftRocksdbOptions gives the write buffer manager 2/3 of the budget, and a 0-byte
+ // write buffer manager means no memtable limit at all
+ static final long MIN_EXPLICIT = MIB;
+ // The value shipped before auto-sizing; a large host never gets more than it had
+ static final long MAX_AUTO = 32_000_000_000L;
+
+ private RocksdbMemoryBudget() {
+ }
+
+ /**
+ * Returns the configured budget in bytes, or an automatic one when the value is
+ * missing, blank or 0.
+ */
+ public static long resolve(String configured) {
+ long value = isAuto(configured) ? 0L : parse(configured);
+ long heap = Runtime.getRuntime().maxMemory();
+ CgroupLimit cgroup = cgroupLimit(Paths.get("/proc/self/cgroup"),
+ Paths.get("/proc/self/mountinfo"),
+ Paths.get("/sys/fs/cgroup"));
+ long physical = effectivePhysical(jvmMemory(), cgroup.bytes);
+ if (value > 0) {
+ if (value < MIN_EXPLICIT) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be 0 (auto) or at least %s bytes, " +
+ "got %s", MIN_EXPLICIT, value));
+ }
+ if (value < MIN_AUTO) {
+ log.warn("rocksdb.total_memory_size={} is below the {} bytes auto-sizing " +
+ "never goes under", value, MIN_AUTO);
+ }
+ if (value == MAX_AUTO) {
+ // Most likely an application-pd.yml kept from before auto-sizing
+ log.warn("rocksdb.total_memory_size={} is the fixed value shipped before " +
+ "auto-sizing; set it to 0 or remove it to size the budget from " +
+ "the available memory", value);
+ }
+ long direct = maxDirectMemory(heap);
+ long left = physical - heap - direct - RESERVED;
+ if (physical > 0 && value > left) {
+ log.warn("rocksdb.total_memory_size={} exceeds the {} bytes left after the " +
+ "heap, direct memory and the {} bytes reserve (physical={}, heap={}, " +
+ "direct={}); the process may be OOM-killed",
+ value, left, RESERVED, physical, heap, direct);
+ }
+ return value;
+ }
+ long direct = maxDirectMemory(heap);
+ long budget = auto(physical, heap, direct);
+ log.info("rocksdb.total_memory_size is unset or 0, auto-sizing to {} bytes " +
+ "(physical={}, heap={}, direct={})", budget, physical, heap, direct);
+ if (cgroup.incomplete) {
+ log.warn("The cgroup mounts in /proc/self/mountinfo don't show this process's " +
+ "whole cgroup hierarchy (an inherited mount in a cgroup namespace, or " +
+ "only cgroup2 subtree mounts), so a tighter limit on the process's " +
+ "cgroup or above the mounted subtree may be missed; set " +
+ "rocksdb.total_memory_size or HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE " +
+ "explicitly");
+ }
+ if (physical > 0 && physical - heap - direct - RESERVED < 2 * MIN_AUTO) {
+ log.warn("Heap, direct memory and the {} bytes reserve leave too little of the " +
+ "{} bytes physical memory, rocksdb gets the {} bytes floor; lower -Xmx " +
+ "or -XX:MaxDirectMemorySize", RESERVED, physical, MIN_AUTO);
+ }
+ return budget;
+ }
+
+ static boolean isAuto(String configured) {
+ return configured == null || configured.trim().isEmpty();
+ }
+
+ static long parse(String configured) {
+ try {
+ long value = Long.parseLong(configured.trim());
+ if (value < 0) {
+ throw new NumberFormatException("negative");
+ }
+ return value;
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be a number of bytes, got '%s'",
+ configured), e);
+ }
+ }
+
+ /**
+ * Half of what is left after the heap, direct memory and the reserve, clamped to
+ * [MIN_AUTO, MAX_AUTO]. Falls back to the heap size, the previous default, when
+ * physical memory is unknown.
+ */
+ static long auto(long physical, long heap, long direct) {
+ if (physical <= 0) {
+ return Math.min(Math.max(heap, MIN_AUTO), MAX_AUTO);
+ }
+ long available = physical - heap - direct - RESERVED;
+ return Math.min(Math.max(available / 2, MIN_AUTO), MAX_AUTO);
+ }
+
+ /**
+ * The raft disruptor buffer size AppConfig derives from the budget when
+ * raft.disruptorBufferSize is 0: the budget in GB, at least 1, rounded to a power
+ * of two, times 32.
+ */
+ public static int disruptorBufferSize(long totalMemory) {
+ // A budget under 1 GB would otherwise give log(0) and a zero-sized buffer
+ int size = (int) Math.max(1, totalMemory / 1000 / 1000 / 1000);
+ return (int) Math.pow(2, Math.round(Math.log(size) / Math.log(2))) * 32;
+ }
+
+ /**
+ * The write_buffer_size AppConfig derives when the key is unset or 0: a thousandth
+ * of the budget, but never below the 1 MiB rocksdb.write_buffer_size accepts
+ * (RocksDBOptions.MEMTABLE_SIZE), which an auto-sized budget under 1 GiB would
+ * otherwise fall under.
+ */
+ public static long writeBufferSize(long totalMemory) {
+ return Math.max(totalMemory / 1000, MIB);
+ }
+
+ private static long jvmMemory() {
+ try {
+ // Reports the cgroup v1/v2 memory limit inside a container, but older JDK 17
+ // updates may not see a limit set only on a parent cgroup
+ OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(
+ OperatingSystemMXBean.class);
+ return os == null ? -1L : os.getTotalMemorySize();
+ } catch (RuntimeException | LinkageError e) {
+ log.warn("Failed to read physical memory size", e);
+ return -1L;
+ }
+ }
+
+ /**
+ * A cgroup memory limit in bytes (-1 for none), and whether the mounts may hide a
+ * stricter one: a cgroup mount existed but none contained this process's cgroup, or
+ * every matching cgroup2 mount was a subtree, so bytes only covers what is visible
+ * from those mounts' roots down.
+ */
+ static final class CgroupLimit {
+
+ final long bytes;
+ final boolean incomplete;
+
+ CgroupLimit(long bytes, boolean incomplete) {
+ this.bytes = bytes;
+ this.incomplete = incomplete;
+ }
+ }
+
+ /**
+ * The smaller of what the JVM reports and the cgroup limit, ignoring unknown (<= 0)
+ * values.
+ */
+ static long effectivePhysical(long jvm, long cgroup) {
+ if (cgroup <= 0) {
+ return jvm;
+ }
+ return jvm <= 0 ? cgroup : Math.min(jvm, cgroup);
+ }
+
+ /**
+ * The lowest memory limit on this process's cgroup or any visible ancestor, or -1
+ * when there is none or it can't be read. The membership path from /proc/self/cgroup
+ * is rebased onto every mount of the controller in /proc/self/mountinfo whose root
+ * contains it, so a cgroupfs mounted from a subtree (root /kubepods at /sys/fs/cgroup)
+ * is walked from the right directory, and the lowest limit over all those mounts is
+ * taken: a leaf-only or unrelated mount listed first can't hide a fuller view listed
+ * later. Without a matching mount the default root is used. cgroup v2
+ * walks memory.max from the process's cgroup up to the mount; cgroup v1 reads
+ * hierarchical_memory_limit, which already accounts for ancestors. A limit above the
+ * mount root, as inside a private cgroup namespace, stays invisible.
+ */
+ static long cgroupMemoryLimit(Path procSelfCgroup, Path mountinfo, Path defaultRoot) {
+ return cgroupLimit(procSelfCgroup, mountinfo, defaultRoot).bytes;
+ }
+
+ static CgroupLimit cgroupLimit(Path procSelfCgroup, Path mountinfo, Path defaultRoot) {
+ List<String[]> v2Mounts = cgroupMounts(mountinfo, true);
+ List<String[]> v1Mounts = cgroupMounts(mountinfo, false);
+ List<String> lines;
+ try {
+ lines = Files.readAllLines(procSelfCgroup, StandardCharsets.UTF_8);
+ } catch (IOException | RuntimeException e) {
+ return new CgroupLimit(-1L, false);
+ }
+ long limit = -1L;
+ boolean incomplete = false;
+ for (String line : lines) {
+ String[] parts = line.split(":", 3);
+ if (parts.length < 3) {
+ continue;
+ }
+ boolean v2 = parts[0].equals("0") && parts[1].isEmpty();
+ if (!v2 && !("," + parts[1] + ",").contains(",memory,")) {
+ continue;
+ }
+ List<String[]> mounts = v2 ? v2Mounts : v1Mounts;
+ boolean matched = false;
+ boolean fullView = false;
+ long lineLimit = -1L;
+ for (String[] mount : mounts) {
+ if (!contains(mount[0], parts[2])) {
+ // A mount of another subtree says nothing about this cgroup
+ continue;
+ }
+ matched = true;
+ Path point = Paths.get(mount[1]);
+ String relative = rebase(parts[2], mount[0]);
+ // A full-hierarchy mount only covers the ancestors if this process's own
+ // directory is actually reachable through it
+ fullView |= mount[0].equals("/") && visible(point, relative);
+ lineLimit = lowest(lineLimit, v2 ? cgroupV2Limit(point, relative) :
+ cgroupV1Limit(point, relative));
+ }
+ limit = lowest(limit, lineLimit);
+ if (v2 && matched && parts[2].equals("/") && lineLimit <= 0) {
Review Comment:
Done in b0d8851e9. Any cgroup2 line whose process path is `/` and that matches a mount is now flagged incomplete, whether the visible `memory.max` is `max` or a number, so the Store logs the warning and asks for an explicit `rocksdb.total_memory_size`. The finite limit is still used as an upper bound. `testNamespaceRootViewIsFlagged` now asserts `incomplete` for the finite 2 GiB case. The not-flagged regression uses a host service cgroup instead (`/system.slice/hg.service` under a full-hierarchy mount, 4 GiB on the service).
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,458 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
Review Comment:
Done in e5bf043cf. The registered Raft log `DBOptions` now carry one `WriteBufferManager` of 256 MiB, charged to the 1 GiB Raft log block cache. `RocksDBLogStorage` and `RocksDBSegmentLogStorage` both get their options from `StorageOptionsFactory.getRocksDBOptions(null, RocksDBLogStorage.class)`. That returns `new DBOptions(registered)`, and the copy keeps the same manager, so every group shares the cap however many partitions are open. Memtable memory then takes room from the cache that `RESERVED` already counts instead of adding to it.
`RaftRocksdbOptionsTest` covers this. It checks that copies made the way JRaft makes them hold the same manager, opens a DB from one of them, and checks that its memtable shows up in the cache's usage. The deployment guide and apache/hugegraph-doc#535 say this now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#issuecomment-6247753293
The `server-rocksdb` failure on 9eaaf577b is in the Arthas CLI check that came in with master's #2945 (`test-arthas-cli.sh`: "Interactive helper did not exit", the helper's 30 s wait timing out). The same job passed on 7feaf6ac2, and 9eaaf577b only changes `RocksdbMemoryBudget` and its Store test, so this looks like a timing flake on a busy runner. I pushed an empty commit to re-run CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4334078216
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,450 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
+ static final long MIN_AUTO = 256 * MIB;
+ // RaftRocksdbOptions gives the write buffer manager 2/3 of the budget, and a 0-byte
+ // write buffer manager means no memtable limit at all
+ static final long MIN_EXPLICIT = MIB;
+ // The value shipped before auto-sizing; a large host never gets more than it had
+ static final long MAX_AUTO = 32_000_000_000L;
+
+ private RocksdbMemoryBudget() {
+ }
+
+ /**
+ * Returns the configured budget in bytes, or an automatic one when the value is
+ * missing, blank or 0.
+ */
+ public static long resolve(String configured) {
+ long value = isAuto(configured) ? 0L : parse(configured);
+ long heap = Runtime.getRuntime().maxMemory();
+ CgroupLimit cgroup = cgroupLimit(Paths.get("/proc/self/cgroup"),
+ Paths.get("/proc/self/mountinfo"),
+ Paths.get("/sys/fs/cgroup"));
+ long physical = effectivePhysical(jvmMemory(), cgroup.bytes);
+ if (value > 0) {
+ if (value < MIN_EXPLICIT) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be 0 (auto) or at least %s bytes, " +
+ "got %s", MIN_EXPLICIT, value));
+ }
+ if (value < MIN_AUTO) {
+ log.warn("rocksdb.total_memory_size={} is below the {} bytes auto-sizing " +
+ "never goes under", value, MIN_AUTO);
+ }
+ if (value == MAX_AUTO) {
+ // Most likely an application-pd.yml kept from before auto-sizing
+ log.warn("rocksdb.total_memory_size={} is the fixed value shipped before " +
+ "auto-sizing; set it to 0 or remove it to size the budget from " +
+ "the available memory", value);
+ }
+ long direct = maxDirectMemory(heap);
+ long left = physical - heap - direct - RESERVED;
+ if (physical > 0 && value > left) {
+ log.warn("rocksdb.total_memory_size={} exceeds the {} bytes left after the " +
+ "heap, direct memory and the {} bytes reserve (physical={}, heap={}, " +
+ "direct={}); the process may be OOM-killed",
+ value, left, RESERVED, physical, heap, direct);
+ }
+ return value;
+ }
+ long direct = maxDirectMemory(heap);
+ long budget = auto(physical, heap, direct);
+ log.info("rocksdb.total_memory_size is unset or 0, auto-sizing to {} bytes " +
+ "(physical={}, heap={}, direct={})", budget, physical, heap, direct);
+ if (cgroup.incomplete) {
+ log.warn("The cgroup mounts in /proc/self/mountinfo don't show this process's " +
+ "whole cgroup hierarchy (an inherited mount in a cgroup namespace, or " +
+ "only cgroup2 subtree mounts), so a tighter limit on the process's " +
+ "cgroup or above the mounted subtree may be missed; set " +
+ "rocksdb.total_memory_size or HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE " +
+ "explicitly");
+ }
+ if (physical > 0 && physical - heap - direct - RESERVED < 2 * MIN_AUTO) {
+ log.warn("Heap, direct memory and the {} bytes reserve leave too little of the " +
+ "{} bytes physical memory, rocksdb gets the {} bytes floor; lower -Xmx " +
+ "or -XX:MaxDirectMemorySize", RESERVED, physical, MIN_AUTO);
+ }
+ return budget;
+ }
+
+ static boolean isAuto(String configured) {
+ return configured == null || configured.trim().isEmpty();
+ }
+
+ static long parse(String configured) {
+ try {
+ long value = Long.parseLong(configured.trim());
+ if (value < 0) {
+ throw new NumberFormatException("negative");
+ }
+ return value;
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be a number of bytes, got '%s'",
+ configured), e);
+ }
+ }
+
+ /**
+ * Half of what is left after the heap, direct memory and the reserve, clamped to
+ * [MIN_AUTO, MAX_AUTO]. Falls back to the heap size, the previous default, when
+ * physical memory is unknown.
+ */
+ static long auto(long physical, long heap, long direct) {
+ if (physical <= 0) {
+ return Math.min(Math.max(heap, MIN_AUTO), MAX_AUTO);
+ }
+ long available = physical - heap - direct - RESERVED;
+ return Math.min(Math.max(available / 2, MIN_AUTO), MAX_AUTO);
+ }
+
+ /**
+ * The raft disruptor buffer size AppConfig derives from the budget when
+ * raft.disruptorBufferSize is 0: the budget in GB, at least 1, rounded to a power
+ * of two, times 32.
+ */
+ public static int disruptorBufferSize(long totalMemory) {
+ // A budget under 1 GB would otherwise give log(0) and a zero-sized buffer
+ int size = (int) Math.max(1, totalMemory / 1000 / 1000 / 1000);
+ return (int) Math.pow(2, Math.round(Math.log(size) / Math.log(2))) * 32;
+ }
+
+ /**
+ * The write_buffer_size AppConfig derives when the key is unset or 0: a thousandth
+ * of the budget, but never below the 1 MiB rocksdb.write_buffer_size accepts
+ * (RocksDBOptions.MEMTABLE_SIZE), which an auto-sized budget under 1 GiB would
+ * otherwise fall under.
+ */
+ public static long writeBufferSize(long totalMemory) {
+ return Math.max(totalMemory / 1000, MIB);
+ }
+
+ private static long jvmMemory() {
+ try {
+ // Reports the cgroup v1/v2 memory limit inside a container, but older JDK 17
+ // updates may not see a limit set only on a parent cgroup
+ OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(
+ OperatingSystemMXBean.class);
+ return os == null ? -1L : os.getTotalMemorySize();
+ } catch (RuntimeException | LinkageError e) {
+ log.warn("Failed to read physical memory size", e);
+ return -1L;
+ }
+ }
+
+ /**
+ * A cgroup memory limit in bytes (-1 for none), and whether the mounts may hide a
+ * stricter one: a cgroup mount existed but none contained this process's cgroup, or
+ * every matching cgroup2 mount was a subtree, so bytes only covers what is visible
+ * from those mounts' roots down.
+ */
+ static final class CgroupLimit {
+
+ final long bytes;
+ final boolean incomplete;
+
+ CgroupLimit(long bytes, boolean incomplete) {
+ this.bytes = bytes;
+ this.incomplete = incomplete;
+ }
+ }
+
+ /**
+ * The smaller of what the JVM reports and the cgroup limit, ignoring unknown (<= 0)
+ * values.
+ */
+ static long effectivePhysical(long jvm, long cgroup) {
+ if (cgroup <= 0) {
+ return jvm;
+ }
+ return jvm <= 0 ? cgroup : Math.min(jvm, cgroup);
+ }
+
+ /**
+ * The lowest memory limit on this process's cgroup or any visible ancestor, or -1
+ * when there is none or it can't be read. The membership path from /proc/self/cgroup
+ * is rebased onto every mount of the controller in /proc/self/mountinfo whose root
+ * contains it, so a cgroupfs mounted from a subtree (root /kubepods at /sys/fs/cgroup)
+ * is walked from the right directory, and the lowest limit over all those mounts is
+ * taken: a leaf-only or unrelated mount listed first can't hide a fuller view listed
+ * later. Without a matching mount the default root is used. cgroup v2
+ * walks memory.max from the process's cgroup up to the mount; cgroup v1 reads
+ * hierarchical_memory_limit, which already accounts for ancestors. A limit above the
+ * mount root, as inside a private cgroup namespace, stays invisible.
+ */
+ static long cgroupMemoryLimit(Path procSelfCgroup, Path mountinfo, Path defaultRoot) {
+ return cgroupLimit(procSelfCgroup, mountinfo, defaultRoot).bytes;
+ }
+
+ static CgroupLimit cgroupLimit(Path procSelfCgroup, Path mountinfo, Path defaultRoot) {
+ List<String[]> v2Mounts = cgroupMounts(mountinfo, true);
+ List<String[]> v1Mounts = cgroupMounts(mountinfo, false);
+ List<String> lines;
+ try {
+ lines = Files.readAllLines(procSelfCgroup, StandardCharsets.UTF_8);
+ } catch (IOException | RuntimeException e) {
+ return new CgroupLimit(-1L, false);
+ }
+ long limit = -1L;
+ boolean incomplete = false;
+ for (String line : lines) {
+ String[] parts = line.split(":", 3);
+ if (parts.length < 3) {
+ continue;
+ }
+ boolean v2 = parts[0].equals("0") && parts[1].isEmpty();
+ if (!v2 && !("," + parts[1] + ",").contains(",memory,")) {
+ continue;
+ }
+ List<String[]> mounts = v2 ? v2Mounts : v1Mounts;
+ boolean matched = false;
+ boolean fullView = false;
+ for (String[] mount : mounts) {
+ if (!contains(mount[0], parts[2])) {
+ // A mount of another subtree says nothing about this cgroup
+ continue;
+ }
+ matched = true;
+ Path point = Paths.get(mount[1]);
+ String relative = rebase(parts[2], mount[0]);
+ // A full-hierarchy mount only covers the ancestors if this process's own
+ // directory is actually reachable through it
+ fullView |= mount[0].equals("/") && visible(point, relative);
Review Comment:
Agreed, fixed in 7a1af80a2. When the process's cgroup2 path is `/` and no finite `memory.max` is visible, the result is now flagged incomplete, so auto-sizing logs the explicit-budget warning instead of treating `max` as unlimited. That's the namespace-root view you describe, where the mount is also rooted at `/`. A finite limit visible at the namespace root still applies without a flag, and v1 isn't flagged, since `hierarchical_memory_limit` already covers the ancestors. One side effect: a process genuinely in the host's root cgroup with no limit also gets the warning; that should be rare, since services run in slices. New regression `testNamespaceRootViewWithoutLimitIsFlagged` covers v2 with `max` (flagged) and with 2 GiB (applied), plus the v1 control; with the new check removed it fails. Store core tests: 135 passed.
##########
hugegraph-store/docs/deployment-guide.md:
##########
@@ -356,14 +356,54 @@ rocksdb:
level0_file_num_compaction_trigger: 4
max_background_jobs: 8 # Background compaction/flush threads
- # Block Cache
- block_cache_size: 16000000000 # Block cache (16GB)
+ # Budget split: this share goes to memtables, the rest to the block cache
+ write_buffer_ratio: 0.66
# SST File Size
target_file_size_base: 134217728 # Target SST size (128MB)
max_bytes_for_level_base: 1073741824 # L1 size (1GB)
```
+`total_memory_size` bounds the memtables (`write_buffer_ratio` of it) and the block cache
+(the rest). Both live outside the Java heap, so `-Xmx` does not limit them. A fixed 1 GiB
+cache for the Raft log comes on top of this budget.
+
+With `total_memory_size: 0`, or the key unset, the Store sizes the budget at startup as half
+of what is left after the heap, the maximum direct memory and a 1.5 GiB reserve, clamped to
+256 MiB–32 GB. Inside a container the starting point is the cgroup memory limit, including a
+limit set only on a parent cgroup when the Store can see that cgroup; inside a private cgroup
+namespace, or when only a subtree of the cgroup2 hierarchy is mounted, a limit above it is not
+visible, so pin the value there if only the pod or slice is limited (the Store logs a warning
+when it detects such a mount). A Store with a 4 GiB limit, `-Xmx1g` and `-XX:MaxDirectMemorySize=512m` gets 512 MiB. Without
+`-XX:MaxDirectMemorySize` the JVM allows as much direct memory as heap, and the budget counts
+it that way. `bin/start-hugegraph-store.sh` sets the flag to the heap it picks; when `-j` sets
+`-Xmx`, `-XX:MaxHeapSize` or `-XX:MaxDirectMemorySize` itself, the launcher leaves the flag out and the JVM's
+default (that heap) or the given cap applies. Either way the budget counts a limit the JVM
+enforces. In small containers set a
+cap below the heap, or the budget falls to the 256 MiB floor.
+
+On a host without a container limit the starting point is physical memory. The default
+`bin/start-hugegraph-store.sh` caps the heap at 2 GiB and direct memory at the same 2 GiB, so a
+16 GiB host gets about 5.25 GiB. When you raise the heap, pass a direct-memory cap with it,
+for example `-j "-Xmx36g -XX:MaxDirectMemorySize=2g"`; otherwise direct memory counts as a
+second heap. With a 2 GiB cap a 16 GiB host with `-Xmx8g` gets 2.25 GiB and a 64 GiB host with
+`-Xmx36g` gets 12.25 GiB. Before this default every host got a fixed `32000000000`, so pin the
+value after upgrading if a large node relied on it.
+
+The chosen value is logged by `RocksdbMemoryBudget` at startup, with a warning when the floor
+applies, or when an explicit value is under 256 MiB or exceeds the memory left after the heap,
+direct memory and the reserve. Set a number of bytes, at least 1048576 (1 MiB), to pin it; a
+smaller value or a unit suffix such as `4GB` is rejected. Before this default existed the
+shipped value was `32000000000`, which can exceed a container limit and get the Store
+OOM-killed. Both caps are soft: RocksDB can briefly go over them under heavy writes, and a
+small budget with many partitions means frequent memtable flushes.
+
+**Upgrading an existing deployment.** Auto-sizing applies only when the loaded configuration
+has `total_memory_size: 0` or no value. A Store that mounts its own `conf/application-pd.yml`,
Review Comment:
Fixed in 7a1af80a2. The upgrade note now explains the copy PD keeps. When PD has none yet, the first Store to start uploads its local `application-pd.yml` (KV key `S:FS`), and any Store whose local file isn't newer than `S:Timestamp` loads it and writes it over its local file. So updating the Store package or a local file doesn't help while PD's copy still says `32000000000`. Operators should set it to `0` or remove it in PD's copy, or set `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` / `-Drocksdb.total_memory_size` on every Store, which the Store keeps above PD's copy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#issuecomment-6465942718
### Review summary for head `7fdd54ab` — score 10/10, no confirmed defects
**Coverage:** this round (R2) is a delta review of the merge commit `7fdd54ab` (parents `1dcabb74` and master `97ecd5e2`). It is not a new full review. The baseline is a full R1 review at base `1216e914` / head `1dcabb74`, which covered all 24 files and found no defects.
The delta work checked:
- **The merge itself.** `git merge-tree 1dcabb74 97ecd5e2` reproduces the head tree, so there was no manual conflict resolution. The PR's added and removed lines are the same as at `1dcabb74`.
- **Files the new base also changed:** `AppConfig.java`, `CoreSuiteTest.java`, `values-cluster.yaml` and the Store docs. Base and PR changes don't overlap, and the 2.5 GiB budget comment in `values-cluster.yaml` still matches the numbers.
- **Non-root images (#3381).** The budget code reads only world-readable `/proc/self/*` and cgroup files. #3381 doesn't touch the entrypoint or the launcher.
- **The segment log storage from #3420.** `RaftSegmentLogStorage` extends `RocksDBSegmentLogStorage`, and JRaft still looks up options by `RocksDBLogStorage.class`, so the shared write-buffer manager still applies.
**Reviewers:** two independent reviewers covered both rounds, Codex (GPT-6) and Claude (`claude-opus-5-5`). Copilot's R1 report is supporting evidence only and doesn't count toward the score. An AGY reviewer is still pending. If a later report changes this result, I'll edit this comment.
**Checks run at `7fdd54ab`:**
- The 5 new Store test classes plus `PartitionLogStorageTest`: 45 tests, 0 failures
- `test-store-docker-entrypoint.sh`: 11 passed
- `test-start-hugegraph-store.sh --gc-only`: passed
- CI policy tests: 66 passed
**Not verified, so not scored:**
- Raft-log memtables, previously uncapped per partition, now share one 256 MiB `WriteBufferManager`. A Store with many partitions may flush the raft log more often. This has not been benchmarked.
- `PdConfigureListener.MEMORY_OVERRIDE_SOURCES` doesn't check `systemEnvironment` or `commandLineArgs`. The shipped launcher passes no program arguments.
- `CgroupMemoryLimit` doesn't catch `InvalidPathException` from a malformed `/proc/self/mountinfo` escape.
**Limits:** no full CoreSuite run, no live cgroup or container run, no image build, and no Helm install.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4295040605
##########
hugegraph-store/docs/deployment-guide.md:
##########
@@ -726,8 +751,15 @@ environment:
HG_STORE_REST_PORT: "8520" # maps to server.port
HG_STORE_RAFT_ADDRESS: store0:8510 # maps to raft.address
HG_STORE_DATA_PATH: /hugegraph-store/storage # maps to app.data-path
+ # optional; maps to rocksdb.total_memory_size (bytes); unset = config file or auto-size
+ # HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE: "2147483648"
```
+`HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` must be a plain number of bytes; the entrypoint refuses
+anything else. When set it outranks `rocksdb.total_memory_size` in a mounted
+`conf/application-pd.yml` and a `-Drocksdb.total_memory_size` in `JAVA_OPTS`. Leave it unset to let the Store size the budget from the container
Review Comment:
⚠️ Important: “Leave it unset” only enables auto-sizing when the loaded config is also 0 or unset. An existing deployment with a mounted `application-pd.yml` can retain the old `32000000000` value; `resolve()` treats any positive value as explicit and only warns, so leaving this variable unset can preserve the original container OOM risk. Please document an upgrade step to remove/set the old key to 0 or set `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE`.
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.lang.management.ManagementFactory;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
+ static final long MIN_AUTO = 256 * MIB;
+ // Direct memory counted when -XX:MaxDirectMemorySize is unset. The JVM then allows
+ // as much as the heap, but the Store's gRPC/Netty buffers do not grow with the heap,
+ // and counting a 36 GB heap twice would leave rocksdb the floor on a large host
+ static final long UNSET_DIRECT = 1024 * MIB;
+ // RaftRocksdbOptions gives the write buffer manager 2/3 of the budget, and a 0-byte
+ // write buffer manager means no memtable limit at all
+ static final long MIN_EXPLICIT = MIB;
+ // The value shipped before auto-sizing; a large host never gets more than it had
+ static final long MAX_AUTO = 32_000_000_000L;
+
+ private RocksdbMemoryBudget() {
+ }
+
+ /**
+ * Returns the configured budget in bytes, or an automatic one when the value is
+ * missing, blank or 0.
+ */
+ public static long resolve(String configured) {
+ long value = isAuto(configured) ? 0L : parse(configured);
+ long heap = Runtime.getRuntime().maxMemory();
+ long physical = physicalMemory();
+ if (value > 0) {
+ if (value < MIN_EXPLICIT) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be 0 (auto) or at least %s bytes, " +
+ "got %s", MIN_EXPLICIT, value));
+ }
+ if (value < MIN_AUTO) {
+ log.warn("rocksdb.total_memory_size={} is below the {} bytes auto-sizing " +
+ "never goes under", value, MIN_AUTO);
+ }
+ long direct = maxDirectMemory(heap);
+ long left = physical - heap - direct - RESERVED;
+ if (physical > 0 && value > left) {
+ log.warn("rocksdb.total_memory_size={} exceeds the {} bytes left after the " +
+ "heap, direct memory and the {} bytes reserve (physical={}, heap={}, " +
+ "direct={}); the process may be OOM-killed",
+ value, left, RESERVED, physical, heap, direct);
+ }
+ return value;
+ }
+ long direct = maxDirectMemory(heap);
+ long budget = auto(physical, heap, direct);
+ log.info("rocksdb.total_memory_size is unset or 0, auto-sizing to {} bytes " +
+ "(physical={}, heap={}, direct={})", budget, physical, heap, direct);
+ if (physical > 0 && physical - heap - direct - RESERVED < 2 * MIN_AUTO) {
+ log.warn("Heap, direct memory and the {} bytes reserve leave too little of the " +
+ "{} bytes physical memory, rocksdb gets the {} bytes floor; lower -Xmx " +
+ "or -XX:MaxDirectMemorySize", RESERVED, physical, MIN_AUTO);
+ }
+ return budget;
+ }
+
+ static boolean isAuto(String configured) {
+ return configured == null || configured.trim().isEmpty();
+ }
+
+ static long parse(String configured) {
+ try {
+ long value = Long.parseLong(configured.trim());
+ if (value < 0) {
+ throw new NumberFormatException("negative");
+ }
+ return value;
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be a number of bytes, got '%s'",
+ configured), e);
+ }
+ }
+
+ /**
+ * Half of what is left after the heap, direct memory and the reserve, clamped to
+ * [MIN_AUTO, MAX_AUTO]. Falls back to the heap size, the previous default, when
+ * physical memory is unknown.
+ */
+ static long auto(long physical, long heap, long direct) {
+ if (physical <= 0) {
+ return Math.min(Math.max(heap, MIN_AUTO), MAX_AUTO);
+ }
+ long available = physical - heap - direct - RESERVED;
+ return Math.min(Math.max(available / 2, MIN_AUTO), MAX_AUTO);
+ }
+
+ /**
+ * The raft disruptor buffer size AppConfig derives from the budget when
+ * raft.disruptorBufferSize is 0: the budget in GB, at least 1, rounded to a power
+ * of two, times 32.
+ */
+ public static int disruptorBufferSize(long totalMemory) {
+ // A budget under 1 GB would otherwise give log(0) and a zero-sized buffer
+ int size = (int) Math.max(1, totalMemory / 1000 / 1000 / 1000);
+ return (int) Math.pow(2, Math.round(Math.log(size) / Math.log(2))) * 32;
+ }
+
+ /**
+ * The write_buffer_size AppConfig derives when the key is unset or 0: a thousandth
+ * of the budget, but never below the 1 MiB rocksdb.write_buffer_size accepts
+ * (RocksDBOptions.MEMTABLE_SIZE), which an auto-sized budget under 1 GiB would
+ * otherwise fall under.
+ */
+ public static long writeBufferSize(long totalMemory) {
+ return Math.max(totalMemory / 1000, MIB);
+ }
+
+ private static long physicalMemory() {
+ try {
+ // Reports the cgroup v1/v2 memory limit inside a container; older JDK 17
+ // updates may not see a limit set only on a parent cgroup
+ OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(
+ OperatingSystemMXBean.class);
+ return os == null ? -1L : os.getTotalMemorySize();
Review Comment:
⚠️ Important: This can still size RocksDB from host RAM on supported Java 17 installs. The adjacent comment acknowledges that older JDK 17 updates may miss a cgroup v2 limit set only on an ancestor; if the JVM reports host memory, `auto()` accepts it (for example, about 30.5 GiB with a 64 GiB host, `-Xmx1g` and a 512 MiB direct-memory cap, even with a 4 GiB container limit). Please detect the effective ancestor limit or require and document a JDK update that reports it, and cover this setup.
##########
.github/workflows/pd-store-ci.yml:
##########
@@ -406,6 +406,10 @@ jobs:
run: |
mvn editorconfig:check -pl hugegraph-store/hg-store-test -am -ntp
+ - name: Run Store docker entrypoint rocksdb memory tests
Review Comment:
⚠️ Important: This step is selected only when the `store` suite runs, but the CI selector does not map `hugegraph-server/hugegraph-dist/src/assembly/travis/test-store-docker-entrypoint.sh` to that suite. A later PR changing only this test script selects the Server/Docker/macOS paths (and distributed dependents), not `store`, so this test is skipped. Please add it to `CONTRACT_SUITES` and cover that mapping in the selector tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#issuecomment-6184873308
## [Codecov](https://app.codecov.io/gh/apache/hugegraph/pull/3355?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 49.27%. Comparing base ([`92af2f8`](https://app.codecov.io/gh/apache/hugegraph/commit/92af2f89c320c3e6fa7c10765a061f3edadea319?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`62376ee`](https://app.codecov.io/gh/apache/hugegraph/commit/62376ee833123aa35b16360350f9e55636b04ce8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 11 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #3355 +/- ##
============================================
- Coverage 54.31% 49.27% -5.04%
+ Complexity 2250 823 -1427
============================================
Files 770 66 -704
Lines 67952 6137 -61815
Branches 9213 658 -8555
============================================
- Hits 36905 3024 -33881
+ Misses 27470 2866 -24604
+ Partials 3577 247 -3330
```
| [Flag](https://app.codecov.io/gh/apache/hugegraph/pull/3355/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [pd](https://app.codecov.io/gh/apache/hugegraph/pull/3355/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.27% <ø> (?)` | |
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/hugegraph/pull/3355?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4305982300
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,433 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
+ static final long MIN_AUTO = 256 * MIB;
+ // RaftRocksdbOptions gives the write buffer manager 2/3 of the budget, and a 0-byte
+ // write buffer manager means no memtable limit at all
+ static final long MIN_EXPLICIT = MIB;
+ // The value shipped before auto-sizing; a large host never gets more than it had
+ static final long MAX_AUTO = 32_000_000_000L;
+
+ private RocksdbMemoryBudget() {
+ }
+
+ /**
+ * Returns the configured budget in bytes, or an automatic one when the value is
+ * missing, blank or 0.
+ */
+ public static long resolve(String configured) {
+ long value = isAuto(configured) ? 0L : parse(configured);
+ long heap = Runtime.getRuntime().maxMemory();
+ CgroupLimit cgroup = cgroupLimit(Paths.get("/proc/self/cgroup"),
+ Paths.get("/proc/self/mountinfo"),
+ Paths.get("/sys/fs/cgroup"));
+ long physical = effectivePhysical(jvmMemory(), cgroup.bytes);
+ if (value > 0) {
+ if (value < MIN_EXPLICIT) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be 0 (auto) or at least %s bytes, " +
+ "got %s", MIN_EXPLICIT, value));
+ }
+ if (value < MIN_AUTO) {
+ log.warn("rocksdb.total_memory_size={} is below the {} bytes auto-sizing " +
+ "never goes under", value, MIN_AUTO);
+ }
+ if (value == MAX_AUTO) {
+ // Most likely an application-pd.yml kept from before auto-sizing
+ log.warn("rocksdb.total_memory_size={} is the fixed value shipped before " +
+ "auto-sizing; set it to 0 or remove it to size the budget from " +
+ "the available memory", value);
+ }
+ long direct = maxDirectMemory(heap);
+ long left = physical - heap - direct - RESERVED;
+ if (physical > 0 && value > left) {
+ log.warn("rocksdb.total_memory_size={} exceeds the {} bytes left after the " +
+ "heap, direct memory and the {} bytes reserve (physical={}, heap={}, " +
+ "direct={}); the process may be OOM-killed",
+ value, left, RESERVED, physical, heap, direct);
+ }
+ return value;
+ }
+ long direct = maxDirectMemory(heap);
+ long budget = auto(physical, heap, direct);
+ log.info("rocksdb.total_memory_size is unset or 0, auto-sizing to {} bytes " +
+ "(physical={}, heap={}, direct={})", budget, physical, heap, direct);
+ if (cgroup.unmatched) {
+ log.warn("No cgroup mount in /proc/self/mountinfo contains this process's " +
+ "cgroup (an inherited mount in a cgroup namespace), so only the " +
+ "limits visible at the mount root are counted and a tighter limit on " +
+ "the process's own cgroup may be missed; set rocksdb.total_memory_size " +
+ "or HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE explicitly");
+ }
+ if (physical > 0 && physical - heap - direct - RESERVED < 2 * MIN_AUTO) {
+ log.warn("Heap, direct memory and the {} bytes reserve leave too little of the " +
+ "{} bytes physical memory, rocksdb gets the {} bytes floor; lower -Xmx " +
+ "or -XX:MaxDirectMemorySize", RESERVED, physical, MIN_AUTO);
+ }
+ return budget;
+ }
+
+ static boolean isAuto(String configured) {
+ return configured == null || configured.trim().isEmpty();
+ }
+
+ static long parse(String configured) {
+ try {
+ long value = Long.parseLong(configured.trim());
+ if (value < 0) {
+ throw new NumberFormatException("negative");
+ }
+ return value;
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be a number of bytes, got '%s'",
+ configured), e);
+ }
+ }
+
+ /**
+ * Half of what is left after the heap, direct memory and the reserve, clamped to
+ * [MIN_AUTO, MAX_AUTO]. Falls back to the heap size, the previous default, when
+ * physical memory is unknown.
+ */
+ static long auto(long physical, long heap, long direct) {
+ if (physical <= 0) {
+ return Math.min(Math.max(heap, MIN_AUTO), MAX_AUTO);
+ }
+ long available = physical - heap - direct - RESERVED;
+ return Math.min(Math.max(available / 2, MIN_AUTO), MAX_AUTO);
+ }
+
+ /**
+ * The raft disruptor buffer size AppConfig derives from the budget when
+ * raft.disruptorBufferSize is 0: the budget in GB, at least 1, rounded to a power
+ * of two, times 32.
+ */
+ public static int disruptorBufferSize(long totalMemory) {
+ // A budget under 1 GB would otherwise give log(0) and a zero-sized buffer
+ int size = (int) Math.max(1, totalMemory / 1000 / 1000 / 1000);
+ return (int) Math.pow(2, Math.round(Math.log(size) / Math.log(2))) * 32;
+ }
+
+ /**
+ * The write_buffer_size AppConfig derives when the key is unset or 0: a thousandth
+ * of the budget, but never below the 1 MiB rocksdb.write_buffer_size accepts
+ * (RocksDBOptions.MEMTABLE_SIZE), which an auto-sized budget under 1 GiB would
+ * otherwise fall under.
+ */
+ public static long writeBufferSize(long totalMemory) {
+ return Math.max(totalMemory / 1000, MIB);
+ }
+
+ private static long jvmMemory() {
+ try {
+ // Reports the cgroup v1/v2 memory limit inside a container, but older JDK 17
+ // updates may not see a limit set only on a parent cgroup
+ OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(
+ OperatingSystemMXBean.class);
+ return os == null ? -1L : os.getTotalMemorySize();
+ } catch (RuntimeException | LinkageError e) {
+ log.warn("Failed to read physical memory size", e);
+ return -1L;
+ }
+ }
+
+ /**
+ * A cgroup memory limit in bytes (-1 for none), and whether a cgroup mount existed
+ * but none of them contained this process's cgroup, in which case bytes only covers
+ * what is visible at those mounts' roots.
+ */
+ static final class CgroupLimit {
+
+ final long bytes;
+ final boolean unmatched;
+
+ CgroupLimit(long bytes, boolean unmatched) {
+ this.bytes = bytes;
+ this.unmatched = unmatched;
+ }
+ }
+
+ /**
+ * The smaller of what the JVM reports and the cgroup limit, ignoring unknown (<= 0)
+ * values.
+ */
+ static long effectivePhysical(long jvm, long cgroup) {
+ if (cgroup <= 0) {
+ return jvm;
+ }
+ return jvm <= 0 ? cgroup : Math.min(jvm, cgroup);
+ }
+
+ /**
+ * The lowest memory limit on this process's cgroup or any visible ancestor, or -1
+ * when there is none or it can't be read. The membership path from /proc/self/cgroup
+ * is rebased onto every mount of the controller in /proc/self/mountinfo whose root
+ * contains it, so a cgroupfs mounted from a subtree (root /kubepods at /sys/fs/cgroup)
+ * is walked from the right directory, and the lowest limit over all those mounts is
+ * taken: a leaf-only or unrelated mount listed first can't hide a fuller view listed
+ * later. Without a matching mount the default root is used. cgroup v2
+ * walks memory.max from the process's cgroup up to the mount; cgroup v1 reads
+ * hierarchical_memory_limit, which already accounts for ancestors. A limit above the
+ * mount root, as inside a private cgroup namespace, stays invisible.
+ */
+ static long cgroupMemoryLimit(Path procSelfCgroup, Path mountinfo, Path defaultRoot) {
+ return cgroupLimit(procSelfCgroup, mountinfo, defaultRoot).bytes;
+ }
+
+ static CgroupLimit cgroupLimit(Path procSelfCgroup, Path mountinfo, Path defaultRoot) {
+ List<String[]> v2Mounts = cgroupMounts(mountinfo, true);
+ List<String[]> v1Mounts = cgroupMounts(mountinfo, false);
+ List<String> lines;
+ try {
+ lines = Files.readAllLines(procSelfCgroup, StandardCharsets.UTF_8);
+ } catch (IOException | RuntimeException e) {
+ return new CgroupLimit(-1L, false);
+ }
+ long limit = -1L;
+ boolean unmatched = false;
+ for (String line : lines) {
+ String[] parts = line.split(":", 3);
+ if (parts.length < 3) {
+ continue;
+ }
+ boolean v2 = parts[0].equals("0") && parts[1].isEmpty();
+ if (!v2 && !("," + parts[1] + ",").contains(",memory,")) {
+ continue;
+ }
+ List<String[]> mounts = v2 ? v2Mounts : v1Mounts;
+ boolean matched = false;
+ for (String[] mount : mounts) {
+ if (!contains(mount[0], parts[2])) {
+ // A mount of another subtree says nothing about this cgroup
+ continue;
+ }
+ matched = true;
+ Path point = Paths.get(mount[1]);
+ String relative = rebase(parts[2], mount[0]);
+ limit = lowest(limit, v2 ? cgroupV2Limit(point, relative) :
+ cgroupV1Limit(point, relative));
+ }
+ if (!matched && !mounts.isEmpty()) {
+ // In a cgroup namespace /proc/self/cgroup says "/" while a mount made
+ // outside it has a root such as "/..": the process's own cgroup can't be
+ // located from here. The limits at each mount root still bound it.
+ unmatched = true;
+ for (String[] mount : mounts) {
+ Path point = Paths.get(mount[1]);
+ limit = lowest(limit, v2 ? readLimit(point.resolve("memory.max")) :
+ cgroupV1Limit(point, ""));
+ }
+ } else if (!matched) {
+ String relative = rebase(parts[2], "/");
+ limit = lowest(limit, v2 ? cgroupV2Limit(defaultRoot, relative) :
+ cgroupV1Limit(defaultRoot.resolve("memory"), relative));
+ }
+ }
+ return new CgroupLimit(limit, unmatched);
+ }
+
+ /**
+ * The {root, mount point} of every cgroup v2 mount, or of every cgroup v1 mount
+ * carrying the memory controller, in /proc/self/mountinfo order.
+ */
+ private static List<String[]> cgroupMounts(Path mountinfo, boolean v2) {
+ List<String[]> mounts = new ArrayList<>();
+ List<String> lines;
+ try {
+ lines = Files.readAllLines(mountinfo, StandardCharsets.UTF_8);
+ } catch (IOException | RuntimeException e) {
+ return mounts;
+ }
+ for (String line : lines) {
+ // id parent major:minor root mount-point options [optional...] - type source super
+ int separator = line.indexOf(" - ");
+ if (separator < 0) {
+ continue;
+ }
+ String[] fields = line.substring(0, separator).split(" ");
+ String[] tail = line.substring(separator + 3).split(" ");
+ if (fields.length < 5 || tail.length < 1) {
+ continue;
+ }
+ boolean match = v2 ? tail[0].equals("cgroup2") :
+ tail[0].equals("cgroup") && tail.length >= 3 &&
+ ("," + tail[2] + ",").contains(",memory,");
+ if (match) {
+ mounts.add(new String[]{unescape(fields[3]), unescape(fields[4])});
+ }
+ }
+ return mounts;
+ }
+
+ // Whether a mount whose root is mountRoot shows the cgroup at membership
+ static boolean contains(String mountRoot, String membership) {
+ String path = membership.startsWith("/") ? membership : "/" + membership;
+ String root = mountRoot.endsWith("/") && mountRoot.length() > 1 ?
+ mountRoot.substring(0, mountRoot.length() - 1) : mountRoot;
+ return root.equals("/") || path.equals(root) || path.startsWith(root + "/");
+ }
+
+ /**
+ * The membership path relative to a mount whose root is mountRoot: /kubepods/pod1/ctr
+ * under root /kubepods is pod1/ctr. A path outside the mount root is kept as is.
+ */
+ static String rebase(String membership, String mountRoot) {
+ String path = membership.startsWith("/") ? membership : "/" + membership;
+ String root = mountRoot.endsWith("/") && mountRoot.length() > 1 ?
+ mountRoot.substring(0, mountRoot.length() - 1) : mountRoot;
+ if (!root.equals("/")) {
+ if (path.equals(root)) {
+ return "";
+ }
+ if (path.startsWith(root + "/")) {
+ path = path.substring(root.length());
+ }
+ }
+ return path.substring(1);
+ }
+
+ // mountinfo writes space, tab, newline and backslash as octal escapes such as \040
+ private static String unescape(String field) {
+ StringBuilder out = new StringBuilder(field.length());
+ for (int i = 0; i < field.length(); i++) {
+ char c = field.charAt(i);
+ if (c == '\\' && i + 4 <= field.length() &&
+ field.substring(i + 1, i + 4).matches("[0-7]{3}")) {
+ out.append((char) Integer.parseInt(field.substring(i + 1, i + 4), 8));
+ i += 3;
+ } else {
+ out.append(c);
+ }
+ }
+ return out.toString();
+ }
+
+ private static long cgroupV2Limit(Path root, String relative) {
+ Path dir = root.resolve(relative).normalize();
+ if (!dir.startsWith(root) || !Files.isDirectory(dir)) {
+ // The process path is not visible from this mount (a cgroup namespace)
+ dir = root;
+ }
+ long limit = -1L;
+ for (Path d = dir; d != null && d.startsWith(root); d = d.getParent()) {
Review Comment:
Agreed, fixed in 7feaf6ac2. If every cgroup2 mount that contains the process's cgroup is a subtree (root other than `/`), the result is now marked incomplete, and auto-sizing logs the explicit-budget warning, unless a full-hierarchy view is also mounted. That view's walk covers the ancestors. v1 is left alone because `hierarchical_memory_limit` already includes them. The flag is renamed from `unmatched` to `incomplete` since it now covers both cases. New fixture `testStricterLimitAboveCgroupV2SubtreeMount` puts 2 GiB on `kubepods`, 4 GiB on the pod and none on the container. With only the `/kubepods` subtree mounted it returns 4 GiB, flagged. With a full view also mounted, in either order, it returns 2 GiB, unflagged. Store core tests: 129 passed. The deployment guide now mentions subtree mounts next to the namespace case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4307980846
##########
hugegraph-store/hg-store-dist/docker/docker-entrypoint.sh:
##########
@@ -54,6 +54,22 @@ require_env "HG_STORE_RAFT_ADDRESS"
: "${HG_STORE_GRPC_PORT:=8500}"
: "${HG_STORE_REST_PORT:=8520}"
: "${HG_STORE_DATA_PATH:=/hugegraph-store/storage}"
+: "${HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE:=}"
+
+# Bytes only: the Store parses the value as a long and refuses to start otherwise
+if [[ -n "${HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE}" &&
+ ! "${HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE}" =~ ^[0-9]+$ ]]; then
+ echo "ERROR: HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE must be a number of bytes," \
+ "got '${HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE}'" >&2
+ exit 2
+fi
+
+# Only emitted when set: SPRING_APPLICATION_JSON outranks conf/application-pd.yml
+ROCKSDB_JSON=""
+if [[ -n "${HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE}" ]]; then
+ ROCKSDB_JSON=",
+ \"rocksdb\": { \"total_memory_size\": \"${HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE}\" }"
Review Comment:
Thanks for confirming it with the real binder, fixed in 8b950e04d. Right after `updatePdConfig()` adds `pd-config` first, `keepMemoryOverride()` puts back on top a source holding only `rocksdb.total_memory_size`. Its value comes from `spring.application.json` (`HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` via the entrypoint) first, then `systemProperties` (`-D`). No other PD key changes precedence, and nothing is written back to PD, since `writeYml` and the PD KV are untouched. `PdConfigureListenerMemoryOverrideTest`, now in `CoreSuiteTest`, binds `rocksdb` as `Map<String, String>` with Boot's `Binder` against a remote 32 GB `pd-config`. An explicit 256 MiB, `0` and a `-D` 512 MiB each win, JSON beats `-D`, and PD's `write_buffer_size` still applies. The unset-override control gets PD's 32 GB, and re-running after a PD reload keeps exactly one override on top. With `keepMemoryOverride()` turned into a no-op, the four override cases fail and only the control passes. Store core tests: 134 pa
ssed. The deployment guide now says the variable also outranks the PD-distributed `application-pd.yml`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#issuecomment-6465512613
@imbajin @bitflicker64 The `JAVA_TOOL_OPTIONS` / `JDK_JAVA_OPTIONS` direct-memory cap fix is in 1dcabb74e, with launcher regression cases, and the thread is resolved. That push dismissed the earlier approval, so could you both take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4279250750
##########
hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/AppConfig.java:
##########
@@ -104,15 +105,11 @@ public void init() {
}
Map<String, String> rocksdb = rocksdbConfig.rocksdb;
- if (!rocksdb.containsKey("total_memory_size")
- || "0".equals(rocksdb.get("total_memory_size"))) {
- rocksdb.put("total_memory_size", Long.toString(rt.maxMemory()));
- }
- long totalMemory = Long.parseLong(rocksdbConfig.rocksdb.get("total_memory_size"));
+ long totalMemory = RocksdbMemoryBudget.resolve(rocksdb.get(RocksdbMemoryBudget.KEY));
+ rocksdb.put(RocksdbMemoryBudget.KEY, Long.toString(totalMemory));
if (raft.getDisruptorBufferSize() == 0) {
Review Comment:
[P2] Keep the derived write buffer above RocksDB's minimum
The smaller automatic budget makes the existing omitted/zero `write_buffer_size` fallback invalid. For a 4 GiB container with `-Xmx1g`, `-XX:MaxDirectMemorySize=512m`, `total_memory_size: 0` and `write_buffer_size: 0`, `resolve()` returns 536870912 bytes; the fallback below then derives 536870 bytes. `RocksDBOptions.MEMTABLE_SIZE` requires at least 1048576 bytes, so Store initialization rejects this configuration through `HgStoreNodeService.init()` → `RaftRocksdbOptions.initRocksdbGlobalConfig()` → `BusinessHandlerImpl.initRocksdb()` → `new HugeConfig(...)`.
Before this change, the same automatic configuration used the 1 GiB heap and derived 1073741 bytes, which passed. Omitting `write_buffer_size` takes the same path; the shipped 32000000 value masks it. Please clamp the derived value to RocksDB's minimum and add a regression covering the AppConfig-to-HugeConfig initialization path. This is source-verified with exact arithmetic, not an independently executed startup test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4300663949
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
+ static final long MIN_AUTO = 256 * MIB;
+ // Direct memory counted when -XX:MaxDirectMemorySize is unset. The JVM then allows
+ // as much as the heap, but the Store's gRPC/Netty buffers do not grow with the heap,
+ // and counting a 36 GB heap twice would leave rocksdb the floor on a large host
+ static final long UNSET_DIRECT = 1024 * MIB;
+ // RaftRocksdbOptions gives the write buffer manager 2/3 of the budget, and a 0-byte
+ // write buffer manager means no memtable limit at all
+ static final long MIN_EXPLICIT = MIB;
+ // The value shipped before auto-sizing; a large host never gets more than it had
+ static final long MAX_AUTO = 32_000_000_000L;
+
+ private RocksdbMemoryBudget() {
+ }
+
+ /**
+ * Returns the configured budget in bytes, or an automatic one when the value is
+ * missing, blank or 0.
+ */
+ public static long resolve(String configured) {
+ long value = isAuto(configured) ? 0L : parse(configured);
+ long heap = Runtime.getRuntime().maxMemory();
+ long physical = physicalMemory();
+ if (value > 0) {
+ if (value < MIN_EXPLICIT) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be 0 (auto) or at least %s bytes, " +
+ "got %s", MIN_EXPLICIT, value));
+ }
+ if (value < MIN_AUTO) {
+ log.warn("rocksdb.total_memory_size={} is below the {} bytes auto-sizing " +
+ "never goes under", value, MIN_AUTO);
+ }
+ if (value == MAX_AUTO) {
+ // Most likely an application-pd.yml kept from before auto-sizing
+ log.warn("rocksdb.total_memory_size={} is the fixed value shipped before " +
+ "auto-sizing; set it to 0 or remove it to size the budget from " +
+ "the available memory", value);
+ }
+ long direct = maxDirectMemory(heap);
+ long left = physical - heap - direct - RESERVED;
+ if (physical > 0 && value > left) {
+ log.warn("rocksdb.total_memory_size={} exceeds the {} bytes left after the " +
+ "heap, direct memory and the {} bytes reserve (physical={}, heap={}, " +
+ "direct={}); the process may be OOM-killed",
+ value, left, RESERVED, physical, heap, direct);
+ }
+ return value;
+ }
+ long direct = maxDirectMemory(heap);
+ long budget = auto(physical, heap, direct);
+ log.info("rocksdb.total_memory_size is unset or 0, auto-sizing to {} bytes " +
+ "(physical={}, heap={}, direct={})", budget, physical, heap, direct);
+ if (physical > 0 && physical - heap - direct - RESERVED < 2 * MIN_AUTO) {
+ log.warn("Heap, direct memory and the {} bytes reserve leave too little of the " +
+ "{} bytes physical memory, rocksdb gets the {} bytes floor; lower -Xmx " +
+ "or -XX:MaxDirectMemorySize", RESERVED, physical, MIN_AUTO);
+ }
+ return budget;
+ }
+
+ static boolean isAuto(String configured) {
+ return configured == null || configured.trim().isEmpty();
+ }
+
+ static long parse(String configured) {
+ try {
+ long value = Long.parseLong(configured.trim());
+ if (value < 0) {
+ throw new NumberFormatException("negative");
+ }
+ return value;
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be a number of bytes, got '%s'",
+ configured), e);
+ }
+ }
+
+ /**
+ * Half of what is left after the heap, direct memory and the reserve, clamped to
+ * [MIN_AUTO, MAX_AUTO]. Falls back to the heap size, the previous default, when
+ * physical memory is unknown.
+ */
+ static long auto(long physical, long heap, long direct) {
+ if (physical <= 0) {
+ return Math.min(Math.max(heap, MIN_AUTO), MAX_AUTO);
+ }
+ long available = physical - heap - direct - RESERVED;
+ return Math.min(Math.max(available / 2, MIN_AUTO), MAX_AUTO);
+ }
+
+ /**
+ * The raft disruptor buffer size AppConfig derives from the budget when
+ * raft.disruptorBufferSize is 0: the budget in GB, at least 1, rounded to a power
+ * of two, times 32.
+ */
+ public static int disruptorBufferSize(long totalMemory) {
+ // A budget under 1 GB would otherwise give log(0) and a zero-sized buffer
+ int size = (int) Math.max(1, totalMemory / 1000 / 1000 / 1000);
+ return (int) Math.pow(2, Math.round(Math.log(size) / Math.log(2))) * 32;
+ }
+
+ /**
+ * The write_buffer_size AppConfig derives when the key is unset or 0: a thousandth
+ * of the budget, but never below the 1 MiB rocksdb.write_buffer_size accepts
+ * (RocksDBOptions.MEMTABLE_SIZE), which an auto-sized budget under 1 GiB would
+ * otherwise fall under.
+ */
+ public static long writeBufferSize(long totalMemory) {
+ return Math.max(totalMemory / 1000, MIB);
+ }
+
+ private static long physicalMemory() {
+ long jvm;
+ try {
+ // Reports the cgroup v1/v2 memory limit inside a container, but older JDK 17
+ // updates may not see a limit set only on a parent cgroup
+ OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(
+ OperatingSystemMXBean.class);
+ jvm = os == null ? -1L : os.getTotalMemorySize();
+ } catch (RuntimeException | LinkageError e) {
+ log.warn("Failed to read physical memory size", e);
+ jvm = -1L;
+ }
+ long cgroup = cgroupMemoryLimit(Paths.get("/proc/self/cgroup"),
+ Paths.get("/proc/self/mountinfo"),
+ Paths.get("/sys/fs/cgroup"));
+ return effectivePhysical(jvm, cgroup);
+ }
+
+ /**
+ * The smaller of what the JVM reports and the cgroup limit, ignoring unknown (<= 0)
+ * values.
+ */
+ static long effectivePhysical(long jvm, long cgroup) {
+ if (cgroup <= 0) {
+ return jvm;
+ }
+ return jvm <= 0 ? cgroup : Math.min(jvm, cgroup);
+ }
+
+ /**
+ * The lowest memory limit on this process's cgroup or any visible ancestor, or -1
+ * when there is none or it can't be read. The membership path from /proc/self/cgroup
+ * is rebased onto every mount of the controller in /proc/self/mountinfo whose root
+ * contains it, so a cgroupfs mounted from a subtree (root /kubepods at /sys/fs/cgroup)
+ * is walked from the right directory, and the lowest limit over all those mounts is
+ * taken: a leaf-only or unrelated mount listed first can't hide a fuller view listed
+ * later. Without a matching mount the default root is used. cgroup v2
+ * walks memory.max from the process's cgroup up to the mount; cgroup v1 reads
+ * hierarchical_memory_limit, which already accounts for ancestors. A limit above the
+ * mount root, as inside a private cgroup namespace, stays invisible.
+ */
+ static long cgroupMemoryLimit(Path procSelfCgroup, Path mountinfo, Path defaultRoot) {
+ List<String[]> v2Mounts = cgroupMounts(mountinfo, true);
+ List<String[]> v1Mounts = cgroupMounts(mountinfo, false);
+ List<String> lines;
+ try {
+ lines = Files.readAllLines(procSelfCgroup, StandardCharsets.UTF_8);
+ } catch (IOException | RuntimeException e) {
+ return -1L;
+ }
+ long limit = -1L;
+ for (String line : lines) {
+ String[] parts = line.split(":", 3);
+ if (parts.length < 3) {
+ continue;
+ }
+ boolean v2 = parts[0].equals("0") && parts[1].isEmpty();
+ if (!v2 && !("," + parts[1] + ",").contains(",memory,")) {
+ continue;
+ }
+ boolean matched = false;
+ for (String[] mount : v2 ? v2Mounts : v1Mounts) {
+ if (!contains(mount[0], parts[2])) {
+ // A mount of another subtree says nothing about this cgroup
+ continue;
+ }
+ matched = true;
+ Path point = Paths.get(mount[1]);
+ String relative = rebase(parts[2], mount[0]);
+ limit = lowest(limit, v2 ? cgroupV2Limit(point, relative) :
+ cgroupV1Limit(point, relative));
+ }
+ if (!matched) {
Review Comment:
⚠️ Important: This fallback misses the process cgroup in the namespace layout documented by [cgroup_namespaces(7)](https://man7.org/linux/man-pages/man7/cgroup_namespaces.7.html): /proc/self/cgroup reports 0::/ while a cgroup2 mount created before the namespace reports root /..; contains rejects that mount, then this code reads only the default mount root. If the namespace root is a child with a tighter memory.max than its parent, the process limit is missed and auto-sizing can allocate an oversized RocksDB budget. Please resolve this mount layout to the process cgroup, or fail safe when it cannot be matched, and add a fixture with an unlimited parent and limited child.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4333220681
##########
hugegraph-store/docs/deployment-guide.md:
##########
@@ -356,14 +356,54 @@ rocksdb:
level0_file_num_compaction_trigger: 4
max_background_jobs: 8 # Background compaction/flush threads
- # Block Cache
- block_cache_size: 16000000000 # Block cache (16GB)
+ # Budget split: this share goes to memtables, the rest to the block cache
+ write_buffer_ratio: 0.66
# SST File Size
target_file_size_base: 134217728 # Target SST size (128MB)
max_bytes_for_level_base: 1073741824 # L1 size (1GB)
```
+`total_memory_size` bounds the memtables (`write_buffer_ratio` of it) and the block cache
+(the rest). Both live outside the Java heap, so `-Xmx` does not limit them. A fixed 1 GiB
+cache for the Raft log comes on top of this budget.
+
+With `total_memory_size: 0`, or the key unset, the Store sizes the budget at startup as half
+of what is left after the heap, the maximum direct memory and a 1.5 GiB reserve, clamped to
+256 MiB–32 GB. Inside a container the starting point is the cgroup memory limit, including a
+limit set only on a parent cgroup when the Store can see that cgroup; inside a private cgroup
+namespace, or when only a subtree of the cgroup2 hierarchy is mounted, a limit above it is not
+visible, so pin the value there if only the pod or slice is limited (the Store logs a warning
+when it detects such a mount). A Store with a 4 GiB limit, `-Xmx1g` and `-XX:MaxDirectMemorySize=512m` gets 512 MiB. Without
+`-XX:MaxDirectMemorySize` the JVM allows as much direct memory as heap, and the budget counts
+it that way. `bin/start-hugegraph-store.sh` sets the flag to the heap it picks; when `-j` sets
+`-Xmx`, `-XX:MaxHeapSize` or `-XX:MaxDirectMemorySize` itself, the launcher leaves the flag out and the JVM's
+default (that heap) or the given cap applies. Either way the budget counts a limit the JVM
+enforces. In small containers set a
+cap below the heap, or the budget falls to the 256 MiB floor.
+
+On a host without a container limit the starting point is physical memory. The default
+`bin/start-hugegraph-store.sh` caps the heap at 2 GiB and direct memory at the same 2 GiB, so a
+16 GiB host gets about 5.25 GiB. When you raise the heap, pass a direct-memory cap with it,
+for example `-j "-Xmx36g -XX:MaxDirectMemorySize=2g"`; otherwise direct memory counts as a
+second heap. With a 2 GiB cap a 16 GiB host with `-Xmx8g` gets 2.25 GiB and a 64 GiB host with
+`-Xmx36g` gets 12.25 GiB. Before this default every host got a fixed `32000000000`, so pin the
+value after upgrading if a large node relied on it.
+
+The chosen value is logged by `RocksdbMemoryBudget` at startup, with a warning when the floor
+applies, or when an explicit value is under 256 MiB or exceeds the memory left after the heap,
+direct memory and the reserve. Set a number of bytes, at least 1048576 (1 MiB), to pin it; a
+smaller value or a unit suffix such as `4GB` is rejected. Before this default existed the
+shipped value was `32000000000`, which can exceed a container limit and get the Store
+OOM-killed. Both caps are soft: RocksDB can briefly go over them under heavy writes, and a
+small budget with many partitions means frequent memtable flushes.
+
+**Upgrading an existing deployment.** Auto-sizing applies only when the loaded configuration
+has `total_memory_size: 0` or no value. A Store that mounts its own `conf/application-pd.yml`,
Review Comment:
⚠️ Important: This upgrade note names a Store-local mounted or retained `conf/application-pd.yml`, but `PdConfigureListener` can load an `application-pd.yml` persisted in PD and add it ahead of the local sources. If that shared file still has `rocksdb.total_memory_size: 32000000000`, updating the Store package does not activate auto-sizing unless the per-Store override is set. Please include updating the PD-managed value to `0` or removing it in the upgrade steps.
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,450 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
+ static final long MIN_AUTO = 256 * MIB;
+ // RaftRocksdbOptions gives the write buffer manager 2/3 of the budget, and a 0-byte
+ // write buffer manager means no memtable limit at all
+ static final long MIN_EXPLICIT = MIB;
+ // The value shipped before auto-sizing; a large host never gets more than it had
+ static final long MAX_AUTO = 32_000_000_000L;
+
+ private RocksdbMemoryBudget() {
+ }
+
+ /**
+ * Returns the configured budget in bytes, or an automatic one when the value is
+ * missing, blank or 0.
+ */
+ public static long resolve(String configured) {
+ long value = isAuto(configured) ? 0L : parse(configured);
+ long heap = Runtime.getRuntime().maxMemory();
+ CgroupLimit cgroup = cgroupLimit(Paths.get("/proc/self/cgroup"),
+ Paths.get("/proc/self/mountinfo"),
+ Paths.get("/sys/fs/cgroup"));
+ long physical = effectivePhysical(jvmMemory(), cgroup.bytes);
+ if (value > 0) {
+ if (value < MIN_EXPLICIT) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be 0 (auto) or at least %s bytes, " +
+ "got %s", MIN_EXPLICIT, value));
+ }
+ if (value < MIN_AUTO) {
+ log.warn("rocksdb.total_memory_size={} is below the {} bytes auto-sizing " +
+ "never goes under", value, MIN_AUTO);
+ }
+ if (value == MAX_AUTO) {
+ // Most likely an application-pd.yml kept from before auto-sizing
+ log.warn("rocksdb.total_memory_size={} is the fixed value shipped before " +
+ "auto-sizing; set it to 0 or remove it to size the budget from " +
+ "the available memory", value);
+ }
+ long direct = maxDirectMemory(heap);
+ long left = physical - heap - direct - RESERVED;
+ if (physical > 0 && value > left) {
+ log.warn("rocksdb.total_memory_size={} exceeds the {} bytes left after the " +
+ "heap, direct memory and the {} bytes reserve (physical={}, heap={}, " +
+ "direct={}); the process may be OOM-killed",
+ value, left, RESERVED, physical, heap, direct);
+ }
+ return value;
+ }
+ long direct = maxDirectMemory(heap);
+ long budget = auto(physical, heap, direct);
+ log.info("rocksdb.total_memory_size is unset or 0, auto-sizing to {} bytes " +
+ "(physical={}, heap={}, direct={})", budget, physical, heap, direct);
+ if (cgroup.incomplete) {
+ log.warn("The cgroup mounts in /proc/self/mountinfo don't show this process's " +
+ "whole cgroup hierarchy (an inherited mount in a cgroup namespace, or " +
+ "only cgroup2 subtree mounts), so a tighter limit on the process's " +
+ "cgroup or above the mounted subtree may be missed; set " +
+ "rocksdb.total_memory_size or HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE " +
+ "explicitly");
+ }
+ if (physical > 0 && physical - heap - direct - RESERVED < 2 * MIN_AUTO) {
+ log.warn("Heap, direct memory and the {} bytes reserve leave too little of the " +
+ "{} bytes physical memory, rocksdb gets the {} bytes floor; lower -Xmx " +
+ "or -XX:MaxDirectMemorySize", RESERVED, physical, MIN_AUTO);
+ }
+ return budget;
+ }
+
+ static boolean isAuto(String configured) {
+ return configured == null || configured.trim().isEmpty();
+ }
+
+ static long parse(String configured) {
+ try {
+ long value = Long.parseLong(configured.trim());
+ if (value < 0) {
+ throw new NumberFormatException("negative");
+ }
+ return value;
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be a number of bytes, got '%s'",
+ configured), e);
+ }
+ }
+
+ /**
+ * Half of what is left after the heap, direct memory and the reserve, clamped to
+ * [MIN_AUTO, MAX_AUTO]. Falls back to the heap size, the previous default, when
+ * physical memory is unknown.
+ */
+ static long auto(long physical, long heap, long direct) {
+ if (physical <= 0) {
+ return Math.min(Math.max(heap, MIN_AUTO), MAX_AUTO);
+ }
+ long available = physical - heap - direct - RESERVED;
+ return Math.min(Math.max(available / 2, MIN_AUTO), MAX_AUTO);
+ }
+
+ /**
+ * The raft disruptor buffer size AppConfig derives from the budget when
+ * raft.disruptorBufferSize is 0: the budget in GB, at least 1, rounded to a power
+ * of two, times 32.
+ */
+ public static int disruptorBufferSize(long totalMemory) {
+ // A budget under 1 GB would otherwise give log(0) and a zero-sized buffer
+ int size = (int) Math.max(1, totalMemory / 1000 / 1000 / 1000);
+ return (int) Math.pow(2, Math.round(Math.log(size) / Math.log(2))) * 32;
+ }
+
+ /**
+ * The write_buffer_size AppConfig derives when the key is unset or 0: a thousandth
+ * of the budget, but never below the 1 MiB rocksdb.write_buffer_size accepts
+ * (RocksDBOptions.MEMTABLE_SIZE), which an auto-sized budget under 1 GiB would
+ * otherwise fall under.
+ */
+ public static long writeBufferSize(long totalMemory) {
+ return Math.max(totalMemory / 1000, MIB);
+ }
+
+ private static long jvmMemory() {
+ try {
+ // Reports the cgroup v1/v2 memory limit inside a container, but older JDK 17
+ // updates may not see a limit set only on a parent cgroup
+ OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(
+ OperatingSystemMXBean.class);
+ return os == null ? -1L : os.getTotalMemorySize();
+ } catch (RuntimeException | LinkageError e) {
+ log.warn("Failed to read physical memory size", e);
+ return -1L;
+ }
+ }
+
+ /**
+ * A cgroup memory limit in bytes (-1 for none), and whether the mounts may hide a
+ * stricter one: a cgroup mount existed but none contained this process's cgroup, or
+ * every matching cgroup2 mount was a subtree, so bytes only covers what is visible
+ * from those mounts' roots down.
+ */
+ static final class CgroupLimit {
+
+ final long bytes;
+ final boolean incomplete;
+
+ CgroupLimit(long bytes, boolean incomplete) {
+ this.bytes = bytes;
+ this.incomplete = incomplete;
+ }
+ }
+
+ /**
+ * The smaller of what the JVM reports and the cgroup limit, ignoring unknown (<= 0)
+ * values.
+ */
+ static long effectivePhysical(long jvm, long cgroup) {
+ if (cgroup <= 0) {
+ return jvm;
+ }
+ return jvm <= 0 ? cgroup : Math.min(jvm, cgroup);
+ }
+
+ /**
+ * The lowest memory limit on this process's cgroup or any visible ancestor, or -1
+ * when there is none or it can't be read. The membership path from /proc/self/cgroup
+ * is rebased onto every mount of the controller in /proc/self/mountinfo whose root
+ * contains it, so a cgroupfs mounted from a subtree (root /kubepods at /sys/fs/cgroup)
+ * is walked from the right directory, and the lowest limit over all those mounts is
+ * taken: a leaf-only or unrelated mount listed first can't hide a fuller view listed
+ * later. Without a matching mount the default root is used. cgroup v2
+ * walks memory.max from the process's cgroup up to the mount; cgroup v1 reads
+ * hierarchical_memory_limit, which already accounts for ancestors. A limit above the
+ * mount root, as inside a private cgroup namespace, stays invisible.
+ */
+ static long cgroupMemoryLimit(Path procSelfCgroup, Path mountinfo, Path defaultRoot) {
+ return cgroupLimit(procSelfCgroup, mountinfo, defaultRoot).bytes;
+ }
+
+ static CgroupLimit cgroupLimit(Path procSelfCgroup, Path mountinfo, Path defaultRoot) {
+ List<String[]> v2Mounts = cgroupMounts(mountinfo, true);
+ List<String[]> v1Mounts = cgroupMounts(mountinfo, false);
+ List<String> lines;
+ try {
+ lines = Files.readAllLines(procSelfCgroup, StandardCharsets.UTF_8);
+ } catch (IOException | RuntimeException e) {
+ return new CgroupLimit(-1L, false);
+ }
+ long limit = -1L;
+ boolean incomplete = false;
+ for (String line : lines) {
+ String[] parts = line.split(":", 3);
+ if (parts.length < 3) {
+ continue;
+ }
+ boolean v2 = parts[0].equals("0") && parts[1].isEmpty();
+ if (!v2 && !("," + parts[1] + ",").contains(",memory,")) {
+ continue;
+ }
+ List<String[]> mounts = v2 ? v2Mounts : v1Mounts;
+ boolean matched = false;
+ boolean fullView = false;
+ for (String[] mount : mounts) {
+ if (!contains(mount[0], parts[2])) {
+ // A mount of another subtree says nothing about this cgroup
+ continue;
+ }
+ matched = true;
+ Path point = Paths.get(mount[1]);
+ String relative = rebase(parts[2], mount[0]);
+ // A full-hierarchy mount only covers the ancestors if this process's own
+ // directory is actually reachable through it
+ fullView |= mount[0].equals("/") && visible(point, relative);
Review Comment:
⚠️ Important: In a private cgroup namespace, `/proc/self/cgroup` can report `/` while a cgroup2 mount created inside the namespace also has mount root `/`. This makes `fullView` true even though any stricter `memory.max` above the namespace root is hidden; if the visible limit is `max`, the code leaves `incomplete` false and may size from host RAM without warning on JDK 17 builds that miss the outer limit. The [Linux cgroup v2 namespace docs](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#namespace) describe both namespace-relative paths and mounts rooted at the namespace root. Please flag this ambiguous root view or otherwise detect hidden parent limits, and add a regression for this mount layout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4303929229
##########
hugegraph-server/hugegraph-dist/src/assembly/travis/start-server.sh:
##########
@@ -42,6 +42,19 @@ function sed_in_place() {
esac
}
+# Leave exactly one key=value line: run-api-test.sh starts the same directory
+# more than once, and a key appended twice reaches HugeConfig as a list.
+function set_property() {
+ local file=$1
+ local key=$2
+ local value=$3
+ local pattern
+
+ pattern=$(printf '%s' "$key" | sed 's/[][\.*^$/]/\\&/g')
Review Comment:
Thanks, fixed in 37597c441. `set_property` no longer escapes the key with sed. The keys it gets are fixed names whose only regex character is `.`, so it uses `${key//./\\.}`, which leaves no bracket expression for BSD sed to misparse. The stock sed on my macOS 26 accepted the old expression, so I couldn't reproduce "unbalanced brackets" locally; the new form has nothing to misparse. `test-start-server.sh` passes 7/7 on Linux (GNU sed) and on macOS with `/usr/bin/sed` under both bash 5 and `/bin/bash` 3.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#issuecomment-6372910626
@imbajin your approval on de37bcb8d was dismissed when I pushed a7dc5c9a7. That commit only touches `hugegraph-store/docs/deployment-guide.md`. It fixes the PD-copy upgrade step you flagged on apache/hugegraph-doc#535: editing `S:FS` alone doesn't apply, so the guide now recommends the per-Store override set to `0`, or changing `S:FS` and then publishing `S:Timestamp`. There are no code changes since your review, and CI is green on a7dc5c9a7. Could you re-approve when you have a moment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4305214822
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,433 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
+ static final long MIN_AUTO = 256 * MIB;
+ // RaftRocksdbOptions gives the write buffer manager 2/3 of the budget, and a 0-byte
+ // write buffer manager means no memtable limit at all
+ static final long MIN_EXPLICIT = MIB;
+ // The value shipped before auto-sizing; a large host never gets more than it had
+ static final long MAX_AUTO = 32_000_000_000L;
+
+ private RocksdbMemoryBudget() {
+ }
+
+ /**
+ * Returns the configured budget in bytes, or an automatic one when the value is
+ * missing, blank or 0.
+ */
+ public static long resolve(String configured) {
+ long value = isAuto(configured) ? 0L : parse(configured);
+ long heap = Runtime.getRuntime().maxMemory();
+ CgroupLimit cgroup = cgroupLimit(Paths.get("/proc/self/cgroup"),
+ Paths.get("/proc/self/mountinfo"),
+ Paths.get("/sys/fs/cgroup"));
+ long physical = effectivePhysical(jvmMemory(), cgroup.bytes);
+ if (value > 0) {
+ if (value < MIN_EXPLICIT) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be 0 (auto) or at least %s bytes, " +
+ "got %s", MIN_EXPLICIT, value));
+ }
+ if (value < MIN_AUTO) {
+ log.warn("rocksdb.total_memory_size={} is below the {} bytes auto-sizing " +
+ "never goes under", value, MIN_AUTO);
+ }
+ if (value == MAX_AUTO) {
+ // Most likely an application-pd.yml kept from before auto-sizing
+ log.warn("rocksdb.total_memory_size={} is the fixed value shipped before " +
+ "auto-sizing; set it to 0 or remove it to size the budget from " +
+ "the available memory", value);
+ }
+ long direct = maxDirectMemory(heap);
+ long left = physical - heap - direct - RESERVED;
+ if (physical > 0 && value > left) {
+ log.warn("rocksdb.total_memory_size={} exceeds the {} bytes left after the " +
+ "heap, direct memory and the {} bytes reserve (physical={}, heap={}, " +
+ "direct={}); the process may be OOM-killed",
+ value, left, RESERVED, physical, heap, direct);
+ }
+ return value;
+ }
+ long direct = maxDirectMemory(heap);
+ long budget = auto(physical, heap, direct);
+ log.info("rocksdb.total_memory_size is unset or 0, auto-sizing to {} bytes " +
+ "(physical={}, heap={}, direct={})", budget, physical, heap, direct);
+ if (cgroup.unmatched) {
+ log.warn("No cgroup mount in /proc/self/mountinfo contains this process's " +
+ "cgroup (an inherited mount in a cgroup namespace), so only the " +
+ "limits visible at the mount root are counted and a tighter limit on " +
+ "the process's own cgroup may be missed; set rocksdb.total_memory_size " +
+ "or HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE explicitly");
+ }
+ if (physical > 0 && physical - heap - direct - RESERVED < 2 * MIN_AUTO) {
+ log.warn("Heap, direct memory and the {} bytes reserve leave too little of the " +
+ "{} bytes physical memory, rocksdb gets the {} bytes floor; lower -Xmx " +
+ "or -XX:MaxDirectMemorySize", RESERVED, physical, MIN_AUTO);
+ }
+ return budget;
+ }
+
+ static boolean isAuto(String configured) {
+ return configured == null || configured.trim().isEmpty();
+ }
+
+ static long parse(String configured) {
+ try {
+ long value = Long.parseLong(configured.trim());
+ if (value < 0) {
+ throw new NumberFormatException("negative");
+ }
+ return value;
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be a number of bytes, got '%s'",
+ configured), e);
+ }
+ }
+
+ /**
+ * Half of what is left after the heap, direct memory and the reserve, clamped to
+ * [MIN_AUTO, MAX_AUTO]. Falls back to the heap size, the previous default, when
+ * physical memory is unknown.
+ */
+ static long auto(long physical, long heap, long direct) {
+ if (physical <= 0) {
+ return Math.min(Math.max(heap, MIN_AUTO), MAX_AUTO);
+ }
+ long available = physical - heap - direct - RESERVED;
+ return Math.min(Math.max(available / 2, MIN_AUTO), MAX_AUTO);
+ }
+
+ /**
+ * The raft disruptor buffer size AppConfig derives from the budget when
+ * raft.disruptorBufferSize is 0: the budget in GB, at least 1, rounded to a power
+ * of two, times 32.
+ */
+ public static int disruptorBufferSize(long totalMemory) {
+ // A budget under 1 GB would otherwise give log(0) and a zero-sized buffer
+ int size = (int) Math.max(1, totalMemory / 1000 / 1000 / 1000);
+ return (int) Math.pow(2, Math.round(Math.log(size) / Math.log(2))) * 32;
+ }
+
+ /**
+ * The write_buffer_size AppConfig derives when the key is unset or 0: a thousandth
+ * of the budget, but never below the 1 MiB rocksdb.write_buffer_size accepts
+ * (RocksDBOptions.MEMTABLE_SIZE), which an auto-sized budget under 1 GiB would
+ * otherwise fall under.
+ */
+ public static long writeBufferSize(long totalMemory) {
+ return Math.max(totalMemory / 1000, MIB);
+ }
+
+ private static long jvmMemory() {
+ try {
+ // Reports the cgroup v1/v2 memory limit inside a container, but older JDK 17
+ // updates may not see a limit set only on a parent cgroup
+ OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(
+ OperatingSystemMXBean.class);
+ return os == null ? -1L : os.getTotalMemorySize();
+ } catch (RuntimeException | LinkageError e) {
+ log.warn("Failed to read physical memory size", e);
+ return -1L;
+ }
+ }
+
+ /**
+ * A cgroup memory limit in bytes (-1 for none), and whether a cgroup mount existed
+ * but none of them contained this process's cgroup, in which case bytes only covers
+ * what is visible at those mounts' roots.
+ */
+ static final class CgroupLimit {
+
+ final long bytes;
+ final boolean unmatched;
+
+ CgroupLimit(long bytes, boolean unmatched) {
+ this.bytes = bytes;
+ this.unmatched = unmatched;
+ }
+ }
+
+ /**
+ * The smaller of what the JVM reports and the cgroup limit, ignoring unknown (<= 0)
+ * values.
+ */
+ static long effectivePhysical(long jvm, long cgroup) {
+ if (cgroup <= 0) {
+ return jvm;
+ }
+ return jvm <= 0 ? cgroup : Math.min(jvm, cgroup);
+ }
+
+ /**
+ * The lowest memory limit on this process's cgroup or any visible ancestor, or -1
+ * when there is none or it can't be read. The membership path from /proc/self/cgroup
+ * is rebased onto every mount of the controller in /proc/self/mountinfo whose root
+ * contains it, so a cgroupfs mounted from a subtree (root /kubepods at /sys/fs/cgroup)
+ * is walked from the right directory, and the lowest limit over all those mounts is
+ * taken: a leaf-only or unrelated mount listed first can't hide a fuller view listed
+ * later. Without a matching mount the default root is used. cgroup v2
+ * walks memory.max from the process's cgroup up to the mount; cgroup v1 reads
+ * hierarchical_memory_limit, which already accounts for ancestors. A limit above the
+ * mount root, as inside a private cgroup namespace, stays invisible.
+ */
+ static long cgroupMemoryLimit(Path procSelfCgroup, Path mountinfo, Path defaultRoot) {
+ return cgroupLimit(procSelfCgroup, mountinfo, defaultRoot).bytes;
+ }
+
+ static CgroupLimit cgroupLimit(Path procSelfCgroup, Path mountinfo, Path defaultRoot) {
+ List<String[]> v2Mounts = cgroupMounts(mountinfo, true);
+ List<String[]> v1Mounts = cgroupMounts(mountinfo, false);
+ List<String> lines;
+ try {
+ lines = Files.readAllLines(procSelfCgroup, StandardCharsets.UTF_8);
+ } catch (IOException | RuntimeException e) {
+ return new CgroupLimit(-1L, false);
+ }
+ long limit = -1L;
+ boolean unmatched = false;
+ for (String line : lines) {
+ String[] parts = line.split(":", 3);
+ if (parts.length < 3) {
+ continue;
+ }
+ boolean v2 = parts[0].equals("0") && parts[1].isEmpty();
+ if (!v2 && !("," + parts[1] + ",").contains(",memory,")) {
+ continue;
+ }
+ List<String[]> mounts = v2 ? v2Mounts : v1Mounts;
+ boolean matched = false;
+ for (String[] mount : mounts) {
+ if (!contains(mount[0], parts[2])) {
+ // A mount of another subtree says nothing about this cgroup
+ continue;
+ }
+ matched = true;
+ Path point = Paths.get(mount[1]);
+ String relative = rebase(parts[2], mount[0]);
+ limit = lowest(limit, v2 ? cgroupV2Limit(point, relative) :
+ cgroupV1Limit(point, relative));
+ }
+ if (!matched && !mounts.isEmpty()) {
+ // In a cgroup namespace /proc/self/cgroup says "/" while a mount made
+ // outside it has a root such as "/..": the process's own cgroup can't be
+ // located from here. The limits at each mount root still bound it.
+ unmatched = true;
+ for (String[] mount : mounts) {
+ Path point = Paths.get(mount[1]);
+ limit = lowest(limit, v2 ? readLimit(point.resolve("memory.max")) :
+ cgroupV1Limit(point, ""));
+ }
+ } else if (!matched) {
+ String relative = rebase(parts[2], "/");
+ limit = lowest(limit, v2 ? cgroupV2Limit(defaultRoot, relative) :
+ cgroupV1Limit(defaultRoot.resolve("memory"), relative));
+ }
+ }
+ return new CgroupLimit(limit, unmatched);
+ }
+
+ /**
+ * The {root, mount point} of every cgroup v2 mount, or of every cgroup v1 mount
+ * carrying the memory controller, in /proc/self/mountinfo order.
+ */
+ private static List<String[]> cgroupMounts(Path mountinfo, boolean v2) {
+ List<String[]> mounts = new ArrayList<>();
+ List<String> lines;
+ try {
+ lines = Files.readAllLines(mountinfo, StandardCharsets.UTF_8);
+ } catch (IOException | RuntimeException e) {
+ return mounts;
+ }
+ for (String line : lines) {
+ // id parent major:minor root mount-point options [optional...] - type source super
+ int separator = line.indexOf(" - ");
+ if (separator < 0) {
+ continue;
+ }
+ String[] fields = line.substring(0, separator).split(" ");
+ String[] tail = line.substring(separator + 3).split(" ");
+ if (fields.length < 5 || tail.length < 1) {
+ continue;
+ }
+ boolean match = v2 ? tail[0].equals("cgroup2") :
+ tail[0].equals("cgroup") && tail.length >= 3 &&
+ ("," + tail[2] + ",").contains(",memory,");
+ if (match) {
+ mounts.add(new String[]{unescape(fields[3]), unescape(fields[4])});
+ }
+ }
+ return mounts;
+ }
+
+ // Whether a mount whose root is mountRoot shows the cgroup at membership
+ static boolean contains(String mountRoot, String membership) {
+ String path = membership.startsWith("/") ? membership : "/" + membership;
+ String root = mountRoot.endsWith("/") && mountRoot.length() > 1 ?
+ mountRoot.substring(0, mountRoot.length() - 1) : mountRoot;
+ return root.equals("/") || path.equals(root) || path.startsWith(root + "/");
+ }
+
+ /**
+ * The membership path relative to a mount whose root is mountRoot: /kubepods/pod1/ctr
+ * under root /kubepods is pod1/ctr. A path outside the mount root is kept as is.
+ */
+ static String rebase(String membership, String mountRoot) {
+ String path = membership.startsWith("/") ? membership : "/" + membership;
+ String root = mountRoot.endsWith("/") && mountRoot.length() > 1 ?
+ mountRoot.substring(0, mountRoot.length() - 1) : mountRoot;
+ if (!root.equals("/")) {
+ if (path.equals(root)) {
+ return "";
+ }
+ if (path.startsWith(root + "/")) {
+ path = path.substring(root.length());
+ }
+ }
+ return path.substring(1);
+ }
+
+ // mountinfo writes space, tab, newline and backslash as octal escapes such as \040
+ private static String unescape(String field) {
+ StringBuilder out = new StringBuilder(field.length());
+ for (int i = 0; i < field.length(); i++) {
+ char c = field.charAt(i);
+ if (c == '\\' && i + 4 <= field.length() &&
+ field.substring(i + 1, i + 4).matches("[0-7]{3}")) {
+ out.append((char) Integer.parseInt(field.substring(i + 1, i + 4), 8));
+ i += 3;
+ } else {
+ out.append(c);
+ }
+ }
+ return out.toString();
+ }
+
+ private static long cgroupV2Limit(Path root, String relative) {
+ Path dir = root.resolve(relative).normalize();
+ if (!dir.startsWith(root) || !Files.isDirectory(dir)) {
+ // The process path is not visible from this mount (a cgroup namespace)
+ dir = root;
+ }
+ long limit = -1L;
+ for (Path d = dir; d != null && d.startsWith(root); d = d.getParent()) {
Review Comment:
⚠️ Important: This walk stops at the cgroup mountpoint, which may itself be a subtree such as `/kubepods`. A stricter `memory.max` above that root is hidden, but `cgroupLimit()` still treats the mount as matched, so startup emits no warning; on JDK 17 builds that report host memory for parent-only limits, auto-sizing can still exceed the effective cgroup budget. Please flag non-root v2 mount roots as incomplete unless another full-hierarchy view is checked, and add a fixture with a stricter parent limit above the mounted subtree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4302779277
##########
hugegraph-server/hugegraph-dist/src/assembly/travis/start-server.sh:
##########
@@ -42,6 +42,19 @@ function sed_in_place() {
esac
}
+# Leave exactly one key=value line: run-api-test.sh starts the same directory
+# more than once, and a key appended twice reaches HugeConfig as a list.
+function set_property() {
+ local file=$1
+ local key=$2
+ local value=$3
+ local pattern
+
+ pattern=$(printf '%s' "$key" | sed 's/[][\.*^$/]/\\&/g')
Review Comment:
⚠️ Important: On Darwin systems using the stock BSD `sed`, the `/` delimiter in this substitution is parsed before the bracket expression closes, so the command returns `unbalanced brackets`. With `set -e`, the unconditional `set_property "$CONF" "schema.sync_deletion"` call exits before server startup. Please escape `/` or use another delimiter and cover the BSD `sed` case. Evidence: the exact expression exits 1 on the local Darwin system; the current-head macOS Actions job passed, so this is specific to `sed` implementations with this parsing behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4301770845
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
+ static final long MIN_AUTO = 256 * MIB;
+ // Direct memory counted when -XX:MaxDirectMemorySize is unset. The JVM then allows
+ // as much as the heap, but the Store's gRPC/Netty buffers do not grow with the heap,
+ // and counting a 36 GB heap twice would leave rocksdb the floor on a large host
+ static final long UNSET_DIRECT = 1024 * MIB;
+ // RaftRocksdbOptions gives the write buffer manager 2/3 of the budget, and a 0-byte
+ // write buffer manager means no memtable limit at all
+ static final long MIN_EXPLICIT = MIB;
+ // The value shipped before auto-sizing; a large host never gets more than it had
+ static final long MAX_AUTO = 32_000_000_000L;
+
+ private RocksdbMemoryBudget() {
+ }
+
+ /**
+ * Returns the configured budget in bytes, or an automatic one when the value is
+ * missing, blank or 0.
+ */
+ public static long resolve(String configured) {
+ long value = isAuto(configured) ? 0L : parse(configured);
+ long heap = Runtime.getRuntime().maxMemory();
+ long physical = physicalMemory();
+ if (value > 0) {
+ if (value < MIN_EXPLICIT) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be 0 (auto) or at least %s bytes, " +
+ "got %s", MIN_EXPLICIT, value));
+ }
+ if (value < MIN_AUTO) {
+ log.warn("rocksdb.total_memory_size={} is below the {} bytes auto-sizing " +
+ "never goes under", value, MIN_AUTO);
+ }
+ if (value == MAX_AUTO) {
+ // Most likely an application-pd.yml kept from before auto-sizing
+ log.warn("rocksdb.total_memory_size={} is the fixed value shipped before " +
+ "auto-sizing; set it to 0 or remove it to size the budget from " +
+ "the available memory", value);
+ }
+ long direct = maxDirectMemory(heap);
+ long left = physical - heap - direct - RESERVED;
+ if (physical > 0 && value > left) {
+ log.warn("rocksdb.total_memory_size={} exceeds the {} bytes left after the " +
+ "heap, direct memory and the {} bytes reserve (physical={}, heap={}, " +
+ "direct={}); the process may be OOM-killed",
+ value, left, RESERVED, physical, heap, direct);
+ }
+ return value;
+ }
+ long direct = maxDirectMemory(heap);
+ long budget = auto(physical, heap, direct);
+ log.info("rocksdb.total_memory_size is unset or 0, auto-sizing to {} bytes " +
+ "(physical={}, heap={}, direct={})", budget, physical, heap, direct);
+ if (physical > 0 && physical - heap - direct - RESERVED < 2 * MIN_AUTO) {
+ log.warn("Heap, direct memory and the {} bytes reserve leave too little of the " +
+ "{} bytes physical memory, rocksdb gets the {} bytes floor; lower -Xmx " +
+ "or -XX:MaxDirectMemorySize", RESERVED, physical, MIN_AUTO);
+ }
+ return budget;
+ }
+
+ static boolean isAuto(String configured) {
+ return configured == null || configured.trim().isEmpty();
+ }
+
+ static long parse(String configured) {
+ try {
+ long value = Long.parseLong(configured.trim());
+ if (value < 0) {
+ throw new NumberFormatException("negative");
+ }
+ return value;
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be a number of bytes, got '%s'",
+ configured), e);
+ }
+ }
+
+ /**
+ * Half of what is left after the heap, direct memory and the reserve, clamped to
+ * [MIN_AUTO, MAX_AUTO]. Falls back to the heap size, the previous default, when
+ * physical memory is unknown.
+ */
+ static long auto(long physical, long heap, long direct) {
+ if (physical <= 0) {
+ return Math.min(Math.max(heap, MIN_AUTO), MAX_AUTO);
+ }
+ long available = physical - heap - direct - RESERVED;
+ return Math.min(Math.max(available / 2, MIN_AUTO), MAX_AUTO);
+ }
+
+ /**
+ * The raft disruptor buffer size AppConfig derives from the budget when
+ * raft.disruptorBufferSize is 0: the budget in GB, at least 1, rounded to a power
+ * of two, times 32.
+ */
+ public static int disruptorBufferSize(long totalMemory) {
+ // A budget under 1 GB would otherwise give log(0) and a zero-sized buffer
+ int size = (int) Math.max(1, totalMemory / 1000 / 1000 / 1000);
+ return (int) Math.pow(2, Math.round(Math.log(size) / Math.log(2))) * 32;
+ }
+
+ /**
+ * The write_buffer_size AppConfig derives when the key is unset or 0: a thousandth
+ * of the budget, but never below the 1 MiB rocksdb.write_buffer_size accepts
+ * (RocksDBOptions.MEMTABLE_SIZE), which an auto-sized budget under 1 GiB would
+ * otherwise fall under.
+ */
+ public static long writeBufferSize(long totalMemory) {
+ return Math.max(totalMemory / 1000, MIB);
+ }
+
+ private static long physicalMemory() {
+ long jvm;
+ try {
+ // Reports the cgroup v1/v2 memory limit inside a container, but older JDK 17
+ // updates may not see a limit set only on a parent cgroup
+ OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(
+ OperatingSystemMXBean.class);
+ jvm = os == null ? -1L : os.getTotalMemorySize();
+ } catch (RuntimeException | LinkageError e) {
+ log.warn("Failed to read physical memory size", e);
+ jvm = -1L;
+ }
+ long cgroup = cgroupMemoryLimit(Paths.get("/proc/self/cgroup"),
+ Paths.get("/proc/self/mountinfo"),
+ Paths.get("/sys/fs/cgroup"));
+ return effectivePhysical(jvm, cgroup);
+ }
+
+ /**
+ * The smaller of what the JVM reports and the cgroup limit, ignoring unknown (<= 0)
+ * values.
+ */
+ static long effectivePhysical(long jvm, long cgroup) {
+ if (cgroup <= 0) {
+ return jvm;
+ }
+ return jvm <= 0 ? cgroup : Math.min(jvm, cgroup);
+ }
+
+ /**
+ * The lowest memory limit on this process's cgroup or any visible ancestor, or -1
+ * when there is none or it can't be read. The membership path from /proc/self/cgroup
+ * is rebased onto every mount of the controller in /proc/self/mountinfo whose root
+ * contains it, so a cgroupfs mounted from a subtree (root /kubepods at /sys/fs/cgroup)
+ * is walked from the right directory, and the lowest limit over all those mounts is
+ * taken: a leaf-only or unrelated mount listed first can't hide a fuller view listed
+ * later. Without a matching mount the default root is used. cgroup v2
+ * walks memory.max from the process's cgroup up to the mount; cgroup v1 reads
+ * hierarchical_memory_limit, which already accounts for ancestors. A limit above the
+ * mount root, as inside a private cgroup namespace, stays invisible.
+ */
+ static long cgroupMemoryLimit(Path procSelfCgroup, Path mountinfo, Path defaultRoot) {
+ List<String[]> v2Mounts = cgroupMounts(mountinfo, true);
+ List<String[]> v1Mounts = cgroupMounts(mountinfo, false);
+ List<String> lines;
+ try {
+ lines = Files.readAllLines(procSelfCgroup, StandardCharsets.UTF_8);
+ } catch (IOException | RuntimeException e) {
+ return -1L;
+ }
+ long limit = -1L;
+ for (String line : lines) {
+ String[] parts = line.split(":", 3);
+ if (parts.length < 3) {
+ continue;
+ }
+ boolean v2 = parts[0].equals("0") && parts[1].isEmpty();
+ if (!v2 && !("," + parts[1] + ",").contains(",memory,")) {
+ continue;
+ }
+ boolean matched = false;
+ for (String[] mount : v2 ? v2Mounts : v1Mounts) {
+ if (!contains(mount[0], parts[2])) {
+ // A mount of another subtree says nothing about this cgroup
+ continue;
+ }
+ matched = true;
+ Path point = Paths.get(mount[1]);
+ String relative = rebase(parts[2], mount[0]);
+ limit = lowest(limit, v2 ? cgroupV2Limit(point, relative) :
+ cgroupV1Limit(point, relative));
+ }
+ if (!matched) {
+ String relative = rebase(parts[2], "/");
+ limit = lowest(limit, v2 ? cgroupV2Limit(defaultRoot, relative) :
+ cgroupV1Limit(defaultRoot.resolve("memory"), relative));
+ }
+ }
+ return limit;
+ }
+
+ /**
+ * The {root, mount point} of every cgroup v2 mount, or of every cgroup v1 mount
+ * carrying the memory controller, in /proc/self/mountinfo order.
+ */
+ private static List<String[]> cgroupMounts(Path mountinfo, boolean v2) {
+ List<String[]> mounts = new ArrayList<>();
+ List<String> lines;
+ try {
+ lines = Files.readAllLines(mountinfo, StandardCharsets.UTF_8);
+ } catch (IOException | RuntimeException e) {
+ return mounts;
+ }
+ for (String line : lines) {
+ // id parent major:minor root mount-point options [optional...] - type source super
+ int separator = line.indexOf(" - ");
+ if (separator < 0) {
+ continue;
+ }
+ String[] fields = line.substring(0, separator).split(" ");
+ String[] tail = line.substring(separator + 3).split(" ");
+ if (fields.length < 5 || tail.length < 1) {
+ continue;
+ }
+ boolean match = v2 ? tail[0].equals("cgroup2") :
+ tail[0].equals("cgroup") && tail.length >= 3 &&
+ ("," + tail[2] + ",").contains(",memory,");
+ if (match) {
+ mounts.add(new String[]{unescape(fields[3]), unescape(fields[4])});
+ }
+ }
+ return mounts;
+ }
+
+ // Whether a mount whose root is mountRoot shows the cgroup at membership
+ static boolean contains(String mountRoot, String membership) {
+ String path = membership.startsWith("/") ? membership : "/" + membership;
+ String root = mountRoot.endsWith("/") && mountRoot.length() > 1 ?
+ mountRoot.substring(0, mountRoot.length() - 1) : mountRoot;
+ return root.equals("/") || path.equals(root) || path.startsWith(root + "/");
+ }
+
+ /**
+ * The membership path relative to a mount whose root is mountRoot: /kubepods/pod1/ctr
+ * under root /kubepods is pod1/ctr. A path outside the mount root is kept as is.
+ */
+ static String rebase(String membership, String mountRoot) {
+ String path = membership.startsWith("/") ? membership : "/" + membership;
+ String root = mountRoot.endsWith("/") && mountRoot.length() > 1 ?
+ mountRoot.substring(0, mountRoot.length() - 1) : mountRoot;
+ if (!root.equals("/")) {
+ if (path.equals(root)) {
+ return "";
+ }
+ if (path.startsWith(root + "/")) {
+ path = path.substring(root.length());
+ }
+ }
+ return path.substring(1);
+ }
+
+ // mountinfo writes space, tab, newline and backslash as octal escapes such as \040
+ private static String unescape(String field) {
+ StringBuilder out = new StringBuilder(field.length());
+ for (int i = 0; i < field.length(); i++) {
+ char c = field.charAt(i);
+ if (c == '\\' && i + 4 <= field.length() &&
+ field.substring(i + 1, i + 4).matches("[0-7]{3}")) {
+ out.append((char) Integer.parseInt(field.substring(i + 1, i + 4), 8));
+ i += 3;
+ } else {
+ out.append(c);
+ }
+ }
+ return out.toString();
+ }
+
+ private static long cgroupV2Limit(Path root, String relative) {
+ Path dir = root.resolve(relative).normalize();
+ if (!dir.startsWith(root) || !Files.isDirectory(dir)) {
+ // The process path is not visible from this mount (a cgroup namespace)
+ dir = root;
+ }
+ long limit = -1L;
+ for (Path d = dir; d != null && d.startsWith(root); d = d.getParent()) {
+ limit = lowest(limit, readLimit(d.resolve("memory.max")));
+ }
+ return limit;
+ }
+
+ private static long cgroupV1Limit(Path base, String relative) {
+ Path dir = base.resolve(relative).normalize();
+ if (!dir.startsWith(base) || !Files.isDirectory(dir)) {
+ dir = base;
+ }
+ long limit = readLimit(dir.resolve("memory.limit_in_bytes"));
+ try {
+ for (String line : Files.readAllLines(dir.resolve("memory.stat"),
+ StandardCharsets.UTF_8)) {
+ if (line.startsWith("hierarchical_memory_limit ")) {
+ limit = lowest(limit, parseLimit(line.substring(line.indexOf(' ') + 1)));
+ }
+ }
+ } catch (IOException | RuntimeException e) {
+ // memory.stat is optional here, memory.limit_in_bytes already counted
+ }
+ return limit;
+ }
+
+ private static long readLimit(Path file) {
+ try {
+ return parseLimit(new String(Files.readAllBytes(file), StandardCharsets.UTF_8));
+ } catch (IOException | RuntimeException e) {
+ return -1L;
+ }
+ }
+
+ // "max" (v2) and the near-Long.MAX_VALUE page-aligned v1 default mean no limit
+ private static long parseLimit(String value) {
+ String trimmed = value.trim();
+ if (trimmed.isEmpty() || trimmed.equals("max")) {
+ return -1L;
+ }
+ try {
+ long limit = Long.parseLong(trimmed);
+ return limit <= 0 || limit >= Long.MAX_VALUE / 2 ? -1L : limit;
+ } catch (NumberFormatException e) {
+ return -1L;
+ }
+ }
+
+ private static long lowest(long a, long b) {
+ if (a <= 0) {
+ return b;
+ }
+ return b <= 0 ? a : Math.min(a, b);
+ }
+
+ private static long maxDirectMemory(long heap) {
+ try {
+ HotSpotDiagnosticMXBean hotspot = ManagementFactory.getPlatformMXBean(
+ HotSpotDiagnosticMXBean.class);
+ return directMemory(Long.parseLong(
+ hotspot.getVMOption("MaxDirectMemorySize").getValue()), heap);
+ } catch (RuntimeException | LinkageError e) {
+ log.warn("Failed to read MaxDirectMemorySize, assuming it is unset", e);
+ return directMemory(0L, heap);
+ }
+ }
+
+ /**
+ * The direct memory the budget leaves room for: an explicit -XX:MaxDirectMemorySize
+ * as given, otherwise (0, unset) the heap size capped at UNSET_DIRECT.
+ */
+ static long directMemory(long configured, long heap) {
+ return configured > 0 ? configured : Math.min(heap, UNSET_DIRECT);
Review Comment:
Agreed, the 1 GiB was an assumption the JVM doesn't enforce. Fixed in 5139e1842, both ways you suggested. `bin/start-hugegraph-store.sh` now passes `-XX:MaxDirectMemorySize` equal to the heap it picks, unless `-j` sets one, since a later flag wins. The budget counts what the JVM actually allows: the configured cap, or the heap when unset. With the launcher's default 2 GiB heap and 2 GiB cap a 16 GiB host gets about 5.25 GiB. I checked the launcher by running it with a stub `java`: the default run gets `-Xmx2048m -XX:MaxDirectMemorySize=2048m`, and `-j "-XX:MaxDirectMemorySize=512m"` ends with 512m. A raised heap with no explicit cap now counts direct memory as a second heap and lands on the floor with a warning; the deployment guide says to pass a smaller cap with a large heap, e.g. `-j "-Xmx36g -XX:MaxDirectMemorySize=2g"` (12.25 GiB on 64 GiB). Store core tests: 128 passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4386515984
##########
hugegraph-store/docs/deployment-guide.md:
##########
@@ -356,14 +356,76 @@ rocksdb:
level0_file_num_compaction_trigger: 4
max_background_jobs: 8 # Background compaction/flush threads
- # Block Cache
- block_cache_size: 16000000000 # Block cache (16GB)
+ # Budget split: this share goes to memtables, the rest to the block cache
+ write_buffer_ratio: 0.66
# SST File Size
target_file_size_base: 134217728 # Target SST size (128MB)
max_bytes_for_level_base: 1073741824 # L1 size (1GB)
```
+`total_memory_size` bounds the memtables (`write_buffer_ratio` of it) and the block cache
+(the rest). Both live outside the Java heap, so `-Xmx` does not limit them. A fixed 1 GiB
+cache for the Raft log comes on top of this budget; the Raft log memtables of all partitions
+share a 256 MiB write buffer charged to that cache, so they don't add to it.
+
+With `total_memory_size: auto` (the shipped value; a missing, blank or `0` value means the same)
+the Store sizes the budget at startup as half of what is left after the heap, the maximum direct
+memory and a 1.5 GiB reserve (the 1 GiB Raft log cache plus 512 MiB for other native memory),
+clamped to 256 MiB–32 GB. Inside a container the starting point is the cgroup memory limit, including a
+limit set only on a parent cgroup when the Store can see that cgroup; inside a private cgroup
+namespace, or when only a subtree of the cgroup2 hierarchy is mounted, a limit above it is not
+visible, so pin the value there if only the pod or slice is limited. The Store uses the lowest
+limit it can see and logs a warning when it can't see the whole hierarchy; it doesn't guess at
+limits it can't read. A Store with a 4 GiB limit, `-Xmx1g` and `-XX:MaxDirectMemorySize=512m`
+gets 512 MiB. Without
+`-XX:MaxDirectMemorySize` the JVM allows as much direct memory as heap, and the budget counts
+it that way. `bin/start-hugegraph-store.sh` sets the flag to the heap it picks; when `-j` sets
+`-Xmx`, `-XX:MaxHeapSize` or `-XX:MaxDirectMemorySize` itself, the launcher leaves the flag out and the JVM's
+default (that heap) or the given cap applies. Either way the budget counts a limit the JVM
+enforces. In small containers set a
+cap below the heap, or the budget falls to the 256 MiB floor.
+
+On a host without a container limit the starting point is physical memory. The default
+`bin/start-hugegraph-store.sh` caps the heap at 2 GiB and direct memory at the same 2 GiB, so a
+16 GiB host gets about 5.25 GiB. When you raise the heap, pass a direct-memory cap with it,
+for example `-j "-Xmx36g -XX:MaxDirectMemorySize=2g"`; otherwise direct memory counts as a
+second heap. With a 2 GiB cap a 16 GiB host with `-Xmx8g` gets 2.25 GiB and a 64 GiB host with
+`-Xmx36g` gets 12.25 GiB. Before this default every host got a fixed `32000000000`, so pin the
+value after upgrading if a large node relied on it.
+
+At startup the Store logs the budget and what it was computed from, for example:
+
+```text
+RocksDB memory budget: 512 MiB (536870912 bytes), mode=auto, physical-limit=4 GiB, source=cgroup-v2, heap=1 GiB, direct=512 MiB, reserve=1.5 GiB, cgroup-view=complete
+```
+
+It warns when the floor applies, or when an explicit value is under 256 MiB or exceeds the
+memory left after the heap, direct memory and the reserve. Set a number of bytes, at least
+1048576 (1 MiB), to pin the budget; a smaller value or a unit suffix such as `4GB` is rejected. Before this default existed the
+shipped value was `32000000000`, which can exceed a container limit and get the Store
+OOM-killed. Both caps are soft: RocksDB can briefly go over them under heavy writes, and a
+small budget with many partitions means frequent memtable flushes.
+
+**Upgrading an existing deployment.** Auto-sizing applies only when the loaded configuration
+has `total_memory_size: auto` (or `0`, or no value). A Store that mounts its own `conf/application-pd.yml`,
+or a tarball install that kept its `conf/` from an earlier release, still has `32000000000` and
+keeps that fixed budget; the Store logs a warning naming the old value at startup. Set the key
+to `auto`, or set `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` in Docker.
+
+PD can also hold the old value. When PD has no shared copy yet, the first Store to start uploads
+its local `application-pd.yml` to PD (KV key `S:FS`). At startup a Store loads that copy over its
+local file only when it has no local file or the file is no newer than PD's `S:Timestamp`. The
+upload sets no timestamp and a missing one counts as 0, so changing `S:FS` alone changes nothing.
+The simplest upgrade is to set `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE=auto` (or
+`-Drocksdb.total_memory_size=auto`) on every Store and restart it; the Store keeps this value
+above both its local file and PD's copy. To change the shared copy instead, set
+`rocksdb.total_memory_size` to `auto` in `S:FS`, then put the current time in
Review Comment:
Right, earlier releases run `Long.parseLong` on anything but `0`. Fixed in 63856ac31. The guide now says to set `0` or remove the key in `S:FS` (earlier releases read `0` as their old heap-sized default), and to put `auto` there only once every Store runs this release. It also covers a copy uploaded from this release's shipped file, which already says `auto`. The same commit notes that the `S:Timestamp` self-restart drops `-j`/`-g`, the point you raised on apache/hugegraph-doc#535.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#issuecomment-6221715720
@imbajin thanks for the thorough rounds on this one. Everything you raised is now addressed at 535c2d01b: the ancestor and subtree cgroup limits, the multi-mount order, the upgrade step for an old `application-pd.yml`, and the `CONTRACT_SUITES` mapping. All 19 checks are green on that commit, and the paired doc PR is apache/hugegraph-doc#535, linked above with `Doc - Done`. Could you take a final look when you have a moment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin merged PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4398825009
##########
hugegraph-store/hg-store-dist/src/assembly/static/bin/start-hugegraph-store.sh:
##########
@@ -169,7 +169,18 @@ if [ "$JAVA_OPTIONS" = "" ]; then
>> ${OUTPUT}
exit 1
fi
- JAVA_OPTIONS="-Xms${MIN_MEM}m -Xmx${XMX}m -XX:MetaspaceSize=256M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${LOGS} ${USER_OPTION} "
+ # Cap direct memory at the heap size picked here, the JVM's own unset default, so the
+ # RocksDB auto-sizing (RocksdbMemoryBudget) counts a limit the JVM actually enforces.
+ # Leave it out when -j sets the heap or the cap itself: a fixed cap from the computed
+ # heap would no longer match a -j -Xmx, and the JVM then defaults it to that heap.
+ DIRECT_MEMORY_OPTION="-XX:MaxDirectMemorySize=${XMX}m"
Review Comment:
Fixed in 1dcabb74e: the launcher now leaves out its default `-XX:MaxDirectMemorySize` when `JAVA_TOOL_OPTIONS` or `JDK_JAVA_OPTIONS` sets one, so that cap applies as given. `test-start-hugegraph-store.sh` gains three cases: a cap in each variable keeps the launcher flag out, and an `-Xmx` in `JAVA_TOOL_OPTIONS` alone still gets the default cap, because the launcher `-Xmx` wins there. The two cap cases fail against the previous launcher. The deployment guide and hugegraph-doc#535 (0cb48506) list both variables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4301752805
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
+ static final long MIN_AUTO = 256 * MIB;
+ // Direct memory counted when -XX:MaxDirectMemorySize is unset. The JVM then allows
+ // as much as the heap, but the Store's gRPC/Netty buffers do not grow with the heap,
+ // and counting a 36 GB heap twice would leave rocksdb the floor on a large host
+ static final long UNSET_DIRECT = 1024 * MIB;
+ // RaftRocksdbOptions gives the write buffer manager 2/3 of the budget, and a 0-byte
+ // write buffer manager means no memtable limit at all
+ static final long MIN_EXPLICIT = MIB;
+ // The value shipped before auto-sizing; a large host never gets more than it had
+ static final long MAX_AUTO = 32_000_000_000L;
+
+ private RocksdbMemoryBudget() {
+ }
+
+ /**
+ * Returns the configured budget in bytes, or an automatic one when the value is
+ * missing, blank or 0.
+ */
+ public static long resolve(String configured) {
+ long value = isAuto(configured) ? 0L : parse(configured);
+ long heap = Runtime.getRuntime().maxMemory();
+ long physical = physicalMemory();
+ if (value > 0) {
+ if (value < MIN_EXPLICIT) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be 0 (auto) or at least %s bytes, " +
+ "got %s", MIN_EXPLICIT, value));
+ }
+ if (value < MIN_AUTO) {
+ log.warn("rocksdb.total_memory_size={} is below the {} bytes auto-sizing " +
+ "never goes under", value, MIN_AUTO);
+ }
+ if (value == MAX_AUTO) {
+ // Most likely an application-pd.yml kept from before auto-sizing
+ log.warn("rocksdb.total_memory_size={} is the fixed value shipped before " +
+ "auto-sizing; set it to 0 or remove it to size the budget from " +
+ "the available memory", value);
+ }
+ long direct = maxDirectMemory(heap);
+ long left = physical - heap - direct - RESERVED;
+ if (physical > 0 && value > left) {
+ log.warn("rocksdb.total_memory_size={} exceeds the {} bytes left after the " +
+ "heap, direct memory and the {} bytes reserve (physical={}, heap={}, " +
+ "direct={}); the process may be OOM-killed",
+ value, left, RESERVED, physical, heap, direct);
+ }
+ return value;
+ }
+ long direct = maxDirectMemory(heap);
+ long budget = auto(physical, heap, direct);
+ log.info("rocksdb.total_memory_size is unset or 0, auto-sizing to {} bytes " +
+ "(physical={}, heap={}, direct={})", budget, physical, heap, direct);
+ if (physical > 0 && physical - heap - direct - RESERVED < 2 * MIN_AUTO) {
+ log.warn("Heap, direct memory and the {} bytes reserve leave too little of the " +
+ "{} bytes physical memory, rocksdb gets the {} bytes floor; lower -Xmx " +
+ "or -XX:MaxDirectMemorySize", RESERVED, physical, MIN_AUTO);
+ }
+ return budget;
+ }
+
+ static boolean isAuto(String configured) {
+ return configured == null || configured.trim().isEmpty();
+ }
+
+ static long parse(String configured) {
+ try {
+ long value = Long.parseLong(configured.trim());
+ if (value < 0) {
+ throw new NumberFormatException("negative");
+ }
+ return value;
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be a number of bytes, got '%s'",
+ configured), e);
+ }
+ }
+
+ /**
+ * Half of what is left after the heap, direct memory and the reserve, clamped to
+ * [MIN_AUTO, MAX_AUTO]. Falls back to the heap size, the previous default, when
+ * physical memory is unknown.
+ */
+ static long auto(long physical, long heap, long direct) {
+ if (physical <= 0) {
+ return Math.min(Math.max(heap, MIN_AUTO), MAX_AUTO);
+ }
+ long available = physical - heap - direct - RESERVED;
+ return Math.min(Math.max(available / 2, MIN_AUTO), MAX_AUTO);
+ }
+
+ /**
+ * The raft disruptor buffer size AppConfig derives from the budget when
+ * raft.disruptorBufferSize is 0: the budget in GB, at least 1, rounded to a power
+ * of two, times 32.
+ */
+ public static int disruptorBufferSize(long totalMemory) {
+ // A budget under 1 GB would otherwise give log(0) and a zero-sized buffer
+ int size = (int) Math.max(1, totalMemory / 1000 / 1000 / 1000);
+ return (int) Math.pow(2, Math.round(Math.log(size) / Math.log(2))) * 32;
+ }
+
+ /**
+ * The write_buffer_size AppConfig derives when the key is unset or 0: a thousandth
+ * of the budget, but never below the 1 MiB rocksdb.write_buffer_size accepts
+ * (RocksDBOptions.MEMTABLE_SIZE), which an auto-sized budget under 1 GiB would
+ * otherwise fall under.
+ */
+ public static long writeBufferSize(long totalMemory) {
+ return Math.max(totalMemory / 1000, MIB);
+ }
+
+ private static long physicalMemory() {
+ long jvm;
+ try {
+ // Reports the cgroup v1/v2 memory limit inside a container, but older JDK 17
+ // updates may not see a limit set only on a parent cgroup
+ OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(
+ OperatingSystemMXBean.class);
+ jvm = os == null ? -1L : os.getTotalMemorySize();
+ } catch (RuntimeException | LinkageError e) {
+ log.warn("Failed to read physical memory size", e);
+ jvm = -1L;
+ }
+ long cgroup = cgroupMemoryLimit(Paths.get("/proc/self/cgroup"),
+ Paths.get("/proc/self/mountinfo"),
+ Paths.get("/sys/fs/cgroup"));
+ return effectivePhysical(jvm, cgroup);
+ }
+
+ /**
+ * The smaller of what the JVM reports and the cgroup limit, ignoring unknown (<= 0)
+ * values.
+ */
+ static long effectivePhysical(long jvm, long cgroup) {
+ if (cgroup <= 0) {
+ return jvm;
+ }
+ return jvm <= 0 ? cgroup : Math.min(jvm, cgroup);
+ }
+
+ /**
+ * The lowest memory limit on this process's cgroup or any visible ancestor, or -1
+ * when there is none or it can't be read. The membership path from /proc/self/cgroup
+ * is rebased onto every mount of the controller in /proc/self/mountinfo whose root
+ * contains it, so a cgroupfs mounted from a subtree (root /kubepods at /sys/fs/cgroup)
+ * is walked from the right directory, and the lowest limit over all those mounts is
+ * taken: a leaf-only or unrelated mount listed first can't hide a fuller view listed
+ * later. Without a matching mount the default root is used. cgroup v2
+ * walks memory.max from the process's cgroup up to the mount; cgroup v1 reads
+ * hierarchical_memory_limit, which already accounts for ancestors. A limit above the
+ * mount root, as inside a private cgroup namespace, stays invisible.
+ */
+ static long cgroupMemoryLimit(Path procSelfCgroup, Path mountinfo, Path defaultRoot) {
+ List<String[]> v2Mounts = cgroupMounts(mountinfo, true);
+ List<String[]> v1Mounts = cgroupMounts(mountinfo, false);
+ List<String> lines;
+ try {
+ lines = Files.readAllLines(procSelfCgroup, StandardCharsets.UTF_8);
+ } catch (IOException | RuntimeException e) {
+ return -1L;
+ }
+ long limit = -1L;
+ for (String line : lines) {
+ String[] parts = line.split(":", 3);
+ if (parts.length < 3) {
+ continue;
+ }
+ boolean v2 = parts[0].equals("0") && parts[1].isEmpty();
+ if (!v2 && !("," + parts[1] + ",").contains(",memory,")) {
+ continue;
+ }
+ boolean matched = false;
+ for (String[] mount : v2 ? v2Mounts : v1Mounts) {
+ if (!contains(mount[0], parts[2])) {
+ // A mount of another subtree says nothing about this cgroup
+ continue;
+ }
+ matched = true;
+ Path point = Paths.get(mount[1]);
+ String relative = rebase(parts[2], mount[0]);
+ limit = lowest(limit, v2 ? cgroupV2Limit(point, relative) :
+ cgroupV1Limit(point, relative));
+ }
+ if (!matched) {
Review Comment:
Agreed, fixed in 0b287bccc. When a cgroup mount exists but none contains the process's cgroup (your `/..` root with membership `/`), the resolver no longer treats it like an unlimited hierarchy. It reads the limit at each such mount's own root, which still bounds the process, and marks the result unmatched. Auto-sizing then logs a WARN saying the process's own, possibly tighter limit can't be seen and to set `rocksdb.total_memory_size` or `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` explicitly. The new fixture has an unlimited parent and a 2 GiB child for v2 (and the same layout for v1): the result is flagged, and an 8 GiB limit on the visible root is still applied. A separate test checks that matched or missing mountinfo isn't flagged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4300663949
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
+ static final long MIN_AUTO = 256 * MIB;
+ // Direct memory counted when -XX:MaxDirectMemorySize is unset. The JVM then allows
+ // as much as the heap, but the Store's gRPC/Netty buffers do not grow with the heap,
+ // and counting a 36 GB heap twice would leave rocksdb the floor on a large host
+ static final long UNSET_DIRECT = 1024 * MIB;
+ // RaftRocksdbOptions gives the write buffer manager 2/3 of the budget, and a 0-byte
+ // write buffer manager means no memtable limit at all
+ static final long MIN_EXPLICIT = MIB;
+ // The value shipped before auto-sizing; a large host never gets more than it had
+ static final long MAX_AUTO = 32_000_000_000L;
+
+ private RocksdbMemoryBudget() {
+ }
+
+ /**
+ * Returns the configured budget in bytes, or an automatic one when the value is
+ * missing, blank or 0.
+ */
+ public static long resolve(String configured) {
+ long value = isAuto(configured) ? 0L : parse(configured);
+ long heap = Runtime.getRuntime().maxMemory();
+ long physical = physicalMemory();
+ if (value > 0) {
+ if (value < MIN_EXPLICIT) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be 0 (auto) or at least %s bytes, " +
+ "got %s", MIN_EXPLICIT, value));
+ }
+ if (value < MIN_AUTO) {
+ log.warn("rocksdb.total_memory_size={} is below the {} bytes auto-sizing " +
+ "never goes under", value, MIN_AUTO);
+ }
+ if (value == MAX_AUTO) {
+ // Most likely an application-pd.yml kept from before auto-sizing
+ log.warn("rocksdb.total_memory_size={} is the fixed value shipped before " +
+ "auto-sizing; set it to 0 or remove it to size the budget from " +
+ "the available memory", value);
+ }
+ long direct = maxDirectMemory(heap);
+ long left = physical - heap - direct - RESERVED;
+ if (physical > 0 && value > left) {
+ log.warn("rocksdb.total_memory_size={} exceeds the {} bytes left after the " +
+ "heap, direct memory and the {} bytes reserve (physical={}, heap={}, " +
+ "direct={}); the process may be OOM-killed",
+ value, left, RESERVED, physical, heap, direct);
+ }
+ return value;
+ }
+ long direct = maxDirectMemory(heap);
+ long budget = auto(physical, heap, direct);
+ log.info("rocksdb.total_memory_size is unset or 0, auto-sizing to {} bytes " +
+ "(physical={}, heap={}, direct={})", budget, physical, heap, direct);
+ if (physical > 0 && physical - heap - direct - RESERVED < 2 * MIN_AUTO) {
+ log.warn("Heap, direct memory and the {} bytes reserve leave too little of the " +
+ "{} bytes physical memory, rocksdb gets the {} bytes floor; lower -Xmx " +
+ "or -XX:MaxDirectMemorySize", RESERVED, physical, MIN_AUTO);
+ }
+ return budget;
+ }
+
+ static boolean isAuto(String configured) {
+ return configured == null || configured.trim().isEmpty();
+ }
+
+ static long parse(String configured) {
+ try {
+ long value = Long.parseLong(configured.trim());
+ if (value < 0) {
+ throw new NumberFormatException("negative");
+ }
+ return value;
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be a number of bytes, got '%s'",
+ configured), e);
+ }
+ }
+
+ /**
+ * Half of what is left after the heap, direct memory and the reserve, clamped to
+ * [MIN_AUTO, MAX_AUTO]. Falls back to the heap size, the previous default, when
+ * physical memory is unknown.
+ */
+ static long auto(long physical, long heap, long direct) {
+ if (physical <= 0) {
+ return Math.min(Math.max(heap, MIN_AUTO), MAX_AUTO);
+ }
+ long available = physical - heap - direct - RESERVED;
+ return Math.min(Math.max(available / 2, MIN_AUTO), MAX_AUTO);
+ }
+
+ /**
+ * The raft disruptor buffer size AppConfig derives from the budget when
+ * raft.disruptorBufferSize is 0: the budget in GB, at least 1, rounded to a power
+ * of two, times 32.
+ */
+ public static int disruptorBufferSize(long totalMemory) {
+ // A budget under 1 GB would otherwise give log(0) and a zero-sized buffer
+ int size = (int) Math.max(1, totalMemory / 1000 / 1000 / 1000);
+ return (int) Math.pow(2, Math.round(Math.log(size) / Math.log(2))) * 32;
+ }
+
+ /**
+ * The write_buffer_size AppConfig derives when the key is unset or 0: a thousandth
+ * of the budget, but never below the 1 MiB rocksdb.write_buffer_size accepts
+ * (RocksDBOptions.MEMTABLE_SIZE), which an auto-sized budget under 1 GiB would
+ * otherwise fall under.
+ */
+ public static long writeBufferSize(long totalMemory) {
+ return Math.max(totalMemory / 1000, MIB);
+ }
+
+ private static long physicalMemory() {
+ long jvm;
+ try {
+ // Reports the cgroup v1/v2 memory limit inside a container, but older JDK 17
+ // updates may not see a limit set only on a parent cgroup
+ OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(
+ OperatingSystemMXBean.class);
+ jvm = os == null ? -1L : os.getTotalMemorySize();
+ } catch (RuntimeException | LinkageError e) {
+ log.warn("Failed to read physical memory size", e);
+ jvm = -1L;
+ }
+ long cgroup = cgroupMemoryLimit(Paths.get("/proc/self/cgroup"),
+ Paths.get("/proc/self/mountinfo"),
+ Paths.get("/sys/fs/cgroup"));
+ return effectivePhysical(jvm, cgroup);
+ }
+
+ /**
+ * The smaller of what the JVM reports and the cgroup limit, ignoring unknown (<= 0)
+ * values.
+ */
+ static long effectivePhysical(long jvm, long cgroup) {
+ if (cgroup <= 0) {
+ return jvm;
+ }
+ return jvm <= 0 ? cgroup : Math.min(jvm, cgroup);
+ }
+
+ /**
+ * The lowest memory limit on this process's cgroup or any visible ancestor, or -1
+ * when there is none or it can't be read. The membership path from /proc/self/cgroup
+ * is rebased onto every mount of the controller in /proc/self/mountinfo whose root
+ * contains it, so a cgroupfs mounted from a subtree (root /kubepods at /sys/fs/cgroup)
+ * is walked from the right directory, and the lowest limit over all those mounts is
+ * taken: a leaf-only or unrelated mount listed first can't hide a fuller view listed
+ * later. Without a matching mount the default root is used. cgroup v2
+ * walks memory.max from the process's cgroup up to the mount; cgroup v1 reads
+ * hierarchical_memory_limit, which already accounts for ancestors. A limit above the
+ * mount root, as inside a private cgroup namespace, stays invisible.
+ */
+ static long cgroupMemoryLimit(Path procSelfCgroup, Path mountinfo, Path defaultRoot) {
+ List<String[]> v2Mounts = cgroupMounts(mountinfo, true);
+ List<String[]> v1Mounts = cgroupMounts(mountinfo, false);
+ List<String> lines;
+ try {
+ lines = Files.readAllLines(procSelfCgroup, StandardCharsets.UTF_8);
+ } catch (IOException | RuntimeException e) {
+ return -1L;
+ }
+ long limit = -1L;
+ for (String line : lines) {
+ String[] parts = line.split(":", 3);
+ if (parts.length < 3) {
+ continue;
+ }
+ boolean v2 = parts[0].equals("0") && parts[1].isEmpty();
+ if (!v2 && !("," + parts[1] + ",").contains(",memory,")) {
+ continue;
+ }
+ boolean matched = false;
+ for (String[] mount : v2 ? v2Mounts : v1Mounts) {
+ if (!contains(mount[0], parts[2])) {
+ // A mount of another subtree says nothing about this cgroup
+ continue;
+ }
+ matched = true;
+ Path point = Paths.get(mount[1]);
+ String relative = rebase(parts[2], mount[0]);
+ limit = lowest(limit, v2 ? cgroupV2Limit(point, relative) :
+ cgroupV1Limit(point, relative));
+ }
+ if (!matched) {
Review Comment:
⚠️ Important: This fallback misses the process cgroup in the namespace layout documented by [cgroup_namespaces(7)](https://man7.org/linux/man-pages/man7/cgroup_namespaces.7.html): /proc/self/cgroup reports 0::/ while a cgroup2 mount created before the namespace reports root /... contains rejects that mount, then this code reads only the default mount root. If the namespace root is a child with a tighter memory.max than its parent, the process limit is missed and auto-sizing can allocate an oversized RocksDB budget. Please resolve this mount layout to the process cgroup, or fail safe when it cannot be matched, and add a fixture with an unlimited parent and limited child.
##########
hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/options/RocksdbMemoryBudget.java:
##########
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hugegraph.store.options;
+
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import com.sun.management.OperatingSystemMXBean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Resolves rocksdb.total_memory_size, the budget RaftRocksdbOptions splits into the
+ * write buffer manager's cache and the block cache. Both caches are off-heap, so an
+ * unset or "0" budget is sized from the memory left after the heap, direct memory and
+ * the fixed raft log cache, instead of a fixed number that can exceed a container limit.
+ */
+@Slf4j
+public final class RocksdbMemoryBudget {
+
+ public static final String KEY = "total_memory_size";
+
+ static final long MIB = 1024L * 1024L;
+ // The 1 GiB raft log block cache (RaftRocksdbOptions) plus thread stacks, metaspace
+ // and other native allocations that sit outside the budget
+ static final long RESERVED = 1024 * MIB + 512 * MIB;
+ static final long MIN_AUTO = 256 * MIB;
+ // Direct memory counted when -XX:MaxDirectMemorySize is unset. The JVM then allows
+ // as much as the heap, but the Store's gRPC/Netty buffers do not grow with the heap,
+ // and counting a 36 GB heap twice would leave rocksdb the floor on a large host
+ static final long UNSET_DIRECT = 1024 * MIB;
+ // RaftRocksdbOptions gives the write buffer manager 2/3 of the budget, and a 0-byte
+ // write buffer manager means no memtable limit at all
+ static final long MIN_EXPLICIT = MIB;
+ // The value shipped before auto-sizing; a large host never gets more than it had
+ static final long MAX_AUTO = 32_000_000_000L;
+
+ private RocksdbMemoryBudget() {
+ }
+
+ /**
+ * Returns the configured budget in bytes, or an automatic one when the value is
+ * missing, blank or 0.
+ */
+ public static long resolve(String configured) {
+ long value = isAuto(configured) ? 0L : parse(configured);
+ long heap = Runtime.getRuntime().maxMemory();
+ long physical = physicalMemory();
+ if (value > 0) {
+ if (value < MIN_EXPLICIT) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be 0 (auto) or at least %s bytes, " +
+ "got %s", MIN_EXPLICIT, value));
+ }
+ if (value < MIN_AUTO) {
+ log.warn("rocksdb.total_memory_size={} is below the {} bytes auto-sizing " +
+ "never goes under", value, MIN_AUTO);
+ }
+ if (value == MAX_AUTO) {
+ // Most likely an application-pd.yml kept from before auto-sizing
+ log.warn("rocksdb.total_memory_size={} is the fixed value shipped before " +
+ "auto-sizing; set it to 0 or remove it to size the budget from " +
+ "the available memory", value);
+ }
+ long direct = maxDirectMemory(heap);
+ long left = physical - heap - direct - RESERVED;
+ if (physical > 0 && value > left) {
+ log.warn("rocksdb.total_memory_size={} exceeds the {} bytes left after the " +
+ "heap, direct memory and the {} bytes reserve (physical={}, heap={}, " +
+ "direct={}); the process may be OOM-killed",
+ value, left, RESERVED, physical, heap, direct);
+ }
+ return value;
+ }
+ long direct = maxDirectMemory(heap);
+ long budget = auto(physical, heap, direct);
+ log.info("rocksdb.total_memory_size is unset or 0, auto-sizing to {} bytes " +
+ "(physical={}, heap={}, direct={})", budget, physical, heap, direct);
+ if (physical > 0 && physical - heap - direct - RESERVED < 2 * MIN_AUTO) {
+ log.warn("Heap, direct memory and the {} bytes reserve leave too little of the " +
+ "{} bytes physical memory, rocksdb gets the {} bytes floor; lower -Xmx " +
+ "or -XX:MaxDirectMemorySize", RESERVED, physical, MIN_AUTO);
+ }
+ return budget;
+ }
+
+ static boolean isAuto(String configured) {
+ return configured == null || configured.trim().isEmpty();
+ }
+
+ static long parse(String configured) {
+ try {
+ long value = Long.parseLong(configured.trim());
+ if (value < 0) {
+ throw new NumberFormatException("negative");
+ }
+ return value;
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(String.format(
+ "rocksdb.total_memory_size must be a number of bytes, got '%s'",
+ configured), e);
+ }
+ }
+
+ /**
+ * Half of what is left after the heap, direct memory and the reserve, clamped to
+ * [MIN_AUTO, MAX_AUTO]. Falls back to the heap size, the previous default, when
+ * physical memory is unknown.
+ */
+ static long auto(long physical, long heap, long direct) {
+ if (physical <= 0) {
+ return Math.min(Math.max(heap, MIN_AUTO), MAX_AUTO);
+ }
+ long available = physical - heap - direct - RESERVED;
+ return Math.min(Math.max(available / 2, MIN_AUTO), MAX_AUTO);
+ }
+
+ /**
+ * The raft disruptor buffer size AppConfig derives from the budget when
+ * raft.disruptorBufferSize is 0: the budget in GB, at least 1, rounded to a power
+ * of two, times 32.
+ */
+ public static int disruptorBufferSize(long totalMemory) {
+ // A budget under 1 GB would otherwise give log(0) and a zero-sized buffer
+ int size = (int) Math.max(1, totalMemory / 1000 / 1000 / 1000);
+ return (int) Math.pow(2, Math.round(Math.log(size) / Math.log(2))) * 32;
+ }
+
+ /**
+ * The write_buffer_size AppConfig derives when the key is unset or 0: a thousandth
+ * of the budget, but never below the 1 MiB rocksdb.write_buffer_size accepts
+ * (RocksDBOptions.MEMTABLE_SIZE), which an auto-sized budget under 1 GiB would
+ * otherwise fall under.
+ */
+ public static long writeBufferSize(long totalMemory) {
+ return Math.max(totalMemory / 1000, MIB);
+ }
+
+ private static long physicalMemory() {
+ long jvm;
+ try {
+ // Reports the cgroup v1/v2 memory limit inside a container, but older JDK 17
+ // updates may not see a limit set only on a parent cgroup
+ OperatingSystemMXBean os = ManagementFactory.getPlatformMXBean(
+ OperatingSystemMXBean.class);
+ jvm = os == null ? -1L : os.getTotalMemorySize();
+ } catch (RuntimeException | LinkageError e) {
+ log.warn("Failed to read physical memory size", e);
+ jvm = -1L;
+ }
+ long cgroup = cgroupMemoryLimit(Paths.get("/proc/self/cgroup"),
+ Paths.get("/proc/self/mountinfo"),
+ Paths.get("/sys/fs/cgroup"));
+ return effectivePhysical(jvm, cgroup);
+ }
+
+ /**
+ * The smaller of what the JVM reports and the cgroup limit, ignoring unknown (<= 0)
+ * values.
+ */
+ static long effectivePhysical(long jvm, long cgroup) {
+ if (cgroup <= 0) {
+ return jvm;
+ }
+ return jvm <= 0 ? cgroup : Math.min(jvm, cgroup);
+ }
+
+ /**
+ * The lowest memory limit on this process's cgroup or any visible ancestor, or -1
+ * when there is none or it can't be read. The membership path from /proc/self/cgroup
+ * is rebased onto every mount of the controller in /proc/self/mountinfo whose root
+ * contains it, so a cgroupfs mounted from a subtree (root /kubepods at /sys/fs/cgroup)
+ * is walked from the right directory, and the lowest limit over all those mounts is
+ * taken: a leaf-only or unrelated mount listed first can't hide a fuller view listed
+ * later. Without a matching mount the default root is used. cgroup v2
+ * walks memory.max from the process's cgroup up to the mount; cgroup v1 reads
+ * hierarchical_memory_limit, which already accounts for ancestors. A limit above the
+ * mount root, as inside a private cgroup namespace, stays invisible.
+ */
+ static long cgroupMemoryLimit(Path procSelfCgroup, Path mountinfo, Path defaultRoot) {
+ List<String[]> v2Mounts = cgroupMounts(mountinfo, true);
+ List<String[]> v1Mounts = cgroupMounts(mountinfo, false);
+ List<String> lines;
+ try {
+ lines = Files.readAllLines(procSelfCgroup, StandardCharsets.UTF_8);
+ } catch (IOException | RuntimeException e) {
+ return -1L;
+ }
+ long limit = -1L;
+ for (String line : lines) {
+ String[] parts = line.split(":", 3);
+ if (parts.length < 3) {
+ continue;
+ }
+ boolean v2 = parts[0].equals("0") && parts[1].isEmpty();
+ if (!v2 && !("," + parts[1] + ",").contains(",memory,")) {
+ continue;
+ }
+ boolean matched = false;
+ for (String[] mount : v2 ? v2Mounts : v1Mounts) {
+ if (!contains(mount[0], parts[2])) {
+ // A mount of another subtree says nothing about this cgroup
+ continue;
+ }
+ matched = true;
+ Path point = Paths.get(mount[1]);
+ String relative = rebase(parts[2], mount[0]);
+ limit = lowest(limit, v2 ? cgroupV2Limit(point, relative) :
+ cgroupV1Limit(point, relative));
+ }
+ if (!matched) {
+ String relative = rebase(parts[2], "/");
+ limit = lowest(limit, v2 ? cgroupV2Limit(defaultRoot, relative) :
+ cgroupV1Limit(defaultRoot.resolve("memory"), relative));
+ }
+ }
+ return limit;
+ }
+
+ /**
+ * The {root, mount point} of every cgroup v2 mount, or of every cgroup v1 mount
+ * carrying the memory controller, in /proc/self/mountinfo order.
+ */
+ private static List<String[]> cgroupMounts(Path mountinfo, boolean v2) {
+ List<String[]> mounts = new ArrayList<>();
+ List<String> lines;
+ try {
+ lines = Files.readAllLines(mountinfo, StandardCharsets.UTF_8);
+ } catch (IOException | RuntimeException e) {
+ return mounts;
+ }
+ for (String line : lines) {
+ // id parent major:minor root mount-point options [optional...] - type source super
+ int separator = line.indexOf(" - ");
+ if (separator < 0) {
+ continue;
+ }
+ String[] fields = line.substring(0, separator).split(" ");
+ String[] tail = line.substring(separator + 3).split(" ");
+ if (fields.length < 5 || tail.length < 1) {
+ continue;
+ }
+ boolean match = v2 ? tail[0].equals("cgroup2") :
+ tail[0].equals("cgroup") && tail.length >= 3 &&
+ ("," + tail[2] + ",").contains(",memory,");
+ if (match) {
+ mounts.add(new String[]{unescape(fields[3]), unescape(fields[4])});
+ }
+ }
+ return mounts;
+ }
+
+ // Whether a mount whose root is mountRoot shows the cgroup at membership
+ static boolean contains(String mountRoot, String membership) {
+ String path = membership.startsWith("/") ? membership : "/" + membership;
+ String root = mountRoot.endsWith("/") && mountRoot.length() > 1 ?
+ mountRoot.substring(0, mountRoot.length() - 1) : mountRoot;
+ return root.equals("/") || path.equals(root) || path.startsWith(root + "/");
+ }
+
+ /**
+ * The membership path relative to a mount whose root is mountRoot: /kubepods/pod1/ctr
+ * under root /kubepods is pod1/ctr. A path outside the mount root is kept as is.
+ */
+ static String rebase(String membership, String mountRoot) {
+ String path = membership.startsWith("/") ? membership : "/" + membership;
+ String root = mountRoot.endsWith("/") && mountRoot.length() > 1 ?
+ mountRoot.substring(0, mountRoot.length() - 1) : mountRoot;
+ if (!root.equals("/")) {
+ if (path.equals(root)) {
+ return "";
+ }
+ if (path.startsWith(root + "/")) {
+ path = path.substring(root.length());
+ }
+ }
+ return path.substring(1);
+ }
+
+ // mountinfo writes space, tab, newline and backslash as octal escapes such as \040
+ private static String unescape(String field) {
+ StringBuilder out = new StringBuilder(field.length());
+ for (int i = 0; i < field.length(); i++) {
+ char c = field.charAt(i);
+ if (c == '\\' && i + 4 <= field.length() &&
+ field.substring(i + 1, i + 4).matches("[0-7]{3}")) {
+ out.append((char) Integer.parseInt(field.substring(i + 1, i + 4), 8));
+ i += 3;
+ } else {
+ out.append(c);
+ }
+ }
+ return out.toString();
+ }
+
+ private static long cgroupV2Limit(Path root, String relative) {
+ Path dir = root.resolve(relative).normalize();
+ if (!dir.startsWith(root) || !Files.isDirectory(dir)) {
+ // The process path is not visible from this mount (a cgroup namespace)
+ dir = root;
+ }
+ long limit = -1L;
+ for (Path d = dir; d != null && d.startsWith(root); d = d.getParent()) {
+ limit = lowest(limit, readLimit(d.resolve("memory.max")));
+ }
+ return limit;
+ }
+
+ private static long cgroupV1Limit(Path base, String relative) {
+ Path dir = base.resolve(relative).normalize();
+ if (!dir.startsWith(base) || !Files.isDirectory(dir)) {
+ dir = base;
+ }
+ long limit = readLimit(dir.resolve("memory.limit_in_bytes"));
+ try {
+ for (String line : Files.readAllLines(dir.resolve("memory.stat"),
+ StandardCharsets.UTF_8)) {
+ if (line.startsWith("hierarchical_memory_limit ")) {
+ limit = lowest(limit, parseLimit(line.substring(line.indexOf(' ') + 1)));
+ }
+ }
+ } catch (IOException | RuntimeException e) {
+ // memory.stat is optional here, memory.limit_in_bytes already counted
+ }
+ return limit;
+ }
+
+ private static long readLimit(Path file) {
+ try {
+ return parseLimit(new String(Files.readAllBytes(file), StandardCharsets.UTF_8));
+ } catch (IOException | RuntimeException e) {
+ return -1L;
+ }
+ }
+
+ // "max" (v2) and the near-Long.MAX_VALUE page-aligned v1 default mean no limit
+ private static long parseLimit(String value) {
+ String trimmed = value.trim();
+ if (trimmed.isEmpty() || trimmed.equals("max")) {
+ return -1L;
+ }
+ try {
+ long limit = Long.parseLong(trimmed);
+ return limit <= 0 || limit >= Long.MAX_VALUE / 2 ? -1L : limit;
+ } catch (NumberFormatException e) {
+ return -1L;
+ }
+ }
+
+ private static long lowest(long a, long b) {
+ if (a <= 0) {
+ return b;
+ }
+ return b <= 0 ? a : Math.min(a, b);
+ }
+
+ private static long maxDirectMemory(long heap) {
+ try {
+ HotSpotDiagnosticMXBean hotspot = ManagementFactory.getPlatformMXBean(
+ HotSpotDiagnosticMXBean.class);
+ return directMemory(Long.parseLong(
+ hotspot.getVMOption("MaxDirectMemorySize").getValue()), heap);
+ } catch (RuntimeException | LinkageError e) {
+ log.warn("Failed to read MaxDirectMemorySize, assuming it is unset", e);
+ return directMemory(0L, heap);
+ }
+ }
+
+ /**
+ * The direct memory the budget leaves room for: an explicit -XX:MaxDirectMemorySize
+ * as given, otherwise (0, unset) the heap size capped at UNSET_DIRECT.
+ */
+ static long directMemory(long configured, long heap) {
+ return configured > 0 ? configured : Math.min(heap, UNSET_DIRECT);
Review Comment:
⚠️ Important: When MaxDirectMemorySize is unset, the JVM permits NIO direct-buffer memory up to Runtime.maxMemory(), but this calculation reserves at most 1 GiB. The default Store launcher does not set a matching direct-memory cap, and the shipped gRPC configuration permits 1000 MB inbound requests, so concurrent direct-buffer use can exceed the amount reserved while the auto-sized RocksDB caches remain available. Please enforce the assumed cap in the default launch configuration or budget against the actual JVM allowance; otherwise direct-buffer-heavy workloads can still exceed the cgroup limit. Evidence: [OpenJDK VM.java](https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/jdk/internal/misc/VM.java) and hugegraph-store/hg-store-node/src/main/resources/application.yml.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#issuecomment-6247642753
@imbajin thanks, both done in 9eaaf577b. The fixture now puts the hidden limit truly above the subtree: 2 GiB on `/outer`, the mount rooted at `/outer/kubepods` with `max`, and 4 GiB on the pod. Subtree-only returns 4 GiB, flagged; with the full view in either order it returns 2 GiB, unflagged. A `/` mount now counts as a full view only when the process's rebased path is actually a directory in it, so an unusable full-root view next to a readable subtree keeps the warning, and a new case covers that. Store core tests: 129 passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#issuecomment-6241538691
@imbajin all checks are green on b9df448a7, your latest master merge, and every review point is resolved. bitflicker64 has approved. The paired doc PR, apache/hugegraph-doc#535, is approved by bitflicker64 and merges cleanly, so it can go in right after this one. Could you take a final look and approve when you have a moment? Thanks for all the careful review rounds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "byteayan (via GitHub)" <gi...@apache.org>.
byteayan commented on PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#issuecomment-6437202450
@imbajin thanks, all seven are in 826302f36. The paired doc PR is updated in apache/hugegraph-doc#535 (13737599).
1. The auto budget is unchanged: heap, direct memory, the cgroup/JVM limit, the reserve and the shared Raft log `WriteBufferManager` all still apply.
2. cgroup detection now lives in `CgroupMemoryLimit`. It returns the lowest visible limit, the cgroup version it came from, and whether the view is incomplete. `RocksdbMemoryBudget` only parses the setting and computes the budget. Nothing guesses at hidden limits: an incomplete view just logs a warning.
3. The shipped config is `total_memory_size: auto`. A missing or blank value, or `0`, still means auto, and positive integers are bytes. `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE=auto` passes the entrypoint, and there's no unit parsing.
4. When `SPRING_APPLICATION_JSON` or `-D` sets the budget, `PdConfigureListener` drops only that key from the PD properties before adding the PD source. The synthetic override source is gone.
5. `disruptorBufferSize()` and `writeBufferSize()` are back in `AppConfig`, with `AppConfigTuningTest`.
6. `RaftRocksdbOptions.RAFT_LOG_CACHE_SIZE` builds the Raft log cache, and the budget reuses it as `RAFT_LOG_RESERVE`, plus a separate 512 MiB `NATIVE_RESERVE`.
7. Startup logs one line, e.g. `RocksDB memory budget: 512 MiB (536870912 bytes), mode=auto, physical-limit=4 GiB, source=cgroup-v2, heap=1 GiB, direct=512 MiB, reserve=1.5 GiB, cgroup-view=complete`.
Store core suite: 162 tests pass, and the entrypoint test has 11 cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] fix(store): size the RocksDB memory budget from available memory [hugegraph]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on code in PR #3355:
URL: https://github.com/apache/hugegraph/pull/3355#discussion_r4390399428
##########
hugegraph-store/docs/deployment-guide.md:
##########
@@ -356,14 +356,82 @@ rocksdb:
level0_file_num_compaction_trigger: 4
max_background_jobs: 8 # Background compaction/flush threads
- # Block Cache
- block_cache_size: 16000000000 # Block cache (16GB)
+ # Budget split: this share goes to memtables, the rest to the block cache
+ write_buffer_ratio: 0.66
# SST File Size
target_file_size_base: 134217728 # Target SST size (128MB)
max_bytes_for_level_base: 1073741824 # L1 size (1GB)
```
+`total_memory_size` bounds the memtables (`write_buffer_ratio` of it) and the block cache
+(the rest). Both live outside the Java heap, so `-Xmx` does not limit them. A fixed 1 GiB
+cache for the Raft log comes on top of this budget; the Raft log memtables of all partitions
+share a 256 MiB write buffer charged to that cache, so they don't add to it.
+
+With `total_memory_size: auto` (the shipped value; a missing, blank or `0` value means the same)
+the Store sizes the budget at startup as half of what is left after the heap, the maximum direct
+memory and a 1.5 GiB reserve (the 1 GiB Raft log cache plus 512 MiB for other native memory),
+clamped to 256 MiB–32 GB. Inside a container the starting point is the cgroup memory limit, including a
+limit set only on a parent cgroup when the Store can see that cgroup; inside a private cgroup
+namespace, or when only a subtree of the cgroup2 hierarchy is mounted, a limit above it is not
+visible, so pin the value there if only the pod or slice is limited. The Store uses the lowest
+limit it can see and logs a warning when it can't see the whole hierarchy; it doesn't guess at
+limits it can't read. A Store with a 4 GiB limit, `-Xmx1g` and `-XX:MaxDirectMemorySize=512m`
+gets 512 MiB. Without
+`-XX:MaxDirectMemorySize` the JVM allows as much direct memory as heap, and the budget counts
+it that way. `bin/start-hugegraph-store.sh` sets the flag to the heap it picks; when `-j` sets
+`-Xmx`, `-XX:MaxHeapSize` or `-XX:MaxDirectMemorySize` itself, the launcher leaves the flag out and the JVM's
+default (that heap) or the given cap applies. Either way the budget counts a limit the JVM
+enforces. In small containers set a
+cap below the heap, or the budget falls to the 256 MiB floor.
+
+On a host without a container limit the starting point is physical memory. The default
+`bin/start-hugegraph-store.sh` caps the heap at 2 GiB and direct memory at the same 2 GiB, so a
+16 GiB host gets about 5.25 GiB. When you raise the heap, pass a direct-memory cap with it,
+for example `-j "-Xmx36g -XX:MaxDirectMemorySize=2g"`; otherwise direct memory counts as a
+second heap. With a 2 GiB cap a 16 GiB host with `-Xmx8g` gets 2.25 GiB and a 64 GiB host with
+`-Xmx36g` gets 12.25 GiB. Before this default every host got a fixed `32000000000`, so pin the
+value after upgrading if a large node relied on it.
+
+At startup the Store logs the budget and what it was computed from, for example:
+
+```text
+RocksDB memory budget: 512 MiB (536870912 bytes), mode=auto, physical-limit=4 GiB, source=cgroup-v2, heap=1 GiB, direct=512 MiB, reserve=1.5 GiB, cgroup-view=complete
+```
+
+It warns when the floor applies, or when an explicit value is under 256 MiB or exceeds the
+memory left after the heap, direct memory and the reserve. Set a number of bytes, at least
+1048576 (1 MiB), to pin the budget; a smaller value or a unit suffix such as `4GB` is rejected. Before this default existed the
+shipped value was `32000000000`, which can exceed a container limit and get the Store
+OOM-killed. Both caps are soft: RocksDB can briefly go over them under heavy writes, and a
+small budget with many partitions means frequent memtable flushes.
+
+**Upgrading an existing deployment.** Auto-sizing applies only when the loaded configuration
+has `total_memory_size: auto` (or `0`, or no value). A Store that mounts its own `conf/application-pd.yml`,
+or a tarball install that kept its `conf/` from an earlier release, still has `32000000000` and
+keeps that fixed budget; the Store logs a warning naming the old value at startup. Set the key
+to `auto`, or set `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` in Docker.
+
+PD can also hold the old value. When PD has no shared copy yet, the first Store to start uploads
+its local `application-pd.yml` to PD (KV key `S:FS`). At startup a Store loads that copy over its
+local file only when it has no local file or the file is no newer than PD's `S:Timestamp`. The
+upload sets no timestamp and a missing one counts as 0, so changing `S:FS` alone changes nothing.
+The simplest upgrade is to set `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE=auto` (or
+`-Drocksdb.total_memory_size=auto`) on every Store and restart it; the Store keeps this value
+above both its local file and PD's copy. To change the shared copy instead, set
+`rocksdb.total_memory_size` to `0` or remove it in `S:FS`, then put the current time in
+milliseconds into `S:Timestamp` through PD's KV service (gRPC `KvService.put`). Use `auto` there
+only once every Store runs this release: an earlier release can't parse `auto` and fails to start,
+while it reads `0` as its old heap-sized default. This also applies to a copy that a Store of this
+release uploaded when PD had none, since the shipped file says `auto`. Every running Store watches
+`S:Timestamp`: it reloads PD's copy, writes it over its local file and restarts itself with
+`bin/restart-hugegraph-store.sh` within 60 seconds. A stopped Store loads the copy at its next
+start. That self-restart passes no arguments, so `-j` and `-g` from the original start are
+dropped, including the `JAVA_OPTS` the Docker entrypoint passes as `-j`, and the launcher's
Review Comment:
Important: in Docker this self-restart never reaches the start step: stopping Java ends `docker-entrypoint.sh` and `dumb-init` (PID 1), so the container exits with 143 and stays down unless a restart policy brings it back through the entrypoint with `JAVA_OPTS`. Please say a Docker Store stops on `S:Timestamp` and point Docker users to `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org