You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hugegraph.apache.org by "LegendPei (via GitHub)" <gi...@apache.org> on 2026/06/04 14:36:16 UTC
[PR] feat(serializer) : make buffer max capacity configurable [hugegraph]
LegendPei opened a new pull request, #3049:
URL: https://github.com/apache/hugegraph/pull/3049
<!--
Thank you very much for contributing to Apache HugeGraph, we are happy that you want to help us improve it!
Here are some tips for you:
1. If this is your first time, please read the [contributing guidelines](https://github.com/apache/hugegraph/blob/master/CONTRIBUTING.md)
2. If a PR fix/close an issue, type the message "close xxx" (xxx is the link of related
issue) in the content, GitHub will auto link it (Required)
3. Name the PR title in "Google Commit Format", start with "feat | fix | perf | refactor | doc | chore",
such like: "feat(core): support the PageRank algorithm" or "fix: wrong break in the compute loop" (module is optional)
skip it if you are unsure about which is the best component.
4. One PR address one issue, better not to mix up multiple issues.
5. Put an `x` in the `[ ]` to mark the item as CHECKED. `[x]` (or click it directly after
published)
-->
## Purpose of the PR
- close #3046
<!--
Please explain more context in this section, clarify why the changes are needed.
e.g:
- If you propose a new API, clarify the use case for a new API.
- If you fix a bug, you can clarify why it is a bug, and should be associated with an issue.
-->
This PR makes the max capacity of one serializer buffer configurable instead of always using the hardcoded default 128MB limit.
The default behavior remains unchanged, while users who need to serialize larger objects can tune the buffer limit through configuration. The new option is still bounded to avoid unbounded memory allocation.
## Main Changes
<!-- Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. These change logs are helpful for better and faster reviews.)
- Add a new core config option: `serializer.buffer_max_capacity`
- Default value: `134217728` bytes, same as the previous hardcoded 128MB limit
- Upper bound: 1GB
- Make `BytesBuffer` use the configured max capacity when:
- creating a buffer
- expanding buffer capacity
- reporting capacity-limit errors
- Make `LZ4Util` respect the configured serializer buffer max capacity during decompression.
- Initialize the serializer buffer max capacity from graph config in `StandardHugeGraph`.
- Add sample config entries in:
- `hugegraph.properties`
- `hstore.properties.template`
- Add focused unit tests for:
- buffer allocation with configured max capacity
- buffer resize limit checks
- invalid max capacity values
- LZ4 decompression limit checks
- loading the max capacity from graph config
## Verifying these changes
<!-- Please pick the proper options below -->
- [ ] Trivial rework / code cleanup without any test coverage. (No Need)
- [ ] Already covered by existing tests, such as *(please modify tests here)*.
- [ ] Need tests and can be verified as follows:
- xxx
## Does this PR potentially affect the following parts?
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] Dependencies ([add/update license](https://hugegraph.apache.org/docs/contribution-guidelines/contribute/#321-check-licenses) info & [regenerate_known_dependencies.sh](../install-dist/scripts/dependency/regenerate_known_dependencies.sh)) <!-- Don't forget to add/update the info in "LICENSE" & "NOTICE" files (both in root & dist module) -->
- [ ] Modify configurations
- [ ] The public API
- [ ] Other affects (typed here)
- [ ] Nope
## Documentation Status
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `Doc - TODO` <!-- Your PR changes impact docs and you will update later -->
- [ ] `Doc - Done` <!-- Related docs have been already added or updated -->
- [ ] `Doc - No Need` <!-- Your PR changes don't impact/need docs -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] feat(serializer) : make buffer max capacity configurable [hugegraph]
Posted by "VGalaxies (via GitHub)" <gi...@apache.org>.
VGalaxies commented on code in PR #3049:
URL: https://github.com/apache/hugegraph/pull/3049#discussion_r3371304518
##########
hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/StandardHugeGraph.java:
##########
@@ -233,6 +234,8 @@ public StandardHugeGraph(HugeConfig config) {
LockUtil.init(this.spaceGraphName());
+ BytesBuffer.initMaxBufferCapacity(
Review Comment:
medium hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/StandardHugeGraph.java:237 - Clean up lock groups when buffer capacity init fails
Evidence: `LockUtil.init(this.spaceGraphName())` runs at line 235, then the new `BytesBuffer.initMaxBufferCapacity(...)` can throw at line 237 on invalid or conflicting `serializer.buffer_max_capacity`. The constructor cleanup begins only inside later `try` blocks, so this exception bypasses `LockUtil.destroy(...)`. `LockManager.create()` throws when a lock group already exists, so a later retry for the same graph can fail before reaching the corrected config.
Impact: A single failed graph open caused by this new option can leave process-wide lock state behind and prevent loading that graph again without restarting the process.
Requested fix: Move the buffer-capacity initialization before `LockUtil.init(...)`, or wrap this initialization path so `LockUtil.destroy(this.spaceGraphName())` runs for any exception after lock initialization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] feat(serializer) : make buffer max capacity configurable [hugegraph]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #3049:
URL: https://github.com/apache/hugegraph/pull/3049#issuecomment-4623203278
## [Codecov](https://app.codecov.io/gh/apache/hugegraph/pull/3049?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 93.25%. Comparing base ([`7f0a44a`](https://app.codecov.io/gh/apache/hugegraph/commit/7f0a44adc0bd8f7445a4ac357633e967955c845c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a08bdd6`](https://app.codecov.io/gh/apache/hugegraph/commit/a08bdd63b54c47d8a0e5680ad3e062face1af97b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #3049 +/- ##
=============================================
+ Coverage 36.08% 93.25% +57.17%
+ Complexity 338 65 -273
=============================================
Files 803 9 -794
Lines 68227 267 -67960
Branches 8963 22 -8941
=============================================
- Hits 24617 249 -24368
+ Misses 40963 8 -40955
+ Partials 2647 10 -2637
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/hugegraph/pull/3049?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] feat(serializer) : make buffer max capacity configurable [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #3049:
URL: https://github.com/apache/hugegraph/pull/3049#discussion_r3407815369
##########
hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/serializer/BytesBuffer.java:
##########
@@ -122,6 +126,46 @@ public static BytesBuffer wrap(byte[] array, int offset, int length) {
return new BytesBuffer(ByteBuffer.wrap(array, offset, length));
}
+ public static int maxBufferCapacity() {
+ Integer capacity = maxBufferCapacity;
+ return capacity != null ? capacity : MAX_BUFFER_CAPACITY;
+ }
+
+ public static synchronized void initMaxBufferCapacity(int capacity) {
+ initMaxBufferCapacity(capacity, true);
+ }
+
+ public static synchronized void initMaxBufferCapacity(int capacity,
+ boolean explicit) {
+ E.checkArgument(capacity >= DEFAULT_CAPACITY &&
+ capacity <= MAX_BUFFER_CAPACITY_UPPER_BOUND,
+ "Max buffer capacity must be in range [%s, %s], " +
+ "but got %s",
+ DEFAULT_CAPACITY, MAX_BUFFER_CAPACITY_UPPER_BOUND,
+ capacity);
+
+ if (!explicit) {
+ if (maxBufferCapacity == null) {
+ maxBufferCapacity = MAX_BUFFER_CAPACITY;
Review Comment:
‼️ **Make the process-wide cap independent of graph load order**
Evidence: the default-path branch now stores `MAX_BUFFER_CAPACITY` when the first graph omits `serializer.buffer_max_capacity`, while an explicit graph opened first still lets later default graphs inherit the custom cap. That means the same set of graph configs can either start or fail depending only on which graph file is opened first.
Impact: a valid explicit process-wide serializer cap can be rejected in multi-graph deployments after any default-config graph initializes first.
Requested fix: select the single process-wide value before opening individual graphs, or let omitted/default graph configs defer initialization until an explicit value is seen; add coverage for both explicit-first/default-second and default-first/explicit-second ordering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] feat(server) : make buffer max capacity configurable [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin merged PR #3049:
URL: https://github.com/apache/hugegraph/pull/3049
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] feat(serializer) : make buffer max capacity configurable [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #3049:
URL: https://github.com/apache/hugegraph/pull/3049#discussion_r3375002901
##########
hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/StandardHugeGraph.java:
##########
@@ -231,15 +232,19 @@ public StandardHugeGraph(HugeConfig config) {
this.readMode = GraphReadMode.OLTP_ONLY;
this.schedulerType = config.get(CoreOptions.SCHEDULER_TYPE);
- LockUtil.init(this.spaceGraphName());
-
+ // Init process-wide static configs before lock, so that validation
+ // failures won't leave stale lock groups in LockManager.
+ BytesBuffer.initMaxBufferCapacity(
Review Comment:
‼️ **Keep the process-wide cap from conflicting with default graph configs**
Evidence: `GraphManager` loads every file under `conf/graphs` in one JVM, and each `StandardHugeGraph` calls `BytesBuffer.initMaxBufferCapacity(config.get(...))`. With this option stored as a graph option, the first graph that sets `serializer.buffer_max_capacity=268435456` freezes the static cap, then any later graph that omits the option still passes the default `134217728` and fails with the new conflicting-value check.
Impact: enabling this option for one graph can prevent a normal multi-graph server from starting unrelated graphs, even though a process-wide setting should be shared by the process.
Requested fix: move the source to server-wide config, or treat an unset/default graph value as inheriting the already-initialized process value, and add a `GraphManager`/`MultiGraphsTest` case covering one custom graph plus one default graph.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] feat(serializer) : make buffer max capacity configurable [hugegraph]
Posted by "VGalaxies (via GitHub)" <gi...@apache.org>.
VGalaxies commented on code in PR #3049:
URL: https://github.com/apache/hugegraph/pull/3049#discussion_r3371304518
##########
hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/StandardHugeGraph.java:
##########
@@ -233,6 +234,8 @@ public StandardHugeGraph(HugeConfig config) {
LockUtil.init(this.spaceGraphName());
+ BytesBuffer.initMaxBufferCapacity(
Review Comment:
**Medium: Clean up lock groups when buffer capacity init fails**
`hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/StandardHugeGraph.java:237`
**Evidence**
- `LockUtil.init(this.spaceGraphName())` runs at line 235, then the new `BytesBuffer.initMaxBufferCapacity(...)` can throw at line 237 on invalid or conflicting `serializer.buffer_max_capacity`. The constructor cleanup begins only inside later `try` blocks, so this exception bypasses `LockUtil.destroy(...)`. `LockManager.create()` throws when a lock group already exists, so a later retry for the same graph can fail before reaching the corrected config.
**Impact**
- A single failed graph open caused by this new option can leave process-wide lock state behind and prevent loading that graph again without restarting the process.
**Requested fix**
- Move the buffer-capacity initialization before `LockUtil.init(...)`, or wrap this initialization path so `LockUtil.destroy(this.spaceGraphName())` runs for any exception after lock initialization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] feat(serializer) : make buffer max capacity configurable [hugegraph]
Posted by "VGalaxies (via GitHub)" <gi...@apache.org>.
VGalaxies commented on code in PR #3049:
URL: https://github.com/apache/hugegraph/pull/3049#discussion_r3400876567
##########
hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/serializer/BytesBuffer.java:
##########
@@ -122,6 +126,40 @@ public static BytesBuffer wrap(byte[] array, int offset, int length) {
return new BytesBuffer(ByteBuffer.wrap(array, offset, length));
}
+ public static int maxBufferCapacity() {
+ Integer capacity = maxBufferCapacity;
+ return capacity != null ? capacity : MAX_BUFFER_CAPACITY;
+ }
+
+ public static synchronized void initMaxBufferCapacity(int capacity) {
+ E.checkArgument(capacity >= DEFAULT_CAPACITY &&
+ capacity <= MAX_BUFFER_CAPACITY_UPPER_BOUND,
+ "Max buffer capacity must be in range [%s, %s], " +
+ "but got %s",
+ DEFAULT_CAPACITY, MAX_BUFFER_CAPACITY_UPPER_BOUND,
+ capacity);
+
+ if (maxBufferCapacity == null) {
Review Comment:
**High: Default config can block later explicit buffer cap**
`hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/serializer/BytesBuffer.java:142`
**Evidence**
- `StandardHugeGraph` initializes `BytesBuffer` for every graph using `config.get(CoreOptions.SERIALIZER_BUFFER_MAX_CAPACITY)`, whose default is `BytesBuffer.MAX_BUFFER_CAPACITY`. `initMaxBufferCapacity()` stores the first value unconditionally when `maxBufferCapacity == null`, and only treats `MAX_BUFFER_CAPACITY` as inherit on later calls. Therefore a default graph opened first fixes the process-wide value at 128 MiB, and a later graph explicitly setting 256 MiB hits the conflict path.
**Impact**
- Multi-graph deployments can succeed or fail based on graph load order; a graph with a valid explicit `serializer.buffer_max_capacity` may fail to load after any graph that omitted the option.
**Requested fix**
- Distinguish omitted/default values from explicit configuration before initializing the process-wide cap, or validate/select the single process-wide value before opening graphs. Add coverage for default-first then custom-second graph initialization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] feat(serializer) : make buffer max capacity configurable [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #3049:
URL: https://github.com/apache/hugegraph/pull/3049#discussion_r3402205068
##########
hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/serializer/BytesBuffer.java:
##########
@@ -122,6 +126,43 @@ public static BytesBuffer wrap(byte[] array, int offset, int length) {
return new BytesBuffer(ByteBuffer.wrap(array, offset, length));
}
+ public static int maxBufferCapacity() {
+ Integer capacity = maxBufferCapacity;
+ return capacity != null ? capacity : MAX_BUFFER_CAPACITY;
+ }
+
+ public static synchronized void initMaxBufferCapacity(int capacity) {
+ initMaxBufferCapacity(capacity, true);
+ }
+
+ public static synchronized void initMaxBufferCapacity(int capacity,
+ boolean explicit) {
+ E.checkArgument(capacity >= DEFAULT_CAPACITY &&
+ capacity <= MAX_BUFFER_CAPACITY_UPPER_BOUND,
+ "Max buffer capacity must be in range [%s, %s], " +
+ "but got %s",
+ DEFAULT_CAPACITY, MAX_BUFFER_CAPACITY_UPPER_BOUND,
+ capacity);
+
+ if (!explicit) {
Review Comment:
‼️ **Freeze the process-wide cap before any graph can use it**
Evidence: `StandardHugeGraph` now passes `explicit=false` when a graph omits `serializer.buffer_max_capacity`, and `BytesBuffer.initMaxBufferCapacity()` returns without setting `maxBufferCapacity`. If that default-config graph is already open, a later graph with an explicit value can still set the static JVM-wide cap, so buffers created after that point for the first graph use the later graph's limit instead of the default it started with.
Impact: the serializer limit for a running graph can change solely because another graph is created later, leaving multi-graph behavior dependent on load order.
Requested fix: select or freeze the single process-wide value before any graph is opened, or reject late explicit initialization once any graph has started using the default process value. Please add a regression test covering a default graph that remains open while a later explicit graph is opened.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] feat(serializer) : make buffer max capacity configurable [hugegraph]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #3049:
URL: https://github.com/apache/hugegraph/pull/3049#discussion_r3360146743
##########
hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/serializer/BytesBuffer.java:
##########
@@ -174,13 +192,15 @@ private void require(int size) {
}
// Extra capacity as buffer
- int newCapacity = size + this.buffer.limit() + DEFAULT_CAPACITY;
- if (newCapacity > MAX_BUFFER_CAPACITY) {
+ long newCapacity = (long) size + this.buffer.limit() +
Review Comment:
⚠️ Non-blocking, but this checks the padded growth target against the configured max, not the actual required size. For example, with `max=128`, a full 64-byte buffer rejects a 1-byte write because `64 + 1 + DEFAULT_CAPACITY == 129`, even though the final content size would only be 65. Consider checking `position + size > maxCapacity` for rejection, then allocating `min(position + size + DEFAULT_CAPACITY, maxCapacity)` when the write itself still fits.
##########
hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/StandardHugeGraph.java:
##########
@@ -233,6 +234,8 @@ public StandardHugeGraph(HugeConfig config) {
LockUtil.init(this.spaceGraphName());
+ BytesBuffer.setMaxBufferCapacity(
Review Comment:
‼️ I think a process-wide buffer limit is the right scope here, because `BytesBuffer` is a static low-level utility and many call sites do not carry graph context. The issue is that this currently behaves like a graph-level config that each `StandardHugeGraph` constructor can overwrite, so the last opened graph silently changes the limit for all graphs in the same JVM. Please make the process-wide semantic explicit: initialize the value once, document `serializer.buffer_max_capacity` as JVM/process-wide, and reject conflicting values when another graph is opened with a different setting. A focused multi-graph test for this conflict/last-opened behavior would be enough; no need to refactor the whole serializer path into per-graph state for this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org