You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "lizhimins (via GitHub)" <gi...@apache.org> on 2026/05/18 12:32:34 UTC
[PR] feat(rocksdb): support native CqCompactionFilter with cross-platform JNI shim [rocketmq]
lizhimins opened a new pull request, #10335:
URL: https://github.com/apache/rocketmq/pull/10335
## Summary
- Add native C++ `CqCompactionFilter` JNI shim for ConsumeQueue compaction, supporting Linux, macOS (x86_64/aarch64), and Windows platforms
- Replace the Java-based `ConsumeQueueCompactionFilterFactory` with a native compaction filter that runs entirely in RocksDB's C++ compaction threads, avoiding JNI callback overhead per key
- Enhance `manualCompaction` logging with `estimateNumKeys` before/after and reduction ratio
- Remove unused `minPhyOffset` parameter from `AbstractRocksDBStorage.manualCompaction`
## Motivation
The existing Java `CompactionFilterFactory` creates a `RemoveConsumeQueueCompactionFilter` per compaction job, which incurs JNI cross-boundary calls for every key during compaction. For ConsumeQueue databases with millions of entries, this overhead is significant.
A native C++ compaction filter:
1. Runs entirely in RocksDB's native compaction threads — zero JNI overhead per key
2. Uses atomic `minPhyOffset` threshold updated from Java via a single JNI call
3. Supports all major platforms (Linux, macOS, Windows)
## Changes
1. **`CqCompactionFilterJni.java`** — JNI loader with platform detection, RocksDB native library co-location for `DT_NEEDED`/`LC_LOAD_DYLIB` resolution
2. **`NativeCqCompactionFilter.java`** — Thin wrapper with `disOwnNativeHandle()` to prevent use-after-free when options close before DB
3. **`cq_compaction_filter.cpp`** — Native C++ filter implementation with `std::atomic<int64_t>` threshold and JNI exports
4. **Pre-built binaries** — `libcq_compaction_filter.so` (Linux), `.dylib` (macOS)
5. **`AbstractRocksDBStorage`** — Enhanced compaction logging, removed unused `minPhyOffset` param
6. **`ConsumeQueueRocksDBStorage`** — Integrates native filter on startup with graceful fallback
7. **`docs/en/Native_RocksDB.md`** — Build instructions for all platforms
## Test Plan
- [x] `CqCompactionFilterJniTest` — verifies native library loading, filter creation, minPhyOffset update, and actual compaction filtering
- [x] Existing `ConsumeQueueRocksDBStorage` integration tests pass
- [ ] Manual verification on macOS and Windows
Closes #10334
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10334] Support native CqCompactionFilter with cross-platform JNI shim [rocketmq]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #10335:
URL: https://github.com/apache/rocketmq/pull/10335#issuecomment-4478221490
## [Codecov](https://app.codecov.io/gh/apache/rocketmq/pull/10335?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `58.62069%` with `48 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 48.85%. Comparing base ([`c6fc39a`](https://app.codecov.io/gh/apache/rocketmq/commit/c6fc39ab5f1661cab5e2d6ff0c215c0add9c6d1d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`4e18a29`](https://app.codecov.io/gh/apache/rocketmq/commit/4e18a2964c68dee50d3ab71a3ae3022ca4530090?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 4 commits behind head on develop.
| [Files with missing lines](https://app.codecov.io/gh/apache/rocketmq/pull/10335?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../rocketmq/store/rocksdb/CqCompactionFilterJni.java](https://app.codecov.io/gh/apache/rocketmq/pull/10335?src=pr&el=tree&filepath=store%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fstore%2Frocksdb%2FCqCompactionFilterJni.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL3JvY2tzZGIvQ3FDb21wYWN0aW9uRmlsdGVySm5pLmphdmE=) | 59.45% | [23 Missing and 7 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/10335?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rocketmq/common/config/AbstractRocksDBStorage.java](https://app.codecov.io/gh/apache/rocketmq/pull/10335?src=pr&el=tree&filepath=common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fcommon%2Fconfig%2FAbstractRocksDBStorage.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vY29uZmlnL0Fic3RyYWN0Um9ja3NEQlN0b3JhZ2UuamF2YQ==) | 0.00% | [12 Missing :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/10335?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...etmq/store/rocksdb/ConsumeQueueRocksDBStorage.java](https://app.codecov.io/gh/apache/rocketmq/pull/10335?src=pr&el=tree&filepath=store%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fstore%2Frocksdb%2FConsumeQueueRocksDBStorage.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL3JvY2tzZGIvQ29uc3VtZVF1ZXVlUm9ja3NEQlN0b3JhZ2UuamF2YQ==) | 77.77% | [3 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/10335?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## develop #10335 +/- ##
=============================================
- Coverage 48.98% 48.85% -0.13%
+ Complexity 13470 13446 -24
=============================================
Files 1375 1376 +1
Lines 100450 100547 +97
Branches 12973 12982 +9
=============================================
- Hits 49202 49127 -75
- Misses 45256 45401 +145
- Partials 5992 6019 +27
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/rocketmq/pull/10335?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10334] Support native CqCompactionFilter with cross-platform JNI shim [rocketmq]
Posted by "lollipopjin (via GitHub)" <gi...@apache.org>.
lollipopjin merged PR #10335:
URL: https://github.com/apache/rocketmq/pull/10335
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10334] Support native CqCompactionFilter with cross-platform JNI shim [rocketmq]
Posted by "leizhiyuan (via GitHub)" <gi...@apache.org>.
leizhiyuan commented on PR #10335:
URL: https://github.com/apache/rocketmq/pull/10335#issuecomment-4485489427
意见 1: loadNativeShim 未捕获 UnsatisfiedLinkError + 缺少 ARM 架构 shim
问题： loadNativeShim() 中 System.load() 调用可能抛出 UnsatisfiedLinkError（例如在 Linux aarch64 上加载 x86-64 的 .so），但当前只捕获了 IOException。由于该方法在 static {} 块中调用，未捕获的 UnsatisfiedLinkError
会导致 ExceptionInInitializerError，使 CqCompactionFilterJni 类在整个 JVM 生命周期内永久不可用，后续任何对该类的引用都会抛出 NoClassDefFoundError。
当前打包的原生库只有：
- .so — 仅 x86-64
- .dylib — 仅 arm64
- .dll — 仅 x86-64
这意味着在 Linux aarch64（云环境中非常常见）加载必然失败，且由于缺少 UnsatisfiedLinkError 的捕获，不会优雅降级，而是直接崩溃。
建议修复：
1. 在 loadNativeShim() 中增加对 UnsatisfiedLinkError 的捕获，确保加载失败时优雅降级（loaded 保持 false，系统正常运行但无 compaction filter）：
```java
System.load(tempLib.getAbsolutePath());
loaded = true;
log.info("[CqCompactionFilterJni] Native library loaded from classpath: {}", tempLib.getAbsolutePath());
} catch (IOException e) {
log.error("[CqCompactionFilterJni] Failed to load native shim", e);
} catch (UnsatisfiedLinkError e) {
log.error("[CqCompactionFilterJni] Failed to load native shim library, "
+ "platform may not be supported: {}", e.getMessage());
}
```
2. 补充 Linux aarch64 的 .so 文件，并在 Java 侧根据 os.arch 选择对应的库文件名。
────────────────────────────────────────────────────────────────────────────────
### English
Issue: System.load() in loadNativeShim() can throw UnsatisfiedLinkError (e.g., when loading an x86-64 .so on Linux aarch64), but only IOException is caught. Since this method is called from a static {} initializer,
an uncaught UnsatisfiedLinkError will propagate as ExceptionInInitializerError, making the CqCompactionFilterJni class permanently unusable for the entire JVM lifetime — any subsequent reference to the class will
throw NoClassDefFoundError.
The bundled native libraries currently only cover:
- .so — x86-64 only
- .dylib — arm64 only
- .dll — x86-64 only
This means on Linux aarch64 (very common in cloud environments) and macOS x86_64, loading will inevitably fail, and without catching UnsatisfiedLinkError, the system won't degrade gracefully — it will crash.
Suggested fix:
1. Catch UnsatisfiedLinkError in loadNativeShim() to ensure graceful degradation when loading fails (loaded stays false, system runs normally without the compaction filter):
```java
System.load(tempLib.getAbsolutePath());
loaded = true;
log.info("[CqCompactionFilterJni] Native library loaded from classpath: {}", tempLib.getAbsolutePath());
} catch (IOException e) {
log.error("[CqCompactionFilterJni] Failed to load native shim", e);
} catch (UnsatisfiedLinkError e) {
log.error("[CqCompactionFilterJni] Failed to load native shim library, "
+ "platform may not be supported: {}", e.getMessage());
}
```
2. Bundle a Linux aarch64 .so ， and select the correct library filename based on os.arch in the Java code.
────────────────────────────────────────────────────────────────────────────────
问题： setMinPhyOffset() 方法在每次调用时以 INFO 级别记录日志。该方法由 cleanExpired() 周期性触发（通常每隔几秒到几十秒一次），在生产环境中会产生大量重复日志，增加日志存储压力且淹没真正有价值的信息。
建议修复： 将日志级别改为 DEBUG：
```java
public static void setMinPhyOffset(long minPhyOffset) {
long ptr = NATIVE_FILTER_PTR.get();
if (ptr != 0) {
setMinPhyOffset0(ptr, minPhyOffset);
log.debug("CqCompactionFilter setMinPhyOffset={}", minPhyOffset);
}
}
```
────────────────────────────────────────────────────────────────────────────────
Issue: setMinPhyOffset() logs at INFO level on every invocation. This method is called periodically by cleanExpired() (typically every few seconds), which produces high-volume repetitive log output in production,
increasing log storage costs and drowning out genuinely useful information.
Suggested fix: Change the log level to DEBUG:
```java
public static void setMinPhyOffset(long minPhyOffset) {
long ptr = NATIVE_FILTER_PTR.get();
if (ptr != 0) {
setMinPhyOffset0(ptr, minPhyOffset);
log.debug("CqCompactionFilter setMinPhyOffset={}", minPhyOffset);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10334] Support native CqCompactionFilter with cross-platform JNI shim [rocketmq]
Posted by "lizhimins (via GitHub)" <gi...@apache.org>.
lizhimins commented on PR #10335:
URL: https://github.com/apache/rocketmq/pull/10335#issuecomment-4485178495
@lizhanhui @leizhiyuan Please help review this pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10334] Support native CqCompactionFilter with cross-platform JNI shim [rocketmq]
Posted by "lizhimins (via GitHub)" <gi...@apache.org>.
lizhimins commented on PR #10335:
URL: https://github.com/apache/rocketmq/pull/10335#issuecomment-4485996387
Added support for the ARM platform
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org