You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "vyalamar (via GitHub)" <gi...@apache.org> on 2026/05/10 05:01:15 UTC
[PR] HDDS-14242. Use native RocksDB bounds for prefix table iterators [ozone]
vyalamar opened a new pull request, #10227:
URL: https://github.com/apache/ozone/pull/10227
## What changed
This change pushes prefix iterator bounds into RocksDB `ReadOptions` for `RDBTable` prefix iteration.
Prefix scans now create native iterator bounds:
- lower bound: the requested prefix
- upper bound: the next lexicographic prefix
The existing Java-side prefix check is retained as a defensive compatibility layer.
## Why
Ozone metadata prefix scans currently seek to the prefix and then rely on Java-side filtering to stop once keys no longer match. Supplying native RocksDB bounds lets RocksDB constrain the iterator range directly and avoid scanning beyond the requested prefix range.
## Implementation notes
- `ManagedReadOptions` owns optional lower/upper bound slices and closes them with the read options lifecycle.
- `RDBTable` creates bounded read options for prefix iterators and transfers ownership to the iterator.
- Iterator close paths now close the RocksDB iterator before the associated read options and bound slices.
- Prefix upper bounds use the next lexicographic prefix, with the all-`0xFF` case falling back to a lower bound only.
## Testing
- `mvn -pl hadoop-hdds/managed-rocksdb test`
- `mvn -pl hadoop-hdds/managed-rocksdb -DskipTests clean install`
- `mvn -pl hadoop-hdds/framework -Dtest=TestRDBStoreAbstractIterator,TestRDBStoreByteArrayIterator,TestRDBStoreCodecBufferIterator,TestRDBTableStore,TestTypedTable test`
- `git diff --check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14242. Use native RocksDB bounds for prefix table iterators [ozone]
Posted by "vyalamar (via GitHub)" <gi...@apache.org>.
vyalamar commented on PR #10227:
URL: https://github.com/apache/ozone/pull/10227#issuecomment-4540944104
@errose28 kindly help with PR review, its stuck on CI for unrelated things.
@smengcl @adoroszlai please consider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14242. Use native RocksDB bounds for prefix table iterators [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #10227: HDDS-14242. Use native RocksDB bounds for prefix table iterators
URL: https://github.com/apache/ozone/pull/10227
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14242. Use native RocksDB bounds for prefix table iterators [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10227:
URL: https://github.com/apache/ozone/pull/10227#issuecomment-4794764356
Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14242. Use native RocksDB bounds for prefix table iterators [ozone]
Posted by "vyalamar (via GitHub)" <gi...@apache.org>.
vyalamar commented on PR #10227:
URL: https://github.com/apache/ozone/pull/10227#issuecomment-4469606258
Thanks . Let me address the comments .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14242. Use native RocksDB bounds for prefix table iterators [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #10227:
URL: https://github.com/apache/ozone/pull/10227#discussion_r3254117788
##########
hadoop-hdds/managed-rocksdb/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedReadOptions.java:
##########
@@ -27,13 +28,98 @@
*/
public class ManagedReadOptions extends ReadOptions {
private final UncheckedAutoCloseable leakTracker = track(this);
+ private final byte[] lowerBound;
+ private final byte[] upperBound;
+ private final ManagedSlice lowerBoundSlice;
+ private final ManagedSlice upperBoundSlice;
+
+ public ManagedReadOptions() {
+ this(null, null);
+ }
+
+ public ManagedReadOptions(byte[] lowerBound, byte[] upperBound) {
+ this.lowerBound = copy(lowerBound);
+ this.upperBound = copy(upperBound);
+
+ ManagedSlice lowerSlice = null;
+ ManagedSlice upperSlice = null;
+ try {
+ lowerSlice = this.lowerBound != null
+ ? new ManagedSlice(this.lowerBound) : null;
+ upperSlice = this.upperBound != null
+ ? new ManagedSlice(this.upperBound) : null;
+
+ if (lowerSlice != null) {
+ setIterateLowerBound(lowerSlice);
+ }
+ if (upperSlice != null) {
+ setIterateUpperBound(upperSlice);
+ }
+ } catch (RuntimeException | Error e) {
+ closeOnConstructorFailure(lowerSlice, upperSlice, e);
+ throw e;
+ }
+
+ this.lowerBoundSlice = lowerSlice;
+ this.upperBoundSlice = upperSlice;
+ }
+
+ private static byte[] copy(byte[] bytes) {
+ return bytes != null ? Arrays.copyOf(bytes, bytes.length) : null;
+ }
+
+ public byte[] getLowerBound() {
+ return copy(lowerBound);
+ }
+
+ public byte[] getUpperBound() {
+ return copy(upperBound);
+ }
+
+ private void closeOnConstructorFailure(ManagedSlice lowerSlice,
+ ManagedSlice upperSlice, Throwable failure) {
+ try {
+ super.close();
+ } catch (RuntimeException | Error e) {
+ failure.addSuppressed(e);
+ }
+ closeAndSuppress(lowerSlice, failure);
+ closeAndSuppress(upperSlice, failure);
+ try {
+ leakTracker.close();
+ } catch (RuntimeException | Error e) {
+ failure.addSuppressed(e);
+ }
+ }
+
+ private static void closeAndSuppress(ManagedSlice slice, Throwable failure) {
+ if (slice != null) {
+ try {
+ slice.close();
+ } catch (RuntimeException | Error e) {
+ failure.addSuppressed(e);
Review Comment:
We can make it more generic, and reuse for closing `leakTracker`.
```suggestion
private static void closeAndSuppress(AutoCloseable closeable, Throwable failure) {
if (closeable != null) {
try {
closeable.close();
} catch (Throwable t) {
failure.addSuppressed(t);
```
If you agree, please move to `IOUtils`.
##########
hadoop-hdds/managed-rocksdb/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedReadOptions.java:
##########
@@ -27,13 +28,98 @@
*/
public class ManagedReadOptions extends ReadOptions {
private final UncheckedAutoCloseable leakTracker = track(this);
+ private final byte[] lowerBound;
+ private final byte[] upperBound;
+ private final ManagedSlice lowerBoundSlice;
+ private final ManagedSlice upperBoundSlice;
+
+ public ManagedReadOptions() {
+ this(null, null);
+ }
+
+ public ManagedReadOptions(byte[] lowerBound, byte[] upperBound) {
+ this.lowerBound = copy(lowerBound);
+ this.upperBound = copy(upperBound);
+
+ ManagedSlice lowerSlice = null;
+ ManagedSlice upperSlice = null;
+ try {
+ lowerSlice = this.lowerBound != null
+ ? new ManagedSlice(this.lowerBound) : null;
+ upperSlice = this.upperBound != null
+ ? new ManagedSlice(this.upperBound) : null;
+
+ if (lowerSlice != null) {
+ setIterateLowerBound(lowerSlice);
+ }
+ if (upperSlice != null) {
+ setIterateUpperBound(upperSlice);
+ }
+ } catch (RuntimeException | Error e) {
+ closeOnConstructorFailure(lowerSlice, upperSlice, e);
+ throw e;
+ }
+
+ this.lowerBoundSlice = lowerSlice;
+ this.upperBoundSlice = upperSlice;
+ }
+
+ private static byte[] copy(byte[] bytes) {
+ return bytes != null ? Arrays.copyOf(bytes, bytes.length) : null;
+ }
+
+ public byte[] getLowerBound() {
+ return copy(lowerBound);
+ }
+
+ public byte[] getUpperBound() {
+ return copy(upperBound);
+ }
+
+ private void closeOnConstructorFailure(ManagedSlice lowerSlice,
+ ManagedSlice upperSlice, Throwable failure) {
+ try {
+ super.close();
+ } catch (RuntimeException | Error e) {
+ failure.addSuppressed(e);
+ }
+ closeAndSuppress(lowerSlice, failure);
+ closeAndSuppress(upperSlice, failure);
+ try {
+ leakTracker.close();
+ } catch (RuntimeException | Error e) {
+ failure.addSuppressed(e);
+ }
+ }
+
+ private static void closeAndSuppress(ManagedSlice slice, Throwable failure) {
+ if (slice != null) {
+ try {
+ slice.close();
+ } catch (RuntimeException | Error e) {
+ failure.addSuppressed(e);
+ }
+ }
+ }
@Override
public void close() {
try {
super.close();
} finally {
- leakTracker.close();
+ try {
+ if (lowerBoundSlice != null) {
+ lowerBoundSlice.close();
+ }
+ } finally {
+ try {
+ if (upperBoundSlice != null) {
+ upperBoundSlice.close();
+ }
+ } finally {
+ leakTracker.close();
+ }
+ }
Review Comment:
Can be simplified?
```java
IOUtils.closeQuietly(lowerBoundSlice, upperBoundSlice, leakTracker);
```
(needs import)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14242. Use native RocksDB bounds for prefix table iterators [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10227:
URL: https://github.com/apache/ozone/pull/10227#issuecomment-4724821733
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14242. Use native RocksDB bounds for prefix table iterators [ozone]
Posted by "vyalamar (via GitHub)" <gi...@apache.org>.
vyalamar commented on PR #10227:
URL: https://github.com/apache/ozone/pull/10227#issuecomment-4544720231
> Please try to avoid force-push when updating the PR. Here are some great articles that explain why:
>
> https://developers.mattermost.com/blog/submitting-great-prs/#4-avoid-force-pushing https://www.freecodecamp.org/news/optimize-pull-requests-for-reviewer-happiness#request-a-review
Thanks, understood. I will avoid force-pushing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14242. Use native RocksDB bounds for prefix table iterators [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10227:
URL: https://github.com/apache/ozone/pull/10227#issuecomment-4541132400
Please try to avoid force-push when updating the PR. Here are some great articles that explain why:
https://developers.mattermost.com/blog/submitting-great-prs/#4-avoid-force-pushing
https://www.freecodecamp.org/news/optimize-pull-requests-for-reviewer-happiness#request-a-review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14242. Use native RocksDB bounds for prefix table iterators [ozone]
Posted by "vyalamar (via GitHub)" <gi...@apache.org>.
vyalamar commented on PR #10227:
URL: https://github.com/apache/ozone/pull/10227#issuecomment-4416637518
@smengcl please help review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org