You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "peterxcli (via GitHub)" <gi...@apache.org> on 2025/03/23 16:17:30 UTC
[PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
peterxcli opened a new pull request, #8141:
URL: https://github.com/apache/ozone/pull/8141
## What changes were proposed in this pull request?
Snapshot Cache should compact the rocksdb for all tables excepting for `FileTable`/`DirectoryTable`/`KeyTable`
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-12503
## How was this patch tested?
CI:
https://github.com/peterxcli/ozone/actions/runs/14020195884
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #8141:
URL: https://github.com/apache/ozone/pull/8141#issuecomment-2924121200
Thanks @Tejaskriya, @jojochuang, @smengcl and @smengcl for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli merged PR #8141:
URL: https://github.com/apache/ozone/pull/8141
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #8141:
URL: https://github.com/apache/ozone/pull/8141#issuecomment-2822669862
The PR looks really good to me. I wonder if compacting to the bottom most is required, because these are column families in the snapshots, so supposedly no change. Compacting to the bottom most for the first time a snapshot get evicted out of cache may make sense, but subsequent compactions for the same snapshot wouldn't make a difference.
Or maybe we should compact column families not tracked in the DAG when a snapshot is created? Do it in the background asynchronously so snapshot create request returns immediately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #8141:
URL: https://github.com/apache/ozone/pull/8141#issuecomment-2844249152
Thanks for the patch @peterxcli .
May I ask what is the motivation behind this? Is this required for correctness, or just an optimization?
cc @hemantk-12 @swamirishi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8141:
URL: https://github.com/apache/ozone/pull/8141#discussion_r2093793989
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -220,6 +246,8 @@ void cleanup() {
LOG.debug("Closing SnapshotId {}. It is not being referenced anymore.", k);
// Close the instance, which also closes its DB handle.
try {
+ compactSnapshotDB(v.get());
Review Comment:
@jojochuang This can block other read operations. Let us perform the snapshot compaction while we still allow reads to go through. Compactions shouldn't block active snapshot operations both background services and active user reads. @peterxcli please write a test case for this as well so that we don't inadvertently block any snapshot operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8141:
URL: https://github.com/apache/ozone/pull/8141#discussion_r2070219760
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java:
##########
@@ -222,6 +222,22 @@ public void compactDB() throws IOException {
}
}
+ @Override
+ public void compactTable(String tableName) throws IOException {
+ try (ManagedCompactRangeOptions options = new ManagedCompactRangeOptions()) {
+ compactTable(tableName, options);
Review Comment:
This is not synchronous. Even if we call this function from snapshot cache to compact db, it wouldn't be helpful since the compaction would be cancelled. We would have to wait for all the compactions to finish before we close it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8141:
URL: https://github.com/apache/ozone/pull/8141#discussion_r2093793989
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -220,6 +246,8 @@ void cleanup() {
LOG.debug("Closing SnapshotId {}. It is not being referenced anymore.", k);
// Close the instance, which also closes its DB handle.
try {
+ compactSnapshotDB(v.get());
Review Comment:
@jojochuang @smengcl This can block other read operations. Let us perform the snapshot compaction while we still allow reads to go through. Compactions shouldn't block active snapshot operations both background services and active user reads. @peterxcli please write a test case for this as well so that we don't inadvertently block any snapshot operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #8141:
URL: https://github.com/apache/ozone/pull/8141#discussion_r2104392494
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java:
##########
@@ -222,6 +222,22 @@ public void compactDB() throws IOException {
}
}
+ @Override
+ public void compactTable(String tableName) throws IOException {
+ try (ManagedCompactRangeOptions options = new ManagedCompactRangeOptions()) {
+ compactTable(tableName, options);
+ }
+ }
+
+ @Override
+ public void compactTable(String tableName, ManagedCompactRangeOptions options) throws IOException {
+ RocksDatabase.ColumnFamily columnFamily = db.getColumnFamily(tableName);
+ if (columnFamily == null) {
+ throw new IOException("Table not found: " + tableName);
+ }
+ db.compactRange(columnFamily, null, null, options);
Review Comment:
Created one to track: https://issues.apache.org/jira/browse/HDDS-13114
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #8141:
URL: https://github.com/apache/ozone/pull/8141#discussion_r2078657515
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -220,6 +241,8 @@ void cleanup() {
LOG.debug("Closing SnapshotId {}. It is not being referenced anymore.", k);
// Close the instance, which also closes its DB handle.
try {
+ compactSnapshotDB(v.get());
Review Comment:
According to https://github.com/facebook/rocksdb/issues/3335 RocksDatabase.compact() is synchronous so it doesn't need to wait?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #8141:
URL: https://github.com/apache/ozone/pull/8141#issuecomment-2924118753
Verified build on my local.
```
gh pr checkout 8141
git merge --no-commit upstream/master
mvn -DskipRecon -DskipShade -DskipTests clean verify
hadoop-ozone/dev-support/checks/checkstyle.sh
hadoop-ozone/dev-support/checks/pmd.sh
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8141:
URL: https://github.com/apache/ozone/pull/8141#discussion_r2093793254
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -220,6 +246,8 @@ void cleanup() {
LOG.debug("Closing SnapshotId {}. It is not being referenced anymore.", k);
// Close the instance, which also closes its DB handle.
try {
+ compactSnapshotDB(v.get());
Review Comment:
We should perform this outside dbMap.compute since this can block snapshot reads. dbMap.compute is blocking. We can just move this above line number 235 same level as the dbMap.compute()
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java:
##########
@@ -222,6 +222,22 @@ public void compactDB() throws IOException {
}
}
+ @Override
+ public void compactTable(String tableName) throws IOException {
+ try (ManagedCompactRangeOptions options = new ManagedCompactRangeOptions()) {
+ compactTable(tableName, options);
+ }
+ }
+
+ @Override
+ public void compactTable(String tableName, ManagedCompactRangeOptions options) throws IOException {
+ RocksDatabase.ColumnFamily columnFamily = db.getColumnFamily(tableName);
+ if (columnFamily == null) {
+ throw new IOException("Table not found: " + tableName);
+ }
+ db.compactRange(columnFamily, null, null, options);
Review Comment:
We should set options.setMaxCompactionBytes() when we open the snapshot rocksdb this could be useful so that we ensure one sub compaction doesn't take up a lot of memory. Also look into making ManagedCompactRangeOptions.setMaxSubCompactions() configurable so that we don't use a lot of CPU for this operation. It is ok if the compactions take time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #8141:
URL: https://github.com/apache/ozone/pull/8141#discussion_r2069983825
##########
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestRDBStore.java:
##########
@@ -151,6 +151,23 @@ public void compactDB() throws Exception {
}
+ @Test
+ public void compactTable() throws Exception {
+ assertNotNull(rdbStore, "DBStore cannot be null");
+
+ for (int j = 0; j <= 20; j++) {
Review Comment:
I have an interesting observation here.
When I play with the number of insertions. I noticed that `metaSizeBeforeCompact` always fluctuates between 2 to 5.
It looks like once it is about to reach 6, it does some compaction **before** the new insertion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #8141:
URL: https://github.com/apache/ozone/pull/8141#discussion_r2083468821
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java:
##########
@@ -222,6 +222,22 @@ public void compactDB() throws IOException {
}
}
+ @Override
+ public void compactTable(String tableName) throws IOException {
+ try (ManagedCompactRangeOptions options = new ManagedCompactRangeOptions()) {
+ compactTable(tableName, options);
Review Comment:
> Consider ManagedCompactRangeOptions.setExclusiveManualCompaction(true)
>
> https://github.com/facebook/rocksdb/wiki/Manual-Compaction
This seems default to true.
https://github.com/facebook/rocksdb/wiki/Manual-Compaction#compactrange
>CompactRangeOptions::exclusive_manual_compaction When set to true, no other compaction will run when this manual compaction is running. **Default value is true**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #8141:
URL: https://github.com/apache/ozone/pull/8141#discussion_r2105744528
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -220,6 +246,8 @@ void cleanup() {
LOG.debug("Closing SnapshotId {}. It is not being referenced anymore.", k);
// Close the instance, which also closes its DB handle.
try {
+ compactSnapshotDB(v.get());
Review Comment:
Done. please take another look, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #8141:
URL: https://github.com/apache/ozone/pull/8141#issuecomment-2845007799
Also when we discussed about this PR, @swamirishi wanted to have a configuration to turn off this behavior, since we're not sure if it's going to help with performance or adding extra overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #8141:
URL: https://github.com/apache/ozone/pull/8141#discussion_r2070409538
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java:
##########
@@ -222,6 +222,22 @@ public void compactDB() throws IOException {
}
}
+ @Override
+ public void compactTable(String tableName) throws IOException {
+ try (ManagedCompactRangeOptions options = new ManagedCompactRangeOptions()) {
+ compactTable(tableName, options);
Review Comment:
Consider ManagedCompactRangeOptions.setExclusiveManualCompaction(true)
https://github.com/facebook/rocksdb/wiki/Manual-Compaction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #8141:
URL: https://github.com/apache/ozone/pull/8141#issuecomment-2840721591
@smengcl maybe you can help review this one? Specifically can we "compact column families not tracked in the DAG when a snapshot is created? Do it in the background asynchronously so snapshot create request returns immediately."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8141:
URL: https://github.com/apache/ozone/pull/8141#discussion_r2070222596
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -220,6 +241,8 @@ void cleanup() {
LOG.debug("Closing SnapshotId {}. It is not being referenced anymore.", k);
// Close the instance, which also closes its DB handle.
try {
+ compactSnapshotDB(v.get());
Review Comment:
Please add a wait after this function or inside the funciton and wait for all compactions to finish before you close the db.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #8141:
URL: https://github.com/apache/ozone/pull/8141#issuecomment-2750036238
Sure, but maybe in another Jira?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12503. Compact snapshot DB before evicting a snapshot out of cache [ozone]
Posted by "Tejaskriya (via GitHub)" <gi...@apache.org>.
Tejaskriya commented on PR #8141:
URL: https://github.com/apache/ozone/pull/8141#issuecomment-2756761418
@swamirishi what is the motivation behind triggering this compaction for snapshots? If it is performance related, then we might need to force bottommost compaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org