You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "keith-turner (via GitHub)" <gi...@apache.org> on 2026/03/23 23:35:01 UTC
[PR] removes CoordinatorSummaryLogger [accumulo]
keith-turner opened a new pull request, #6237:
URL: https://github.com/apache/accumulo/pull/6237
This logger was useful in 2.1. However in 4.0 its redundant with functionality in the monitor and is just extra code to maintain. Removing it also removes a usage of the running cache which may be helpful for #6217
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] removes CoordinatorSummaryLogger [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on PR #6237:
URL: https://github.com/apache/accumulo/pull/6237#issuecomment-4126019765
@keith-turner - I like your approach above to capture the information in the SystemInformation class as it's processing the information for the current compactions. I would suggest making that change in this PR and then creating a follow-on issue (assign to me if you want) to integrate that information into an endpoint for the Monitor display.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] removes CoordinatorSummaryLogger [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #6237:
URL: https://github.com/apache/accumulo/pull/6237#issuecomment-4122418344
It may be easy to compute the per table counts in the monitor. The following is a quick try.
```diff
diff --git a/server/monitor/src/main/java/org/apache/accumulo/monitor/next/SystemInformation.java b/server/monitor/src/main/java/org/apache/accumulo/monitor/next/SystemInformation.java
index 27b22d1700..7594e0d1f9 100644
--- a/server/monitor/src/main/java/org/apache/accumulo/monitor/next/SystemInformation.java
+++ b/server/monitor/src/main/java/org/apache/accumulo/monitor/next/SystemInformation.java
@@ -400,6 +400,9 @@ public class SystemInformation {
protected final Map<String,TimeOrderedRunningCompactionSet> longRunningCompactionsByRg =
new ConcurrentHashMap<>();
+ protected final Map<TableId,AtomicLong> runningCompactionsPerTable = new ConcurrentHashMap<>();
+ protected final Map<String,AtomicLong> runningCompactionsPerGroup = new ConcurrentHashMap<>();
+
// Table Information
private final Map<TableId,TableSummary> tables = new ConcurrentHashMap<>();
private final Map<TableId,List<TabletInformation>> tablets = new ConcurrentHashMap<>();
@@ -538,6 +541,12 @@ public class SystemInformation {
}
public void processExternalCompaction(TExternalCompaction tec) {
+
+ var tableId = KeyExtent.fromThrift(tec.getJob().extent).tableId();
+ runningCompactionsPerTable.computeIfAbsent(tableId, t -> new AtomicLong()).incrementAndGet();
+ runningCompactionsPerGroup.computeIfAbsent(tec.getGroupName(), t -> new AtomicLong())
+ .incrementAndGet();
+
this.longRunningCompactionsByRg.computeIfAbsent(tec.getGroupName(),
k -> new TimeOrderedRunningCompactionSet(rgLongRunningCompactionSize)).add(tec);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] removes CoordinatorSummaryLogger [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #6237:
URL: https://github.com/apache/accumulo/pull/6237#issuecomment-4122014171
> accumulo.compaction.majc.in_progress gives number of running compactions, but I don't think it has a queue.id tag, emitted by the compactor
It does seem to set the queue.id [here](https://github.com/apache/accumulo/blob/a1a05ff6dad6d71f56567826e8f4a47ad6803ef0/server/compactor/src/main/java/org/apache/accumulo/compactor/Compactor.java#L285)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] removes CoordinatorSummaryLogger [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #6237:
URL: https://github.com/apache/accumulo/pull/6237#issuecomment-4121982430
> I don't think there is a metric or tag for table, so there is no way to see number of running compactions per table in the metrics.
The monitor also scans the metadata table, could it compute those counts from the metadata table scan?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] removes CoordinatorSummaryLogger [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #6237:
URL: https://github.com/apache/accumulo/pull/6237#issuecomment-4122017953
@dlmarion do you know if the monitor currently displays this information or if there is an issue open for doing so? Wondering if this PR needs a follow on issue before it can be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] removes CoordinatorSummaryLogger [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #6237:
URL: https://github.com/apache/accumulo/pull/6237#issuecomment-4127573837
@dlmarion I added the changes to system information and opened #6247
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] removes CoordinatorSummaryLogger [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #6237:
URL: https://github.com/apache/accumulo/pull/6237#issuecomment-4122150121
Instead of scanning the metadata table, another way the monitor could potentially compute per table info is if the data it gets from each compactor about its running compaction contains the table id.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] removes CoordinatorSummaryLogger [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on code in PR #6237:
URL: https://github.com/apache/accumulo/pull/6237#discussion_r2989373026
##########
server/monitor/src/main/java/org/apache/accumulo/monitor/next/SystemInformation.java:
##########
@@ -398,6 +399,9 @@ public Stream<TExternalCompaction> stream() {
protected final Map<String,TimeOrderedRunningCompactionSet> longRunningCompactionsByRg =
new ConcurrentHashMap<>();
+ protected final Map<TableId,LongAdder> runningCompactionsPerTable = new ConcurrentHashMap<>();
+ protected final Map<String,LongAdder> runningCompactionsPerGroup = new ConcurrentHashMap<>();
+
Review Comment:
Should include these in the `clear()` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] removes CoordinatorSummaryLogger [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner merged PR #6237:
URL: https://github.com/apache/accumulo/pull/6237
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] removes CoordinatorSummaryLogger [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on PR #6237:
URL: https://github.com/apache/accumulo/pull/6237#issuecomment-4117512513
The CoordinatorSummaryLogger prints number of queued and running compactions per queue and number of running compactions per table. The related queue metrics are:
* `accumulo.compaction.queue.jobs.queued` for number of queued compactions, has a `queue.id` tag for each queue, emitted by the coordinator
* `accumulo.compaction.majc.in_progress` gives number of running compactions, but I don't think it has a `queue.id` tag, emitted by the compactor
* I don't think there is a metric or tag for table, so there is no way to see number of running compactions per table in the metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org