You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "keith-turner (via GitHub)" <gi...@apache.org> on 2026/03/25 00:35:27 UTC
[PR] Removes running cache from coordinator [accumulo]
keith-turner opened a new pull request, #6244:
URL: https://github.com/apache/accumulo/pull/6244
This contains uncommitted changes from many other PRs related to the running cache, so opening as a draft for now. This is the last PR in the series that actually removes the running cache now that most of its usages are removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Removes running cache from coordinator [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #6244:
URL: https://github.com/apache/accumulo/pull/6244#discussion_r2990094949
##########
server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java:
##########
@@ -1127,26 +1012,6 @@ public void cleanUpInternalState() {
return;
}
- // Compaction jobs are created in the TabletGroupWatcher and added to the Coordinator
- // via the addJobs method which adds the job to the CompactionJobQueues object.
- final Set<ResourceGroupId> groupsWithJobs = jobQueues.getQueueIds();
-
- final Set<ResourceGroupId> jobGroupsNotInConfiguration =
- Sets.difference(groupsWithJobs, groupsInConfiguration);
-
- if (jobGroupsNotInConfiguration != null && !jobGroupsNotInConfiguration.isEmpty()) {
- RUNNING_CACHE.values().forEach(tec -> {
- if (jobGroupsNotInConfiguration.contains(ResourceGroupId.of(tec.getGroupName()))) {
- LOG.warn(
- "External compaction {} running in group {} on compactor {},"
- + " but group not found in current configuration. Failing compaction...",
- tec.getJob().getExternalCompactionId(), tec.getGroupName(), tec.getCompactor());
Review Comment:
Removed this functionality and not sure it should be carried forward in anyway. If a user has a compactor group A that is running long running compactions. They start a new compactor group B up and configure new long running compactions to go there. They want the compactions running in A to finish and will kill all processes in A when that happens. This code would interfere w/ that intent. Seems like if this done it should be user directed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Removes running cache from coordinator [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner merged PR #6244:
URL: https://github.com/apache/accumulo/pull/6244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Removes running cache from coordinator [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #6244:
URL: https://github.com/apache/accumulo/pull/6244#discussion_r2991616493
##########
server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java:
##########
@@ -1098,23 +996,10 @@ public void cleanUpInternalState() {
ThreadPools.resizePool(reservationPools.get(DataLevel.USER), config,
Property.COMPACTION_COORDINATOR_RESERVATION_THREADS_USER);
- // grab a snapshot of the ids in the set before reading the metadata table. This is done to
- // avoid removing things that are added while reading the metadata.
- final Set<ExternalCompactionId> idsSnapshot = Set.copyOf(RUNNING_CACHE.keySet());
-
// grab the ids that are listed as running in the metadata table. It important that this is done
// after getting the snapshot.
final Set<ExternalCompactionId> idsInMetadata = readExternalCompactionIds();
LOG.trace("Current ECIDs in metadata: {}", idsInMetadata.size());
Review Comment:
No, missed that. Removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Removes running cache from coordinator [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #6244:
URL: https://github.com/apache/accumulo/pull/6244#discussion_r2991517376
##########
server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java:
##########
Review Comment:
Well there is notifying the user about it and then there is taking action on it. Seems useful to still inform about it, which I think the monitor is doing already. I moved some of the other informing logs to the monitor in 8249686. That simplified this method to just the thread pool resize, which is helpful for #6217.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Removes running cache from coordinator [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #6244:
URL: https://github.com/apache/accumulo/pull/6244#discussion_r2991537354
##########
server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java:
##########
Review Comment:
I will do that as a follow on, would be good to remove the RPC and the compactor calls to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Removes running cache from coordinator [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #6244:
URL: https://github.com/apache/accumulo/pull/6244#discussion_r2991614649
##########
server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java:
##########
Review Comment:
Opened #6252
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Removes running cache from coordinator [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on code in PR #6244:
URL: https://github.com/apache/accumulo/pull/6244#discussion_r2991137777
##########
server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java:
##########
@@ -27,7 +27,6 @@
import static org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.OPID;
import static org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.PREV_ROW;
import static org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.SELECTED;
-import static org.apache.accumulo.core.util.threads.ThreadPoolNames.COMPACTOR_RUNNING_COMPACTIONS_POOL;
Review Comment:
Can this thread pool name be removed?
##########
server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java:
##########
Review Comment:
Seems like you could remove this for the same reason as above. The user changed the configuration but has not stopped the running compactors.
##########
server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java:
##########
Review Comment:
Agree with todo below. This can be removed. It was to update the status of the entry in the running cache, and to centralize logging of status updates in the coordinator.
##########
server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java:
##########
@@ -1098,23 +996,10 @@ public void cleanUpInternalState() {
ThreadPools.resizePool(reservationPools.get(DataLevel.USER), config,
Property.COMPACTION_COORDINATOR_RESERVATION_THREADS_USER);
- // grab a snapshot of the ids in the set before reading the metadata table. This is done to
- // avoid removing things that are added while reading the metadata.
- final Set<ExternalCompactionId> idsSnapshot = Set.copyOf(RUNNING_CACHE.keySet());
-
// grab the ids that are listed as running in the metadata table. It important that this is done
// after getting the snapshot.
final Set<ExternalCompactionId> idsInMetadata = readExternalCompactionIds();
LOG.trace("Current ECIDs in metadata: {}", idsInMetadata.size());
Review Comment:
Is this still needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Removes running cache from coordinator [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #6244:
URL: https://github.com/apache/accumulo/pull/6244#discussion_r2991506926
##########
server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java:
##########
@@ -27,7 +27,6 @@
import static org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.OPID;
import static org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.PREV_ROW;
import static org.apache.accumulo.core.metadata.schema.TabletMetadata.ColumnType.SELECTED;
-import static org.apache.accumulo.core.util.threads.ThreadPoolNames.COMPACTOR_RUNNING_COMPACTIONS_POOL;
Review Comment:
Its still used by ExternalCompactionUtil
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org