You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "dlmarion (via GitHub)" <gi...@apache.org> on 2025/08/22 11:45:44 UTC
[PR] Add TransactionRunner to set when created, not when first run [accumulo]
dlmarion opened a new pull request, #5817:
URL: https://github.com/apache/accumulo/pull/5817
The TransactionRunner instance was being added to the runningTxRunners when the TransactionRunner.run method is called, not when it's created. FateExecutor.initiateShutdown iterates over runningTxRunners and calls flagStop to tell the TransactionRunners to exit. Before this fix it's possible that a TransactionRunner could be created, but not run by the ThreadPoolExecutor, and therefore miss the stop signal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2293704935
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -412,7 +418,9 @@ public void run() {
}
} finally {
log.trace("A TransactionRunner is exiting...");
- Preconditions.checkState(runningTxRunners.remove(this));
+ synchronized (runningTxRunners) {
+ Preconditions.checkState(runningTxRunners.remove(this));
+ }
Review Comment:
no need to sync here since `this.runningTxRunners = Collections.synchronizedSet(new HashSet<>());`
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -105,15 +105,22 @@ public FateExecutor(Fate<T> fate, T environment, Set<Fate.FateOperation> fateOps
protected void resizeFateExecutor(Map<Set<Fate.FateOperation>,Integer> poolConfigs,
long idleCheckIntervalMillis) {
final var pool = transactionExecutor;
- final var runningTxRunners = getRunningTxRunners();
+ final var runningTxRunnersCopy = getRunningTxRunners();
final int configured = poolConfigs.get(fateOps);
ThreadPools.resizePool(pool, () -> configured, poolName);
- final int needed = configured - runningTxRunners.size();
+ final int needed = configured - runningTxRunnersCopy.size();
if (needed > 0) {
// If the pool grew, then ensure that there is a TransactionRunner for each thread
for (int i = 0; i < needed; i++) {
try {
- pool.execute(new TransactionRunner());
+ final TransactionRunner tr = new TransactionRunner();
+ synchronized (runningTxRunners) {
+ if (pool.isShutdown() || pool.isTerminating()) {
+ return;
+ }
+ runningTxRunners.add(tr);
+ pool.execute(tr);
+ }
Review Comment:
I like this switch to add the runner before calling execute, this is safer.
Some comments:
1) Should we maybe just avoid the use of the copy and just lock the runningTxRunners for the entire if-elseif-else block? Using the copy and the real seems a bit sketchy, but I'm not sure if it's actually wrong/a problem. The copy was sketching me out when I was working on #5813 so I removed it there as well.
2) `pool.isShutdown() || pool.isTerminating()` --> `pool.isShutdown()` might be clearer. I'm pretty sure `pool.isTerminating()` is true only if `pool.isShutdown()` is true
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -105,15 +105,22 @@ public FateExecutor(Fate<T> fate, T environment, Set<Fate.FateOperation> fateOps
protected void resizeFateExecutor(Map<Set<Fate.FateOperation>,Integer> poolConfigs,
long idleCheckIntervalMillis) {
final var pool = transactionExecutor;
- final var runningTxRunners = getRunningTxRunners();
+ final var runningTxRunnersCopy = getRunningTxRunners();
final int configured = poolConfigs.get(fateOps);
ThreadPools.resizePool(pool, () -> configured, poolName);
- final int needed = configured - runningTxRunners.size();
+ final int needed = configured - runningTxRunnersCopy.size();
if (needed > 0) {
// If the pool grew, then ensure that there is a TransactionRunner for each thread
for (int i = 0; i < needed; i++) {
try {
- pool.execute(new TransactionRunner());
+ final TransactionRunner tr = new TransactionRunner();
+ synchronized (runningTxRunners) {
+ if (pool.isShutdown() || pool.isTerminating()) {
+ return;
+ }
+ runningTxRunners.add(tr);
+ pool.execute(tr);
+ }
} catch (RejectedExecutionException e) {
// RejectedExecutionException could be shutting down
if (pool.isShutdown()) {
Review Comment:
I was thinking this case should probably be removed, but it might still be possible if the pool is shutdown after the `if (pool.isShutdown() || pool.isTerminating())` check but before `pool.execute(tr);`... So I think we still need it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2294029178
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -105,15 +105,22 @@ public FateExecutor(Fate<T> fate, T environment, Set<Fate.FateOperation> fateOps
protected void resizeFateExecutor(Map<Set<Fate.FateOperation>,Integer> poolConfigs,
long idleCheckIntervalMillis) {
final var pool = transactionExecutor;
- final var runningTxRunners = getRunningTxRunners();
+ final var runningTxRunnersCopy = getRunningTxRunners();
final int configured = poolConfigs.get(fateOps);
ThreadPools.resizePool(pool, () -> configured, poolName);
- final int needed = configured - runningTxRunners.size();
+ final int needed = configured - runningTxRunnersCopy.size();
if (needed > 0) {
// If the pool grew, then ensure that there is a TransactionRunner for each thread
for (int i = 0; i < needed; i++) {
try {
- pool.execute(new TransactionRunner());
+ final TransactionRunner tr = new TransactionRunner();
+ synchronized (runningTxRunners) {
+ if (pool.isShutdown() || pool.isTerminating()) {
+ return;
+ }
+ runningTxRunners.add(tr);
+ pool.execute(tr);
+ }
Review Comment:
Seems like it may be good to get 5813 merged and then make this change to avoid the copy change.
Checking the pool for shutdown here has a race condition like in the following.
```java
if (pool.isShutdown() || pool.isTerminating()) {
return;
}
runningTxRunners.add(tr);
// pool could be shutdown here and a rejected excecution exception thrown
pool.execute(tr);
```
So maybe could do something like the following.
```java
try {
runningTxRunners.add(tr);
pool.execute(tr);
}catch (RejectedExecutionException e){
runningTxRunners.remove(tr);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2298709857
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -107,30 +106,27 @@ public FateExecutor(Fate<T> fate, T environment, Set<Fate.FateOperation> fateOps
*/
protected void resizeFateExecutor(Map<Set<Fate.FateOperation>,Integer> poolConfigs,
long idleCheckIntervalMillis) {
- final var pool = transactionExecutor;
final int configured = poolConfigs.get(fateOps);
- ThreadPools.resizePool(pool, () -> configured, poolName);
+ ThreadPools.resizePool(transactionExecutor, () -> configured, poolName);
synchronized (runningTxRunners) {
final int running = runningTxRunners.size();
final int needed = configured - running;
log.trace("resizing pools configured:{} running:{} needed:{} fateOps:{}", configured, running,
needed, fateOps);
-
if (needed > 0) {
// If the pool grew, then ensure that there is a TransactionRunner for each thread
for (int i = 0; i < needed; i++) {
+ if (transactionExecutor.isShutdown()) {
+ log.trace("Not adding TransactionRunner, FateExecutor is shutdown.");
+ break;
+ }
+ final TransactionRunner tr = new TransactionRunner();
try {
- pool.execute(new TransactionRunner());
+ runningTxRunners.add(tr);
+ transactionExecutor.execute(tr);
} catch (RejectedExecutionException e) {
- // RejectedExecutionException could be shutting down
- if (pool.isShutdown()) {
- // The exception is expected in this case, no need to spam the logs.
- log.trace("Expected error adding transaction runner to FaTE executor pool. "
- + "The pool is shutdown.", e);
- } else {
Review Comment:
It could still be the case that the pool is shutdown if it is shutdown between the shutdown check and the execute call. So, the `if (transactionExecutor.isShutdown()) {` in the try doesn't really help us
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2298662294
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -107,23 +106,27 @@ public FateExecutor(Fate<T> fate, T environment, Set<Fate.FateOperation> fateOps
*/
protected void resizeFateExecutor(Map<Set<Fate.FateOperation>,Integer> poolConfigs,
long idleCheckIntervalMillis) {
- final var pool = transactionExecutor;
final int configured = poolConfigs.get(fateOps);
- ThreadPools.resizePool(pool, () -> configured, poolName);
+ ThreadPools.resizePool(transactionExecutor, () -> configured, poolName);
synchronized (runningTxRunners) {
final int running = runningTxRunners.size();
final int needed = configured - running;
log.trace("resizing pools configured:{} running:{} needed:{} fateOps:{}", configured, running,
needed, fateOps);
-
if (needed > 0) {
// If the pool grew, then ensure that there is a TransactionRunner for each thread
for (int i = 0; i < needed; i++) {
+ if (transactionExecutor.isShutdown()) {
+ log.trace("Not adding TransactionRunner, FateExecutor is shutdown.");
+ break;
+ }
try {
- pool.execute(new TransactionRunner());
+ final TransactionRunner tr = new TransactionRunner();
+ runningTxRunners.add(tr);
+ transactionExecutor.execute(tr);
} catch (RejectedExecutionException e) {
// RejectedExecutionException could be shutting down
- if (pool.isShutdown()) {
+ if (transactionExecutor.isShutdown()) {
// The exception is expected in this case, no need to spam the logs.
log.trace("Expected error adding transaction runner to FaTE executor pool. "
+ "The pool is shutdown.", e);
Review Comment:
Could probably remove the shutdown check in the loop. If it is shutdown will get a rejected execution exception and then it will log and break.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2298629371
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -107,23 +106,27 @@ public FateExecutor(Fate<T> fate, T environment, Set<Fate.FateOperation> fateOps
*/
protected void resizeFateExecutor(Map<Set<Fate.FateOperation>,Integer> poolConfigs,
long idleCheckIntervalMillis) {
- final var pool = transactionExecutor;
final int configured = poolConfigs.get(fateOps);
- ThreadPools.resizePool(pool, () -> configured, poolName);
+ ThreadPools.resizePool(transactionExecutor, () -> configured, poolName);
synchronized (runningTxRunners) {
final int running = runningTxRunners.size();
final int needed = configured - running;
log.trace("resizing pools configured:{} running:{} needed:{} fateOps:{}", configured, running,
needed, fateOps);
-
if (needed > 0) {
// If the pool grew, then ensure that there is a TransactionRunner for each thread
for (int i = 0; i < needed; i++) {
+ if (transactionExecutor.isShutdown()) {
+ log.trace("Not adding TransactionRunner, FateExecutor is shutdown.");
+ break;
+ }
try {
- pool.execute(new TransactionRunner());
+ final TransactionRunner tr = new TransactionRunner();
+ runningTxRunners.add(tr);
+ transactionExecutor.execute(tr);
} catch (RejectedExecutionException e) {
// RejectedExecutionException could be shutting down
- if (pool.isShutdown()) {
+ if (transactionExecutor.isShutdown()) {
// The exception is expected in this case, no need to spam the logs.
log.trace("Expected error adding transaction runner to FaTE executor pool. "
+ "The pool is shutdown.", e);
Review Comment:
Just realized the current shutdown check could show not shutdown, then it could be shutdown right before we call `execute`, so we could still see `RejectedExecutionException` due to shutdown
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#issuecomment-3215519641
I need to merge in `main` and clean this up a little before re-review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2298294247
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -105,19 +105,26 @@ public FateExecutor(Fate<T> fate, T environment, Set<Fate.FateOperation> fateOps
*/
protected void resizeFateExecutor(Map<Set<Fate.FateOperation>,Integer> poolConfigs,
long idleCheckIntervalMillis) {
- final var pool = transactionExecutor;
final int configured = poolConfigs.get(fateOps);
- ThreadPools.resizePool(pool, () -> configured, poolName);
- synchronized (runningTxRunners) {
- final int needed = configured - runningTxRunners.size();
- if (needed > 0) {
- // If the pool grew, then ensure that there is a TransactionRunner for each thread
+ ThreadPools.resizePool(transactionExecutor, () -> configured, poolName);
+ final int needed = configured - runningTxRunners.size();
+ if (needed > 0) {
+ // If the pool grew, then ensure that there is a TransactionRunner for each thread
+ log.trace("FateExecutor {} needs {} more TransactionRunners", fateOps, needed);
+ synchronized (runningTxRunners) {
Review Comment:
I still think it would be best to keep the entire if-elseif-else locked under runningTxRunners to prevent any potential concurrency issues. This code will lock to get the runningTxRunners.size(), unlock, then lock again under the assumption that runningTxRunners.size() is the same which it might not be
##########
core/src/main/java/org/apache/accumulo/core/fate/user/UserFateStore.java:
##########
@@ -112,6 +112,8 @@ public UserFateStore(ClientContext context, String tableName, ZooUtil.LockID loc
super(lockID, isLockHeld, maxDeferred, fateIdGenerator);
this.context = Objects.requireNonNull(context);
this.tableName = Objects.requireNonNull(tableName);
+ Preconditions.checkArgument(this.context.tableOperations().exists(tableName),
+ "user fate store table does not exist.");
Review Comment:
Could add the table here
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -201,6 +209,12 @@ protected Set<Fate.FateOperation> getFateOps() {
return fateOps;
}
+ public void printInfo() {
+ synchronized (runningTxRunners) {
+ runningTxRunners.forEach(r -> r.printInfo());
+ }
+ }
+
Review Comment:
#5824 does a similar thing, but adds the info to the existing toString. I think the toString additions are a bit cleaner
##########
test/src/main/java/org/apache/accumulo/test/fate/FlakyFate.java:
##########
@@ -39,11 +39,6 @@ public class FlakyFate<T> extends Fate<T> {
public FlakyFate(T environment, FateStore<T> store, Function<Repo<T>,String> toLogStrFunc,
AccumuloConfiguration conf) {
super(environment, store, false, toLogStrFunc, conf, new ScheduledThreadPoolExecutor(2));
- }
-
- @Override
- protected void startFateExecutors(T environment, AccumuloConfiguration conf,
- Set<FateExecutor<T>> fateExecutors) {
for (var poolConfig : getPoolConfigurations(conf).entrySet()) {
fateExecutors.add(
new FlakyFateExecutor<>(this, environment, poolConfig.getKey(), poolConfig.getValue()));
Review Comment:
Is any of this still needed? I think it can be removed if the FatePoolWatcher runs for FlakyFate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2293910839
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -412,7 +418,9 @@ public void run() {
}
} finally {
log.trace("A TransactionRunner is exiting...");
- Preconditions.checkState(runningTxRunners.remove(this));
+ synchronized (runningTxRunners) {
+ Preconditions.checkState(runningTxRunners.remove(this));
+ }
Review Comment:
We are only synchronizing elsewhere when doing compound ops
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2293869415
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -412,7 +418,9 @@ public void run() {
}
} finally {
log.trace("A TransactionRunner is exiting...");
- Preconditions.checkState(runningTxRunners.remove(this));
+ synchronized (runningTxRunners) {
+ Preconditions.checkState(runningTxRunners.remove(this));
+ }
Review Comment:
But we are synchronizing on it elsewhere...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2293908283
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -412,7 +418,9 @@ public void run() {
}
} finally {
log.trace("A TransactionRunner is exiting...");
- Preconditions.checkState(runningTxRunners.remove(this));
+ synchronized (runningTxRunners) {
+ Preconditions.checkState(runningTxRunners.remove(this));
+ }
Review Comment:
understood, just wondering if we should remove the synchronization in the other places too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2298607086
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -107,23 +106,27 @@ public FateExecutor(Fate<T> fate, T environment, Set<Fate.FateOperation> fateOps
*/
protected void resizeFateExecutor(Map<Set<Fate.FateOperation>,Integer> poolConfigs,
long idleCheckIntervalMillis) {
- final var pool = transactionExecutor;
final int configured = poolConfigs.get(fateOps);
- ThreadPools.resizePool(pool, () -> configured, poolName);
+ ThreadPools.resizePool(transactionExecutor, () -> configured, poolName);
synchronized (runningTxRunners) {
final int running = runningTxRunners.size();
final int needed = configured - running;
log.trace("resizing pools configured:{} running:{} needed:{} fateOps:{}", configured, running,
needed, fateOps);
-
if (needed > 0) {
// If the pool grew, then ensure that there is a TransactionRunner for each thread
for (int i = 0; i < needed; i++) {
+ if (transactionExecutor.isShutdown()) {
+ log.trace("Not adding TransactionRunner, FateExecutor is shutdown.");
+ break;
+ }
try {
- pool.execute(new TransactionRunner());
+ final TransactionRunner tr = new TransactionRunner();
+ runningTxRunners.add(tr);
+ transactionExecutor.execute(tr);
} catch (RejectedExecutionException e) {
// RejectedExecutionException could be shutting down
- if (pool.isShutdown()) {
+ if (transactionExecutor.isShutdown()) {
// The exception is expected in this case, no need to spam the logs.
log.trace("Expected error adding transaction runner to FaTE executor pool. "
+ "The pool is shutdown.", e);
Review Comment:
I'm still not sure about these changes.
1) I'm not sure if we should be checking if the executor is shutdown, I think that would be best handled with the `RejectedExecutionException`
2) I don't think this current code handles the case where the transaction runner is added, but the `execute` results in RejectedExecutionException. In this current code, if the `transactionExecutor.isShutdown()` the tr would never be added, but if the `RejectedExecutionException` occurs for some other reason, the tr wouldn't be removed from runningTxRunners.
Maybe fix is always add the tr to runningTxRunners, call execute, if RejectedExecutionException remove the tr from runningTxRunners. Those that start successfully will cleanup after they finish running (removed in their `run` method) and those that cannot be executed will immediately be removed (in handling RejectedExecutionException).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2293903345
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -412,7 +418,9 @@ public void run() {
}
} finally {
log.trace("A TransactionRunner is exiting...");
- Preconditions.checkState(runningTxRunners.remove(this));
+ synchronized (runningTxRunners) {
+ Preconditions.checkState(runningTxRunners.remove(this));
+ }
Review Comment:
But remove will synchronize internally. Am I missing something
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2294029178
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -105,15 +105,22 @@ public FateExecutor(Fate<T> fate, T environment, Set<Fate.FateOperation> fateOps
protected void resizeFateExecutor(Map<Set<Fate.FateOperation>,Integer> poolConfigs,
long idleCheckIntervalMillis) {
final var pool = transactionExecutor;
- final var runningTxRunners = getRunningTxRunners();
+ final var runningTxRunnersCopy = getRunningTxRunners();
final int configured = poolConfigs.get(fateOps);
ThreadPools.resizePool(pool, () -> configured, poolName);
- final int needed = configured - runningTxRunners.size();
+ final int needed = configured - runningTxRunnersCopy.size();
if (needed > 0) {
// If the pool grew, then ensure that there is a TransactionRunner for each thread
for (int i = 0; i < needed; i++) {
try {
- pool.execute(new TransactionRunner());
+ final TransactionRunner tr = new TransactionRunner();
+ synchronized (runningTxRunners) {
+ if (pool.isShutdown() || pool.isTerminating()) {
+ return;
+ }
+ runningTxRunners.add(tr);
+ pool.execute(tr);
+ }
Review Comment:
Seems like it may be good to get 5813 merged and then make this change to avoid the copy change.
Checking the pool for shutdown here has a race condition like in the following.
```java
if (pool.isShutdown() || pool.isTerminating()) {
return;
}
runningTxRunners.add(tr);
// pool could be shutdown here and a rejected excecution exception thrown
pool.execute(tr);
```
So maybe could do something like the following and not check for shutdown here.
```java
try {
runningTxRunners.add(tr);
pool.execute(tr);
}catch (RejectedExecutionException e){
runningTxRunners.remove(tr);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2298709857
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -107,30 +106,27 @@ public FateExecutor(Fate<T> fate, T environment, Set<Fate.FateOperation> fateOps
*/
protected void resizeFateExecutor(Map<Set<Fate.FateOperation>,Integer> poolConfigs,
long idleCheckIntervalMillis) {
- final var pool = transactionExecutor;
final int configured = poolConfigs.get(fateOps);
- ThreadPools.resizePool(pool, () -> configured, poolName);
+ ThreadPools.resizePool(transactionExecutor, () -> configured, poolName);
synchronized (runningTxRunners) {
final int running = runningTxRunners.size();
final int needed = configured - running;
log.trace("resizing pools configured:{} running:{} needed:{} fateOps:{}", configured, running,
needed, fateOps);
-
if (needed > 0) {
// If the pool grew, then ensure that there is a TransactionRunner for each thread
for (int i = 0; i < needed; i++) {
+ if (transactionExecutor.isShutdown()) {
+ log.trace("Not adding TransactionRunner, FateExecutor is shutdown.");
+ break;
+ }
+ final TransactionRunner tr = new TransactionRunner();
try {
- pool.execute(new TransactionRunner());
+ runningTxRunners.add(tr);
+ transactionExecutor.execute(tr);
} catch (RejectedExecutionException e) {
- // RejectedExecutionException could be shutting down
- if (pool.isShutdown()) {
- // The exception is expected in this case, no need to spam the logs.
- log.trace("Expected error adding transaction runner to FaTE executor pool. "
- + "The pool is shutdown.", e);
- } else {
Review Comment:
It could still be the case that the pool is shutdown if it is shutdown between the shutdown check and the execute call. So, the isShutdown check doesn't really help us
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2298530774
##########
test/src/main/java/org/apache/accumulo/test/fate/FlakyFate.java:
##########
@@ -39,11 +39,6 @@ public class FlakyFate<T> extends Fate<T> {
public FlakyFate(T environment, FateStore<T> store, Function<Repo<T>,String> toLogStrFunc,
AccumuloConfiguration conf) {
super(environment, store, false, toLogStrFunc, conf, new ScheduledThreadPoolExecutor(2));
- }
-
- @Override
- protected void startFateExecutors(T environment, AccumuloConfiguration conf,
- Set<FateExecutor<T>> fateExecutors) {
for (var poolConfig : getPoolConfigurations(conf).entrySet()) {
fateExecutors.add(
new FlakyFateExecutor<>(this, environment, poolConfig.getKey(), poolConfig.getValue()));
Review Comment:
Oh right. Yeah this is still needed. And I think the test does not modify the FATE config at all so this should be all that is needed.
Would be good to run the Flaky FATE tests to make sure this is still good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2298672573
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -107,23 +106,27 @@ public FateExecutor(Fate<T> fate, T environment, Set<Fate.FateOperation> fateOps
*/
protected void resizeFateExecutor(Map<Set<Fate.FateOperation>,Integer> poolConfigs,
long idleCheckIntervalMillis) {
- final var pool = transactionExecutor;
final int configured = poolConfigs.get(fateOps);
- ThreadPools.resizePool(pool, () -> configured, poolName);
+ ThreadPools.resizePool(transactionExecutor, () -> configured, poolName);
synchronized (runningTxRunners) {
final int running = runningTxRunners.size();
final int needed = configured - running;
log.trace("resizing pools configured:{} running:{} needed:{} fateOps:{}", configured, running,
needed, fateOps);
-
if (needed > 0) {
// If the pool grew, then ensure that there is a TransactionRunner for each thread
for (int i = 0; i < needed; i++) {
+ if (transactionExecutor.isShutdown()) {
+ log.trace("Not adding TransactionRunner, FateExecutor is shutdown.");
+ break;
+ }
try {
- pool.execute(new TransactionRunner());
+ final TransactionRunner tr = new TransactionRunner();
+ runningTxRunners.add(tr);
+ transactionExecutor.execute(tr);
} catch (RejectedExecutionException e) {
// RejectedExecutionException could be shutting down
- if (pool.isShutdown()) {
+ if (transactionExecutor.isShutdown()) {
// The exception is expected in this case, no need to spam the logs.
log.trace("Expected error adding transaction runner to FaTE executor pool. "
+ "The pool is shutdown.", e);
Review Comment:
Yeah I think
```
try {
tr = new TransactionRunner()
runningTxRunners.add(tr)
executor.execute(tr)
} catch (REE) {
runningTxRunners.remove(tr)
...
}
```
would be good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2299025419
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -107,30 +106,27 @@ public FateExecutor(Fate<T> fate, T environment, Set<Fate.FateOperation> fateOps
*/
protected void resizeFateExecutor(Map<Set<Fate.FateOperation>,Integer> poolConfigs,
long idleCheckIntervalMillis) {
- final var pool = transactionExecutor;
final int configured = poolConfigs.get(fateOps);
- ThreadPools.resizePool(pool, () -> configured, poolName);
+ ThreadPools.resizePool(transactionExecutor, () -> configured, poolName);
synchronized (runningTxRunners) {
final int running = runningTxRunners.size();
final int needed = configured - running;
log.trace("resizing pools configured:{} running:{} needed:{} fateOps:{}", configured, running,
needed, fateOps);
-
if (needed > 0) {
// If the pool grew, then ensure that there is a TransactionRunner for each thread
for (int i = 0; i < needed; i++) {
+ if (transactionExecutor.isShutdown()) {
+ log.trace("Not adding TransactionRunner, FateExecutor is shutdown.");
+ break;
+ }
+ final TransactionRunner tr = new TransactionRunner();
try {
- pool.execute(new TransactionRunner());
+ runningTxRunners.add(tr);
+ transactionExecutor.execute(tr);
} catch (RejectedExecutionException e) {
- // RejectedExecutionException could be shutting down
- if (pool.isShutdown()) {
- // The exception is expected in this case, no need to spam the logs.
- log.trace("Expected error adding transaction runner to FaTE executor pool. "
- + "The pool is shutdown.", e);
- } else {
Review Comment:
I should have been more explicit: this pool.isShowdown() check is still needed to avoid the ERROR log when it an expected shutdown occurred. Created #5829 to add this back
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2298508064
##########
test/src/main/java/org/apache/accumulo/test/fate/FlakyFate.java:
##########
@@ -39,11 +39,6 @@ public class FlakyFate<T> extends Fate<T> {
public FlakyFate(T environment, FateStore<T> store, Function<Repo<T>,String> toLogStrFunc,
AccumuloConfiguration conf) {
super(environment, store, false, toLogStrFunc, conf, new ScheduledThreadPoolExecutor(2));
- }
-
- @Override
- protected void startFateExecutors(T environment, AccumuloConfiguration conf,
- Set<FateExecutor<T>> fateExecutors) {
for (var poolConfig : getPoolConfigurations(conf).entrySet()) {
fateExecutors.add(
new FlakyFateExecutor<>(this, environment, poolConfig.getKey(), poolConfig.getValue()));
Review Comment:
If we remove this, then `FlakyFateExecutor`'s won't be used, right? The FatePoolWatcher adds `FateExecutor` instances.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion merged PR #5817:
URL: https://github.com/apache/accumulo/pull/5817
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2298594273
##########
core/src/main/java/org/apache/accumulo/core/fate/Fate.java:
##########
@@ -187,8 +189,8 @@ public void run() {
+ "the FateExecutor.", getFateConfigProp(), fateExecutor);
fateExecutor.initiateShutdown();
} else if (fateExecutor.isShutdown() && fateExecutor.isAlive()) {
- log.debug("{} has been shutdown, but is still actively working on transactions.",
- fateExecutor);
+ log.debug("{} has been shutdown, but is still actively working on {} transactions.",
+ fateExecutor, fateExecutor.getNumRunningTxRunners());
Review Comment:
The toString for FateExecutor will include the count of the number of running thread, so do not need to add it again here.
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -107,23 +106,27 @@ public FateExecutor(Fate<T> fate, T environment, Set<Fate.FateOperation> fateOps
*/
protected void resizeFateExecutor(Map<Set<Fate.FateOperation>,Integer> poolConfigs,
long idleCheckIntervalMillis) {
- final var pool = transactionExecutor;
final int configured = poolConfigs.get(fateOps);
- ThreadPools.resizePool(pool, () -> configured, poolName);
+ ThreadPools.resizePool(transactionExecutor, () -> configured, poolName);
synchronized (runningTxRunners) {
final int running = runningTxRunners.size();
final int needed = configured - running;
log.trace("resizing pools configured:{} running:{} needed:{} fateOps:{}", configured, running,
needed, fateOps);
-
if (needed > 0) {
// If the pool grew, then ensure that there is a TransactionRunner for each thread
for (int i = 0; i < needed; i++) {
+ if (transactionExecutor.isShutdown()) {
+ log.trace("Not adding TransactionRunner, FateExecutor is shutdown.");
+ break;
+ }
try {
- pool.execute(new TransactionRunner());
+ final TransactionRunner tr = new TransactionRunner();
+ runningTxRunners.add(tr);
+ transactionExecutor.execute(tr);
} catch (RejectedExecutionException e) {
// RejectedExecutionException could be shutting down
- if (pool.isShutdown()) {
+ if (transactionExecutor.isShutdown()) {
Review Comment:
Need to remove the runner when execution rejected because the finally block in the background thread that removes it will never run.
```suggestion
runningTxRunners.remove(tr);
if (transactionExecutor.isShutdown()) {
```
##########
core/src/main/java/org/apache/accumulo/core/fate/Fate.java:
##########
@@ -207,6 +209,7 @@ public void run() {
synchronized (fateExecutors) {
if (fateExecutors.stream().map(FateExecutor::getFateOps)
.noneMatch(fo -> fo.equals(configFateOps))) {
+ log.debug("Adding FateExecutor for {}", configFateOps);
Review Comment:
Would be useful to also include the store type here prior to the ops. Then we know if its meta or user fate that is adding a FateExecutor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add TransactionRunner to set when created, not when first run [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on code in PR #5817:
URL: https://github.com/apache/accumulo/pull/5817#discussion_r2298707190
##########
core/src/main/java/org/apache/accumulo/core/fate/FateExecutor.java:
##########
@@ -107,30 +106,27 @@ public FateExecutor(Fate<T> fate, T environment, Set<Fate.FateOperation> fateOps
*/
protected void resizeFateExecutor(Map<Set<Fate.FateOperation>,Integer> poolConfigs,
long idleCheckIntervalMillis) {
- final var pool = transactionExecutor;
final int configured = poolConfigs.get(fateOps);
- ThreadPools.resizePool(pool, () -> configured, poolName);
+ ThreadPools.resizePool(transactionExecutor, () -> configured, poolName);
synchronized (runningTxRunners) {
final int running = runningTxRunners.size();
final int needed = configured - running;
log.trace("resizing pools configured:{} running:{} needed:{} fateOps:{}", configured, running,
needed, fateOps);
-
if (needed > 0) {
// If the pool grew, then ensure that there is a TransactionRunner for each thread
for (int i = 0; i < needed; i++) {
+ if (transactionExecutor.isShutdown()) {
+ log.trace("Not adding TransactionRunner, FateExecutor is shutdown.");
+ break;
+ }
Review Comment:
```suggestion
```
I still think this should be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org