You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2021/04/26 14:34:53 UTC
[GitHub] [accumulo] keith-turner opened a new issue #2030: Make delete table operations cancel user compactions
keith-turner opened a new issue #2030:
URL: https://github.com/apache/accumulo/issues/2030
**Is your feature request related to a problem? Please describe.**
User compactions start a FATE op that gets a table lock. Delete table operations wait to get the table lock. They may wait on a user compaction. If they do, it would be nice if they didn't.
**Describe the solution you'd like**
There is already an existing cancel user compaction API call. It may be that a delete table operation could call this (client side) prior to initiating a delete table FATE op.
**Describe alternatives you've considered**
A more general alternative may be to add canceling FATE ops as a high level FATE operation that could be called before delete table to cancel any FATE op running against that table (like merge).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2030: Make delete table operations cancel user compactions
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2030:
URL: https://github.com/apache/accumulo/issues/2030#issuecomment-827677007
I did see the CompactionCanceledException thrown in 1.10 while testing #1872. But it looks like the Delete doesn't take any action to cancel the compaction. I guess it depends on where in the compaction process the Delete occurs as to whether the compaction will see the table deleted before it kicks off. I created an IT to test a typical internal majc and saw the compaction hang, timing out the test #2038
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #2030: Make delete table operations cancel user compactions
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #2030:
URL: https://github.com/apache/accumulo/issues/2030#issuecomment-852206136
Nevermind, on further review, the DELETE is trying to get a Write lock, and it can't because the COMPACT operation has a read lock. I should have logged the lock type in the retry message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #2030: Make delete table operations cancel user compactions
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #2030:
URL: https://github.com/apache/accumulo/issues/2030#issuecomment-852109921
I wonder if [CompactionCheck ](https://github.com/apache/accumulo/blob/main/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableImpl.java#L524) should return `false` if `Tablet.isBeingDeleted()` is `true`. It already returns `false` if the tablet is closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion closed issue #2030: Make delete table operations cancel user compactions
Posted by GitBox <gi...@apache.org>.
dlmarion closed issue #2030:
URL: https://github.com/apache/accumulo/issues/2030
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #2030: Make delete table operations cancel user compactions
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #2030:
URL: https://github.com/apache/accumulo/issues/2030#issuecomment-827515740
So, in one of the external compaction tests that I wrote the delete command was waiting to get the FATE lock which was held by the user compaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #2030: Make delete table operations cancel user compactions
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #2030:
URL: https://github.com/apache/accumulo/issues/2030#issuecomment-852190551
So, that didn't work. Looking at the Manager logs I see a constant stream of:
```
2021-06-01T14:35:54,613 [tableOps.Utils] INFO : namespace +default (55e4a47017a35a48) locked for read operation: DELETE
```
I noticed that this is the success case when the getLock method is successful. I added some logging to the `else` case, when the lock is not successful and the method returns `100`. It appears that the DELETE fate operation cannot get the table lock. The delete call in your test is waiting until the compaction is complete before executing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #2030: Make delete table operations cancel user compactions
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #2030:
URL: https://github.com/apache/accumulo/issues/2030#issuecomment-852198441
Here's the snippet from the manager log (with my log changes):
```
2021-06-01T14:47:54,043 [tableOps.Utils] INFO : namespace +default (69c29b96a0cc24bc) locked for read operation: COMPACT
2021-06-01T14:47:54,060 [tableOps.Utils] INFO : table 1 (69c29b96a0cc24bc) locked for read operation: COMPACT
2021-06-01T14:47:54,507 [accumulo.audit] INFO : operation: permitted; user: root; client: 127.0.0.1:59340; action: authenticate;
2021-06-01T14:47:54,516 [accumulo.audit] INFO : operation: permitted; user: root; client: 127.0.0.1:59340; action: authenticate;
2021-06-01T14:47:54,542 [accumulo.audit] INFO : operation: permitted; user: root; client: 127.0.0.1:59340; action: authenticate;
2021-06-01T14:47:54,542 [accumulo.audit] INFO : operation: permitted; user: root; client: 127.0.0.1:59340; action: deleteTable; targetTable: DeleteCancelsMajCIT_test0;
2021-06-01T14:47:54,556 [accumulo.audit] INFO : operation: permitted; user: root; client: 127.0.0.1:59340; action: authenticate;
2021-06-01T14:47:54,557 [tableOps.Utils] INFO : namespace +default (252a9562646ec62e) locked for read operation: DELETE
2021-06-01T14:47:54,561 [tableOps.Utils] INFO : table 1 (252a9562646ec62e) locked by another operation, will retry
```
Is this the intended design for the DistributedReadWriteLock? Only 1 concurrent READ lock? This seems to deviate from it's parent class, https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/locks/ReadWriteLock.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2030: Make delete table operations cancel user compactions
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2030:
URL: https://github.com/apache/accumulo/issues/2030#issuecomment-827123141
I thought this is already done? https://github.com/apache/accumulo/blob/e6cc640216e22075c205efeb6ba784a5af720c60/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Compactor.java#L391-L403
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion closed issue #2030: Make delete table operations cancel user compactions
Posted by GitBox <gi...@apache.org>.
dlmarion closed issue #2030:
URL: https://github.com/apache/accumulo/issues/2030
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #2030: Make delete table operations cancel user compactions
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #2030:
URL: https://github.com/apache/accumulo/issues/2030#issuecomment-861678284
Maybe we should have delete call cancel compactions on the table before trying to get the write lock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org