You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2020/06/15 20:14:05 UTC
[GitHub] [accumulo] keith-turner opened a new issue #1628: Add test where compaction selector sporadically throws an exception.
keith-turner opened a new issue #1628:
URL: https://github.com/apache/accumulo/issues/1628
Add test that does the following to org.apache.accumulo.test.CompactionIT
* Create a compaction selector that only compacts file names that match a pattern and throws an exception randomly 50% of the time.
* Initiate a compaction of a subset of file using above selector. Confirm subset if compacted.
* Add files and run above step a few times.
This test will ensure that compaction selector is retried when it throws an exception and a single exception does not cause compactions to hang.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ivakegg commented on issue #1628: Add test where compaction selector sporadically throws an exception.
Posted by GitBox <gi...@apache.org>.
ivakegg commented on issue #1628:
URL: https://github.com/apache/accumulo/issues/1628#issuecomment-721861746
@tynyttie As far as I can tell this ticket is still relevant. You would still be creating a CompactionSelector implementation as described and initiating the compaction.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ivakegg edited a comment on issue #1628: Add test where compaction selector sporadically throws an exception.
Posted by GitBox <gi...@apache.org>.
ivakegg edited a comment on issue #1628:
URL: https://github.com/apache/accumulo/issues/1628#issuecomment-721861746
@tynyttie As far as I can tell this ticket is still relevant. You would still be creating a CompactionSelector implementation as described and configuring this to be used on the table. The expected result would be to "ensure that compaction selector is retried when it throws an exception and a single exception does not cause compactions to hang." So essentially you will need to verify that after a failed compaction, that a successful compaction is completed. @keith-turner Any other guidance?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] tynyttie commented on issue #1628: Add test where compaction selector sporadically throws an exception.
Posted by GitBox <gi...@apache.org>.
tynyttie commented on issue #1628:
URL: https://github.com/apache/accumulo/issues/1628#issuecomment-718127854
@keith-turner Is this ticket still relevant base on your new updates to Compaction? If so, what components do I directly affect to accomplish the task? Plus, what was your expected result?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ivakegg edited a comment on issue #1628: Add test where compaction selector sporadically throws an exception.
Posted by GitBox <gi...@apache.org>.
ivakegg edited a comment on issue #1628:
URL: https://github.com/apache/accumulo/issues/1628#issuecomment-721861746
@tynyttie As far as I can tell this ticket is still relevant. You would still be creating a CompactionSelector implementation as described and configuring this to be used on the table. The expected result would be to "ensure that compaction selector is retried when it throws an exception and a single exception does not cause compactions to hang." So essentially you will need to verify that after a failed compaction, that a successful compaction is completed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] tynyttie commented on issue #1628: Add test where compaction selector sporadically throws an exception.
Posted by GitBox <gi...@apache.org>.
tynyttie commented on issue #1628:
URL: https://github.com/apache/accumulo/issues/1628#issuecomment-694993657
I will look at it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on issue #1628: Add test where compaction selector sporadically throws an exception.
Posted by GitBox <gi...@apache.org>.
keith-turner commented on issue #1628:
URL: https://github.com/apache/accumulo/issues/1628#issuecomment-723359231
> My issue is when the unchecked exception is thrown, where in Compaction Manger hierarchy should I appropriately handle it in order for Accumulo to re-kick and select a new compact-able?
@tynyttie my hope is that it already works, but the test will let us know if it actually works. The following code is where files are selected.
https://github.com/apache/accumulo/blob/18ba45e144ea8213a9637fe36bf0d8a68c3f15f5/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableImpl.java#L328
If an exception is thrown, the following should catch and hopefully reset things to a good state so that future compactions can run if an exception is not thrown.
https://github.com/apache/accumulo/blob/18ba45e144ea8213a9637fe36bf0d8a68c3f15f5/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableImpl.java#L353
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii closed issue #1628: Add test where compaction selector sporadically throws an exception.
Posted by GitBox <gi...@apache.org>.
ctubbsii closed issue #1628:
URL: https://github.com/apache/accumulo/issues/1628
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner edited a comment on issue #1628: Add test where compaction selector sporadically throws an exception.
Posted by GitBox <gi...@apache.org>.
keith-turner edited a comment on issue #1628:
URL: https://github.com/apache/accumulo/issues/1628#issuecomment-722759100
@tynyttie I would add the test to the following class.
https://github.com/apache/accumulo/blob/main/test/src/main/java/org/apache/accumulo/test/CompactionExecutorIT.java
The test could do something like the following. Could create BuggySelector as a static public class in CompactionExecutorIT
```java
@Test
public void testBuggySelector(){
//TODO create table bst and Accumulo client
// add 3 files to table...
addFile(client, "bst", 1, 1000, false);
addFile(client, "bst", 1, 500, false)
addFile(client, "bst", 1, 250, false)
// get the current files
var files = getFiles(client, "bst");
// TODO somehow select a random subset of files from above set
randomSubset = ....
String rsl = .... somehow join randomSubset into a comma sep list
// configure compaction to use BuggySelector. BuggySelector should select the files passed in the option fileToCompact and it should sometimes randomly throw an exception.
var cc1 = new CompactionConfig()
.setSelector(
new PluginConfig(BuggySelector.class.getName(), Map.of("filesToCompact", rls)))
.setWait(true);
client.tableOperations().compact("bst", cc1);
var files2 = getFiles(client, "bst");
// TODO verify that files2 does not contain anything from randomSubset
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on issue #1628: Add test where compaction selector sporadically throws an exception.
Posted by GitBox <gi...@apache.org>.
keith-turner commented on issue #1628:
URL: https://github.com/apache/accumulo/issues/1628#issuecomment-722759100
@tynyttie I would add the test to the following class.
https://github.com/apache/accumulo/blob/main/test/src/main/java/org/apache/accumulo/test/CompactionExecutorIT.java
The test could do something like the following. Could create BuggySelector as a static public class in CompactionExecutorIT
```java
@Test
public void testBuggySelector(){
//TODO create table bst and Accumulo client
// add 3 files to table...
addFile(client, "bst", 1, 1000, false);
addFile(client, "bst", 1, 500, false)
addFile(client, "bst", 1, 250, false)
// get the current files
var files = getFiles(client, "bst");
// TODO somehow select a random subset of files from above set
randomSubset = ....
String rsl = .... somehow join randomSubset into a comma sep list
// configure compaction to use BuggySelector. BuggySelector should select the files passed in the option fileToCompact and it should sometimes randomly throw an exception.
var cc1 = new CompactionConfig()
.setSelector(
new PluginConfig(BuggySelector.class.getName(), Map.of("filesToCompact", rls)))
.setWait(true);
client.tableOperations().compact("bst", cc1);
var files2 = getFiles(client, "bst");
// TODO verify that files does not contain anything from randomSubset
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] tynyttie commented on issue #1628: Add test where compaction selector sporadically throws an exception.
Posted by GitBox <gi...@apache.org>.
tynyttie commented on issue #1628:
URL: https://github.com/apache/accumulo/issues/1628#issuecomment-723145270
@keith-turner Thank you for replying back to me. My issue is when the unchecked exception is thrown, where in Compaction Manger hierarchy should I appropriately handle it in order for Accumulo to re-kick and select a new compact-able? Should it be in the Compaction Manger Class, Compaction Executor class or Compaction Service class?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org