You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "dlmarion (via GitHub)" <gi...@apache.org> on 2023/07/27 22:56:16 UTC
[GitHub] [accumulo] dlmarion opened a new pull request, #3665: Deduplicate compaction tests
dlmarion opened a new pull request, #3665:
URL: https://github.com/apache/accumulo/pull/3665
Removed tests from CompactionIT that had a matching test in ExternalCompaction*IT. Updated tests in ExternalCompaction*IT that were missing some recent updates in CompactionIT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on a diff in pull request #3665: Deduplicate compaction tests
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #3665:
URL: https://github.com/apache/accumulo/pull/3665#discussion_r1276976544
##########
test/src/main/java/org/apache/accumulo/test/functional/CompactionIT.java:
##########
@@ -189,114 +170,6 @@ public void configureMiniCluster(MiniAccumuloConfigImpl cfg, Configuration hadoo
hadoopCoreSite.set("fs.file.impl", RawLocalFileSystem.class.getName());
}
- @Test
- public void testBadSelector() throws Exception {
- try (AccumuloClient c = Accumulo.newClient().from(getClientProps()).build()) {
- final String tableName = getUniqueNames(1)[0];
- NewTableConfiguration tc = new NewTableConfiguration();
- // Ensure compactions don't kick off
- tc.setProperties(Map.of(Property.TABLE_MAJC_RATIO.getKey(), "10.0"));
- c.tableOperations().create(tableName, tc);
- // Create multiple RFiles
- try (BatchWriter bw = c.createBatchWriter(tableName)) {
- for (int i = 1; i <= 4; i++) {
- Mutation m = new Mutation(Integer.toString(i));
- m.put("cf", "cq", new Value());
- bw.addMutation(m);
- bw.flush();
- // flush often to create multiple files to compact
- c.tableOperations().flush(tableName, null, null, true);
- }
- }
-
- CompactionConfig config = new CompactionConfig()
- .setSelector(new PluginConfig(ErrorThrowingSelector.class.getName(), Map.of()))
- .setWait(true);
- assertThrows(AccumuloException.class, () -> c.tableOperations().compact(tableName, config));
-
- List<String> rows = new ArrayList<>();
- c.createScanner(tableName).forEach((k, v) -> rows.add(k.getRow().toString()));
- assertEquals(List.of("1", "2", "3", "4"), rows);
-
- assertNoCompactionMetadata(tableName);
- }
- }
-
- @Test
- public void testCompactionWithTableIterator() throws Exception {
- String table1 = this.getUniqueNames(1)[0];
- try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
- client.tableOperations().create(table1);
- try (BatchWriter bw = client.createBatchWriter(table1)) {
- for (int i = 1; i <= 4; i++) {
- Mutation m = new Mutation(Integer.toString(i));
- m.put("cf", "cq", new Value());
- bw.addMutation(m);
- bw.flush();
- // flush often to create multiple files to compact
- client.tableOperations().flush(table1, null, null, true);
- }
- }
-
- IteratorSetting setting = new IteratorSetting(50, "delete", DevNull.class);
- client.tableOperations().attachIterator(table1, setting, EnumSet.of(IteratorScope.majc));
- client.tableOperations().compact(table1, new CompactionConfig().setWait(true));
-
- try (Scanner s = client.createScanner(table1)) {
- assertFalse(s.iterator().hasNext());
- }
-
- assertNoCompactionMetadata(table1);
- }
- }
-
- @Test
- public void testUserCompactionCancellation() throws Exception {
- final String table1 = this.getUniqueNames(1)[0];
- try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
- client.tableOperations().create(table1);
- try (BatchWriter bw = client.createBatchWriter(table1)) {
- for (int i = 1; i <= MAX_DATA; i++) {
- Mutation m = new Mutation(Integer.toString(i));
- m.put("cf", "cq", new Value());
- bw.addMutation(m);
- bw.flush();
- // flush often to create multiple files to compact
- client.tableOperations().flush(table1, null, null, true);
- }
- }
-
- final AtomicReference<Exception> error = new AtomicReference<>();
- Thread t = new Thread(() -> {
- try {
- IteratorSetting setting = new IteratorSetting(50, "sleepy", SlowIterator.class);
- setting.addOption("sleepTime", "3000");
- setting.addOption("seekSleepTime", "3000");
- var cconf = new CompactionConfig().setWait(true).setIterators(List.of(setting));
- client.tableOperations().compact(table1, cconf);
- } catch (AccumuloSecurityException | TableNotFoundException | AccumuloException e) {
- error.set(e);
- }
- });
- t.start();
- // when the compaction starts it will create a selected files column in the tablet, wait for
- // that to happen
- while (countTablets(table1, tm -> tm.getSelectedFiles() != null) == 0) {
- Thread.sleep(1000);
- }
- client.tableOperations().cancelCompaction(table1);
- t.join();
- Exception e = error.get();
- assertNotNull(e);
- assertEquals(TableOperationsImpl.COMPACTION_CANCELED_MSG, e.getMessage());
Review Comment:
The variant of the test in ExternalCompaction_2_It does not run the compaction in backgound thread and make sure it throws an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on a diff in pull request #3665: Deduplicate compaction tests
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on code in PR #3665:
URL: https://github.com/apache/accumulo/pull/3665#discussion_r1277444473
##########
test/src/main/java/org/apache/accumulo/test/functional/CompactionIT.java:
##########
@@ -189,114 +170,6 @@ public void configureMiniCluster(MiniAccumuloConfigImpl cfg, Configuration hadoo
hadoopCoreSite.set("fs.file.impl", RawLocalFileSystem.class.getName());
}
- @Test
- public void testBadSelector() throws Exception {
- try (AccumuloClient c = Accumulo.newClient().from(getClientProps()).build()) {
- final String tableName = getUniqueNames(1)[0];
- NewTableConfiguration tc = new NewTableConfiguration();
- // Ensure compactions don't kick off
- tc.setProperties(Map.of(Property.TABLE_MAJC_RATIO.getKey(), "10.0"));
- c.tableOperations().create(tableName, tc);
- // Create multiple RFiles
- try (BatchWriter bw = c.createBatchWriter(tableName)) {
- for (int i = 1; i <= 4; i++) {
- Mutation m = new Mutation(Integer.toString(i));
- m.put("cf", "cq", new Value());
- bw.addMutation(m);
- bw.flush();
- // flush often to create multiple files to compact
- c.tableOperations().flush(tableName, null, null, true);
- }
- }
-
- CompactionConfig config = new CompactionConfig()
- .setSelector(new PluginConfig(ErrorThrowingSelector.class.getName(), Map.of()))
- .setWait(true);
- assertThrows(AccumuloException.class, () -> c.tableOperations().compact(tableName, config));
-
- List<String> rows = new ArrayList<>();
- c.createScanner(tableName).forEach((k, v) -> rows.add(k.getRow().toString()));
- assertEquals(List.of("1", "2", "3", "4"), rows);
-
- assertNoCompactionMetadata(tableName);
- }
- }
-
- @Test
- public void testCompactionWithTableIterator() throws Exception {
- String table1 = this.getUniqueNames(1)[0];
- try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
- client.tableOperations().create(table1);
- try (BatchWriter bw = client.createBatchWriter(table1)) {
- for (int i = 1; i <= 4; i++) {
- Mutation m = new Mutation(Integer.toString(i));
- m.put("cf", "cq", new Value());
- bw.addMutation(m);
- bw.flush();
- // flush often to create multiple files to compact
- client.tableOperations().flush(table1, null, null, true);
- }
- }
-
- IteratorSetting setting = new IteratorSetting(50, "delete", DevNull.class);
- client.tableOperations().attachIterator(table1, setting, EnumSet.of(IteratorScope.majc));
- client.tableOperations().compact(table1, new CompactionConfig().setWait(true));
-
- try (Scanner s = client.createScanner(table1)) {
- assertFalse(s.iterator().hasNext());
- }
-
- assertNoCompactionMetadata(table1);
- }
- }
-
- @Test
- public void testUserCompactionCancellation() throws Exception {
- final String table1 = this.getUniqueNames(1)[0];
- try (AccumuloClient client = Accumulo.newClient().from(getClientProps()).build()) {
- client.tableOperations().create(table1);
- try (BatchWriter bw = client.createBatchWriter(table1)) {
- for (int i = 1; i <= MAX_DATA; i++) {
- Mutation m = new Mutation(Integer.toString(i));
- m.put("cf", "cq", new Value());
- bw.addMutation(m);
- bw.flush();
- // flush often to create multiple files to compact
- client.tableOperations().flush(table1, null, null, true);
- }
- }
-
- final AtomicReference<Exception> error = new AtomicReference<>();
- Thread t = new Thread(() -> {
- try {
- IteratorSetting setting = new IteratorSetting(50, "sleepy", SlowIterator.class);
- setting.addOption("sleepTime", "3000");
- setting.addOption("seekSleepTime", "3000");
- var cconf = new CompactionConfig().setWait(true).setIterators(List.of(setting));
- client.tableOperations().compact(table1, cconf);
- } catch (AccumuloSecurityException | TableNotFoundException | AccumuloException e) {
- error.set(e);
- }
- });
- t.start();
- // when the compaction starts it will create a selected files column in the tablet, wait for
- // that to happen
- while (countTablets(table1, tm -> tm.getSelectedFiles() != null) == 0) {
- Thread.sleep(1000);
- }
- client.tableOperations().cancelCompaction(table1);
- t.join();
- Exception e = error.get();
- assertNotNull(e);
- assertEquals(TableOperationsImpl.COMPACTION_CANCELED_MSG, e.getMessage());
Review Comment:
good catch, added in 369f8d8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion merged pull request #3665: Deduplicate compaction tests
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion merged PR #3665:
URL: https://github.com/apache/accumulo/pull/3665
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org