You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "kevinrr888 (via GitHub)" <gi...@apache.org> on 2025/06/25 16:43:01 UTC
[I] Need ComprehensiveTableOperationsIT_SimpleSuite.test_compact() functionality [accumulo]
kevinrr888 opened a new issue, #5679:
URL: https://github.com/apache/accumulo/issues/5679
**Is your feature request related to a problem? Please describe.**
The test was originally written as shown in additional context but was deleted. This test needs to test compactions on all the system tables, but the approach below is either flawed or is demonstrating a bug with the Manager.
**Describe the solution you'd like**
Need to determine if the issue is with the test itself or the Manager. If the issue is with the test itself, need to fix the test below or rewrite the test while still testing basic functionality of compactions on all system tables. If the issue is with the Manager, need to resolve the issue and hopefully can use the same test below.
**Describe alternatives you've considered**
N/A
**Additional context**
See https://github.com/apache/accumulo/pull/5474#discussion_r2042452359 which details the issue seen in the below test.
```
@Test
public void test_compact() throws Exception {
// compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
// test exists
assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
// disable the GC to prevent automatic compactions on METADATA and ROOT tables
getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
try {
// test basic functionality for system tables
userTable = getUniqueNames(1)[0];
ops.create(userTable);
// create some RFiles for the METADATA and ROOT tables by creating some data in the user
// table, flushing that table, then the METADATA table, then the ROOT table
for (int i = 0; i < 3; i++) {
try (var bw = client.createBatchWriter(userTable)) {
var mut = new Mutation("r" + i);
mut.put("cf", "cq", "v");
bw.addMutation(mut);
}
ops.flush(userTable, null, null, true);
ops.flush(SystemTables.METADATA.tableName(), null, null, true);
ops.flush(SystemTables.ROOT.tableName(), null, null, true);
}
for (var sysTable : SystemTables.tableNames()) {
// create some RFiles for FATE and SCAN_REF tables
if (sysTable == SystemTables.SCAN_REF) {
createScanRefTableRow();
ops.flush(SystemTables.SCAN_REF.tableName(), null, null, true);
} else if (sysTable == SystemTables.FATE) {
createFateTableRow(userTable);
ops.flush(SystemTables.FATE.tableName(), null, null, true);
}
Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);
log.info("Compacting " + sysTable);
ops.compact(sysTable.tableName(), null, null, true, true);
log.info("Finished compacting " + sysTable);
// RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
// with 'A' that was not present before the compaction.
Wait.waitFor(() -> {
var stfsAfterCompact = getStoredTabFiles(sysTable);
String regex = "^A.*\\.rf$";
var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
.filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
var A_stfsAfterCompaction = stfsAfterCompact.stream()
.filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
});
}
} finally {
getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Need ComprehensiveTableOperationsIT_SimpleSuite.test_compact() functionality [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on issue #5679:
URL: https://github.com/apache/accumulo/issues/5679#issuecomment-3201977239
1) Need to have data in the FATE table when the flush() occurs. This will write an RFile for the FATE table
2) Then, compact(SystemTables.FATE) should compact that file giving the 'A' file
Are you sure when you are using the slow iterator that data exists in the FATE table when the flush occurs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Need ComprehensiveTableOperationsIT_SimpleSuite.test_compact() functionality [accumulo]
Posted by "ArbaazKhan1 (via GitHub)" <gi...@apache.org>.
ArbaazKhan1 commented on issue #5679:
URL: https://github.com/apache/accumulo/issues/5679#issuecomment-3103294953
I can take a look at this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Need ComprehensiveTableOperationsIT_SimpleSuite.test_compact() functionality [accumulo]
Posted by "ArbaazKhan1 (via GitHub)" <gi...@apache.org>.
ArbaazKhan1 commented on issue #5679:
URL: https://github.com/apache/accumulo/issues/5679#issuecomment-3201929315
I'm running into two issues while running `test_compact`on system tables.
1. In its current form the test uses a slow iterator, which gets stuck in `CompcationDriver.isReady`rather than proceeding to `call()` as expected. This unexpectedly causes other system table compactions to hang as well.
2. writing the test without slow iterator or using a low sleep time it seems compactions complete but don't generate 'A' files for the FATE table, causing the test to time out in `Wait.waitFor()`
It seems the FATE table might have special compaction criteria that aren't being met in the test. What's the recommended approach here? Any guidance on properly testing system table compactions would be appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Need ComprehensiveTableOperationsIT_SimpleSuite.test_compact() functionality [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on issue #5679:
URL: https://github.com/apache/accumulo/issues/5679#issuecomment-3233621616
#5798 is merged and introduced a new way to have a slow running FATE operation in the Manager. I think this would help in this issue. Instead of attaching a slow iterator and compacting a user table, could use the new SlowFateSplitManager and add splits to a user table. This split operation will take >= 10sec to complete. Can flush FATE table while this is running.
This doesn't completely answer why compactions on other tables were affected by the slow compaction on the user table, but maybe that can be a new issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Need ComprehensiveTableOperationsIT_SimpleSuite.test_compact() functionality [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 closed issue #5679: Need ComprehensiveTableOperationsIT_SimpleSuite.test_compact() functionality
URL: https://github.com/apache/accumulo/issues/5679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Need ComprehensiveTableOperationsIT_SimpleSuite.test_compact() functionality [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on issue #5679:
URL: https://github.com/apache/accumulo/issues/5679#issuecomment-3377698706
> This doesn't completely answer why compactions on other tables were affected by the slow compaction on the user table, but maybe that can be a new issue.
Test needed two compactors instead of the default of one. Resolved in #5881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org