You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2022/10/25 19:35:08 UTC
[GitHub] [accumulo] keith-turner opened a new pull request, #3044: Prevents bulk import from hanging.
keith-turner opened a new pull request, #3044:
URL: https://github.com/apache/accumulo/pull/3044
Fixes a bug where bulk import would hang when the first row of the file was equal to the last row of the first tablet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] DomGarguilo commented on a diff in pull request #3044: Prevents bulk import from hanging.
Posted by GitBox <gi...@apache.org>.
DomGarguilo commented on code in PR #3044:
URL: https://github.com/apache/accumulo/pull/3044#discussion_r1004908040
##########
test/src/main/java/org/apache/accumulo/test/functional/BulkNewIT.java:
##########
@@ -480,6 +480,33 @@ public void testEmptyDirWithIgnoreOption() throws Exception {
}
}
+ /*
+ * This test imports a file where the first row of the file is equal to the last row of the first
+ * tablet. There was a bug where this scenario would cause bulk import to hang forever.
+ */
+ @Test
+ public void testEndOfFirstTablet() throws Exception {
+ try (AccumuloClient c = Accumulo.newClient().from(getClientProps()).build()) {
+ String dir = getDir("/testBulkFile-");
+ FileSystem fs = getCluster().getFileSystem();
+ fs.mkdirs(new Path(dir));
+
+ addSplits(c, tableName, "0333");
+
+ var h1 = writeData(dir + "/f1.", aconf, 333, 333);
+
+ c.tableOperations().importDirectory(dir).to(tableName).load();
+
+ verifyData(c, tableName, 333, 333, false);
+
+ Map<String,Set<String>> hashes = new HashMap<>();
+ hashes.put("0333", new HashSet<>());
+ hashes.get("0333").add(h1);
Review Comment:
Could this be condensed?
```suggestion
hashes.put("0333", Set.of(h1));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner merged pull request #3044: Prevents bulk import from hanging.
Posted by GitBox <gi...@apache.org>.
keith-turner merged PR #3044:
URL: https://github.com/apache/accumulo/pull/3044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on a diff in pull request #3044: Prevents bulk import from hanging.
Posted by GitBox <gi...@apache.org>.
keith-turner commented on code in PR #3044:
URL: https://github.com/apache/accumulo/pull/3044#discussion_r1004993310
##########
test/src/main/java/org/apache/accumulo/test/functional/BulkNewIT.java:
##########
@@ -480,6 +480,32 @@ public void testEmptyDirWithIgnoreOption() throws Exception {
}
}
+ /*
+ * This test imports a file where the first row of the file is equal to the last row of the first
+ * tablet. There was a bug where this scenario would cause bulk import to hang forever.
+ */
+ @Test
+ public void testEndOfFirstTablet() throws Exception {
+ try (AccumuloClient c = Accumulo.newClient().from(getClientProps()).build()) {
+ String dir = getDir("/testBulkFile-");
+ FileSystem fs = getCluster().getFileSystem();
+ fs.mkdirs(new Path(dir));
+
+ addSplits(c, tableName, "0333");
+
+ var h1 = writeData(dir + "/f1.", aconf, 333, 333);
+
+ c.tableOperations().importDirectory(dir).to(tableName).load();
+
+ verifyData(c, tableName, 333, 333, false);
+
+ Map<String,Set<String>> hashes = new HashMap<>();
+ hashes.put("0333", Set.of(h1));
+ hashes.put("null", new HashSet<>());
Review Comment:
```suggestion
hashes.put("null", Set.of());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org