You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "keith-turner (via GitHub)" <gi...@apache.org> on 2025/08/04 16:50:44 UTC
[PR] fixes log recovery by creating parent directories [accumulo]
keith-turner opened a new pull request, #5776:
URL: https://github.com/apache/accumulo/pull/5776
Changes in #5743 made RFileOperations use a HDFS builder to create an output stream. Prior to using the builder the calls to HDFS to create an output stream would also create parent directories if they did not exist. Seems the log recovery code in accumulo was relying on this behavior. Added an option to the builder to create parent dirs if they do not exists.
This problem was detected by TotalQueuedIT which was failing in log recovery and the tserver had exception about a parent directory not existing. With these changes TotalQueuedIT now passes.
Could not find javadoc on the old HDFS API calls that mentioned that the parent directory was created. Wrote some test code that called the previous APIs and confirmed that parent dirs were created if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fixes log recovery by creating parent directories [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner merged PR #5776:
URL: https://github.com/apache/accumulo/pull/5776
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fixes log recovery by creating parent directories [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #5776:
URL: https://github.com/apache/accumulo/pull/5776#issuecomment-3151762431
Found some javadoc about the expected behavior. The createNonRecursive methods document the expected behavior of the create methods.
https://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#createNonRecursive-org.apache.hadoop.fs.Path-boolean-int-short-long-org.apache.hadoop.util.Progressable-
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fixes log recovery by creating parent directories [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #5776:
URL: https://github.com/apache/accumulo/pull/5776#issuecomment-3151544701
This is the test code used to verify the old HDFS API call would create parent dirs.
```diff
diff --git a/test/src/main/java/org/apache/accumulo/test/TotalQueuedIT.java b/test/src/main/java/org/apache/accumulo/test/TotalQueuedIT.java
index 9b99c5f24e..a66a7ce242 100644
--- a/test/src/main/java/org/apache/accumulo/test/TotalQueuedIT.java
+++ b/test/src/main/java/org/apache/accumulo/test/TotalQueuedIT.java
@@ -40,6 +40,7 @@ import org.apache.accumulo.miniclusterImpl.MiniAccumuloConfigImpl;
import org.apache.accumulo.server.ServerContext;
import org.apache.accumulo.test.functional.ConfigurableMacBase;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
import org.junit.jupiter.api.Test;
public class TotalQueuedIT extends ConfigurableMacBase {
@@ -65,6 +66,27 @@ public class TotalQueuedIT extends ConfigurableMacBase {
@Test
public void test() throws Exception {
try (AccumuloClient c = Accumulo.newClient().from(getClientProperties()).build()) {
+
+ var statusBefore = getCluster().getMiniDfs().getFileSystem().listStatus(new Path("/"));
+ for (var status : statusBefore) {
+ System.out.println("before " + status);
+ }
+
+ // call the HDFS API that RFileOperations used to call
+ var os = getCluster().getMiniDfs().getFileSystem().create(new Path("/testcp/file.txt"), false,
+ 4096, (short) 3, 1048576);
+ os.writeUTF("test data");
+ os.close();
+
+ var statusAfter = getCluster().getMiniDfs().getFileSystem().listStatus(new Path("/"));
+ for (var status : statusAfter) {
+ System.out.println("after " + status);
+ }
+
+ var is = getCluster().getMiniDfs().getFileSystem().open(new Path("/testcp/file.txt"));
+ var data = is.readUTF();
+ System.out.println("read:" + data);
+
String tableName = getUniqueNames(1)[0];
c.tableOperations().create(tableName);
c.tableOperations().setProperty(tableName, Property.TABLE_MAJC_RATIO.getKey(), "9999");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fixes log recovery by creating parent directories [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #5776:
URL: https://github.com/apache/accumulo/pull/5776#issuecomment-3151716141
Locally I ran `mvn clean verify -Psunny` and that was fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fixes log recovery by creating parent directories [accumulo]
Posted by "ddanielr (via GitHub)" <gi...@apache.org>.
ddanielr commented on PR #5776:
URL: https://github.com/apache/accumulo/pull/5776#issuecomment-3151628191
Should this PR be re-based against 2.1 since #5743 was merged into 2.1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fixes log recovery by creating parent directories [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #5776:
URL: https://github.com/apache/accumulo/pull/5776#issuecomment-3151702854
> Should this PR be re-based against 2.1 since https://github.com/apache/accumulo/pull/5743 was merged into 2.1?
Yeah locally I made the changes against 2.1, forgot to switch the branch on the PR. I pushed 317eaf3 to force GH actions to rerun against 2.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org